Add workflow basis to fetch HEBIS Data RPB-225 #105

TobiasNx · 2024-12-02T12:15:08Z

Reuse lobid transformation for Hebis Mapping

reused almost all of the lobid transformation. Not used stuff is overhead but it makes it more easy to adjust.
I put all maps static and test in the maps folder.

TODO:

adjust to HEBIS and rpb specific stuff
simple holdings are provided in 924, they are now included.
Check: Do they provide rpb specific infos via sru? It seems not
Introduce bash script that creates the sru query and reuses them in the flux
This PR is depending on #129

I reuse the hole transfomration, sofar no adjustments were done for specifc hebis data source. e.g. almaMmsId should be renamed hebisId

TobiasNx · 2025-01-17T14:31:46Z

@fsteeg and @acka47 have a look at my transformed data from hebis including holdings from 924. This could be your basis for the strapi import.

…formHebis

This needs to be adjusted with regard to the strapi indexing.

TobiasNx · 2025-02-26T10:18:44Z

conf/output/test-hebis-to-lobid-output-0.json

@acka47 have a look at the files named conf/output/test-hebis-to-lobid-output-n.json

TobiasNx · 2025-02-26T10:20:19Z

conf/hebisMarc2lobid-transformation/comparisonRpbRecords/rpb-hebis-records-0.json

@acka47 files named conf/hebisMarc2lobid-transformation/comparisonRpbRecords/rpb-hebis-records-n.json are for only for comparison they are records from the productive rpb.

acka47 · 2025-02-28T07:55:43Z

I tested the files against the JSON schema from https://github.com/hbz/lobid-resources/blob/master/src/test/resources/schemas/resource.json Overall, this looks good but there are apparently two subjects in the test files that do not get sufficient information in the resulting JSON. (I will add additional comments as pointers.) Here is the validation output:

$ sh validateJsonTestFiles.sh 
Testing version: draft
alma-fix/test-hebis-to-lobid-output-0.json passed test
alma-fix/test-hebis-to-lobid-output-1.json passed test
alma-fix/test-hebis-to-lobid-output-10.json failed test
[
  {
    instancePath: '/subject/2',
    schemaPath: 'complexSubject.json/required',
    keyword: 'required',
    params: { missingProperty: 'label' },
    message: "must have required property 'label'"
  },
  {
    instancePath: '/subject/2',
    schemaPath: 'complexSubject.json/required',
    keyword: 'required',
    params: { missingProperty: 'componentList' },
    message: "must have required property 'componentList'"
  },
  {
    instancePath: '/subject/2/type/0',
    schemaPath: 'complexSubject.json/properties/type/items/const',
    keyword: 'const',
    params: { allowedValue: 'ComplexSubject' },
    message: 'must be equal to constant'
  },
  {
    instancePath: '/subject/2',
    schemaPath: 'skosConcept.json/required',
    keyword: 'required',
    params: { missingProperty: 'source' },
    message: "must have required property 'source'"
  },
  {
    instancePath: '/subject/2',
    schemaPath: '#/properties/subject/items/anyOf',
    keyword: 'anyOf',
    params: {},
    message: 'must match a schema in anyOf'
  }
]
alma-fix/test-hebis-to-lobid-output-11.json failed test
[
  {
    instancePath: '/subject/2',
    schemaPath: 'complexSubject.json/required',
    keyword: 'required',
    params: { missingProperty: 'label' },
    message: "must have required property 'label'"
  },
  {
    instancePath: '/subject/2',
    schemaPath: 'complexSubject.json/required',
    keyword: 'required',
    params: { missingProperty: 'componentList' },
    message: "must have required property 'componentList'"
  },
  {
    instancePath: '/subject/2/type/0',
    schemaPath: 'complexSubject.json/properties/type/items/const',
    keyword: 'const',
    params: { allowedValue: 'ComplexSubject' },
    message: 'must be equal to constant'
  },
  {
    instancePath: '/subject/2',
    schemaPath: 'skosConcept.json/required',
    keyword: 'required',
    params: { missingProperty: 'source' },
    message: "must have required property 'source'"
  },
  {
    instancePath: '/subject/2',
    schemaPath: '#/properties/subject/items/anyOf',
    keyword: 'anyOf',
    params: {},
    message: 'must match a schema in anyOf'
  }
]
alma-fix/test-hebis-to-lobid-output-12.json passed test
alma-fix/test-hebis-to-lobid-output-13.json passed test
error:  /home/acka47/git/lobid-resources/src/test/resources/alma-fix/test-hebis-to-lobid-output-14.json: Unexpected end of JSON input
-e Test FAILED

We might have to add a mapping there or to ignore these subjects.

However, it would probably make sense to set up an automatic test that fetches the schema from the lobid-resources repo an validates the files against it, wouldn' it?

acka47 · 2025-02-28T07:57:24Z

conf/output/test-hebis-to-lobid-output-10.json

+      "label" : "Systematik der DNB (bis 2003)",
+      "id" : "https://bartoc.org/en/node/18497"
+    }
+  }, {


This subject entry is to spars wrt the lobid-resources JSON schema, see #105 (comment)

acka47 · 2025-02-28T07:58:22Z

conf/output/test-hebis-to-lobid-output-11.json

+      "label" : "Systematik der DNB (bis 2003)",
+      "id" : "https://bartoc.org/en/node/18497"
+    }
+  }, {


This subject entry also is to sparse wrt the lobid-resources JSON schema, see #105 (comment)

Same situation as above.

TobiasNx · 2025-04-01T10:08:08Z

conf/fetchAndTransformHebisRecord.flux

+| batch-reset(batchsize="1")
+| encode-json(prettyPrinting="true")
+| write(FLUX_DIR + "output/test-hebis-to-lobid-output-${i}.json")
+;


For the productive workflow @fsteeg you would need to adjust the process here.

fsteeg · 2025-04-01T13:53:30Z

conf/output/test-hebis-to-lobid-output-1.json

+    "type" : [ "Item", "PhysicalObject" ],
+    "callNumber" : "Mog m 7188"
+  }, {
+    "heldBy" : {


The structure here seems to be wrong: heldBy is in a separate object, but should be in the object above?

Fixed this.

To merge external hebis data with strapi record for indexing

fsteeg · 2025-04-01T15:03:48Z

Great, this works in principle, see comment in Jira.

Assigning:

@TobiasNx for comment above (and open TODOs in original comment)
@dr0i for review of 4a8c98f

dr0i · 2025-04-03T07:37:24Z

I added a comment in 4a8c98f , but cannot see it ... has GitHUb changed its behaviour, can you see my comment?

fsteeg · 2025-04-03T08:08:51Z

I added a comment in 4a8c98f , but cannot see it ... has GitHUb changed its behaviour, can you see my comment?

I don't see it. Maybe you added it to a review, but didn't submit the review?

fsteeg · 2025-04-03T08:53:13Z

I don't see it. Maybe you added it to a review, but didn't submit the review?

Found the comment and replied here:

4a8c98f#r154800490

app/controllers/nwbib/Application.java

fsteeg · 2025-04-03T15:50:24Z

Now under functional review by LBZ in https://jira.hbz-nrw.de/browse/RPB-225.

fsteeg · 2025-04-03T15:54:45Z

Re-requested review from @acka47 to make sure requested changes are addressed.

acka47

+1

TobiasNx added 8 commits December 2, 2024 11:18

Add first workflow to fetch HEBIS Data RPB-225

3153738

Reuse lobid-transformation for hebis data RPD-225

39d4908

I reuse the hole transfomration, sofar no adjustments were done for specifc hebis data source. e.g. almaMmsId should be renamed hebisId

Adjust transformation and add tests for hebis RPB-225

d52a7b6

Adjust transformation for fetching hebis records

c3d159e

Add workflow to fetch data for comparison from rpb

6127f70

Bump Metafacture Version

eaa1d0e

Update to current lobid transformation #225

5fdfe75

Add holdings from 924 #225

9fca285

TobiasNx requested a review from fsteeg January 17, 2025 14:33

TobiasNx assigned fsteeg Jan 17, 2025

TobiasNx added 5 commits January 17, 2025 18:08

Merge remote-tracking branch 'origin/main' into rpb-225-fetchAndTrans…

dcfa0c3

…formHebis

Update github actions build

c366a29

Add a suggestion for the workflow RPB-225

9f9e982

Rename folder for hebis transformation RPB-225

07fe5d8

Outcomment describedBy transformation RPB-225

deedcd5

This needs to be adjusted with regard to the strapi indexing.

TobiasNx requested a review from acka47 February 26, 2025 10:16

TobiasNx changed the title ~~Add first workflow to fetch HEBIS Data RPB-225~~ Add workflow basis to fetch HEBIS Data RPB-225 Feb 26, 2025

TobiasNx marked this pull request as ready for review February 26, 2025 10:16

TobiasNx commented Feb 26, 2025

View reviewed changes

acka47 assigned acka47 and unassigned fsteeg Feb 26, 2025

acka47 requested changes Feb 28, 2025

View reviewed changes

acka47 assigned TobiasNx and unassigned acka47 Feb 28, 2025

TobiasNx added 2 commits February 28, 2025 14:04

Update transformation to current lobid fix RPB-225

bb13c82

Add SRU Links to describedBy RPB-225

f73b85b

TobiasNx force-pushed the rpb-225-fetchAndTransformHebis branch from 702074f to f73b85b Compare February 28, 2025 13:14

TobiasNx commented Apr 1, 2025

View reviewed changes

fsteeg reviewed Apr 1, 2025

View reviewed changes

Get hebis data and call transformation in PUT route (RPB-225)

4a8c98f

To merge external hebis data with strapi record for indexing

fsteeg assigned dr0i and TobiasNx and unassigned fsteeg Apr 1, 2025

Fix holding infos RPB-225

49064c5

TobiasNx requested a review from fsteeg April 1, 2025 15:53

dr0i removed their assignment Apr 3, 2025

Add hasItem.id and retain items in merged hebis records (RPB-225)

58dd0f4

fsteeg requested review from dr0i and removed request for acka47 April 3, 2025 09:12

fsteeg assigned dr0i Apr 3, 2025

dr0i approved these changes Apr 3, 2025

View reviewed changes

app/controllers/nwbib/Application.java Outdated Show resolved Hide resolved

fsteeg added 3 commits April 3, 2025 14:52

Update test output for current hebis transformation (RPB-225)

9f916e9

Remove almaMmsId from hebis data, null values in general (RPB-225)

3e41238

Rename single-record lobid and hebis transformation files (RPB-225)

d174dce

fsteeg approved these changes Apr 3, 2025

View reviewed changes

fsteeg unassigned dr0i and TobiasNx Apr 3, 2025

fsteeg requested a review from acka47 April 3, 2025 15:53

fsteeg assigned acka47 Apr 3, 2025

acka47 approved these changes Apr 7, 2025

View reviewed changes

fsteeg merged commit 9074c38 into main Apr 8, 2025
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add workflow basis to fetch HEBIS Data RPB-225 #105

Add workflow basis to fetch HEBIS Data RPB-225 #105

TobiasNx commented Dec 2, 2024 •

edited

Loading

TobiasNx commented Jan 17, 2025 •

edited

Loading

TobiasNx Feb 26, 2025

TobiasNx Feb 26, 2025

acka47 commented Feb 28, 2025

acka47 Feb 28, 2025

acka47 Feb 28, 2025

TobiasNx Feb 28, 2025

TobiasNx Apr 1, 2025

fsteeg Apr 1, 2025

TobiasNx Apr 1, 2025

fsteeg commented Apr 1, 2025

dr0i commented Apr 3, 2025

fsteeg commented Apr 3, 2025

fsteeg commented Apr 3, 2025

fsteeg commented Apr 3, 2025

fsteeg commented Apr 3, 2025

acka47 left a comment

Add workflow basis to fetch HEBIS Data RPB-225 #105

Add workflow basis to fetch HEBIS Data RPB-225 #105

Conversation

TobiasNx commented Dec 2, 2024 • edited Loading

TobiasNx commented Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acka47 commented Feb 28, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fsteeg commented Apr 1, 2025

dr0i commented Apr 3, 2025

fsteeg commented Apr 3, 2025

fsteeg commented Apr 3, 2025

fsteeg commented Apr 3, 2025

fsteeg commented Apr 3, 2025

acka47 left a comment

Choose a reason for hiding this comment

TobiasNx commented Dec 2, 2024 •

edited

Loading

TobiasNx commented Jan 17, 2025 •

edited

Loading