Skip to content

OAI_ORE and Archival Bag updates #12063

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
qqmyers wants to merge 30 commits into
base: develop
Choose a base branch
from
Open

OAI_ORE and Archival Bag updates #12063

qqmyers wants to merge 30 commits into from
+878 −193

Conversation

@qqmyers
Copy link
Member

@qqmyers qqmyers commented Dec 19, 2025 •
edited
Loading

What this PR does / why we need it: This PR contains multiple updates to the OAI-ORE metadata export and archival Bag output:

OAI-ORE

  • now uses URIs for checksum algorithms, as required to create valid JSON-LD
  • a bug causing failures with deaccessioned versions when the deaccession note ("Deaccession Reason" in the UI) was null (which has been allowed via the API).
  • the "https://schema.org/additionalType" is updated to "Dataverse OREMap Format v1.0.2" to indicate that the out has changed

Archival Bag

  • for dataset versions with no files, the (empty) manifest-.txt file created will now use the default algorithm defined by the "FileFixityChecksumAlgorithm" setting rather than always defaulting to "md5"
  • a bug causing the bag-info.txt to not have information on contacts when the dataset version has more than one contact has been fixed
  • values used in the bag-info.txt file that may be multi-line (with embedded CR or LF characters) are now properly indented/formatted per the BagIt specification (i.e. Internal-Sender-Identifier, External-Description, Source-Organization, Organization-Address).
  • the name of the dataset is no longer used as a subdirectory under the data directory (dataset names can be long enough to cause failures when unzipping)
  • a new key, "Dataverse-Bag-Version" has been added to bag-info.txt with a value "1.0", allowing tracking of changes to Dataverse's acrhival bag generation
  • improvements to handling file retrieval errors and throttling

Which issue(s) this PR closes:

(I assume these don't work to automate closing, but they're here to let DANS see the correlations)

  • Closes #DD-2109
  • Closes #DD-2082
  • Closes #DD1508
  • Closes #DD2112

Special notes for your reviewer: FWIW: Some of the simpler fixes are in a single commit if you want to look at them separately.

Suggestions on how to test this: There are new tests for the multiline wrapping functionality and overall generation of the bag-info-txt file. Testing overall bag generation means configuring an archiver per the guides (probably the local file one) and verifying the output. The main changes to check are in the OAI-ORE file, bag-info file, and that there is no data/* dir using the dataset title. One could also verify that the manifest file name with no files matches the configured algorithm.
Given that DANS is interested in these, I assume they'll also be testing - could coordinate.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?: note added

Additional documentation:

qqmyers and others added 30 commits December 6, 2025 18:26
@qqmyers
add checksum URI values and methods
c9f728b
@qqmyers
update version and use checksum URIs
a25e47b
@qqmyers
handle multiline descriptions and org names
6c0cb49
@qqmyers
drop blank lines in multiline values
7a34db8 
Spec doesn't allow empty lines, dropping whitespace-only lines seems
reasonable as well (users can't see from the Dataverse display whether
an empty line would appear in bag-info.txt or not if we all whotespace
only lines (or whitespace beyond the 78 char wrap limit)
@qqmyers
remove title as a folder
b0daad7 
affects manifest and pid-mapping files as well as data file placement
@qqmyers
handle null deaccession reason
e5457a8
@qqmyers
use static to simplify testing
10b0556
@qqmyers
Merge remote-tracking branch 'IQSS/develop' into OREBag1.0.2
d6cf1e2
@qqmyers
Sanitize/split multiline catalog entry, add Dataverse-Bag-Version
6d24185
@janvanmansum
Added unit tests for multilineWrap
c4daf28
@janvanmansum
Removed unnecessary repeat helper method
e76bc91
@janvanmansum
Alined test names with actual test being done
108c912
@qqmyers
Merge pull request #48 from janvanmansum/OREBag1.0.2-amend
62ea9d9 
Added unit tests for multilineWrap
@qqmyers
DD-2098 - allow archivalstatus calls on deaccessioned versions
884b81b
@qqmyers
Merge remote-tracking branch 'IQSS/develop' into OREBag1.0.2
5e4e90a
@qqmyers
set array properly
3076d69
@qqmyers
Merge remote-tracking branch 'IQSS/develop' into OREBag1.0.2
cbdc15f
@qqmyers
DD-2212 - use configured checksum when no files are present
1a7dafa
@qqmyers
Revert "DD-2098 - allow archivalstatus calls on deaccessioned versions"
7eea57c 
This reverts commit 884b81b.
@qqmyers
add Source-Org as a potential multiline case, remove change to Int Id
2477cf9
@qqmyers
release note
3f3908f
@qqmyers
use constants, pass labelLength to wrapping, start custom lineWrap
aa44c08
@qqmyers
update to handle overall 79 char length
8227edf
@qqmyers
wrap any other potentially long values
d0749fc
@qqmyers
cleanup deprecated code, auto-gen comments
24a625f
@qqmyers
update comment
bf036f3
@qqmyers
add tests
be65611
@qqmyers
Merge remote-tracking branch 'IQSS/develop' into OREBag1.0.2
2516cf4
@qqmyers
QDR updates to apache 5, better fault tolerance for file retrieval
24d098a
@qqmyers
release note update
b4a3799
@qqmyers qqmyers added this to the 6.10 milestone Dec 22, 2025
@qqmyers qqmyers added Size: 10 A percentage of a sprint. 7 hours. GDCC: DANS related to GDCC work for DANS TDL of interest to the Texas Digital Library GDCC: QDR of interest to QDR labels Dec 22, 2025
@qqmyers qqmyers marked this pull request as ready for review December 22, 2025 22:16
@coveralls
Copy link

coveralls commented Dec 22, 2025 •
edited
Loading

Coverage Status

coverage: 24.418% (+0.2%) from 24.232%
when pulling b4a3799 on GlobalDataverseCommunityConsortium:OREBag1.0.2
into 96e96f4 on IQSS:develop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

GDCC: DANS related to GDCC work for DANS GDCC: QDR of interest to QDR Size: 10 A percentage of a sprint. 7 hours. TDL of interest to the Texas Digital Library

Projects

IQSS Dataverse Project
Status: No status

Milestone

6.10

Development

Successfully merging this pull request may close these issues.

3 participants

@qqmyers @coveralls @janvanmansum