The work in #2805 introduces a way to recover resource maps when they've failed to upload after the metadata document was successfully saved (i.e. an orphaned EML doc was created). DataPackageRecovery can reconstruct a missing resource map from the last saved resource map (this is the fallback method when we can't find a resource map saved in browser storage). The current reconstruction starts with the previous resource map and advances members already in it to their latest versions. It cannot recover a new data object uploaded during the interrupted save because that PID was never in the previous map.
Data objects are uploaded before metadata, and the resource map is uploaded last. If the updated EML was saved before the resource map failed, its entity pids can provide evidence of new data objects that also committed.
For EML metadata, server recovery should:
- Fetch the EML doc version that was successfully saved
- Read PIDs from entities in the EML
- Verify each PID by getting the system metadata (use bounded concurrency like other parts of the data package loading code)
- May need to go back in version chain to see which version in the resource map the given PID obsoletes
- In the reconsructed resource map, use the PID referenced by the EML for that entity (instead of automatically using the latest version)
- Add verified data objects to the reconstructed resource map and add the reciprocal documentation relationship.
- Preserve the previous resource map's other members, locations, provenance, etc. PIDs found in EML should be additive. An entity's absence from EML is not proof that we should remove an existing resource map member. Only a 404 should mean that a referenced object is absent.
This recovery strategy may still lack some of the original resource map information, like prov:atLocation or other provenance relationships.
Authentication, network, and other temporary sysMeta lookup failures should leave recovery inconclusive and retryable.
We should determine what to do when a PID that is referenced in the EML does not exist. Probably could warn the user and allow them to continue with the recovery, but not include that object in the reconstructed resource map, or we could fail the recovery and require data team support to resolve the issue.
Example case: An interrupted save adds new.csv with pid abc123. The data object and updated EML are saved successfully, but the resource map does not. The user tries to access the dataset from another browser where there is no local storage of the attempted resource map. During recovery, the EML is downloaded and we see it contains the new PID abc123. We successfully fetch the system metadata for abc123 and add it to the reconstructed resource map.
The work in #2805 introduces a way to recover resource maps when they've failed to upload after the metadata document was successfully saved (i.e. an orphaned EML doc was created).
DataPackageRecoverycan reconstruct a missing resource map from the last saved resource map (this is the fallback method when we can't find a resource map saved in browser storage). The current reconstruction starts with the previous resource map and advances members already in it to their latest versions. It cannot recover a new data object uploaded during the interrupted save because that PID was never in the previous map.Data objects are uploaded before metadata, and the resource map is uploaded last. If the updated EML was saved before the resource map failed, its entity pids can provide evidence of new data objects that also committed.
For EML metadata, server recovery should:
This recovery strategy may still lack some of the original resource map information, like
prov:atLocationor other provenance relationships.Authentication, network, and other temporary sysMeta lookup failures should leave recovery inconclusive and retryable.
We should determine what to do when a PID that is referenced in the EML does not exist. Probably could warn the user and allow them to continue with the recovery, but not include that object in the reconstructed resource map, or we could fail the recovery and require data team support to resolve the issue.
Example case: An interrupted save adds
new.csvwith pidabc123. The data object and updated EML are saved successfully, but the resource map does not. The user tries to access the dataset from another browser where there is no local storage of the attempted resource map. During recovery, the EML is downloaded and we see it contains the new PIDabc123. We successfully fetch the system metadata forabc123and add it to the reconstructed resource map.