Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add emmet-archival namespace package #1190

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

esoteric-ephemera
Copy link
Collaborator

Summary

Adds an archival namespace package to emmet to establish MP's data archival formats moving forward. This package should be flexible enough to permit new data formats as they become relevant (e.g., data from phonopy, LOBSTER, etc.)

  • Base Archiver class that defines a schema for how data is to be archived / retrieved from HDF5 + zarr files
  • VASP RawArchive class:
    • Stores only relevant VASP files as string / bytes archives in hierarchical format
    • Automatically strips copyright protected info from VASP files
    • Constructs TaskDoc from HDF5/zarr data
  • VASP VolumetricArchive + separate volumetric data classes: formats for converting CHGCAR-structured files, bandstructures, and DOS to HDF5/zarr
  • TrajArchive for storing trajectory information. Agnostic to traj format type

To Do's:

  • Tests tests tests
  • Extend TrajArchive to strip ionic step info from a TaskDoc
  • Test formats on a subset of VASP data on S3 to estimate storage size, partial retrieval, etc.
  • Parquet extensions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant