cdffs

A file system interface (cdffs) to allow users to work with CDF Files using the fsspec supported/compatible python packages (pandas, xarray etc).

fsspec provides an abstract file system interface to work with local/cloud storages and based on the protocol name (example, s3 or abfs) provided in the path, fsspec translates the incoming requests to storage specific implementations and send the responses back to the upstream package to work with the desired data.

Refer fsspec documentation to get the list of all supported/compatible python packages.

Installation

cdffs is available on PyPI. Install using,

pip install cognite-cdffs

Usage

Important steps to follow when working with CDF Files using the fsspec supported python packages.

1. Import cdffs package

  from cognite import cdffs  # noqa

2. Follow instructions from Authentication to authenticate.

3. Read/write the files from/to CDF using fsspec supported packages. Example,

   • Read zarr files using using xarray.

   ds = xarray.open_zarr("cdffs://sample_data/test.zarr")
   • Write zarr files using xarray.

   ds.to_zarr("cdffs://sample_data/test.zarr", storage_options={"file_metadata": metadata})

Refer cdffs.readthedocs.io for more details.

Vendoring

cdffs uses pydandic.v1 package using vendoring. It was mainly introduced to overcome version conflicts related to the Cognite Notebooks.

Contributing

Want to contribute? Check out CONTRIBUTING.