Add function to store (grid) data as NetCDF

[peanutfun]

Early versions of #898 #857 provided functions to store the exceedance map data as NetCDF files. We decided to remove this feature from the PR(s).

@DahyannAraya has also been working on NetCDF output of Impacts. It might be worthwhile to consolidate these efforts.

Minimal Viable Feature
Provide a function that "rasterizes" a GeoDataFrame into an xarray DataArray or Dataset, and writes its data into a NetCDF file. Throw an error if the data cannot be rasterized (apparently is not gridded).

[bguillod]

This would be useful feature indeed.

My suggestion would be to write method to convert data (exceedance, hazard, impact, or whatever object is applies to) to and from xarray (with fixed conventions for each class) and let xarray deal with the reading/writing of the file.

More specifically: I suggest there are methods to_xarray() and from_xarray (or similar) only. The user can then read/write that xarray using xarray itself (e.g., xarray.Dataset.to_netcdf or xarray.Dataset.to_hdf5 or xarray.Dataset.to_zarr, xarray.open_dataset, ...). This would leave the user the option to store data in its preferred format.

[peanutfun]

@bguillod Thanks for the input. Indeed, xarray and its extensive functionality for storing and loading data seems to be the natural choice for that. As became apparent in #898, it is feasible to return computed data as GeoDataFrame. As Climada already uses this data structure extensively, we can start with a "conversion" from GeoDataFrame to xarray DataArray or Dataset and should cover most cases with that.

To consolidate any efforts already undertaken: Do you, by any chance, already have a code available that brings any Climada data structure into an xarray structure?

[hsteptoe]

I'm also very much in favour of an xarray interface. In the absence of any other volunteers I can probably find some to start this development...

[chahank]

That would be very much welcome! One thing to keep in mind overall is that xarray and netcdf do not really support sparse data. CLIMADA's success however relies on the use of sparse data structures for hazard events, impacts and exposures. If this can be somehow brought together, that would be very useful.

[peanutfun]

@DahyannAraya Please upload the functions you wrote for writing Impact to NetCDF here, as we just discussed personally 🙏

@chahank The feature request focuses on writing NetCDF data, not necessarily transforming Climada data structures into equivalent xarray structures and back. xarray will mostly be used as "proxy" to store the data. That being said, we already use xarray in conjunction with sparse within Hazard.from_raster_xarray, to store an intermediate, sparse representation of the raster data. Finally, NetCDF itself does not support sparse data structures. However, it supports "fill values" for missing data (typically NaN) and can store these very efficiently when compression is applied.

@hsteptoe Thanks a lot for volunteering! 😊 Let's clarify the exact feature request first, and then draft an implementation plan.

[spjuhel]

@ValentinGebhart possibly related to #1035 ?

[ValentinGebhart]

Thanks @spjuhel! Yes, this is the same problem, I was not aware of this issue. I assume there was no further progress about the implementation somewhere else @peanutfun @chahank ?

[chahank]

No that I know of. @bguillod can you provide such functionalities maybe?

[DahyannAraya]

Hey everyone,

As @peanutfun mentioned, I've already implemented a function called save_impact_data_to_NetCDF, which saves the impact: impact matrix (imp_mat) and environmental data (eai_exp) to a NetCDF file. It supports optional logarithmic scaling for both data types. This function was developed as part of one of the tutorials on scenario data sourced from Copernicus.

The function uses xarray for handling the data and writing it to NetCDF, with the time dimension stored via the time attribute in the impact object. Coordinates for latitude and longitude are taken from impact_object.coord_exp.

Additionally, I have also written reader functions to visualize and process this data once saved to NetCDF.

I haven’t pushed the code to CLIMADA yet, but this tutorial has already been reviewed. I can share the code or discuss it further. Let me know.

[DahyannAraya]

Hello, just for your information: I mentioned this to @chahank — I’ll be working with @NicolasColombi next week on the function I have in my project (planning, push, and pull request (PR) process) to see how much of it we can use to create something useful in line with what was requested. Everyone who wants to join is welcome. :)

[ValentinGebhart]

Thank you @DahyannAraya! In your functions, do you do any regridding (or even interpolation) of the spatial coordinates, or does it store the spatial information in 1D? I think in some way it makes sense to combine this with #1035.

[peanutfun]

Implementing #1035 would make it very easy to output data as NetCDF