Writing data as a stream

[davepruitt]

I am aware that PureHDF currently does not support "streaming", but instead uses "deferred writing" (and the final structure of the file must be known beforehand).

I am writing an application in which I need an HDF5 library that is 100% managed code (like this library) because I need it to run on any platform, so I don't want dependencies on the compiled DLLs. I also need to be able to stream data to a file without knowing how big the file will be at the end (I don't know how many samples/frames I am streaming beforehand), and if possible I'd like the file to still maintain data integrity in case of a crash.

Is there any possibility that this library may be able to handle writing data as a stream of an unknown length anytime in the near future?

[Apollo3zehn]

The problem is that when you want to use compression filters, it is necessary to chunk the data. And to later be able to find these chunks in the file we have to create an index for these chunks. The index type depends on the data shape (there are 5 different index types). The size of the index depends on the number of chunks and will grow with more chunk coming in. And if the index growths, it needs to be moved from its current position in the file to the end of the file, creating a hole. With many datasets this becomes a problem and needs free space management and also internal pointers in the file need to be updated which is in total too much for my spare time. This limitation applies when you want to close the file and reopen it later to add a new chunk.

However, PureHDF currently keeps the chunk index in memory and only flushes it to the file when the file stream is about to be closed which is a good precondition for allowing an unlimited number of chunks, i.e. streaming of data.

I think for compressed datasets PureHDF is always using the fixed array chunk index which is being instantiated here in the Dispose method of H5D_Chunk4_FixedArray. This method is called when the file is about to be closed. The FixedArrayHeader depends on the variable elements which is an array of pointers to the chunks in the file. The size of the array is WriteChunkInfos.Length, i.e. it depends on the size of WriteChunkInfos which is defined in the base class. The size of WriteChunkInfos in turn depends on the dimensionality of the dataset and the chunk dimensions.

Minimal changes required to support a growing dataset:

• Convert the fixed size WriteChunkInfos array into a List<> which can grow
• The elements of the WriteChunkInfos array are initialized whenever a chunk is being accessed for writing and that element is not yet initialized. So we would need to think about what should happen if the user accesses a chunk with an index larger than the current size of the list. E.g. the list of chunk infos has 10 elements but we want to access the 20th element. I think it is necessary to fill the in-between indices with dummy elements with the element address set to Superblock.UndefinedAddress
• When the user creates a dataset, check for the presence of H5Constants.Unlimited in one of its dimensions. If H5Constants.Unlimited is present in the fastest changing dimension (i.e. the last dimension (example: [10, 20, H5Constants.Unlimited])) we will continue as usual. But when the file is about to be closed, the final number of chunks has to be provided to the DataspaceMessage so that the actual dataset size is being written to the file instead of H5Constants.Unlimited. However, I think the dataspace message is being flushed to disk before the chunk index so we have to overwrite the dimension field later by seeking to the correct position in the file which is ugly but possible.
• There are probably some array boundary checks in the public API which need to be adapted to account for H5Constants.Unlimited being allowed in the last dimension.

Unfortunately I have some high priority tasks at work right now so that I can not invest into PureHDF for a few months :-(

Of course you can try implementing it yourself, depending how big your motivation is to solve this problem :-)