Expanding a dataset

[vinnieg]

The documentation mentioned 'reshape' for a dataset. How do you do this? Otherwise, can you add a dataset to a file once it is created? (i.e. not with an initialization list)

Thanks

[Apollo3zehn]

Sorry, I overlooked your message in my mail inbox. With reshape I want to say that you can have a 1-D array with e.g. 10.000 elements but treat this array as e.g. 2x5000 2D array. It is just another view of the same underlying data, no copying or rearranging of data is involved. In terms of PureHDF this means that you can change the shape of your input array so that in the HDF5 file the shape is different. As long as the total number of array elements does not change, you can provide any dimensions you want.

Otherwise, can you add a dataset to a file once it is created? (i.e. not with an initialization list)

You cannot add a dataset to the file once it has been created because that would means that internal HDF5 structures would have to be modified which is quite a complex task (because the modified structure is likely bigger than before and so it will have to be moved to the end of the file leaving a hole in the file and references need to be updated).

The only thing supported is to create a chunked dataset and write the chunk contents later to the file.

[DSanchen]

Hi,

You cannot add a dataset to the file once it has been created....

The only thing supported is to create a chunked dataset and write the chunk contents later to the file.

And with a chunked dataset, is it possible, to define a "unlimited" dimension along the file can grow as large as it needs to be ? E.g. I would like to add an unknown number of fixed sized images to the file, so file dimensions would be ["unlimited", 100, 200] for 100x200 pixel images. And then I would use an own counter to select the current image I would write with a hyperslab at [imageCounter, 0, 0], block[imageCounter, 100, 200] to write one single image.

Is this possible with the current implementation ?

[Apollo3zehn]

This would also mean that new chunks have to be created which in turn modifies the chunk index which is not supported and probably never will as this requires a huge investment :-( The current design philosophy is that everything is known at the time of writing so all internal file addresses have a real value and it can be flushed to disk in one go.