HDF5.Open() stream does not support asynchronous reading.

[roach-korea]

Hi.

I am trying to read data using PureHDF in a WebAssembly (WASM) environment.

In a browser, data is always read asynchronously. However, PureHDF attempts to read the data synchronously when using a stream, which causes an exception.

Are there any plans to support asynchronous reading?

[Apollo3zehn]

Hi @roach-korea, this is a very nice use case I was looking for as well and I put high amount of effort in supporting async but it was a rabbit hole and made the existing code base really really complex and caused headaches so that I finally removed everthing async related again in this commit: 3b6e152

I was never able to test this, but running .NET in a webworker should allow you to use the synchronuous API.

[TheCollegedude]

Hi, since the ...Async methods are still there but throwing NotImplementedExceptions: Are there any plans to bring it back?

[Apollo3zehn]

Sorry, I did not see your post until now. The async APIs are there because there are two implementations of the HDF5 API, one for native HDF5 files and another one for HSDS which supports async. This was a try to mimic what the HDF5 lib does with its different virtual file drivers but it seems to be more confusing than helpful. I do not plan to bring async support back because I was unable to implement it properly. IIRC the main problem was with the interaction of Memory<T> (async) and Span<T> (sync). It is easy to e.g. cast Span<int> to Span<byte> and vice versa but inefficient to do the same with Memory<T> because it requires an allocation for every cast. This plus problems with async streams made me remove the async support at some point. However, I would accept a PR if somebody is willing to give it another try to find an elegant solution.

[roach-korea]

Great news! @Apollo3zehn

HDF5 2.0 now officially supports native asynchronous operations. I was wondering if this update would be helpful or relevant for the async support roadmap in PureHDF?

[Apollo3zehn]

Hi @roach-korea thanks for the hint but it unfortunately does not change the implementation challenges in PureHDF. The native library does not use generics and they use no safe language which makes the async implementation for C a bit different compared to C#. Some things are harder and some are simpler. Currently I do not see how to implement it in an elegant way. See my answer to @TheCollegedude for a bit more details. My hope for WASM lies in simple support for Webworkers as they allow for synchronous stream access: dotnet/aspnetcore#17730 (comment)

[Apollo3zehn]

Maybe PureHDF can be used in combination with https://github.com/Tewr/BlazorWorker ?