Performance issues when reading large string arrays

[cbergemann]

Dear Vincent,

we have been experiencing performance issues when reading very large string arrays. Writing them happens in reasonable speeds that are similar to hdf5 native performance, but reading them is extremely slow. This is some test code that demonstrates the problem:

using PureHDF;

var dim1 = 100_000;
var dim2 = 200;

var data = new string[dim1, dim2];
for (var it1 = 0; it1 < dim1; it1++)
{
    for (var it2 = 0; it2 < dim2; it2++)
    {
        data[it1, it2] = $"String at position ({it1}, {it2})";
    }
}

var file = new H5File
{
    ["Dummy"] = data,
};

var stopwatch = System.Diagnostics.Stopwatch.StartNew();
file.Write("test_string_array.h5");
var time1 = stopwatch.Elapsed;
Console.WriteLine("Write Time: " + time1);

stopwatch.Restart();

using (var h5 = H5File.OpenRead("test_string_array.h5"))
{
    var readData = h5.Dataset("Dummy").Read<string[,]>();
    var time2 = stopwatch.Elapsed;
    Console.WriteLine("Read Time: " + time2);

    for (var it1 = 0; it1 < dim1; it1++)
    {
        for (var it2 = 0; it2 < dim2; it2++)
        {
            if (data[it1, it2] != readData[it1, it2])
            {
                throw new Exception("Data mismatch!");
            }
        }
    }

    Console.WriteLine("Data verified successfully.");
}

Using the current dev-branch, this results in the following output:

Write Time: 00:00:37.7613047
Read Time: 00:04:58.3444982
Data verified successfully.

It takes a good 5 minutes to read the dataset. Using h5read in MATLAB to do the same takes about 20 seconds on my machine.

Most of the time is spent (expectedly) in the DataTypeMessaging.GetDecodeInforForVariableLengthString::decode method. It does many calls to NativeCache.GetGlobalHeapObject and to OffsetStream.ReadDataset.
It does this for every single string, doing multiple read operations per string on the stream.

I have created a branch in which I do the following:

• Add a buffer in GlobalHeapCollection.Decode (which I think you already planned, judging by your TODO comment :))
• Add a buffered decoder for types where the total size is known beforehand (currently only for variable length strings, some of the other types can probably be sped up like this as well)

With these changes, the output is now

Write Time: 00:00:36.0785450
Read Time: 00:00:20.3966121
Data verified successfully.

I will create a pull request with the changes for you to review. It is all a bit crude, and I might be missing some edge-cases, but feel free to take it as a starting point. I am also open for feedback and can probably spend some more time on this.

Best,
Constantin