Navigating External Links?

[esaMark]

First, thanks so much for pureHDF - I have VB.NET code which I've been maintaining and extending for the last 20 years and when a colleague asked me to add HDF5 to the list of files the code can process, pureHDF got me off to a quick start on the problem. The file structure (over which I have no control) consists of a root file containing a number of external links (unknown in advance) which link to a set of groups containing metadata. Embedded in those groups are further external links to files containing the (image stack) data sets to be processed. So far I've been able to read out all the link names in all of the files and present them in a TreeView object. This is done using nested for each loops like the simple example below

for each link0 in file.Children
      'code to add link0.name to TreeNodes
       if  TypeOf link0 is IH5Group Then
                 NextGroup=CType(link0, IH5Group)
                 for each link1 in NextGroup.Children
                         'code to add link1.name to Child TreeNodes
                        ' .... more loops
                  Next
         End If
Next

The loops implicitly navigate the external links somehow so, at some level, the relative paths embedded in the files must be available to them.
However, I've had no success at reading attributes and datasets except from the root file for two reasons:

1. I can't see how to tell if a link is external - there seems to be no property which tells me this.
2. I can't see how to read out the relative paths embedded in the h5 files (other than writing code to read the file as bytes and find them myself) so that I can set up an H5LinkAccess object and use something like

Dim DataSet = group.Dataset(path, LinkAccess)

Any help with what I'm missing would be welcome.

[Apollo3zehn]

Hi, thanks for your report. I am not sure if I fully understand the problem. Why do you need a H5LinkAccess object? Is the external link path prefix depending on the relative path embedded in the external link, i.e. is it not constant? Or do you only want to provide a LinkAccess object when it is really an external link? It should have no effect when using it on other link types.

If you are allowed to, you could send a minimal sample to purehdf-151@m1.apollo3zehn.net which might help me to understand the problem and, if required, expose the missing data in the API.

[esaMark]

The files I have are huge so I've sent you the original root file plus a simplified external file with the same name and number of groups that there are external links in the root file, containing scalar string data. I didn't make the fake point to any further external files.
I'm probably missing something simple.

What I've called the root file contains a set of n external links to different groups in one external file in a sub-directory of the one containing the root file. (In general, it might not be safe to assume that all the groups were in the same external file but provided the paths are in the root file, that does not matter.) In the real case, n sub-groups in the external file each contain a further external link to an h5 file in a sub-sub directory containing a data file.

With nested loops as described in the first post I can read out all the local names of the groups in the entire file set and display them in a TreeView object. I can then read and display the attributes of each link using code like

 Private Shared Sub GetAttributes(ByRef AtrProp() As AttributeProperties,
                                  ByRef APC As Integer,
                                  ByRef Output() As String,
                                  ByRef NLines As Integer, link As IH5Object)
     For Each item In link.Attributes
         ReDim Preserve AtrProp(APC)
         AtrProp(APC) = New AttributeProperties(item)
         ReDim Preserve Output(NLines)
         Output(NLines) = AtrProp(APC).Name
         NLines += 1
         'If AtrProp(APC).TypeClass = H5DataTypeClass.VariableLength Or AtrProp(APC).TypeClass = H5DataTypeClass.String Then
         '    ReDim Preserve Output(NLines)
         '    Output(NLines) = item.Read(Of String)()
         '    NLines += 1
         'End If
         APC += 1
     Next
 End Sub

where AtrProp is an object of class AttributeProperties which looks like this:

Friend Class AttributeProperties
    Friend Name As String
    Friend Rank As Byte                             '=IH5DataSpace.Rank - Rank (dimensionality) of the data object
    Friend Dimensions As ULong()                      '=Ulong(Rank) - size in items of each dimension
    Friend DataSpaceType As H5DataspaceType '=H5DataSpaceType - Null=2, Scalar =0, Simple =1
    Friend TypeClass As H5DataTypeClass          'byte enumeration: Array = 10, Bitfield=4, Compound=6, Enumerated = 8, 
    '                                                                       'Fixed point = 0, FloatingPoint = 1, Opaque=5, Reference = 7, String = 3, Time = 2, 
    '                                                                       'VariableLength = 9
    Friend Size As Integer                                     '=IH5DataType.Size - Size of the data type in bytes

    Friend Sub New(A As IH5Attribute)
        Name = A.Name
        Rank = A.Space.Rank         
        Dimensions = A.Space.Dimensions
        DataSpaceType = A.Space.Type
        TypeClass = A.Type.Class
        Size = A.Type.Size
    End Sub
End Class

from which I should have enough information to read Attribute data (or with a similar Object, Dataset data) into the correct types.

However, I can get no further - for example, uncommenting the lines in the sub above throws an inner exception "file is too small" as soon as it has navigated outside the root file. I can't read any attributes or data except from the root file or a simple h5 file with no external links. For the root file I assume that to read the data or attributes at the end of a particular link I reconstruct the path from the root "/" by string concatenation like link0.name & "/" & link1.name & "/" ... or by using the TreeNode.FullPath property with the Path Separators set to "/". That seems to work. For an external file, I was assuming that I need to create a Link Access object and add it to the path from the root of the external file - something like

Dim LinkAccess = New H5LinkAccess("External\Link\Prefix")

And then

Dim DataSet = group.Dataset("path/from/externalfile/root", LinkAccess)

I can't make that work because I can't find any method of explicitly obtaining the external link paths from the root file although I know they are there, and they are embedded in the link objects as non public properties.

[Apollo3zehn]

Thanks for the sample. When I run the following code, I can successfully read the data of the datasets where the links point to.

using PureHDF;

string rootFilePath = "/path/to/blc16728_CeO2_NIST_Standard1.h5";
using var root = H5File.OpenRead(rootFilePath);

Console.WriteLine("Children from root:");

foreach (var child in root.Children())
{
    Console.WriteLine($"  Name: '{child.Name}', Type: {child.GetType().Name}");
    
    if (child is IH5Group group)
    {
        Console.WriteLine($"    Contents:");
        foreach (var item in group.Children())
        {
            Console.WriteLine($"      - {item.Name}");
            
            if (item is IH5Dataset dataset)
            {
                try
                {
                    var data = dataset.Read<string>();
                    Console.WriteLine($"        = '{data}'");
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"        Error: {ex.Message}");
                }
            }
        }
    }
}

Result:

Children from root:
  Name: '1.1', Type: NativeGroup
    Contents:
      - Data1
        = 'This is a String'
  Name: '2.1', Type: NativeGroup
    Contents:
      - Data1
        = 'This is a String'
  Name: '3.1', Type: NativeGroup
    Contents:
      - Data1
        = 'This is a String'
  Name: '4.1', Type: NativeGroup
    Contents:
      - Data1
        = 'This is a String'
  Name: '5.1', Type: NativeGroup
    Contents:
      - Data1
        = 'This is a String'
  Name: '6.1', Type: NativeGroup
    Contents:
      - Data1
        = 'This is a String'

Here is the content of the first external link:

The link consists of two main properties: the path to the external file (Ce02_NIST_Standard1_0001/CeO2_NIST_Standard1_0001.h5), and the path to the object within the external file (/1.1).

So the external file path is relative to the root file and thus no explicit external link prefix is needed. You can find here the file resolution logic, maybe it helps to understand what is happening.

So you are right that currently PureHDF gives no access to the link details. Do you still need them or does the code above also work for you?

[Apollo3zehn]

I forgot to mention that I had to rename the file the external link points to because I am on Linux with a case sensitive file system. The link expects CeO2_NIST_Standard1_0001.h5 and the actual file name in your zip file is CeO2_Nist_Standard1_0001.h5 (NIST vs Nist).

[esaMark]

Thanks for all that and sorry re the file name. I understand the H5LinkAccess object and link navigation a lot better and I ran your C# code and a VB version of it on the test structure I sent you with the same results as you got.

Yes, it would be nice to have access to the link details - the name and the link. The HDF5 viewer shows the name against the relevant group, as does the silx viewer and I spent a day or two trying to do the same so the C and python APIs presumably have accessors for these. They also mark the links as 'External' so there may be a boolean property for that also.

There's still a problem though. There's no obvious reason why the C# console app should not work on the first two levels of my original file set provided it does not try to read other data into strings. So I put an if statement around the read

if (dataset.Type.Class == H5DataTypeClass.VariableLength)
{
    var data = dataset.Read<string>();
    Console.WriteLine($"        = '{data}'");
}

and ran the code with strange results. It turned out that I had mistakenly identified my problem as navigating external links when it fact it was the reading of certain strings which was throwing exceptions: Since this is a new topic I'm going to mark your info above as answer and start again.