writing to MultiContainerInterface extension results in orphaned container

[bendichter]

Am I doing this wrong? I may be doing this wrong. Here's the minimal example:

from pynwb.spec import NWBNamespaceBuilder, NWBGroupSpec, NWBAttributeSpec
from pynwb import load_namespaces, NWBFile, NWBHDF5IO, get_class


ext_name = 'test_ext'
ns_path = ext_name + ".namespace.yaml"
ext_source = ext_name + ".extensions.yaml"

elem = NWBGroupSpec(neurodata_type_inc='NWBDataInterface',
                    neurodata_type_def='Elem',
                    quantity='+',
                    doc='doc',
                    attributes=[NWBAttributeSpec(name='data', doc='data',
                                                 dtype='double', required=True)])

grouper = NWBGroupSpec(neurodata_type_inc='NWBDataInterface',
                       neurodata_type_def='Grouper',
                       quantity='?',
                       doc='doc',
                       groups=[elem])

ns_builder = NWBNamespaceBuilder(ext_name + ' extensions', ext_name)

ns_builder.add_spec(ext_source, grouper)
ns_builder.export(ns_path)
load_namespaces(ns_path)

Grouper = get_class('Grouper', ext_name)
Elem = get_class('Elem', ext_name)

elem = Elem(data=2.0, name='elem', source='source')
group = Grouper(elems=elem, name='group', source='source')

nwbfile = NWBFile("source", "a file with header data", "NB123A", '2018-06-01T00:00:00')
mod = nwbfile.create_processing_module(name='module', source='source', 
                                       description='description')

mod.add_container(group)

with NWBHDF5IO('test_group.nwb', 'w') as io:
    io.write(nwbfile)

Using HDFView, there is no elem, and no elem.data. Same issue if I write the Grouper class myself.

[ageorgou]

When I run this and open the file in HDFView (v. 2.14), I do see an elem inside the group, but it has no attributes. It seems that it is written as a soft link. Here is the output from h5dump (running h5dump -g processing/module/group test_group.nwb):

HDF5 "test_group.nwb" {
GROUP "processing/module/group" {
   ATTRIBUTE "namespace" {
      DATATYPE  H5T_STRING {
         STRSIZE H5T_VARIABLE;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_UTF8;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "test_ext"
      }
   }
   ATTRIBUTE "neurodata_type" {
      DATATYPE  H5T_STRING {
         STRSIZE H5T_VARIABLE;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_UTF8;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "Grouper"
      }
   }
   ATTRIBUTE "source" {
      DATATYPE  H5T_STRING {
         STRSIZE H5T_VARIABLE;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_UTF8;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "source"
      }
   }
   SOFTLINK "elem" {
      LINKTARGET "/elem"
   }
}
}

There are no other contents called elem in the file.

[ajtritt]

I see the same thing @ageorgou sees.

The autogenerated class Grouper is not making the Elems children (i.e. not setting parent), hence the OrphanContainerWarning. When the parent of a child (in our case None) does not match the container holding reference to said child, it is assumed to be an external child, so a link is set. In this case, a link is made to something that doesn't exist. The Elem instance (i.e. elem) doesn't exist because it doesn't get visited again during the recursive building of containers. You don't see any attributes on elem because it's a broken soft-link.

[ajtritt]

The fix for this is to detect the inclusion of multiple Elems in Grouper, and set include "child": True in the __nwbfields__ class attribute of the autogenerated class. You can see the note to myself to do this here:

pynwb/src/pynwb/form/build/map.py

Line 989 in 6d2cc9f

# TODO: set __nwbfields__

The "work around" would be to define your own class. This is not just a work around, but it's the recommended way for extending. Writing your own classes is the safest way to extend NWB.

[bendichter]

I was able to get it working by building my own class

from pynwb.spec import NWBNamespaceBuilder, NWBGroupSpec, NWBAttributeSpec
from pynwb import load_namespaces, NWBFile, NWBHDF5IO, get_class
from pynwb.file import MultiContainerInterface, register_class

ext_name = 'test_ext'
ns_path = ext_name + ".namespace.yaml"
ext_source = ext_name + ".extensions.yaml"

elem = NWBGroupSpec(neurodata_type_inc='NWBDataInterface',
                    neurodata_type_def='Elem',
                    quantity='+',
                    doc='doc',
                    attributes=[NWBAttributeSpec(name='data', doc='data',
                                                 dtype='double', required=True)])

grouper = NWBGroupSpec(neurodata_type_inc='NWBDataInterface',
                       neurodata_type_def='Grouper',
                       quantity='?',
                       doc='doc',
                       groups=[elem])

ns_builder = NWBNamespaceBuilder(ext_name + ' extensions', ext_name)

ns_builder.add_spec(ext_source, grouper)
ns_builder.export(ns_path)


load_namespaces(ns_path)

Elem = get_class('Elem', ext_name)


@register_class('Grouper', ext_name)
class Grouper(MultiContainerInterface):

    __clsconf__ = {
        'attr': 'elems',
        'type': Elem,
        'add': 'add_elem',
        'get': 'get_elem',
        'create': 'create_elem',
    }

    __help = 'test grouper'


elem1 = Elem(data=2.0, name='elem', source='source')
elem2 = Elem(data=3.0, name='elem2', source='source')

group = Grouper(name='group', source='source')
group.add_elem(elem1)
group.add_elem(elem2)

nwbfile = NWBFile("source", "a file with header data", "NB123A", '2018-06-01T00:00:00')
mod = nwbfile.create_processing_module(name='module', source='source',
                                       description='description')

mod.add_container(group)

with NWBHDF5IO('test_group', 'w') as io:
    io.write(nwbfile)

I think the best approach here would be to correct get_class so it generates the class correctly. I know you prefer custom classes but in cases where it's a straightforward constructor I think the auto-generated classes make sense because that means the nwb file can be entirely self-contained, and no external python libraries need to be maintained and distributed. If we can't support certain types of classes that's ok, but then I don't think we should be auto-generating classes that don't work. We should catch these cases and send a message to the user that they need to create a constructor themselves- at least until we get around to correcting the auto-generation feature for these types of classes.