Packaging OpenEphys data: "Your data may be transposed"

[JustinKillebrew]

I'm using an ElectricalSeries to save an OpenEphys stream:

        import spikeinterface.extractors as se
        recording = se.read_openephys(folder_path, stream_name=s, load_sync_timestamps=True)
        numSeg = recording.get_num_segments()
       
        for segNum in range(numSeg):
            # create an ElectricalSeries for raw voltage data
            ts = ElectricalSeries(
                    name=f'{s}_{segNum}',
                    description=s,
                    data=recording.get_traces(segNum),
                    timestamps=recording.get_times(segNum),
                    electrodes=all_table_region
            )

        # add the TimeSeries to the NWB 
        nwbfile.add_acquisition(ts)

But the data structure is a n x 1 ndarray which causes a warning "The second dimension of data does not match the length of electrodes. Your data may be transposed."

Is this an acceptable method for repackaging OpenEphys data? Do I need to rebuild the data so that it has the correct 2nd dimension? If so, how would I do that? Simply reshape the ndarray so that it has n x numElectrodes? Recording.get_times(segNum) also returns an n x 1 ndarray.

There's also a warning for some of the streams:
" UserWarning: Could not load synchronized timestamps for Record_Node_101#NI-DAQmx-104.PXIe-6341"

I there a problem with the OpenEphys data structure?

Are there any examples/tutorials for converting multi-stream, multi-segment OpenEphys data?

Thanks,
JK

[h-mayorquin]

Hi, what does the data represents? If the data is indeed (samples x 1) because there is only one electrode that should be fine.

I suggest that you use the interface of neuroconv which will take care of the multi-segment.Plus, you will get compression and the right chunking by default.

We don't have a multi-stream converter yet.

[JustinKillebrew]

Hello and thanks. The data are from a 384-channel neuropixel probe. The streams are action potentials, LFP, events, and 2 sync streams.

When I run the neuroconv OpenEphysRecordingInterface example (modified for my data) I get:
ValueError: More than one stream is detected! Please specify which stream you wish to load with the stream_name argument. To see what streams are available, call OpenEphysRecordingInterface.get_stream_names(folder_path=...).

So I get the available streams and then for each stream, try to run_conversion() but there are duplicate names:

interface.run_conversion(nwbfile_path=nwbfile_path, metadata=metadata, append_on_disk_nwbfile=True)

ValueError: Cannot add <class 'pynwb.ecephys.ElectricalSeries'> 'ElectricalSeries0' at 0x139109841655744 to dict attribute 'acquisition' in <class 'pynwb.file.NWBFile'> 'root'. <class 'pynwb.ecephys.ElectricalSeries'> 'ElectricalSeries0' at 0x139109859644368 already exists in 'acquisition' and has the same name.

How do I convert multi-stream, multi-segment OpenEphys data?

Thanks again!
JK

[h-mayorquin]

Is there any way you can share your data? I might be able to put together a converter

[JustinKillebrew]

I need permission, but that shouldn't be a problem. I think I'll need your email address to share the data via OneDrive. Or is there another upload site?

Thank you.
JK

[h-mayorquin]

My email is h.mayorquin@gmail.com One Drive should work just fine.

[h-mayorquin]

ValueError: Cannot add <class 'pynwb.ecephys.ElectricalSeries'> 'ElectricalSeries0' at 0x139109841655744 to dict attribute 'acquisition' in <class 'pynwb.file.NWBFile'> 'root'. <class 'pynwb.ecephys.ElectricalSeries'> 'ElectricalSeries0' at 0x139109859644368 already exists in 'acquisition' and has the same name.

How do I convert multi-stream, multi-segment OpenEphys data?

Can you open an issue about this in neuroconv?

[h-mayorquin]

Here, let's continue it here:

catalystneuro/neuroconv#1686

[h-mayorquin]

It would be great if you can install the current branch on that PR and test it so we know if it works for your case.

[JustinKillebrew]

Ok, I created a fresh conda env and then did:
pip install git+https://github.com/catalystneuro/neuroconv.git@add_open_ephys_binary_converter

This runs with a warning:

from neuroconv.converters import OpenEphysBinaryConverter
folder_path = "/home/justin/testData/2025-03-06_14-30-33/Record_Node_101/"

converter = OpenEphysBinaryConverter(folder_path=folder_path)

nwbfile = converter.create_nwbfile()
nwbfile 
 

pynwb/file.py:492: UserWarning: Date is missing timezone information. Updating to local timezone.
  args_to_set['session_start_time'] = _add_missing_timezone(session_start_time)

converter._stream_names returns the 5 streams contained in the dataset.
Now I'm running it with

configure_and_write_nwbfile(
    nwbfile, nwbfile_path="./JK_test5.nwb", backend="hdf5"

It's been running for over an hour but the nwb file is growing so I think it's working.

What other tests should I run?

Thank you!
JK

[h-mayorquin]

Yes, check that everything in the file is as you expect. The number of streams, the number of samples, the start times or timestamps of the data on the series, number of electrodes.

Basically, one you convert the file, read the NWB file and check that 1) all the metadata and the experimental data that you expect is present and 2) it is sound.

[JustinKillebrew]

I was able to create an nwb file and then add metadata from mat files and export it. I think everything is where it needs to be.

Thanks for creating the new converter.

JK