How to modify the items of an Electrodes DynamicTable?

[TommasoLambresa]

I have two questions regarding the handling of the Electrodes table in NWB, for which I could not find answers in the documentation:

1. Modifying the Electrodes Table

• After converting an Intan file to NWB, I want to modify the 'location' column of the Electrodes table by updating its values with a different string.
• Since the DynamicTable does not support direct assignment (as it has no setter), is there a way to modify the items or even remove columns from the table?
• This tutorial does not seem to cover this use case. Could this functionality be documented for future users?

2. Managing Multiple Electrodes Tables

• Is the Electrodes table intended to be a single table containing all electrodes from an experiment, or can multiple tables be used?
• Specifically, when computing local field potentials (LFPs) from high-density probes (e.g., Neuropixels, SiNAPS), signals are often averaged across electrodes along a specific spatial direction (e.g., x-direction of a shank). This results in LFP signals that are no longer tied to individual electrodes but rather to groups of electrodes.
• Given this, is there a way to create a separate Electrodes table to reference these "virtual" electrodes created by averaging? If not, what would be the recommended approach for managing such derived signals?

[stephprince]

@TommasoLambresa Thanks for the outlining the questions so clearly - there are a couple of different points here so I'll try to address them fully.

After converting an Intan file to NWB, I want to modify the 'location' column of the Electrodes table by updating its values with a different string.

You can replace the data in a column in it's entirety with the following code:

with NWBHDF5IO("test_modify_electrodes_table.nwb", mode="a") as io:
    nwbfile = io.read()
    new_locations = ["brain_area_a", "brain_area_b"]
    nwbfile.electrodes.location.data[:] = new_locations  # must match the length of the existing location data

Since the DynamicTable does not support direct assignment (as it has no setter), is there a way to modify the items or even remove columns from the table?

It is possible to add new columns or rows to the electrodes DynamicTable. However removing columns with pynwb is a bit more difficult and is an open issue. There are ways to do this with h5py directly, but it would need to be done carefully. Let me know if you have a specific use case and we can discuss more.

# in append mode, add a new row
nwbfile.add_electrode(group=electrode_group, location="brain_area_c",)

# in append mode, add a new column
nwbfile.add_electrode_column("label", "description of electrodes", data=["ch0", "ch1", "ch2"])

This tutorial does not seem to cover this use case. Could this functionality be documented for future users?

Good point, I think a section in that tutorial on editing DynamicTables would be great to add!

Is the Electrodes table intended to be a single table containing all electrodes from an experiment, or can multiple tables be used?

Yes, the electrodes table is intended to contain all electrodes. When creating ElectricalSeries you can select subsets of these electrodes by creating a DynamicTableRegion that references specific rows in the electrodes table. However, it is best practice that the number of electrodes referenced by an ElectricalSeries and the second dimension of the data are the same.

To follow this best practice while addressing the specific case you mentioned about computing LFPs from high-density probes, one approach would be to create a separate entry in the electrodes table for each "virtual" electrode. You could add an additional column to the electrodes table that describes which electrodes were included in this virtual electrode and how they were linked.

Below is a script to highlight the different functionality discussed above. Let me know if you have any further questions!

import numpy as np

from datetime import datetime
from uuid import uuid4
from dateutil import tz

from pynwb import NWBHDF5IO, NWBFile, H5DataIO
from pynwb.ecephys import LFP, ElectricalSeries

# setup nwbfile
nwbfile = NWBFile(
    session_description="Mouse exploring an open field",
    identifier=str(uuid4()),
    session_start_time=datetime(2025, 3, 7, 3, 30, 3, tzinfo=tz.gettz("US/Pacific")),
)

# create device
device = nwbfile.create_device(name="array", description="an array", manufacturer="company")

# create an electrode group
electrode_group = nwbfile.create_electrode_group(
    name="shank0",
    description="electrode group for shank 0",
    device=device,
    location="brain area",
)

# add electrodes to the electrode table
n_electrodes = 4
for ch_ix in range(n_electrodes):
    nwbfile.add_electrode(
        group=electrode_group,
        location="brain area",
    )

# make the fields expandable 
# NOTE: this step is only required if you want to append additional rows later on
# appending additional columns or modifying existing column data is possible without this step
nwbfile.electrodes.id.set_data_io(H5DataIO, {'maxshape': (None,)})
nwbfile.electrodes.group.set_data_io(H5DataIO, {'maxshape': (None,)})
nwbfile.electrodes.group_name.set_data_io(H5DataIO, {'maxshape': (None,)})
nwbfile.electrodes.location.set_data_io(H5DataIO, {'maxshape': (None,),})

# create an table region with all the electrodes
all_table_region = nwbfile.create_electrode_table_region(
    region=list(range(n_electrodes)),  # reference row indices 0 to N-1
    description="all electrodes",
)

# add raw electrical series to the nwbfile
raw_electrical_series = ElectricalSeries(
    name="series0",
    description="Raw acquisition traces",
    data=np.random.randn(50, n_electrodes),
    electrodes=all_table_region,  # references all electrodes in the table
    starting_time=0.0,  # timestamp of the first sample in seconds relative to the session start time
    rate=20000.0,  # in Hz
)
nwbfile.add_acquisition(raw_electrical_series)

# write initial file
with NWBHDF5IO("test_modify_electrodes_table.nwb", "w") as io:
    io.write(nwbfile)

# open the file in append mode to modify the electrodes table
with NWBHDF5IO("test_modify_electrodes_table.nwb", mode="a") as io:
    nwbfile = io.read()
    
    # replace data in the location column
    new_locations = ["brain_area_a", "brain_area_a", 'brain_area_b', 'brain_area_b']
    nwbfile.electrodes.location.data[:] = new_locations
    
    # add a new column
    nwbfile.add_electrode_column("label", "description of electrodes", data=["ch0", "ch1", "ch2", "ch3"])
    nwbfile.electrodes.label.set_data_io(H5DataIO, {'maxshape': (None,)})

    # add a new row
    nwbfile.add_electrode(
        group=nwbfile.electrode_groups["shank0"],
        location="brain_area_c",
        label="ch4"
    )

    # add virtual electrodes for each brain area
    nwbfile.add_electrode_column(name="linked_electrodes", 
                                 description="electrodes averaged together for LFP from each brain area", 
                                 data=[[]]*5, index=True)
    nwbfile.electrodes["linked_electrodes"].set_data_io(H5DataIO, {'maxshape': (None,)})  # make expandable
    brain_areas = np.unique(nwbfile.electrodes.location[:])
    for i, loc in enumerate(brain_areas):
        linked_electrodes = np.where(nwbfile.electrodes.location[:] == loc)[0].tolist()
        nwbfile.add_electrode(group=nwbfile.electrode_groups["shank0"], 
                              location=loc, 
                              label=f'virtual_ch{i}', 
                              linked_electrodes=linked_electrodes)

    # create an electrical series of LFP
    virtual_electrodes_region = nwbfile.create_electrode_table_region(region=[5, 6, 7], description="virtual LFP channels")
    lfp_electrical_series = ElectricalSeries(
        name="series1",
        description="LFP data",
        data=np.random.randn(50, 3),
        electrodes=virtual_electrodes_region,  # reference the virtual electrodes
        starting_time=0.0,
        rate=200.0,
    )
    lfp = LFP(electrical_series=lfp_electrical_series)
    ecephys_module = nwbfile.create_processing_module(
        name="ecephys", description="processed extracellular electrophysiology data"
    )
    ecephys_module.add(lfp)
    nwbfile.add_acquisition(lfp_electrical_series)

    io.write(nwbfile)

with NWBHDF5IO("test_modify_electrodes_table.nwb", "r") as io:
    nwbfile = io.read()
    el_df = nwbfile.electrodes.to_dataframe()
    print(f'All electrodes: {el_df.label.to_list()}')

    elecrodes_raw = nwbfile.acquisition["series0"].electrodes.to_dataframe()
    print(f'Raw electrical series electrodes: {elecrodes_raw.label.to_list()}')

    elecrodes_lfp = nwbfile.acquisition["series1"].electrodes.to_dataframe()
    print(f'LFP electrodes: {elecrodes_lfp.label.to_list()}')

Returns:

All electrodes: ['ch0', 'ch1', 'ch2', 'ch3', 'ch4', 'virtual_ch0', 'virtual_ch1', 'virtual_ch2']
Raw electrical series electrodes: ['ch0', 'ch1', 'ch2', 'ch3']
LFP electrodes: ['virtual_ch0', 'virtual_ch1', 'virtual_ch2']

[TommasoLambresa]

Thank you for your comprehensive response! I will treasure your advice, and if anything remains unclear, I will return here for further clarification.
Thanks!

[FrancescoNegri]

Many thanks for this super detailed answer @stephprince!

[stephprince]

I will close this issue for now but please reopen if there are any other questions!

[FrancescoNegri]

@stephprince sorry to bother you again!
Unluckily, the solution you provided to make the electrodes table editable is not applicable in our case.
Specifically, we use the run_conversion() function from neuroconv to convert Plexon's .pl2 raw files to .nwb. That function immediately saves the generated NWBFile on disk, making the electrodes table columns shape not editable.
Later, we use pynwb and/or h5py to edit and add some metadata to the .nwb file, but I did not manage to find a way to let the NWBFile.add_electrode() function to work.

Given our scenario, how would you suggest us to proceed?
Thanks in advance! :)

[FrancescoNegri]

I managed to solve this last issue by exploting h5py and the following workaround.

import h5py

with h5py.File(file_path, "r+") as f:
    for colname in f["/general/extracellular_ephys/electrodes"].keys():
        old_dset = f["/general/extracellular_ephys/electrodes/{}".format(colname)]
        new_dset = f.create_dataset(
            "/general/extracellular_ephys/electrodes/_{}".format(colname),
            shape=old_dset.shape,
            maxshape=(None,),
            dtype=old_dset.dtype
        )
        new_dset[:] = old_dset[:]
        for key, value in old_dset.attrs.items():
            new_dset.attrs[key] = value
        del f["/general/extracellular_ephys/electrodes/{}".format(colname)]
        f["/general/extracellular_ephys/electrodes/{}".format(colname)] = new_dset
        del f["/general/extracellular_ephys/electrodes/_{}".format(colname)]

I used f["/general/extracellular_ephys/electrodes"].keys() instead of f["/general/extracellular_ephys/electrodes"].attrs["colnames"] in order to edit the id column as well.

If a better solution is available please let me know! 😊

[stephprince]

@FrancescoNegri

Unluckily, the solution you provided to make the electrodes table editable is not applicable in our case. Specifically, we use the run_conversion() function from neuroconv to convert Plexon's .pl2 raw files to .nwb. That function immediately saves the generated NWBFile on disk, making the electrodes table columns shape not editable.

I see, unfortunately it is more complicated if the data was originally written without chunking enabled. The approach you describe above to delete the old dataset and create a new expandable dataset by setting the maxshape to None is likely the best workaround in this case.

There is some ongoing work in pynwb/hdmf to making DynamicTables like electrodes table expandable by default (linked below for reference if you want to follow updates). In the meantime I can open an issue on neuroconv to see if it can create the electrodes table with chunked columns.

Ongoing work / related issues for expandable DynamicTables:

• Make all DynamicTables chunked and resizable by default #1067
• [Feature]: DecompositionSeries BandsTable should be typed #2044
• [Feature]: Expandable datasets (HDF5) hdmf-dev/hdmf#1153
• [Bug]: Dimension Labels and Expandable Datasets Edge Cases hdmf-dev/hdmf#1163
• VectorData Refactor Expandable (#1158) hdmf-dev/hdmf#1180

[FrancescoNegri]

Thank you for linking related issues!
For the time being I will stick to this workaround then :)