Skip to content

Add remote-read support to read_nwb#2190

Open
h-mayorquin wants to merge 2 commits into
devfrom
add_remote_reading_support_to_read_nwb
Open

Add remote-read support to read_nwb#2190
h-mayorquin wants to merge 2 commits into
devfrom
add_remote_reading_support_to_read_nwb

Conversation

@h-mayorquin

Copy link
Copy Markdown
Contributor

Closes #2150.

pynwb.read_nwb(path) now accepts remote URLs (s3://, gs://, abfs://, https://, etc.) in addition to local paths. Anonymous public remote files just work; for credentialed remote access, configure credentials in the environment (AWS profile, GOOGLE_APPLICATION_CREDENTIALS, Azure managed identity, etc.) and fsspec picks them up automatically. The signature stays at one parameter to honor the original read_nwb design from #1974. Power-user scenarios (e.g. forced h5py ROS3 driver or custom S3-compatible endpoints) continue to require dropping to NWBHDF5IO or NWBZarrIO directly.

To move this forward I have decided not to use the can_read methods for dispatch, like I did in #1994 for local file support. What we have here is unverified URL pattern matching: routing based on conventions like the .zarr suffix and DANDI's /zarr/<uuid>/ URL layout, without opening the file. It seems to me that can_read's contract is stronger and the matching we have here would not fit well there. Especially the convention regarding DANDI Zarr files, which is very NWB-specific and in my view lives naturally here. I think we can move it there later if we want. There is also a secondary performance cost: can_read works by opening the file, so using it for dispatch would double the network requests on every remote read, which we are avoiding at the moment.

The Zarr test depends on hdmf-zarr PR #348 (a one-line resolve_ref fix for fsspec self-references that I opened upstream), so requirements-opt.txt temporarily pins hdmf-zarr to that PR's branch via PEP 508 git notation. Once #348 lands in an hdmf-zarr release, revert that line back to plain hdmf-zarr; the in-file comment documents the cleanup step.

I am also fixing a pre-existing scheme bug in NWBHDF5IO.read_nwb's streaming branch: the fsspec filesystem was hard-coded to "http" regardless of URL scheme, so s3://, gs://, and abfs:// paths silently failed for non-HTTP backends.

Checklist

  • Did you update CHANGELOG.md with your changes?
  • Have you checked our Contributing document?
  • Have you ensured the PR clearly describes the problem and the solution?
  • Is your contribution compliant with our coding style? This can be checked running ruff check . && codespell from the source directory.
  • Have you checked to ensure that there aren't other open Pull Requests for the same change?
  • Have you included the relevant issue number using "Fix #XXX" notation where XXX is the issue number? By including "Fix #XXX" you allow GitHub to close issue #XXX when the PR is merged.

@codecov

codecov Bot commented May 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.20%. Comparing base (9231882) to head (0ccff58).

Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #2190      +/-   ##
==========================================
+ Coverage   95.18%   95.20%   +0.01%     
==========================================
  Files          30       30              
  Lines        2991     3003      +12     
  Branches      444      446       +2     
==========================================
+ Hits         2847     2859      +12     
  Misses         86       86              
  Partials       58       58              
Flag Coverage Δ
integration 73.22% <100.00%> (+0.17%) ⬆️
unit 85.28% <7.14%> (-0.31%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@h-mayorquin h-mayorquin changed the title Add remote-read support to pynwb.read_nwb Add remote-read support to read_nwb May 14, 2026
@h-mayorquin h-mayorquin marked this pull request as ready for review May 14, 2026 03:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Support for remote paths in pynwb.read_nwb

1 participant