Defining the scope

[kushalkolar]

I think this is perhaps the most important thing to start with. What types of visualizations do we want to support?

We look at this for some ideas and try to have at least everything they have: https://github.com/holoviz-topics/neuro

The vague idea of our scope is simultaneous visualization of calcium/electrophysiology data integrated with behavioral data. Maybe we can build a list of all the visual elements that are possible here (feel free to edit this issue and add to the list):

calcium imaging video

• unlimited size
• multiple FOV
• with spatial footprints
  • shown as contours or transparent overlays

calcium or ephys traces

• as lines
  • support thousands in a line stack
    • subsample if extremely large since line rendering is expensive
    • if the entire line-stack shows subsampled data, use a LinearRegionSelector to zoom-in on sections and show the selected region in another subplot. Sync any graphic features between the full-graphic and selection-graphic (maybe later we can provide a way in fpl to auto-support this, perhaps implemented in LinearRegionSelector?).
• colors based on a categorical feature (ex. behavior states)
• as a heatmap, also have the option to use a LinearRegionSelector or RectangleRegionSelector

behavior video

• overlay entire track as a line
  • display a single track (ex. center of mass) or all tracks (ex. head, tail, paws etc.)
  • allow making a window of time around the current timepoint visible (alpha = 1) and the rest transparent
  • if multiple tracks are displayed (ex. paw, tail), easy way to choose alpha for each
  • auto-sync with calcium/ephys using pynapple magic 🍍 🪄
  • BallSlider to allow scrolling through time by moving the mouse along the tracks.
    • If there are mutliple tracks, maybe use a keyboard key to denote the "active" track for the BallSlider

Downstream analysis, ex: PCA, behavior (velocity? etc.)

• All downstream analysis results should be converted to a pynapple array and have a time axis so they can be synchronized with all other visualizations.
• Allow user to display downstream analysis with a graphic of their choice, line, linestack, heatmap, scatter or video, and the corresponding tools are auto-available for each. Ex. if it's a PCA they could use line(s) and a BallSlider
  • This might help us create a base abstraction from which all the visualization are made

[gviejo]

I would add to this the ability to automatically create pynapple IntervalSet with the LinearRegionSelector

[BalzaniEdoardo]

Some comments:

• For the auto-sync, we need to take care of maintaining the sync when controllers move the cameras of various scenes as well as when sliders or other mechanisms update the frame of videos.
• For the downstream analysis, do we need specialized functions? nemos has a decorator that casts back to pynapple if the time-axis is preserved, we could implement something like that. once it is a pynapple Ts of any kind we should be able to visualize. Or, alternatively, we should not care about the anlaysis, if people want to visualize output time series, they should convert back to pynapple or use compatible tools, like nemos