Abstraction layer with fpl #3
Description
Ideas for the abstraction layer between pynaviz and fastplotlib
I think the core ideas is very simple, mapping pynapple array dimensions to fastplotlib GraphicFeatures. Time axis in pynapple arrays are the synchronization glue 😄 . There are n-dimensions to visualize, first 1 or 2 dims are sliders, time and z-axis (if relevant). The rest are mapped to GraphicFeatures.
I think this has the potential to do something ex. napari cannot. Handle arbitrary timeseries and other data types (scatter etc) are abstracted. The idea is sort of an interactive "super-seaborn" for neuroscience. Napari is just mapping dims to images, but I think we can come up with an abstraction that maps dims to a number of different graphics and different aspects of those graphics, geometries, colors, etc.
There probably has to be a common time axis "per neuro widget". You can give the widget a number of diff pynapple arrays, but the time axis must be the same between them. And then all other axes are mapped to graphic buffers across the various graphic and visuals (ex. video, traces, a downstream PCA, behavior tracks etc.).
If we try to think of each type of visual:
- video (calcium or behavior) - time is first dim, rest just map to a 2D frame
- traces - time is x-axis, we need to map y-values, states (ex. ethogram) to colors, and index (an individual line in the stack).
- tracks, scatters, downstream analysis (pca etc.): time is a parametric variable, we can probably use a
BallSlider, other dims map to x and y coordinates for each point on the line and one dim can map to the line color at each point.
One possible way to cleanly handle time could be a global time state object per-widget. This maintains the time across all visuals/graphics in a widget.
The concept of various "selected component indices" could also be global state objects per-widget. Examples:
- calcium: selected cell footprints and temporal components
- ephys: selected traces? Or selected epochs (this might be a bit diff actually)
- behavior: selected parts (ex. tail, head, paw etc.)