Simplifying Panel-based dashboard

[jbednar]

Until release 0.6.9, Datashader shipped with a Bokeh-based example dashboard with widgets to control various aspects of the plotting process (dataset selection, aggregation method, colormaps, spreading, opacity of map and of data, etc.). In release 0.6.9, this 500-line file was replaced with dashboard.ipynb, which does nearly the same thing with 70 lines of Panel-based code.

The new Panel version is much easier to maintain and understand, but it is very densely packed and requires certain bits of magic that are difficult to explain, as you can see in the extended description at the end of the notebook. It would be great if we could further streamline this example, which would require improvements to HoloViews, Param, or Panel.

The complexity mainly comes from wanting to avoid re-running expensive computations, so that simple operations can be fast and responsive, forcing the user to wait only for things that genuinely require deeply re-computing everything. To do that, the various widgets and Bokeh plot interactions need to be hooked up directly to just the right bit of computation, triggering that computation if and only if it is required for the update. Doing so is sometimes simple, but often still very tricky. Can we eliminate any of the following gotchas, limitations, and tricky bits?

1. (Minor) People will be confused if their parameters come out in a different order than they were defined, so I explicitly added precedence values, but that's verbose and somewhat confusing. We could at least rename precedence to be shorter, but using definition order is probably even more useful.
2. (Minor) Can we make it simpler to take the first object as the default for an ObjectSelector? Right now the default of an ObjectSelector is None unless specified, but it's tricky to pull out one of the items from an arbitrary object list to set the default when None isn't appropriate. E.g. for an OrderedDict objects list, the first item in the dict is next(iter(objs.items()))[1], which is very obscure and awkward. It would be nice if Param can compute the default, preferably given a list or an OrderedDict so that the default is well defined while still extractable automatically from the objects. There would be some implications for backwards compatibility, as right now if no default is specified I believe the default is None. If that's a major issue, we could require the user to pass allow_None=False to get this automatic-default behavior, but that would be ugly, so hopefully we don't need that.
3. (Major) Can we avoid having to make the ds.rasterize() call have dynamic=False? We're forced to do that because the rasterize method is wrapped in a DynamicMap and thus cannot return a DynamicMap itself, but doing that cuts off the RangeXY stream that normally makes the rasterization dependent on the zoom and pan events in the plot. As a result we have to explicitly add RangeXY as a stream later, in viewable(), which is mysterious and tricky for the user to figure out, as they otherwise never need to know RangeXY exists at all.
4. (Major) Can we make there be fewer ways to make DynamicMaps in this example? Right now we have various operations that build DynamicMaps (spread, shade, rasterize), the hv.DynamicMap constructor for zero-argument methods, and hv.util.Dynamic for methods with arguments. Can't we make hv.DynamicMap accept arguments with methods, using an optional argument to specify a HoloViews object to which the method will be applied if one is supplied? It's very hard to explain to people why they need these different mechanisms and how to use them.
5. (Medium) Having to create c_stream is confusing. Can we avoid it altogether by having HoloViews operations accept a list of parameters to watch, e.g. watch=[self.param.cmap,self.param.normalization] or maybe streams=[self.param.cmap,self.param.normalization]? Would presumably require per-instance Parameter objects, and not sure how the renaming required for s_stream would be achieved if so.

[jbednar]

Current code, for reference:

plots  = {source.metadata['plots'][p].get('label',p):p for p in source.plots}
fields = odict([(v.get('label',k),k) for k,v in source.metadata['fields'].items()])
field  = next(iter(fields.items()))[1]
aggfns = odict([(f.capitalize(),getattr(ds,f)) for f in ['count','sum','min','max','mean','var','std']])

norms  = {'Histogram_Equalization': 'eq_hist', 'Linear': 'linear', 'Log': 'log', 'Cube root': 'cbrt'}
cmaps  = {n: colorcet.palette[n] for n in ['fire', 'bgy', 'bgyw', 'bmy', 'gray', 'kbc']}

maps   = ['CartoMidnight', 'StamenWatercolor', 'StamenTonerBackground', 'EsriImagery', 'EsriUSATopo', 'EsriTerrain']
bases  = {name: ts.relabel(name) for name, ts in gts.tile_sources.items() if name in maps}

gopts  = hv.opts.WMTS(width=800, height=650, xaxis=None, yaxis=None, bgcolor='black', show_grid=False)

class Explorer(pm.Parameterized):
    plot          = pm.ObjectSelector( precedence=0.10, default=source.plots[0], objects=plots)
    field         = pm.ObjectSelector( precedence=0.11, default=field,           objects=fields)
    agg_fn        = pm.ObjectSelector( precedence=0.12, default=ds.count,        objects=aggfns)
    
    normalization = pm.ObjectSelector( precedence=0.13, default='eq_hist',       objects=norms)
    cmap          = pm.ObjectSelector( precedence=0.14, default=cmaps['fire'],   objects=cmaps)
    spreading     = pm.Integer(        precedence=0.16, default=0,               bounds=(0, 5))
    
    basemap       = pm.ObjectSelector( precedence=0.18, default=bases['EsriImagery'], objects=bases)
    data_opacity  = pm.Magnitude(      precedence=0.20, default=1.00, doc="Alpha value for the data")
    map_opacity   = pm.Magnitude(      precedence=0.22, default=0.75, doc="Alpha value for the map")
    show_labels   = pm.Boolean(        precedence=0.24, default=True)
    
    @pm.depends('plot')
    def elem(self):
        return getattr(source.plot, self.plot)()

    @pm.depends('field', 'agg_fn')
    def rasterize(self, element, x_range=None, y_range=None):
        field = None if self.field == "counts" else self.field
        return rasterize(element, width=800, height=600, aggregator=self.agg_fn(field),
                         x_range=x_range, y_range=y_range, dynamic=False)

    @pm.depends('map_opacity','basemap')
    def tiles(self):
        return self.basemap.opts(gopts).opts(alpha=self.map_opacity)

    @pm.depends('show_labels')
    def labels(self):
        return gts.StamenLabels.options(level='annotation', alpha=1 if self.show_labels else 0)
    
    @pm.depends('data_opacity')
    def apply_opacity(self, shaded):
        return shaded.opts(alpha=self.data_opacity, show_legend=False)

    def viewable(self,**kwargs):
        data_dmap  = hv.DynamicMap(self.elem)
        rasterized = hv.util.Dynamic(data_dmap, operation=self.rasterize, streams=[hv.streams.RangeXY])
        
        c_stream   = hv.streams.Params(self, ['cmap', 'normalization'])
        s_stream   = hv.streams.Params(self, ['spreading'], rename={'spreading': 'px'})
        shaded     = spread(shade(rasterized, streams=[c_stream]), streams=[s_stream], how="add")
        shaded     = hv.util.Dynamic(shaded, operation=self.apply_opacity)
        
        return hv.DynamicMap(self.tiles) * shaded * hv.DynamicMap(self.labels)

[jbednar]

Sketch of possible code with above extensions to HoloViews/Param/Panel:

plots  = odict([(source.metadata['plots'][p].get('label',p),p) for p in source.plots])
fields = odict([(v.get('label',k),k) for k,v in source.metadata['fields'].items()])
aggfns = odict([(f.capitalize(),getattr(ds,f)) for f in ['count','sum','min','max','mean','var','std']])
norms  = odict(Histogram_Equalization='eq_hist', Linear='linear', Log='log', Cube_root='cbrt')
cmaps  = odict([(n,colorcet.palette[n]) for n in ['fire', 'bgy', 'bgyw', 'bmy', 'gray', 'kbc']])
maps   = ['EsriImagery', 'EsriUSATopo', 'EsriTerrain', 'CartoMidnight', 'StamenWatercolor', 'StamenTonerBackground']
bases  = odict([(name,ts.relabel(name)) for name, ts in gts.tile_sources.items() if name in maps])
gopts  = hv.opts.WMTS(width=800, height=650, xaxis=None, yaxis=None, bgcolor='black', show_grid=False)

class Explorer(pm.Parameterized):
    plot          = pm.ObjectSelector(objects=plots)
    field         = pm.ObjectSelector(objects=fields)
    agg_fn        = pm.ObjectSelector(objects=aggfns)
    
    normalization = pm.ObjectSelector(objects=norms)
    cmap          = pm.ObjectSelector(objects=cmaps)
    spreading     = pm.Integer(0, bounds=(0,5))
    
    basemap       = pm.ObjectSelector(objects=bases)
    data_opacity  = pm.Magnitude(1.00, doc='Alpha value for the data')
    map_opacity   = pm.Magnitude(0.75, doc='Alpha value for the map')
    show_labels   = pm.Boolean(True)
    
    @pm.depends('plot')
    def elem(self):
        return getattr(source.plot, self.plot)()

    @pm.depends('field', 'agg_fn')
    def rasterize(self, element, x_range=None, y_range=None):
        field = None if self.field =='counts' else self.field
        return rasterize(element, width=800, height=600, aggregator=self.agg_fn(field),
                         x_range=x_range, y_range=y_range)

    @pm.depends('map_opacity','basemap')
    def tiles(self):
        return self.basemap.opts(gopts).opts(alpha=self.map_opacity)

    @pm.depends('show_labels')
    def labels(self):
        return gts.StamenLabels.options(level='annotation', alpha=1 if self.show_labels else 0)
    
    @pm.depends('data_opacity')
    def apply_opacity(self, shaded):
        return shaded.opts(alpha=self.data_opacity, show_legend=False)

    def viewable(self,**kwargs):
        rasterized = hv.DynamicMap(self.rasterize, obj=hv.DynamicMap(self.elem))
        shaded     = shade(rasterized, cmap=self.param.cmap, normalization= self.param.normalization])
        spreaded   = spread(shaded, px=self.param.spreading, how='add')
        dataplot   = hv.DynamicMap(self.apply_opacity, obj=shaded)
        
        return hv.DynamicMap(self.tiles) * dataplot * hv.DynamicMap(self.labels)

Some bits like rename and what happens to RangeXY are very speculative and up for discussion, but if we can do something like this, I think it will be much easier for people to follow and adapt as an example.

[philippjfr]

I'll put together some actual proposals myself soon, but a DynamicMap will definitely never get an obj kwarg, proposals to replace the Dynamic utility in the example will either involve a generic operation or a method.

[jbednar]

Let's brainstorm possibilities that you would agree to soon!

I'm not sure if I truly think this is a good idea, but pm.ObjectSelector(objects=plots) could be further simplified to the more obvious pm.ObjectSelector(plots). The one thing that must be given to an object selector is the list of objects, since it's proposed that we be able to take the default from it. So if objects is None, we could just treat the default as if it were the objects list/odict. Obviously no one would ever pass default=plots, but as a positional argument it would work. The code would be very readable, though the docstring could be a bit confusing. There's also some current meaning for objects=None that gives it an empty list and allows it to be set to an arbitrary item, but I'm not sure if that's actually useful.

[philippjfr]

Sure, my proposals are just tiny wrappers around the Dynamic utility, which is how all methods and operations on the DynamicMap return values are implemented. I'll make a PR for both of them and we can meet to discuss them and other options tomorrow.

[jbednar]

I think apply makes sense as an operation, as in:

rasterized = apply(self.rasterize, hv.DynamicMap(self.elem))

Not sure if it makes sense to allow both HoloViews objects (as here) or also callables that return them:

rasterized = apply(self.rasterize, self.elem)

In this latter form it's a type of pipeline...

[philippjfr]

After a long meeting about this, here is what we would eventually like the API to be:

plots  = odict([(source.metadata['plots'][p].get('label',p),p) for p in source.plots])
fields = odict([(v.get('label',k),k) for k,v in source.metadata['fields'].items()])
aggfns = odict([(f.capitalize(),getattr(ds,f)) for f in ['count','sum','min','max','mean','var','std']])
norms  = odict(Histogram_Equalization='eq_hist', Linear='linear', Log='log', Cube_root='cbrt')
cmaps  = odict([(n,colorcet.palette[n]) for n in ['fire', 'bgy', 'bgyw', 'bmy', 'gray', 'kbc']])
maps   = ['EsriImagery', 'EsriUSATopo', 'EsriTerrain', 'CartoMidnight', 'StamenWatercolor', 'StamenTonerBackground']
bases  = odict([(name,ts.relabel(name)) for name, ts in gts.tile_sources.items() if name in maps])
gopts  = hv.opts.WMTS(width=800, height=650, xaxis=None, yaxis=None, bgcolor='black', show_grid=False)

class Explorer(pm.LocalParameterized):
    plot          = pm.Selector(plots)
    field         = pm.Selector(fields)
    agg_fn        = pm.Selector(aggfns)
    
    normalization = pm.Selector(norms)
    cmap          = pm.Selector(cmaps)
    spreading     = pm.Integer(0, bounds=(0, 5))
    
    basemap       = pm.Selector(bases)
    data_opacity  = pm.Magnitude(1.00, doc="Alpha value for the data")
    map_opacity   = pm.Magnitude(0.75, doc="Alpha value for the map")
    show_labels   = pm.Boolean(True)

    @pm.depends('plot')
    def elem(self):
        return getattr(source.plot, self.plot)()

    @pm.depends('field', 'agg_fn')
    def aggregator(self):
        field = None if self.field == "counts" else self.field
        return self.agg_fn(field)

    @pm.depends('map_opacity', 'basemap')
    def tiles(self):
        return self.basemap.opts(gopts).opts(alpha=self.map_opacity)

    @pm.depends('show_labels')
    def labels(self):
        return gts.StamenLabels.options(level='annotation', alpha=1 if self.show_labels else 0)
    
    @pm.depends('data_opacity')
    def apply_opacity(self, shaded):
        return shaded.opts(alpha=self.data_opacity, show_legend=False)

    def viewable(self,**kwargs):
        rasterized = rasterize(hv.DynamicMap(self.elem), aggregator=self.aggregator, 
                               width=800, height=400, cmap=self.param.cmap)
        shaded     = shade(rasterized, cmap=self.param.cmap)
        spreaded   = spread(shaded, px=self.param.spreading, how="add")
        dataplot   = spreaded.map(self.apply_opacity)

        return hv.DynamicMap(self.tiles) * shaded * hv.DynamicMap(self.labels)

[jbednar]

Getting there will require:

• Making Param support instance parameters (possibly via a new subclass of Parameterized as above, which enables having Parameter objects per instance) (Add support for instance parameters param#306)
• Generalize map method to allow chaining (Add an apply method to apply functions to elements holoviews#3474)
• Sort parameters in definition order (Support class attribute definition order param#185)
• Rename pm.Selector ABC to pm.SelectorBase, then subclass to make new pm.Selector class that's like ObjectSelector but with a saner interface, then replace ObjectSelector in pyviz and hv and Panel examples, then deprecate ObjectSelector eventually (Updated Selector to be a user friendly ObjectSelector param#316)

Other optional changes that might help:

• Allowing all parameters to be Dynamic (probably not relevant here as that's a pull rather than push model of dynamic updates)
• Allowing dynamic parameters to accept a method on their object (maybe already supported?)
• Making holoviews operations (and other calls?) pull out the dependencies of those methods

[jlstevens]

@jbednar If you make your initial list of suggested improvements into checkboxes, you can tick off the first one now as holoviz/panel#238 has been merged.

#707 is the first dashboard using this behavior afaik but that PR hasn't been merged just yet.