general dev chat

[mdsumner]

Hello, following up from #34 where we discussed a broad range of issues around future prospects and generalization/s.

Some topics we discussed below, please pull me up on details or things I've missed if you want anything changed or added.

1. prospects for Python bindings to mirror the R package bindings (Add Python Bindings #34 itself)
2. prospects for putting features in the R package upstream (re-using the index for multi-layers, building histograms)
3. abstract output mode, where input is as general as xmin,xmax,ymin,ymax,nx,ny as the grid spec, and return value is simply cell index and coverage fraction - this is proved by Coverage fraction #40 (and much less polished experiments in https://github.com/mdsumner/exactextractr/tree/MIKE-decouple-terra-raster)
4. whether and where we should pursue generalizations from the heavily optimized exactextract model, which is polygon overlay of boundary-crossed pixels, and a flood-fill completion for completely-covered cells
5. the relationship of exactextract to fasterize which is non-exact, simply centre-of-cell in-polygon test, but which has similar more abstract utility related to 3
6. prospects for ultimately having exactextract as part of GDAL itself, a natural home for raster/vector and options for exact vs. approx extraction and grouping

1, 2

Dependent on funding but clear and in scope for exactextract and its R, Python interface.

3

Seems readily available and of interest especially to @hellkite500 and @mdsumner, for further discussion ... There's a few nuances, about whether an actual raster is the input or just its grid specification, and variations on the output - these could potentially be as little as individual cell index and their fraction, with row/start/stop indexes defining the fully covered fill as an dense optimization, or more tied to actual values and calculations with groupings from the polygon layer.

The variations possible both for input and output make this a fairly open ended topic in terms of the core scope.

4

It's possible that a quad-mesh model (a structured mesh, rectangular in index space nx*ny) could fit the optimized case for regular-raster/polygons. The boundary-affected cells input as geos polygons, and the remaining flood-fill coverage using the grid index rather than the geometry of the mesh. (this is @mdsumner thinking aloud)

Prospects for unstructured meshes are probably much more a general meshing problem, but with similar optimizations to identify boundary-affected cells vs. the flood-fill remainder. (Experiments with Triangle library certainly show that the raster-cell as polygon case does not scale well, and so it would be similar for general meshes - possibly faster with CGAL, but still there's the scale problem of how-big-a-mesh and how optimizations are handled, which gets us quite far from the current scope of exactextract).

[jdalrym2]

Broad plan for me is to work with @hellkite500 and @jduckerOWP to integrate coverage fraction into the existing Python bindings (#34) so it can be used right away in some respect...

Purely volunteer for me atm, so hard to commit to any timing. Open to funding if a given timeline is desired.

[BwL1289]

thank you for all your hard work, @jdalrym2!

[dbaston]

If you have time, I would try running your data through exactextractr to see how the coverage fractions are reused in that interface. I recognize there may be a case where storing the coverage fractions makes sense (though it would be great to see it, with inputs X and Y), but I don't expect it to be part of the typical workflow.

[jduckerOWP]

@mdsumner @dbaston thanks for starting this thread and summarizing our previous meeting we had earlier this week. We'll keep everyone up to date regarding potential funding options that are on the table in the near future. I'm currently evaluating the coverage fraction pull request and what will be required to integrate into the Python bindings. It appears the challenge will be to integrate the coverage fraction writer code into the map writer bindings as a separate initialization phase for the Python bindings. My C+ code knowledge is limited (meteorology background is biased towards FORTRAN unfortunately) but I will reach out to you by the end of the week @jdalrym2 to inquire a few questions regarding your Python bindings development. It doesn't look like a significant hurdle based on your initial Python bindings setup, but I may be underestimating a few factors here.

[jdalrym2]

@jduckerOWP @hellkite500 Quick glance looks like integration of (#40) into my Python bindings (#34) doesn't seem like much of a hassle, so I'll take a crack at it when time allows and make a branch for us to play with.

@dbaston Great point about looking at the R bindings. Ultimately we want a common-ish interface between the two language bindings so I'll spend some time playing around. I admit though I probably have a total experience of 1 hour in R, lol.