Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom aggregation of dynamic branches #1385

Open
wlandau opened this issue Nov 21, 2024 · 1 comment
Open

Custom aggregation of dynamic branches #1385

wlandau opened this issue Nov 21, 2024 · 1 comment
Assignees

Comments

@wlandau
Copy link
Member

wlandau commented Nov 21, 2024

Originally discussed in ropensci/tarchetypes#204.

Dynamic branch aggregation in targets is not as efficient as it could be. First you download and read every single branch sequentially, then apply the aggregation method determined by the iteration method of the target. The technique proposed in ropensci/tarchetypes#204 is much faster, but it's a low-level hack. It would be nice to arrive at something that fits well into targets natively.

It's tricky because this seems to break a lot of assumptions around how I designed targets, and it's hard to know where to fit it in at the interface level. If we could assume all files are on disk, then we could add it to tar_format(). But if you have native cloud storage, or if you have CAS storage backed by a database, then it's trickier. For maximum flexibility, we might consider adding custom aggregation to tar_repository_cas(), but then you would need to know how to read every branch, and that information has to be consistent with the choice of storage format. We might have to stick with user-defined wrappers like in ropensci/tarchetypes#204, but it's worth thinking about all these options.

@wlandau wlandau self-assigned this Nov 21, 2024
@Aariq
Copy link

Aariq commented Dec 9, 2024

This would be interesting for geotargets as there are many potential ways to iterate on rasters—e.g. by layer or by tile—and different corresponding ways to re-combine the results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants