Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store data in a local database cache #19

Open
kinverarity1 opened this issue Mar 28, 2020 · 1 comment
Open

Store data in a local database cache #19

kinverarity1 opened this issue Mar 28, 2020 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@kinverarity1
Copy link
Owner

Is your feature request related to a problem? Please describe.
Every time you make a request for some data that you've already done, it re-downloads it.

Describe the solution you'd like
A locally stored database cache. Ideally this would be allowed to have a maximum size, and therefore we would benefit from an ORM so that we easily do bulk removes, updates, and inserts. The data model should be based only on what's available via the bulk CSV downloads. The JSON service queries are an extra goodie which could be mapped across later. So SQLAlchemy would be an additional dependency. SQLite would be the obvious choice for the cache itself.

The default behaviour of whether to rely solely on the cached results, or update from Groundwater Data, should be configurable, but an easy rule of thumb would be that for some queries like water_levels and salinities, an update should be run if the query is on a new day. Others like well_summary, and so on, should probably be more like a month. In any case: add a keyword argument to the query methods: update_cache=False/True/"auto".

@kinverarity1 kinverarity1 added the enhancement New feature or request label Mar 28, 2020
@kinverarity1 kinverarity1 self-assigned this Mar 28, 2020
@kinverarity1
Copy link
Owner Author

FYI #31 adds a framework to add this on to.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant