Reinstate top-level update_time? (and it was removed and the source dates are all Null).

[danmanix]

When looking to keep local records up to date with Overture, the update_time is a simple way of doing deltas without having to rely on something like Athena to do joins on the changelog (which also is only relevant to the latest release rather a proper timestamp)

If the objective of the Overture project is to combine and improve upon the data provided by multiple sources, would it not in the future be the case that a record is processed and updated, not just at the point of an update from a source, but by further automated processes that are Overture specific?

It feels like a top-level update_time makes sense:

1. to have a shorthand field to query when running updates on local data from the parquet files (avoiding more costly WHERE (SELECT) queries to dig in to the sources struct and find the most recent update time)
2. but to also identify updates triggered directly?

or, in the future is there likely to be 'Overture' as a source?

Many thanks if you can share the thinking here..

[jwass]

Hi @danmanix - So I'm responsible for #237 :). This was all pretty heavily debated internally and I regret us not having more of a public description of that decision.

One of the main reasons we removed the top-level update_time field was because we didn't actually yet have any definition of how it's supposed to be populated and used. Should it change to be the latest time any attribute is modified? Should it reflect the time the underlying data was updated - e.g. satellite image collection time, or latest OSM timestamp? Since this might be confusing for downstream users (as it could be here) we decided to remove it until a later time when we'd have a clearer use and definition.

So related to that - can you provide some more information on how you'd be using the time and/or changelog? And what kind of local data you'd be keeping up to date?

BTW - in your title "the source dates are all Null" is not quite right. It's populated on a number of themes/types. For some ML buildings the update time is the satellite image collection time.

[danmanix]

Heya - thanks for the quick response! (and yes, I should have specified I've been working with Divisions/ Areas mostly, and keeping an eye on Places..)

Regarding what I want to keep up to date, it really is just that - We track 'Location' of something, at varying degrees of precision, from the deepest 'Locality' (our language for a Division) we can determine, to an address, a specific lat/lng, or to a Place (directly equivalent).

I've updated our queries to work with the source->update_time, which I'm sure will work fine when Divisions gets those datetimes in the dataset, but as you go in to detail on, I think I was unclear what would constitute a new update time and it sounds like you have been thinking on that a fair bit.

So to that: hierarchies, names, points and polys are the things that matter to us most when it comes to being up to date.

Thanks again.