RFC: Replacing "admins" theme with a refactored "divisions" theme

[vcschapp]

In the upcoming months, Overture is marching toward a v1.0.0 schema release which, once released, we hope will be forward-compatible with new features and free from backwardly-incompatible schema changes...

As part of our effort to get to v1.0.0, we are looking at refactoring our current "admins" theme to achieve a number of design goals, some smaller, and some larger. The following are the main goals of the refactor:

Request for Comments

We would love to get your feedback and field your questions on this proposed refactor!

Goals of Refactor

1. Support different political perspectives as a first-class item in the schema.
2. Orient the schema toward vending data from which conclusions can be drawn rather than the conclusions themselves, where an example of conclusions in the "admins" schema as at v0.7.0 is geopolDisplay=hidden.
3. Make the schema more friendly for querying, especially when considering the ability to query the Parquet data easily using big-data SQL solutions. e.g. "Give me the features contained in this province" or "Give me the borders between these two countries" should be easy.
4. Fix some conceptual problems, for example the current localityType field is both confusing from a naming perspective and also confused as to what it is trying to represent—a local perspective, or a canonical value.
5. Sand off some rough edges, clear up some naming inconsistencies and misnomers, and generally bring the schema into closer alignment with the other themes.

Proposed Feature Types

The proposed refactored theme would be named "divisions" (alternatives considered: boundaries, administration, administrative). It would initially consist of three primary feature types:

1. A division is a Point feature representing the approximate position of some kind of recognized official or non-official organization of people: country, province, city, neighborhood, etc., as seen from a given political perspective.
2. An area is a Polygon or MultiPolygon feature capturing the shape of the land area, or the land area + territorial sea belonging to a division feature, as seen from that division's political perspective.
3. A boundary is a LineString feature capturing a shared border line between two division features, where the geometry is either wholly maritime or wholly non-maritime.

The properties initially available on these three feature types are shown in the matrix table below. The property is described in short form, and a longer description is given below. Please be advised that this proposal leans on, and in some places shamelessly copies, prior art and ideas from the Who's on First ("WOF") schema. In particular, the subType property borrows heavily from WOF ideas about placetypes, and the hierarchies property borrows from WOF the idea of replicating multiple hierarchies on each feature.

Property	division	area	boundary	Property Type	Short Description
subType	✅	✅		string (enum)	Required. Type, e.g. country, region, locality, neighborhood similar to a WOF placetype, from a finite, ordered, hierarchical, enumerated list.
localType	✅			string	Optional. What subType is called locally within the division.
names	✅	✅		object	Required. Standard Overture names object.
perspectives	✅			array	Optional. Non-empty array of political perspectives from which the division is seen.
hierarchies	✅			array	Required. Array of hierarchy arrays containing the full ancestor hierarchy or hierarchies of the division.
parentDivisionId	✅			string	Optional. Foreign key reference to default parent division.
country	✅	✅		string	Required. ISO 3166-1 alpha-2 country code of division or its default ancestor country.
subDivision	✅	✅		string	Optional. ISO 3166-2 principal subdivision code of division or its default ancestor principal subdivision.
norms	✅			object	Optional. Optional norms object with information about locals rules and customs within the division.
demographics	✅			object	Optional. Optional demograhics object with information about the division's demographics, e.g. population.
flags		✅	✅	array	Optional. Array of boolean flags.
divisionIds		✅	✅	array	Required. Two-element array of foreign key references to divisions on the left (0'th entry) and right (1'th entry) of the boundary line.

Detail of Selected Properties

subType

The subType property of a division or area contains a normalized string value from the enumeration whose members are listed in the table below. These enumeration members are taken from a simplified version of the full Who's on First placetypes hierarchy. These are described in the table below.

#	subType	Description	Example
1	country	Largest unit of independent sovereignty.	United States
2	dependency	A place that is not exactly a sub-region of a country but is dependent on a parent company for defence, passport control, subsidies, etc.	Puerto Rico
3	macroregion	A bundle of regions. These exist mainly in Europe.	Scotland; Île-de-France
4	region	States, provinces, regions. Largest sub-country administrative unit most countries, unless they have dependencies/macroregions.	Alaska; Alberta
5	macrocounty	A bundle of counties. Again, these exist mainly in Europe.	Inverness
6	county	Counties... Largest sub-region administrative unit in most countries, unless they have macrocounties.	Kings County, NY
7	localadmin	A level of government available in some parts of the world that contains localities or populated places that themselves have no authority. Often but not exclusively found in Europe.	Paris
8	locality	A populated place that may or may not have its own administrative authority. (It won't if it belongs to a localadmin.)	Taipei
9	borough	A local government unit, below the locality placetype.	Brooklyn, Queens, etc.
10	macrohood	A super-neighborhood that contains smaller divisions of type neighborhood.	BoCoCa (Boerum Hill, Cobble Hill, and Carroll Gardens)
11	neighborhood	A neighborhood. Most neighborhoods will be just this, unless there's enough granular division to warrant introducing macrohood and/or microhood divisions.	Cobble Hill
12	microhood	A mini-neighborhood that is contained within a division of type neighborhood.	Gätjensort in Hamburg

The allowed containment of the subType fields is shown in the diagram below, where an arrow from a subordinate sub-type to a superior sub-type indicates that the subordinate sub-type may be contained by the superior one. We may be missing some arrows.

perspectives

The optional perspectives property of a division feature is a non-empty array of perspective objects documenting the political perspective(s) from which the division is viewed. A perspective is a simple object with two mandatory fields, type and value. The below example documents India's political perspective:

{
  "perspectives": [
    {
      "type": "country",
      "value": "IN"
    }
  ]
}

The type field contains a string value from an enumeration. Currently the only enumeration member is "country". The value field contains a string value; if type is "country" then value must contain an ISO 3166-1 alpha-2 country code string.

If the perspectives property is omitted, it means the division is seen from a "default" political perspective. Otherwise, it is a non-empty array of perspective objects indicating the non-default perspectives the division corresponds to. The term "default" does not imply any value-judgments, but is typically the most commonly-accessed perspective.

hierarchies

Every division feature has a hierarchies property containing a non-empty array of hierarchies.

• Each hierarchy is an ordered, non-empty, array of hierarchy member objects. There is always one hierarchy member for the division itself, and one for each of its ancestor divisions.
• Typically there is only one hierarchy, but if the division or any of its ancestor divisions has multiple parents, then there will be one extra hierarchy for each additional permutation.
• If there is more than one hierarchy, then hierarchies[0] will be the default hierarchy and the order of the others will be "arbitrary", in that we don't specifically say what it will be, other than that we will strive to keep the order consistent.
• If the multiple hierarchies arise because of multiple political perspectives, and one of the political perspectives is the default one, then the default hierarchy will include the default political perspective at the topmost (closest to root) level of the tree where it occurs.

By containing a materialized view of hierarchies in this way, this structure enables an individual division to be understood in its full context, and enables the full ancestor tree structure (looking upward) to be queried against without needing to construct an obscure/complex SQL statement (like WITH RECURSIVE in Trino/Amazon Athena). This structure is inspired by wof:hierarchies from Who's on First.

{
  "hierarchies": [
    [
      {
        "divisionId": "foo-123",
        "primaryName": "Canada",
        "subType": "country",
        "localType": "country"
      },
      {
       "divisionId": "bar-456",
       "primaryName": "Ontario",
       "subType": "region",
       "localType": "province"
      }
    ]
  ]
}

parentDivisionId

For divisions that are countries, parentDivisionId is omitted. Otherwise, it contains the ID of the division feature that constitutes the division's immediate parent in the administrative hierarchy; or, if the division has multiple parents according to different hierarchies, then it is the default parent.

The simplifying decision to have parentDivisionId be a scalar property instead of an array and thus reflect at most one parent, was made to make the most common use cases simple and easy, while the more complex use cases remain possible via hierarchies.

country

For divisions that are have their own dedicated ISO 3166-1 alpha-2 country code, the country property contains that code. Not all such features will have subType=country, e.g. Puerto Rico.

For divisions that do not have their own dedicated ISO 3166-1 alpha-2 country code but where an ancestor feature does, this property duplicates the country code of the ancestor feature. If there are multiple ancestors due to multiple political perspectives, then this property contains the country code of the ancestor seen from the default political perspective.

A scalar property rather than an array is used for the same reason as for parentDivisionId: to make the most common cases easy, knowing that complex cases are possible using alternative means.

subDivision

For divisions that have their own dedicated ISO 3166-2 principal subdivision code, the subDivision property contains that code.

For divisions that do not have their own dedicated principal subdivision code, but where an ancestor feature does, this property duplicates the principal sub-division code of the ancestor feature. If there are multiple ancestors due to multiple different political perspectives, then this property contains the principal subdivision code of the ancestor seen from the default political perspective.

A scalar property rather than an array is used for the same reason as for parentDivisionId: to make the most common cases easy, knowing that complex cases are possible using alternative means.

norms

The optional norms property of a division feature contains a norms object which specifies a small set of local rules and customs that are important to mapping use cases.

{
  "norms": {
    "drivingSide": "right",
    "defaultLanguage": "en-us"
  }
}

demographics

The optional demographics property of a division contains a demographics object which specifies a small set of demographics sub-properties that are important to mapping use cases.

{
  "demographics": {
    "population": 100000
  }
}

flags

The flags property is an array property containing a set of flags that may apply to the feature. The following flag options are available:

• area feature type:
  1. The isMaritime flag, if present, indicates that the feature includes not only the division's land area but also its territorial sea. If absent, the feature contains only the land area.
• boundary feature type:
  1. The isMaritime flag, if present, indicates that the boundary geometry lies wholly in a water body.
  2. The isDisputed flag, if present, indicates that location of the current boundary is not controversial, but that there is a dispute about where it should be.
  3. The isAmbiguous flag, if present, indicates that there is a dispute about where the boundary should be and that its current location is also controversial.

divisionIds

The divisionIds property of a boundary line feature is an array of length exactly two, where the first element is the ID of the division to the boundary's left, and the second element is the ID of the division to the boundary's right, as seen from a person oriented in the same direction as the boundary's line string geometry.

[Spothedog1]

Howdy - I really like this refactor, I've found the current naming extremely confusing while trying to wrap my head around what everything represents.

Some notes:

Theme

I like the theme name of divisions

Division

For the first feature, I think "division" makes me thing of a line or polygon, not really a Point. As something is being divided". But I get what you mean and don't hate calling it division

Area

Area is good, I like this feature

Boundary

For boundary, I don't quite understand why its needed so some explanation would be good. Can this not also be captured by running a calculation over the Area feature?

Properties

SubType

How would very minute divisons like a political district be represented? I'm thinking like the district of a House Representative. Possibly a locality?

Perspectives

Can you give some more examples of this one? It's saying India is a country, but don't we already know that from the subType?

Hierarchies

Same comment as perspective, can't this information nbe dervied with the parentDivisonId?

Norms & Demographics

Is a nested dictionary the best way to represent this? I feel like it might make more sense to break into separate tables or objects as I could see these growing to extremely large. Will breaking these out in the future break backwards compatability?

Another comment for all themes, what is the thinking with temporal dimensions? How will this refactor allow us to model changes in divisions over time?

Thanks for the hard work on this, I'm excited to see it come to life!

[nvkelso]

Thanks for the thoughtful proposal and the 🎩 tip to WOF :) Some questions and comments...

1. Consider adding disputed as a division unit enum. Including optional disputed territories could be useful both for map display (like with the World Bank's curated set of top-level admin which includes multiple "it's complicated" areas) and for reverse geocoding as a quick flattened view of "dragons be here" from all the various point-of-view alternate polygon geometries for a given feature. A thought experiment: Is Antarctica a country or a dependency or a disputed territory? Another way to think about this is why limit the flags isDisputed and isAmbiguous to boundary parts only instead of also allowing them for areas when isMaritime is already allowed to be for both?
2. For "perspectives", how would you indicate that Country Y does not recognize country X at all? Do you need another enum besides country like unrecognized? Israel, Taiwan, and Kosovo are common examples.
3. Should "perspectives" also be allowed for boundary line features? Right now it reads as if divisionIds are only way of representing the default "facts" about the line, but there are also real world "perspectives" on these... India and China both recognize each other, but disagree on the frontier boundary line between their territories in multiple, unconnected segments.
4. For division "hierarchies" property arrays, have you considered labeling them in addition to them having an arbitrary order after the [0] element? How is an implementor supposed to interprete a multiple hierarchy list to know if the end-user is from Country Z looking at Country Y which is disputed by Country X which hierarchy to use? Especially when only incomplete portions of Country X and Country Y are disputed? Or are they supposed to see the array length is more than 1 so "anything goes" / "you're on your own", but then it's all noise and no signal?
5. For norms, is "defaultLanguage" allowed to be a list or limited to a single value? It's somewhat common for countries to have multiple default languages (like Canada) and while resolving to finer admin levels like region and locality can segment that down to just 1, there are multiple capital cities that have more than one official language.
6. A word of caution on definition of areas land &/or land+sea extent: Consider recommending that "land only" is preferred and consider later supporting the full land+sea as an alternate geometry (not yet modeled here, important for reverse geocoding). This proposal on the whole seems optimized for map display, which presumes land area only. The worst case is like with US Census 2000 when some states provided sea centerline matched boundaries and others provided coastline land boundaries. The land+sea areas will have smaller file size, however.
7. Placetype enums: WOF has considered adding a new microcounty placetype for countries like India where an additional level is needed to better refelct how admin divisions in extremely populous countries play out IRL. WOF doesn't yet have data for these, but some other projects like GADM or geoBoundaries might. The numbers on the left of the table are formatting, not the enum value? The danger of using numbers is people make assumptions that they can be read as numerical admin levels, but then if you insert a level later, like microcounty, then the numbers don't stack logically anymore and downstream code might have unexpected output. The "Europe" comments in the table could be dropped as all of those rows are as likely to be found outside of Europe in real life because of colonial histories as inside Europe.

[vcschapp]

Stopping by to say a huge thanks to @nvkelso and @Spothedog1 for the feedback. There are some very good pointed questions and good points in this. Hopefully I'll be able to put together a more complete/substantive response in the near future.

[vcschapp]

Hi @Spotthedog1! We've had a chance to digest some of your feedback and I wanted to share some thoughts.

Before sharing them, I want to note that our first version of the divisions schema was (quietly) released on 2024-03-12 as part of schema v0.9.0. Please feel free to check it out here. Much is the same as describe above, but some has also evolved.

Boundary

For boundary, I don't quite understand why its needed so some explanation would be good. Can this not also be captured by running calculation over the Area feature?

The main use case for the boundary feature type is map rendering. More obvious examples include showing border lines between countries or principal sub-divisions (provinces, states, etc.). Less obvious ones include being able to identify disputed boundaries so you can assess whether to render them differently on the map. It's common to show such boundaries as dotted lines.

To some limited degree, boundaries could be generated by cutting up the polygon of the area feature. However, there are places where this isn't really possible. Boundaries carry dispute information and, since not all parts of a boundary are necessarily disputed, the boundary feature type provides a higher level of detail. Similarly, the boundary feature type allows the maritime parts of a border to be
easily detected and treated differently than the land parts.

SubType

How would very minute divisons like a political district be represented? I'm thinking like the district of a House Representative. Possibly a locality?

The divisions schema isn't intended for arbitrary spatial divisions of a country, so the most accurate answer is that we don't ever intend to
use divisions to host things like constituency boundaries. That being said, I believe the schema would technically work at least in the House district case if structured the way you suggest. Since I suspect that House districts are a state matter and likely don't cross state lines, it might be that treating them as a special county or macrocounty may make the most sense. This is also an area where adding additional "extension properties", which are supported by the Overture schema, may make sense to carry other metadata specific to your use case.

As a point of interest, a similar case people like to raise is postal delivery areas. We have made an explicit decision within Overture that these don't belong in the the divisions schema and would be included in the upcoming addresses schema, as they are very postal address
specific.

Perspectives

Can you give some more examples of this one? It's saying India is a country, but don't we already know that from the subType?

The idea is that India's from India's perspective looks different than India from Pakistan's perspectie due to various border disputes between the countries. In a certain sense there are several Indias, not just one. The perspectives property helps capture this nuance.

Based on some of @nvkelso's feedback below, I think we need to tweak the perspectives concept, but we'll endeavour to post some examples once we've figured out how to make some acceptable changes!

Hierarchies

Same comment as perspective, can't this information [be] dervied with the parentDivisonId?

Yes and no.

Let's start with the yes part. If there's only one hierarchy then yes, you can technically generate this information by walking up the parent_division_id chain. We made a calculated choice to introduce some redundancy here to make certain common data query patterns easier to execute. For example, if you want to know how many administrative levels down a certain feature is, you can just look at
COUNT(hierarchies[0]). Similarly, it makes an ancestry check between two features possible without having to write a recursive query.

Moving to the no part, a division can technically participate in multiple hierarchies. We again made a calculated choice for simplicity to make parent_division_id a flat scalar value, not an array, so it can only reference one parent, the "default" parent. If you really need access to all the parents, they're available from hierarchies.

Multiple hierarchies can occur both with and without political perspectives. An example without a political perspectives angle is that the NYC borough of The Bronx technically belongs to Bronx County, a county subordinate to the state, and to New York City. It has both parents at the same time. An example with a political perspective angle is the city of Taipei or a neighborhood therein. From the Taiwanese perspective, the ancestor chain for these entities runs up to the Republic of China. From the "One China" perspective, it runs up to the PRC. The hierarchies property enables deeply-nested descendant divisions to capture all such scenarios.

Norms & Demographics

Is a nested dictionary the best way to represent this? I feel like it might make more sense to break into separate tables or objects as > I could see these growing to extremely large. Will breaking these out in the future break backwards compatability?

Separate tables or objects aren't on the agenda because we have made the decision that every Overture object is a feature and must be respresentible as GeoJSON.

In the version of the divisions schema we released in schema v0.9.0, linked above, we eliminated the demographics property altogether,
promoting population to the top level. This is because we feel that otherwise, demographics becomes a Trojan horse into which all manner of random data could be inserted, and we wish to keep things simple. Other datasets can be joined with Overture divisions via GERS, country codes, etc., to get miscellaneous demographic information, and those data should never be in Overture.

We retained norms with the caveat that only data directly related to key mapping use cases will be placed into it. I don't anticipate it growing large, but in any event, it is hard to imagine what "too large" would look like. Storage is cheap, and Parquet and dereference
pushdown are powerful, so nested structures may safely grow large.

Since all properties must be attached to features, the only alternative to nesting is pulling everything from the nested object up to the top level. This mutes the natural explanatory power of grouping and nesting, tends to cause humans to get lost, and in the SQL context results in SELECT * FROM overture returning an overwhelming number of columns. We still think nesting is a good fit for norms.

Another comment for all themes, what is the thinking with temporal dimensions? How will this refactor allow us to model changes in
divisions over time?

We don't currently anticipate adding temporal dimensions to the schema.

However, we do believe that our novel data release structure will help with temporal analysis, and that in the future as the schema stabilizes (soon!) and we begin to publish release-over-release GERS ID change information, temporal analysis will get even easier.

The nice thing about our release structure is that every release is a full point-in-time snapshot in analysis-friendly cloud-native Parquet. Since all the data are available for all time, and the releases are very easy to compare to each other (e.g. by OUTER JOIN in Amazon Athena), we hope consumers can derive the temporal data they need without much effort.

We could potentially add a bit more temporally useful release metadata by "wrapping" the releases in a technology like Iceberg. I'm curious to hear thoughts there.

[vcschapp]

@nvkelso this was a key question:

For "perspectives", how would you indicate that Country Y does not recognize country X at all? Do you need another enum besides country like unrecognized? Israel, Taiwan, and Kosovo are common examples.

Current approach

It seems at first that this can be achieved in the current schema by creating a dedicated perspective for country Y and expressly including in that perspective only those countries it does recognize and not those that it doesn't recognize.

Let's say for the sake of argument that the country of Bar is not recognized by the country of Foo, but Foo recognizes every other country. Basically there would need to be a division for Bar that contains a perspective entry for every country level division except Foo.

id: 456
type: Feature
geometry: ...
properties:
  theme: divisions
  type: division
  subtype: country
  names:
    primary: Bar
  country: BR            # Country = Bar (not Brazil, this example is fictitious!)
  perspectives:          # Basically every country except Foo is in this list 👇
    - type: country
      holder: BZ         # Country = Baz (not Belize, this example is fictitious!)
    - type: country
      holder: QX         # Country = Qux
  - ...

The biggest drawback of this approach is you need to duplicate the country Bar, because you would need one instance of Bar in the default perspective (no perspectives property, because we say that the absence of a perspectives property makes a division part of the default perspective); and one instance of Bar that is included in the perspective of everyone except Foo.

So you end up with something like:

	Division 123	Division 456
Description	Bar in the default perspective	Bar from Foo's perspective
perspectives	Absent, which indicates default.	Bar, Baz, Qux, basically everyone but Foo

Suggested alternative approaches

A PR to schema was raised last week to try to tackle this issue: #162. Everyone is welcome to join the PR discussion!

[nvkelso]

Great solution!

[plaflamme]

FYI: BigQuery does not support arrays of arrays so the new hierarchies field cannot be represented directly. They recommend making arrays of structs which have array fields, e.g.: ARRAY<STRUCT<hierachy ARRAY<STRUCT<...>>>>.

This is related to point 4. above:

For division "hierarchies" property arrays, have you considered labeling them in addition to them having an arbitrary order after the [0] element?

If hierarchy were objects instead of arrays, then a hierarchies field would naturally be an array of structs.

Obviously, leaking BigQuery concerns into the schema design is not desirable. But perhaps this is something worth keeping in mind if this limitation exists in other similar engines making if harder for (some) users to use directly.

For reference for other BigQuery users, when creating an external tables, you need to set enable_list_inference to false. It successfully infers the schema, but is quite clunky to work with:

[tjwebb]

As a BQ user, I appreciate this info. I've made an attempt at systematizing Overture import into BQ; if it's of interest to the project let me know: https://github.com/tjwebb/bigquery-overture

[plaflamme]

Thanks, that looks useful!

I've used a less fancy approach with rclone (to avoid having to create an AWS account) and one external BQ table per theme/type combination (since the schema can vary by type). I let it infer the schema, but, unfortunately, I did have to special-case the divisions theme to set enable_list_inference to false.

For reference, the rclone commands you can use to sync to GCS:

GCS_BUCKET=my-fantastic-gcs-bucket
OVERTURE_RELEASE="2024-04-16-beta.0"
docker run --rm -v "$HOME/.config/gcloud:/gcp/config:ro" \
  -e CLOUDSDK_CONFIG=/gcp/config \
  -e GOOGLE_APPLICATION_CREDENTIALS=/gcp/config/application_default_credentials.json \
  rclone/rclone:latest \
  sync -v \
    --s3-provider=AWS --s3-env-auth=false :s3:overturemaps-us-west-2/release/${OVERTURE_RELEASE} \
    --gcs-bucket-policy-only :gcs:${GCS_BUCKET}/release/${OVERTURE_RELEASE}