Introduce osm_nodes annotations for 64-bit OSM Node Ids

[SiarheiFedartsou]

Issue

potentially closes #5970 and #5716

The question here is how to preserve backward compatibility, that's why I want to discuss this before doing final "polishing".

What I am doing at the moment I add additional field osm_nodes which in some cases may be returned along with old nodes field which is kind of ugly and wasteful(but we cannot just remove this old nodes without breaking backward compatibility). Another option would be to add parameter(e.g. use_64bit_node_ids=true) and start returning node ids as strings in this case.

Also worth noting that JSON itself seems to support 64-bit numbers and we could just fix our JSON serialization to not convert numbers to double before serialization, but JS itself safely supports only integers up to 53-bit(https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER) and

Tasklist

• CHANGELOG.md entry (How to write a changelog entry)
• update relevant Wiki pages
• add tests (see testing documentation)
• review
• adjust for comments
• cherry pick to release branch

Requirements / Relations

Link any requirements here. Other pull requests this PR is based on?

[SiarheiFedartsou]

In the nearest API we unconditionally add this new osm_nodes array, i.e. always have kind of duplication.

[SiarheiFedartsou]

In case of annotations we add this osm_nodes if it was explicitly requested(annotations=osm_nodes,...) or in case of annotations=true

[SiarheiFedartsou]

flatbuffers implementation is not ready yet, but general idea will be the same

[SiarheiFedartsou]

@mjjbell would be great if you will take a look and share your opinion

[SiarheiFedartsou]

On another hand as per this site the largest node id at the moment is only >2^33, i.e. we seems to be okay with 2^53 limitation of JS(that's the probably main reason why we would need to migrate to string from number). I.e. we can probably only make fix in flatbuffers.

But on another hand I personally know the cases when companies enrich OSM data with their own data and in order to avoid conflicts they start their IDs "from the end"(i.e. 2^64 - 1 goes first).

[SiarheiFedartsou]

Another option I see is to explicitly enable/disable this via CLI parameter/config(and have it disabled by default).

[SiarheiFedartsou]

OSM Node IDs are 34-bit internally

osrm-backend/include/extractor/packed_osm_ids.hpp

Line 14 in bfb74c2

using PackedOSMIDs = util::detail::PackedVector<OSMNodeID, 34, Ownership>;

Related issue #5644

[mjjbell]

Let me summarise to see if I understand.

• OSM Node IDs are currently < 2^34

• JSON Numbers have precision up to 2^53, so our JSON API will be fine for a while / forever.

• Our Flatbuffers API is using uint for nodes and is therefore already overflowing. A breaking change is required to fix.

• Some people hack OSM IDs that are larger than JSON number precision can support. They would like to be able to return node IDs as strings.

Is this right?

[SiarheiFedartsou]

Let me summarise to see if I understand.

• OSM Node IDs are currently < 2^34
• JSON Numbers have precision up to 2^53, so our JSON API will be fine for a while / forever.
• Our Flatbuffers API is using uint for nodes and is therefore already overflowing. A breaking change is required to fix.
• Some people hack OSM IDs that are larger than JSON number precision can support. They would like to be able to return node IDs as strings.

Is this right?

Exactly. At the same time for some reason initially I was thinking that OSM ids are already > 2^53, now it turned out that the problem is not so serious - after all those people can start from 2^53 - 1, but not from 2^64-1. But when it comes to Flatbuffers I think we need to fix that.