Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update OSM documentation #574

Open
tyrauber opened this issue Jan 29, 2023 · 1 comment
Open

Update OSM documentation #574

tyrauber opened this issue Jan 29, 2023 · 1 comment
Labels
documentation Documentation related stuff good first issue Good for newcomers help wanted Extra attention is needed

Comments

@tyrauber
Copy link

Very nice project! I was hoping to get some more details about the openstreetmaps implementation. Specifically:

  • importer: Is the importer custom to this package, or using an existing library? What is the performance of that import process for the world pbf, compared to Osm2pgsql, Osmium, Osmosis and Imposm.
  • data structure: It sounds like this is a full import? That it maintains all nodes, ways and relations?
  • normalization: In order to save denormalized geometries in PostGIS (e.g. linestring, polygon, multi-polygon, etc.), Baremaps creates a cache for nodes, ways and relations. [LMDB](https://symas.com/lmdb/) is used under the hood to achieve great performance. It sounds like this normalization is happening in the server and not in postgres. What is the performance impact of this normalization? Is there a cache warming period on server restart?
  • updating: Any plans on supporting periodic updates from OSM changesets from something like osmupdate?
  • caching: It appears to me like the server is caching tiles. It would be helpful to know how that caching is happening, what are the space requirements, and how one would purge tiles from cache (especially after a changeset has been imported).
@bchapuis
Copy link
Member

bchapuis commented Jan 29, 2023

Good questions.

  • importer: Is the importer custom to this package, or using an existing library? What is the performance of that import process for the world pbf, compared to Osm2pgsql, Osmium, Osmosis and Imposm.

The pbf parser and database importer are custom to this project. The parser use the Java Stream api to deserialise the pbf in parallel. The importer uses the postgres COPY api to achieve great performance. They should be in par with what other projects are doing.

Our current performance bottleneck is related to geometry simplification (merging highways, creating simplified polygons for landuse, etc.). This currently occurs once the data has been imported in the database. For the planet, this part of the process takes about two days and we are currently trying to optimise this part of the workflow.

  • data structure: It sounds like this is a full import? That it maintains all nodes, ways and relations?

Yes, we import all node, ways, relations and tags.

  • normalization: In order to save denormalized geometries in PostGIS (e.g. linestring, polygon, multi-polygon, etc.), Baremaps creates a cache for nodes, ways and relations. [LMDB](https://symas.com/lmdb/) is used under the hood to achieve great performance. It sounds like this normalization is happening in the server and not in postgres. What is the performance impact of this normalization? Is there a cache warming period on server restart?

This part of the documentation is outdated. We removed LMDB and now rely on memory mapped files to cache the nodes and references. As the pbf files are ordered, we are creating these caches on the fly and read the pbf files only once.

  • updating: Any plans on supporting periodic updates from OSM changesets from something like osmupdate?

Yes, we do have preliminary support for osm updates but it is experimental and not extensively tested.

  • caching: It appears to me like the server is caching tiles. It would be helpful to know how that caching is happening, what are the space requirements, and how one would purge tiles from cache (especially after a changeset has been imported).

Yes, the server uses Caffeine to cache tiles in the RAM

The configuration of the cache can be tweaked with the --cache flag of the map serve command. It takes a caffeine specification as an argument.

@bchapuis bchapuis added help wanted Extra attention is needed good first issue Good for newcomers documentation Documentation related stuff labels Mar 5, 2023
@bchapuis bchapuis added this to the Apache Baremaps v0.7.2 milestone May 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Documentation related stuff good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants