sparsearray vs array

[hyperknot]

In PLANET.md, it says two differing options for big RAM servers.

If your system has at least 1.5x as much memory as the input OSM file size, run this command to store node location cache in-memory:

java -Xmx110g \
  `# return unused heap memory to the OS` \
  -XX:MaxHeapFreeRatio=40 \
  -jar planetiler.jar \
  `# Download the latest planet.osm.pbf from s3://osm-pds bucket` \
  --area=planet --bounds=planet --download \
  `# Accelerate the download by fetching the 10 1GB chunks at a time in parallel` \
  --download-threads=10 --download-chunk-size-mb=1000 \
  `# Also download name translations from wikidata` \
  --fetch-wikidata \
  --output=output.mbtiles \
  `# Store temporary node locations in memory` \
  --nodemap-type=array --storage=ram

later on:

Then I added a script runworld.sh to run with 100GB of RAM:

java -Xmx100g \
  -jar planetiler.jar \
  `# Download the latest planet.osm.pbf from s3://osm-pds bucket` \
  --area=planet --bounds=world --download \
  `# Accelerate the download by fetching the 10 1GB chunks at a time in parallel` \
  --download-threads=10 --download-chunk-size-mb=1000 \
  `# Also download name translations from wikidata` \
  --fetch-wikidata \
  --output=output.mbtiles \
  --nodemap-type=sparsearray --nodemap-storage=ram

From comparing these, the difference is --nodemap-type=sparsearray vs array and -XX:MaxHeapFreeRatio=40. Can you tell me which of these should be used on a 128 GB server? Also, what's the right way to find the -Xmx value? Just try and look for a warning?

[hyperknot]

I made a full run with array at the end, with the first version. Had to add --force.

command:

java -Xmx110g \
  `# return unused heap memory to the OS` \
  -XX:MaxHeapFreeRatio=40 \
  -jar planetiler.jar \
  `# Download the latest planet.osm.pbf from s3://osm-pds bucket` \
  --area=planet --bounds=planet --download \
  `# Accelerate the download by fetching the 10 1GB chunks at a time in parallel` \
  --download-threads=10 --download-chunk-size-mb=1000 \
  `# Also download name translations from wikidata` \
  --fetch-wikidata \
  --output=output.mbtiles \
  `# Store temporary node locations in memory` \
  --nodemap-type=array --storage=ram \
  --force

Log: https://gist.github.com/hyperknot/eaad0705b4cd7171ea57bd39b3437520

The server was a Hetzner 32 vCPU / 128 core. Amazing deal compared to AWS and even DO. About € 0.4 / hr, so full planet is < $2.

[msbarry]

Oops! Sorry about that, it should be --nodemap-type=array for any of the planet examples.

They both map node with ID X to a number Y that encodes the lat/lon --nodemap-type=array creates a file where the value at index X is Y. --nodemap-type=sparsearray stores lists of X/Y pairs and does a binary search through X IDs and gets the Y value adjacent to it. So array is best for a full planet import that contains most nodes, and sparsearray is best for an extract that is missing a large portion of node IDs.

For a machine with 128GB RAM I'd actually suggest --nodemap-type=array --storage=mmap and -Xmx20g so the remaining 108GB of RAM outside the JVM heap can be used for the OS to cache memory-mapped node location lookups.

[hyperknot]

Thanks. It seems to have just completed with -Xmx110g + --force. If it completed it means that it's correct, right? I mean there would have been some kind of errors thrown if it run out of memory. I think 128 is just on the very limit with todays OSM planet files.

[hyperknot]

Log here: https://gist.github.com/hyperknot/eaad0705b4cd7171ea57bd39b3437520

[msbarry]

Looks good! You're mostly CPU bound and I don't see much GC so tweaking memory settings won't make a big difference. The benefit of mmap in the future is that instead of running out of memory when OSM planet file grows it just runs a few seconds slower but it looks like you're fine either way for now.

[hyperknot]

Thanks. If you want I can add the file to the logs, it's a very economical server, only 0.4 USD per hour or so. So a full planet takes <$2, which is quite amazing! I used to rent huge servers for a month to run OpenMapTiles, now it's 3 hours and $2. Thank you for creating this library!

[hyperknot]

Hi, doing an other run, now with the recommended settings. 128 GB server, using -Xmx20g

It terminates like this now:

Exception in thread "main" java.lang.IllegalStateException: Insufficient resources for read phase, use the --force argument to continue anyway:
23G JVM heap requested for read phase, 21G available
        at com.onthegomap.planetiler.util.ResourceUsage.checkAgainstLimits(ResourceUsage.java:162)
        at com.onthegomap.planetiler.Planetiler.checkMemory(Planetiler.java:833)
        at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:744)
        at org.openmaptiles.OpenMapTilesMain.run(OpenMapTilesMain.java:54)
        at org.openmaptiles.OpenMapTilesMain.main(OpenMapTilesMain.java:14)
        at com.onthegomap.planetiler.Main.main(Main.java:88)

Settings are identical as in PLANET.md:

java -Xmx20g \
  -jar /data/planetiler/bin/planetiler.jar \
  `# Download the latest planet.osm.pbf from s3://osm-pds bucket` \
  --area=planet --bounds=planet --download \
  `# Accelerate the download by fetching the 10 1GB chunks at a time in parallel` \
  --download-threads=10 --download-chunk-size-mb=1000 \
  `# Also download name translations from wikidata` \
  --fetch-wikidata \
  --output=output.mbtiles \
  `# Store temporary node locations at fixed positions in a memory-mapped file` \
  --nodemap-type=array --storage=mmap \
  > "output_$DATE.log"

[msbarry]

As the planet osm file grows, the recommended memory settings do too, so you could probably resolve this by increasing to -Xmx22g or -Xmx23g. It might be fine with 20g, just run with --force to bypass these memory warnings.

[hyperknot]

Thanks for the reply. Making it too big (60 GB) or too small (10 GB) both gives a warning. I've found the best performance with 30 GB on a 128 GB machine. No warnings, quite fast as well.