Planet generation failed at 64Gb

[tmshv]

Hello! I am trying to generate planet mbtiles on 64Gb RAM machine with config according to PLANET.md file:

#!/usr/bin/env bash
set -e
java -Xmx60g \
  -jar planetiler-openmaptiles-3.14.0-with-deps.jar \
  `# Download the latest planet.osm.pbf from s3://osm-pds bucket` \
  --area=planet --bounds=planet --download \
  `# Accelerate the download by fetching the 10 1GB chunks at a time in parallel` \
  --download-threads=10 --download-chunk-size-mb=1000 \
  `# Also download name translations from wikidata` \
  --fetch-wikidata \
  --output=output.mbtiles \
  `# Store temporary node locations at fixed positions in a memory-mapped file` \
  --nodemap-type=array --storage=mmap 2>&1 | tee logs.txt

But it just stops to run at some point when all the memory are wasted.

tail -n 20 logs.txt:

 -> (164/202) -> worker(26% 27% 29% 31% 33% 27% 26% 15%)
4:58:27 INF [sort] -  chunks: [  22 / 193  11% ] 219G
    cpus: 5.5 gc:  0% heap: 52G/64G direct: 51M postGC: 49G
    -> (163/202) -> worker(37% 35% 22% 31% 34% 33% 37% 41%)
4:58:37 INF [sort] -  chunks: [  24 / 193  12% ] 218G
    cpus: 3.3 gc:  6% heap: 54G/64G direct: 51M postGC: 53G
    -> (161/202) -> worker(20% 22% 54%  0% 22% 19% 17% 14%)
4:58:52 INF [sort] -  chunks: [  24 / 193  12% ] 220G
    cpus: 5.2 gc:  5% heap: 57G/64G direct: 51M postGC: 56G
    -> (161/202) -> worker(47%  9% 25% 44% 26% 25% 24% 29%)
4:59:08 INF [sort] -  chunks: [  27 / 193  14% ] 218G
    cpus: 5.3 gc: 13% heap: 62G/64G direct: 51M postGC: 61G
    -> (158/202) -> worker(23% 38% 31% 23% 32%  9% 13% 34%)
4:59:19 INF [sort] -  chunks: [  27 / 193  14% ] 219G
    cpus: 3.7 gc:  4% heap: 62G/64G direct: 51M postGC: 62G
    -> (158/202) -> worker(18% 11% 26% 19% 20% 52%  0% 18%)
4:59:29 INF [sort] -  chunks: [  29 / 193  15% ] 218G
    cpus: 3.6 gc:  9% heap: 64G/64G direct: 51M postGC: 64G
    -> (156/202) -> worker(15% 14% 18% 19%  3% 11% 55% 16%)

Please help find me where I might be wrong with the config?

[wipfli]

I think you are using too much memory. Try it with less. Here is some explanation: #468 (comment)

[tmshv]

Thank you. Now I've figured out that tmux was killed not plantiler itself.

[msbarry]

Great! Those logs look normal for the sort phase. @wipfli is right, with mmap storage you only want to give the jvm -Xmx20g max. That will leave plenty of space for the OS to cache memory mapped files so it will go a lot faster.

[tmshv]

Thank you! It works with -Xmx20g.

[wipfli]

@tmshv nice! If you want you can make a pull request to make it more clear in the planet.md documentation that one should not use all the memory of the machine with xmx...

[phanecak-maptiler]

I believe such information is already there, see https://github.com/onthegomap/planetiler/blob/main/PLANET.md#2-run-planetiler

If your system has at least 1.5x as much memory as the input OSM file size ...

...

If your system has less than 1.5x as much memory as the input OSM file size ...

... -Xmx20g ...