SSD decay

[gszabo]

Since we started playing around with planetiler, SSDs seem to be dying. We bought a separate SSD for running planetiler, and it died after about converting the planet twice. We thought it was bad luck with the hardware, so went on to use the SSD the OS lives on as a fallback. After 1-2 conversions that one also seems to be dying.

Is it just bad luck, or can it be related to running planetiler? Has anyone else experience something like this? Do you know anything particular about the disk usage of planetiler that can be harmful for SSDs? Considering the size of the output data the SSD should last at least thousands of write cycles.

[bdon]

How are you detecting problems with the drive and what filesystem are you using?

[msbarry]

Also would you mind sharing the command you're running with? Disk usage will vary drastically with the memory and configuration options.

[gszabo]

Thank you for the replies. Here are some details:

It is an Ubuntu desktop system. Both SSDs use the ext4 filesystem. One particular symptom is a complete OS failure, causing a restart and then the disks are mounted as read-only. The separate disk had SMART diagnostic errors and we had to unplug it from the computer as it caused OS crashes even without being used.

When the OS disk started showing problems and became read-only, we were able to reboot Ubuntu in recovery mode and run fsck to fix errors in the filesystem, and reboot into normal mode. Then after using planetiler the OS disk became read-only once again and now even the recovery console cannot boot. We were able to rescue some data from it with booting from a live usb stick.

This machine is used mostly for running planetiler, so interference with other applications are unlikely.

The command we use to run is the following:

java -Xmx40g -jar planetiler-0.5.0.jar \
--osm-path="something.osm.pbf" \
--mbtiles_attribution="..." \
--mbtiles_description="..." \
--mbtiles="something.mbtiles" \
--fetch-wikidata \
--building-merge-z13=false \
--download \
--force

We give 40GB heap to java. The machine has 48 GB memory.

[msbarry]

Ah yes I can see what's happening... Planetiler hits the disk for:

1. sequential scan through all input files (60gb read)
2. on the OSM first-pass, it stores node locations in a memory-mapped file (80gb written)
3. then it scans through OSM again, reading node locations from the memory-mapped file (60gb read + 80gb * disk cache miss factor)
4. while scanning through input files it does sequential writes to intermediate feature files (200gb written)
5. then to sort the intermediate features it reads, sorts, then writes the feature files out to disk (200gb read, 200gb written)
6. then it reads the 200gb sorted feature files and writes the ~100gb output file more or less sequentially (200gb read, 80gb written)

So at a bare minimum this will read 600gb and write 560gb per planet run but the "disk cache miss factor" in step 3 goes up a lot as the memory available outside the JVM for the OS to use for caching pages of the node location file decreases. If you have 80gb available outside the JVM, the factor should be 1 since each page can be cached in memory, but you only have 8gb free so it might be 10 or more.

If you run planetiler with between -Xmx10g and -Xmx20g it will give the OS a lot more space to cache node locations more efficiently and should reduce your disk usage a lot. Can you run track disk i/o with these settings vs. -Xmx40g and report back what the difference in disk usage is?

You can also buy more memory, if you had 128GB then it would reduce SSD reads/writes to ~600gb per run. If you really want to take it easy on your SSD, you can get 512GB of RAM, mount 400GB of that as ramdisk and use that for temp storage. Then you could get away with just writing the 80gb output file to your persistent SSD when planetiler finishes.

Also I'd recommend using --nodemap-type=array and --storage=mmap as described on https://github.com/onthegomap/planetiler/blob/main/PLANET.md, and make sure to disable swap. And you might want to try version 0.6, it improves performance with smaller heaps.

[gszabo]

Thank you for the detailed explanation! I was wondering where I'd find a consumer PC motherboard that can take hundreds of GBs of memory 😅 Once we have a running system again with a new disk we'll definitely apply your suggestions, thanks! 🙏

[ImreSamu]

where I'd find a consumer PC motherboard that can take hundreds of GBs of memory ..

The latest consumer motherboards from AMD and Intel now support up to 192GB of RAM (4x48GB DIMM).

[Firefishy]

Some tips about SSD:

• Ensure you are running the latest vendor firmware. The updates often fix heinous firmware bugs.
• Make sure to monitor the SSD with smartmontools and watch the wear level counter(s). Daily or Weekly SMART tests are also a good idea.
• If you are running consumer level SSDs (or SD cards) it might be worth tuning the filesystem mount commit= eg: commit=60 in an attempt to lower the forced syncs which can lower lifetime of the SSD. This option can increase risk of filesystem corruption or data loss in the event of an unexpected power loss.

[gszabo]

Thank you for these tips!

[gszabo]

I'm closing this discussion as you have been very helpful and provided many insights for us. Now the ball is in our court, but it will probably take some time to implement and experiment with the suggestions, and I don't want to leave this as an open/unresolved question for a long time.

[ImreSamu]

Just out of curiosity, what size and type of SSD did this occur with?

[gszabo]

The separate newly bought disk was a WD Blue SA510 SATA 1TB 2.5" .

[ImreSamu]

Thank you,
I'm looking forward to the results of the new tests.

I think it's important to keep an eye on disk usage during processing.
This means monitoring the current usage and free space while processing is happening.
Because the less free space there is, the more the SSD cells get used up. At least, that's what I suspect.
( https://serverfault.com/a/883952 )

The current consumer-grade "WD Blue SA510 SATA 1TB 2.5" has an Endurance (TBW) of 400,
while a "2TB WD Red SA500 SATA" has a TBW of 1300.

But in this case, I think a 2TB DataCenter SSD might be better:
for example, a "Kingston DC500M 1920 GB" has a TBW of 4555.
https://www.kingston.com/en/ssd/dc500-data-center-solid-state-drive
benchmarks:

• https://www.anandtech.com/print/14541/the-kingston-dc500-series-enterprise-sata-ssds-review (2019 )
• https://www.storagereview.com/review/kingston-dc500m-enterprise-ssd-review ( 2019 )
• https://hothardware.com/reviews/kingston-data-center-dc500-solid-state-drive-review (2019 )
• https://www.funkykit.com/reviews/storage/kingston-dc500m-1-92tb-mixed-use-datacenter-ssd-review-raid0-and-raid1/ ( 2021 )

[gszabo]

Oh well... We ordered a Samsung 980 Pro 1 TB with 600 TBW. We shall see.