Add benchmark crud --json-lines #9165
Description
/kind enhancement
Following the discussion in #9104 it's quite hard to properly benchmark Borg's various encryption modes right now. First, it's hard to benchmark multiple modes for comparison, see #9167. Second, it's hard to use Borg's benchmark results to draw conclusions, because results are only available in a human readable format.
I thus like to suggest adding formatting options to both benchmark crud (both Borg 1.4 and Borg 2) and benchmark cpu (Borg 2 only). This issue is about benchmark crud, see #9166 for benchmark cpu.
The results shown don't change, they are just presented differently, and the format tries to be as close as possible to Borg's code to reduce the work required to implement this suggestion.
Since benchmark crud is a single comparison of different commands with different samples, it's predestined for a --json-lines option, as used by many other Borg commands as well. The --json-lines option should be added to both Borg 1.4 and Borg 2.
It's fully intentional that the output of --json-lines is pretty repetitive: Borg could indeed use --json instead and print a single JSON object with grouped results (like for benchmark cpu, see #9166) and named samples with characteristics shown in a separate section. However, besides such aggregation logic being harder to implement, I chose this simple but repetitive format because it makes subsequent usage easier. For example, I was thinking about also suggesting a --csv option, but if we keep things simple with --json-lines, the same can be achieved with a simple jq one-liner (which we could add to the docs).
Let's assume the following original results:
C-Z-BIG 224.32 MB/s (10 * 100.00 MB all-zero files: 4.46s)
R-Z-BIG 167.19 MB/s (10 * 100.00 MB all-zero files: 5.98s)
U-Z-BIG 56.77 MB/s (10 * 100.00 MB all-zero files: 17.62s)
D-Z-BIG 393.29 MB/s (10 * 100.00 MB all-zero files: 2.54s)
C-R-BIG 62.85 MB/s (10 * 100.00 MB random files: 15.91s)
R-R-BIG 174.57 MB/s (10 * 100.00 MB random files: 5.73s)
U-R-BIG 53.61 MB/s (10 * 100.00 MB random files: 18.65s)
D-R-BIG 515.74 MB/s (10 * 100.00 MB random files: 1.94s)
C-Z-MEDIUM 209.08 MB/s (1000 * 1.00 MB all-zero files: 4.78s)
R-Z-MEDIUM 267.10 MB/s (1000 * 1.00 MB all-zero files: 3.74s)
U-Z-MEDIUM 53.56 MB/s (1000 * 1.00 MB all-zero files: 18.67s)
D-Z-MEDIUM 490.60 MB/s (1000 * 1.00 MB all-zero files: 2.04s)
C-R-MEDIUM 81.81 MB/s (1000 * 1.00 MB random files: 12.22s)
R-R-MEDIUM 185.68 MB/s (1000 * 1.00 MB random files: 5.39s)
U-R-MEDIUM 63.56 MB/s (1000 * 1.00 MB random files: 15.73s)
D-R-MEDIUM 305.80 MB/s (1000 * 1.00 MB random files: 3.27s)
C-Z-SMALL 19.85 MB/s (10000 * 10.00 kB all-zero files: 5.04s)
R-Z-SMALL 116.56 MB/s (10000 * 10.00 kB all-zero files: 0.86s)
U-Z-SMALL 6.11 MB/s (10000 * 10.00 kB all-zero files: 16.38s)
D-Z-SMALL 52.83 MB/s (10000 * 10.00 kB all-zero files: 1.89s)
C-R-SMALL 17.71 MB/s (10000 * 10.00 kB random files: 5.65s)
R-R-SMALL 109.28 MB/s (10000 * 10.00 kB random files: 0.92s)
U-R-SMALL 6.62 MB/s (10000 * 10.00 kB random files: 15.11s)
D-R-SMALL 51.17 MB/s (10000 * 10.00 kB random files: 1.95s)
With --json-lines it should print the following. I/O should be printed in bytes per second, time in seconds with the highest available precision.
{"id": "C-Z-BIG", "command": "create1", "sample": "Z-BIG", "sample_count": 10, "sample_size": 100000000, "sample_random": false, "time": 4.460000, "io": 224320000}
{"id": "R-Z-BIG", "command": "extract", "sample": "Z-BIG", "sample_count": 10, "sample_size": 100000000, "sample_random": false, "time": 5.980000, "io": 167190000}
{"id": "U-Z-BIG", "command": "create2", "sample": "Z-BIG", "sample_count": 10, "sample_size": 100000000, "sample_random": false, "time": 17.620000, "io": 56770000}
{"id": "D-Z-BIG", "command": "delete", "sample": "Z-BIG", "sample_count": 10, "sample_size": 100000000, "sample_random": false, "time": 2.540000, "io": 393290000}
{"id": "C-R-BIG", "command": "create1", "sample": "R-BIG", "sample_count": 10, "sample_size": 100000000, "sample_random": true, "time": 15.910000, "io": 62850000}
{"id": "R-R-BIG", "command": "extract", "sample": "R-BIG", "sample_count": 10, "sample_size": 100000000, "sample_random": true, "time": 5.730000, "io": 174570000}
{"id": "U-R-BIG", "command": "create2", "sample": "R-BIG", "sample_count": 10, "sample_size": 100000000, "sample_random": true, "time": 18.650000, "io": 53610000}
{"id": "D-R-BIG", "command": "delete", "sample": "R-BIG", "sample_count": 10, "sample_size": 100000000, "sample_random": true, "time": 1.940000, "io": 515740000}
{"id": "C-Z-MEDIUM", "command": "create1", "sample": "Z-MEDIUM", "sample_count": 1000, "sample_size": 1000000, "sample_random": false, "time": 4.780000, "io": 209080000}
{"id": "R-Z-MEDIUM", "command": "extract", "sample": "Z-MEDIUM", "sample_count": 1000, "sample_size": 1000000, "sample_random": false, "time": 3.740000, "io": 267100000}
{"id": "U-Z-MEDIUM", "command": "create2", "sample": "Z-MEDIUM", "sample_count": 1000, "sample_size": 1000000, "sample_random": false, "time": 18.670000, "io": 53560000}
{"id": "D-Z-MEDIUM", "command": "delete", "sample": "Z-MEDIUM", "sample_count": 1000, "sample_size": 1000000, "sample_random": false, "time": 2.040000, "io": 490600000}
{"id": "C-R-MEDIUM", "command": "create1", "sample": "R-MEDIUM", "sample_count": 1000, "sample_size": 1000000, "sample_random": true, "time": 12.220000, "io": 81810000}
{"id": "R-R-MEDIUM", "command": "extract", "sample": "R-MEDIUM", "sample_count": 1000, "sample_size": 1000000, "sample_random": true, "time": 5.390000, "io": 185680000}
{"id": "U-R-MEDIUM", "command": "create2", "sample": "R-MEDIUM", "sample_count": 1000, "sample_size": 1000000, "sample_random": true, "time": 15.730000, "io": 63560000}
{"id": "D-R-MEDIUM", "command": "delete", "sample": "R-MEDIUM", "sample_count": 1000, "sample_size": 1000000, "sample_random": true, "time": 3.270000, "io": 305800000}
{"id": "C-Z-SMALL", "command": "create1", "sample": "Z-SMALL", "sample_count": 10000, "sample_size": 10000, "sample_random": false, "time": 5.040000, "io": 19850000}
{"id": "R-Z-SMALL", "command": "extract", "sample": "Z-SMALL", "sample_count": 10000, "sample_size": 10000, "sample_random": false, "time": 0.860000, "io": 116560000}
{"id": "U-Z-SMALL", "command": "create2", "sample": "Z-SMALL", "sample_count": 10000, "sample_size": 10000, "sample_random": false, "time": 16.380000, "io": 6110000}
{"id": "D-Z-SMALL", "command": "delete", "sample": "Z-SMALL", "sample_count": 10000, "sample_size": 10000, "sample_random": false, "time": 1.890000, "io": 52830000}
{"id": "C-R-SMALL", "command": "create1", "sample": "R-SMALL", "sample_count": 10000, "sample_size": 10000, "sample_random": true, "time": 5.650000, "io": 17710000}
{"id": "R-R-SMALL", "command": "extract", "sample": "R-SMALL", "sample_count": 10000, "sample_size": 10000, "sample_random": true, "time": 0.920000, "io": 109280000}
{"id": "U-R-SMALL", "command": "create2", "sample": "R-SMALL", "sample_count": 10000, "sample_size": 10000, "sample_random": true, "time": 15.110000, "io": 6620000}
{"id": "D-R-SMALL", "command": "delete", "sample": "R-SMALL", "sample_count": 10000, "sample_size": 10000, "sample_random": true, "time": 1.950000, "io": 51170000}Converting this output to CSV is easy with jq (could be added to the docs):
$ borg benchmark crud --json-lines REPOSITORY PATH \
| jq -sr '(first | keys_unsorted | @tsv), (.[] | [.[]] | @tsv)'
id command sample sample_count sample_size sample_random time io
C-Z-BIG create1 Z-BIG 10 100000000 false 4.460000 224320000
R-Z-BIG extract Z-BIG 10 100000000 false 5.980000 167190000
U-Z-BIG create2 Z-BIG 10 100000000 false 17.620000 56770000
D-Z-BIG delete Z-BIG 10 100000000 false 2.540000 393290000
C-R-BIG create1 R-BIG 10 100000000 true 15.910000 62850000
R-R-BIG extract R-BIG 10 100000000 true 5.730000 174570000
U-R-BIG create2 R-BIG 10 100000000 true 18.650000 53610000
D-R-BIG delete R-BIG 10 100000000 true 1.940000 515740000
C-Z-MEDIUM create1 Z-MEDIUM 1000 1000000 false 4.780000 209080000
R-Z-MEDIUM extract Z-MEDIUM 1000 1000000 false 3.740000 267100000
U-Z-MEDIUM create2 Z-MEDIUM 1000 1000000 false 18.670000 53560000
D-Z-MEDIUM delete Z-MEDIUM 1000 1000000 false 2.040000 490600000
C-R-MEDIUM create1 R-MEDIUM 1000 1000000 true 12.220000 81810000
R-R-MEDIUM extract R-MEDIUM 1000 1000000 true 5.390000 185680000
U-R-MEDIUM create2 R-MEDIUM 1000 1000000 true 15.730000 63560000
D-R-MEDIUM delete R-MEDIUM 1000 1000000 true 3.270000 305800000
C-Z-SMALL create1 Z-SMALL 10000 10000 false 5.040000 19850000
R-Z-SMALL extract Z-SMALL 10000 10000 false 0.860000 116560000
U-Z-SMALL create2 Z-SMALL 10000 10000 false 16.380000 6110000
D-Z-SMALL delete Z-SMALL 10000 10000 false 1.890000 52830000
C-R-SMALL create1 R-SMALL 10000 10000 true 5.650000 17710000
R-R-SMALL extract R-SMALL 10000 10000 true 0.920000 109280000
U-R-SMALL create2 R-SMALL 10000 10000 true 15.110000 6620000
D-R-SMALL delete R-SMALL 10000 10000 true 1.950000 51170000