Integrate Sprandom in to FIO

[steven-sprouse-sndk]

As SSD capacities increase to 64TB, 128TB and beyond the time to randomly precondition these SSDs is increasing from hours to sometimes day. At an OCP Storage Tech Talk in May '25 we presented Sprandom (SanDisk Pseudo-Random) preconditioning method to accelerate the random preconditioning.

We propose to integrate the Sprandom method in to FIO and are looking to work with the FIO developers to help us accelerate this development. @vincentkfu

[vincentkfu]

Thanks for initiating this discussion here. Happy to discuss the implementation. When you are ready to share patches we can move the discussion to a pull request.

[DanHelmick]

@steven-sprouse-sndk
I think it would be nice to have relevant defaults. I think we need a value for OP and a value for the number of segments. I understood from your presentation that the algorithm is not very sensitive to the number of segments. Is it recommendable to perfectly estimate OP, over estimate OP, or under estimate OP? I suspect under estimating is very poor due to the need to continue preconditioning after Sprandom is complete. I also have suspicions over estimating is non-optimal but acceptable. If this is the case, I put forth a proposal to make FIO's default an over estimate for most drives on the market.

For the sake of argument, I will put forth the following estimates depending on the OP estimate style we want to select.
Under estimate = 12%
"Perfect" estimate = 15%
Over estimate = 17%

What do you recommend for useful defaults of the OP and the segments?

[steven-sprouse-sndk]

@DanHelmick , I think it is better to slightly underestimate vs. overestimate the OP. Underestimating OP will lead to an underestimate of the physical drive capacity. The downside of this is that there is some un-written OP in the drive that will require either additional random write time, or some ramp time during the test workload while the drive works through the extra OP.

Overestimating the OP can cause sprandom to begin over-writing the physical media which can mess up the carefully crafted OP distribution we're trying to create.

As for a useful default, probably something like 15% (write amp ~4.0) sounds reasonable.

[DanHelmick]

@steven-sprouse-sndk
Sounds great. Let's use your recommended default of 15%. Additionally, I think we should put your recommendation in the documentation of this feature. Something to the effect of:
"If the precise drive OP is not known, it is recommendable to underestimate the OP. Then one should follow up with a reduced random write preconditioning to finalize the steady state preconditioning."

Of course, feel free to word-tweak what specifically is in the documentation. I'm just calling out that this topic is valuable knowledge for a future user of the feature.

[steven-sprouse-sndk]

@DanHelmick , @vincentkfu The initial PR for integrating sprandom is up for review.
#1954

Would appreciate feedback from @vincentkfu on the approach.

[vincentkfu]

@steven-sprouse-sndk I have been doing some tests on a small device with sprandom and am hoping for some feedback from you.

I ran the following jobs (full seq write followed by rand write) for traditional preconditioning:

fio \
                --name=seqwrite \
                --ioengine=io_uring \
                --filename=/dev/nvme1n1 \
                --direct=1 \
                --norandommap=1 \
                --rw=write \
                --iodepth=64 \
                --output=seqwrite.out \
                --bandwidth-log
fio \
                --name=randwrite \
                --ioengine=io_uring \
                --filename=/dev/nvme1n1 \
                --group_reporting=1 \
                --direct=1 \
                --norandommap=1 \
                --rw=randwrite \
                --iodepth=128 \
                --numjobs=4 \
                --time_based \
                --runtime=2h \
                --output=randwrite.out \
                --bandwidth-log

The first job ran for 1832s and the device required an additional 2000s to reach steady state. The plot below shows bandwidth for the random write job.

In total this required about 65 min to reach steady state.

I then formatted the device and ran the following jobs (sprandom followed by same rand write job as above) to try out sprandom:

fio \
                --name=sprandom \
                --ioengine=io_uring \
                --filename=/dev/nvme1n1 \
                --group_reporting=1 \
                --direct=1 \
                --norandommap=1 \
                --rw=randwrite \
                --iodepth=64 \
                --sprandom=1 \
                --spr_op=0.14 \
                --spr_num_regions=100 \
                --output=sprandom.out \
                --bandwidth-log
fio \
                --name=randwrite \
                --ioengine=io_uring \
                --filename=/dev/nvme1n1 \
                --group_reporting=1 \
                --direct=1 \
                --norandommap=1 \
                --rw=randwrite \
                --iodepth=128 \
                --numjobs=4 \
                --time_based \
                --runtime=2h \
                --output=randwrite.out \
                --bandwidth-log

The sprandom job ran for 2240s and the random write job ran for about 1500s before attaining steady state for a total of 3740s or only a couple minutes faster than the amount of time traditional preconditioning required. The plot below shows bandwidth for the random write job.

Do you have any thoughts about how to tweak the sprandom job to make it more effective?

[steven-sprouse-sndk]

@vincentkfu , A couple things to try. What are the bs and ba values you use during sprandom and the random write? To make Sprandom go a bit faster you can try increasing the bs value to be a plane page (16KiB), or even die page (32K/64K). This is the case if you have IOPs limitations.

Another thing to try is to increase your spr_op setting. It looks like the random write is still burning through OP for 50-60 secs at the start of the test, so maybe try .15 or .16? If you can post a plot of the BW during the sprandom phase that would be useful to see if there is any performance drop near the end of the sprandom precon.

In the steady state measurement from the conventional precondition, are you able to measure the actual write amp? if so, you can use that to determine the exact OP value to use.

[vincentkfu]

@vincentkfu , A couple things to try. What are the bs and ba values you use during sprandom and the random write? To make Sprandom go a bit faster you can try increasing the bs value to be a plane page (16KiB), or even die page (32K/64K). This is the case if you have IOPs limitations.

I am using the default 4KiB settings for ba and bs. For a fair comparison I would use larger transfer sizes for traditional preconditioning as well.

Another thing to try is to increase your spr_op setting. It looks like the random write is still burning through OP for 50-60 secs at the start of the test, so maybe try .15 or .16? If you can post a plot of the BW during the sprandom phase that would be useful to see if there is any performance drop near the end of the sprandom precon.

At the tail end of the op=0.14 sprandom run bandwidth does drop:

For op=0.15 I obtained the following (sprandom job first, randwrite job second):

In the steady state measurement from the conventional precondition, are you able to measure the actual write amp? if so, you can use that to determine the exact OP value to use.

I sought to determine OP by using bw to pinpoint when garbage collection begins. For the randwrite job following the 100% full seq write bandwidth dropped from ~3000 MiB and bottoms out at around ~500 MiB/s. So I totalled all of the traffic until bandwidth bottomed out as my estimate of overprovisioning. I have not used the SMART log values for measuring write amplification but I see how they could be used.

[steven-sprouse-sndk]

@vincentkfu , the premature drop in performance during the sprandom phase actually implies that the OP setting is too high, so try dialing it back from .14 to .13 or .12

[vincentkfu]

On a whim I actually ran with op=0.13 overnight and your prompting reminded me to look at the results.

The sprandom run still shows a drop at the very end. I also set qd=512 to match the randwrite job instead of 64 in the earlier runs.

The random write plot suggests that steady state begins at around 500-1000s into the run which is an improvement.

[steven-sprouse-sndk]

@vincentkfu If you increase the bs used during sprandom from 4K to 16K you should see the precon bandwidth improve to be closer to the seq write bandwidth and still correlate pretty well with the conventional 4K random write precon. E.g. sprandom precon with 16K, but still do random write test with 4K.

[xzc-yy]

@steven-sprouse-sndk I have some questions
Q1: about the bs (block size) setting. When testing an 8TB drive (Gen3 X4), if I set bs=4k, the entire test will take a very long time. But if I increase the block size to bs=512k to improve performance, it will negatively impact the distribution of fragmented invalid data, which undermines the test's effectiveness. Could you provide some suggestions on the optimal value of bs for disks of different capacities

Q2: I have conducted some FIO SPrandom experiments, where the address offsets recorded in write_iolog significantly deviated from the expected values, I set the size to 5G and the maximum address recorded by parameter write_iolog is less than 1G.
fio run command:
fio --filename=/dev/nvme0n1 --size=5g --write_iolog=fio_sprandom_write_iolog.txt --debug=sprandom examples/sprandom.fio
fio_sprandom_write_iolog.txt

Thanks for your reply

[steven-sprouse-sndk]

@xzc-yy Q1: Some drives have CPU bottlenecks and can only do so many commands/s (IO/s). As you've observed increasing the bs to 512k may undermine your test preconditioning. You may want to try with smaller bs like bs=16k or bs=64k and then see how that correlates with the expected performance with conventional preconditioning.
Q2: Are you saying that fio with sprandom wrote to a range outside of the logical range you specified?

[Y-cuteBoy]

@steven-sprouse-sndk Hello, I am using the sprandom feature to accelerate a drive into its steady state, followed by a 2-hour random write test. The drive has a 6.7% Over-Provisioning (OP) and a capacity of 61.44TB. However, the results I'm getting are significantly higher than the official specification of 45K IOPS.
My parameters are:

--sprandom=1

--spr_op=0.05

--spr_num_regions=100

test result

[steven-sprouse-sndk]

@Y-cuteBoy The issue is most likely that the drive has much more OP than you are specifying. 6.7% OP is usually the difference between what we call the binary and decimal capacity of the drive. But your drive's probably have more than that: like >10% would be typical. So what you are seeing in the high bandwidth is the drive consuming the >3% of the OP. You probably need to wait for 3-4 hours before you see a drop in the performance.

Try increasing the spr_op (try a binary search) until you see the performance drop within the first 5 minutes of the post sprandom random write test. This will tell you the true OP inside the drive.

[Sairamb9742]

Hello @vincentkfu,

I am evaluating Sprandom in fio (v3.41) as part of an SSD validation effort and wanted to ask for your guidance regarding the compute_gc_validity check and the assert(waf > 1.0) condition.

I have tested Sprandom on multiple enterprise SSDs with the recommended configuration parameters, including:

Samsung enterprise SSDs (8 TB and 16 TB)
Solidigm D5‑P5336 (61.44 TB, QLC)

In all cases, Sprandom aborts very early with:
sprandom.c: compute_gc_validity: Assertion `waf > 1.0' failed.

What I am observing on these drives appears to be firmware behavior that aggressively defers garbage collection (large hidden OP, write buffering, endurance optimization), so the predicted WAF remains ≤ 1.0 during the Sprandom validation phase.

I wanted to confirm a few points with you:

  1. Is this behavior expected for modern enterprise SSDs (including high‑capacity QLC devices) that defer GC aggressively?
  2. Is Sprandom intended to be applicable only to SSDs whose firmware exposes GC pressure early enough for WAF > 1.0 to be observable during preconditioning?
  3. Is there any recommended way (or any intention) to relax or parameterize the waf > 1.0 requirement, or is this a hard correctness gate by design?
  4. For SSDs that violate this assumption, would you agree that fio’s empirical steady‑state detection (bandwidth/latency stabilization) is the correct alternative approach?

I am mainly trying to understand the intended scope of applicability and ensure I am using the right methodology for these classes of devices.

[steven-sprouse-sndk]

@Sairamb9742

1. There was a recent pull request that tried to address the case where the SSD has a write cache that delays GC. Look at SPRandom Cache Size Behavior Implementation #2044 and find an fio release where that has been integrated. Sprandom splits the drive in to N physical regions. You'll want to ensure that the region size is greater than the cache size.
2. Sprandom solves the problem of how to distribute invalid data across the drive for a drive that uses the greedy algorithm. There are still many behaviors in the drive (caching, damped throttling) that may prevent sprandom from having the desired effect.
3. It is this way by design. WAF is computed from the logical capacity of the drive and the OP, and by definition must be >= 1. Look at the derivation of WA=f(op) here: https://ieeexplore.ieee.org/document/6167472.
4. I can't comment on whether the steady state detection is the best way.