engines/io_uring_cmd: extend bsg cmd_type with cdb_len, verify, and prefetch by ljw8161 · Pull Request #2123 · axboe/fio

ljw8161 · 2026-07-31T07:27:58Z

Summary

This series extends the bsg cmd_type of the io_uring_cmd engine, which
until now hardcoded READ(10)/WRITE(10) and supported only plain
read/write as the data-direction operations. The changes let users
select the SCSI CDB length and issue VERIFY and PRE-FETCH commands,
which is useful for benchmarking devices whose capacity or per-I/O
transfer size exceeds the 10-byte CDB limits, and for measuring the
cost of medium verification and read-cache warming.

The series also consolidates the bsg completion decoding into a single
place and drops a dead assignment on the bsg submission path. All new
options default to the current behavior, so existing bsg jobs are
unaffected.

Changes

Drop unused write_opcode assignment on the bsg path — cleanup.
ld->write_opcode is only consumed on the nvme path; the bsg path
fills the CDB directly from io_u->ddir, so the assignment never
had any effect. Simplifies the surrounding write-mode check.
Consolidate bsg status decoding into errdetails — cleanup.
The bsg completion was decoded in two places: fio_ioring_cmd_event()
split the packed SCSI status from cqe->big_cqe[0] into
io_u->error, and fio_ioring_cmd_errdetails() re-derived the SCSI
and host status from it. The nvme path only does this breakdown once,
in errdetails. Move the bsg path to the same model: event now
stores the raw cqe->big_cqe[0] in io_u->error, leaving
errdetails as the single place that splits the SCSI status
(bits 0-7) and host status (bits 16-23).
Add cdb_len option — selects the CDB length for READ/WRITE
(and SYNCHRONIZE CACHE). Supported values are 10, 16, and 32; the
32-byte CDB uses the variable-length CDB (opcode 0x7F). The
default is 0 (auto): the smallest CDB whose LBA and
transfer-length fields can hold the request is chosen, escalating
from 10 to 16 as the LBA or transfer length grows. An explicit
cdb_len rejects requests whose LBA or transfer length would
overflow the selected CDB's fields rather than silently
promoting them, so a chosen CDB length is preserved for
benchmarking.
Support write_mode=verify — extends the existing (nvme-only)
option to bsg, issuing SCSI VERIFY as the write-side operation. A
new verify_bytchk option maps to the CDB BYTCHK field:
0 = medium verification only (no host transfer), 1 = full
byte-by-byte comparison, 3 = single-block comparison against the
range. writefua is rejected in this mode since FUA is not defined
for VERIFY.
Add read_mode=prefetch — a new option symmetric to
write_mode, issuing SCSI PRE-FETCH for DDIR_READ to warm the
device read cache without transferring data to the host. Unlike
SYNCHRONIZE CACHE (exposed via DDIR_SYNC), PRE-FETCH moves data
toward the cache and belongs on the read side. readfua and
cdb_len=32 are rejected in this mode: the FUA bit is not defined
for PRE-FETCH, and SBC defines no 32-byte PRE-FETCH service action.

Non-default read_mode with cmd_type=nvme, and verify_bytchk
without write_mode=verify (bsg), are rejected at engine init so
misconfigurations are surfaced instead of silently ignored.

Documentation (HOWTO.rst, fio.1) is updated for all new options.

fio_ioring_cmd_init() assigns ld->write_opcode = bsg_cmd_write_10 when cmd_type=bsg, but that field is never read on the bsg submission path. fio_bsg_uring_cmd_prep() fills bc->cdb[0] directly from io_u->ddir, and ld->write_opcode is only consumed by fio_nvme_uring_cmd_prep() on the nvme path. The assignment has therefore never had any effect. Drop the dead assignment. The surrounding `if (write_mode == WRITE)` branch existed only to hold it; once the assignment is gone the branch is empty, so the check collapses to a plain rejection of unsupported write modes. Fixes: d77ed2b ("engines/io_uring: Add bsg support for io_uring_cmd engine") Signed-off-by: Jungwon Lee <jjung1.lee@samsung.com>

The bsg completion was being parsed in two places: fio_ioring_cmd_event() decoded the packed SCSI status from cqe->big_cqe[0] into io_u->error, and fio_ioring_cmd_errdetails() re-derived the SCSI status and host status from io_u->error. The NVMe path only does this breakdown once, in errdetails. Drop the bsg-specific decoding from event and store the raw cqe->big_cqe[0] in io_u->error, leaving errdetails as the single place that splits the SCSI status (bits 0-7) and host status (bits 16-23). Fixes: d77ed2b ("engines/io_uring: Add bsg support for io_uring_cmd engine") Signed-off-by: Jungwon Lee <jjung1.lee@samsung.com>

The bsg cmd_type of the io_uring_cmd engine has so far hardcoded READ(10) and WRITE(10) for DDIR_READ and DDIR_WRITE. That caps the supported LBA range at 32 bits and the transfer length at 65535 blocks, which prevents benchmarking devices whose capacity or per-I/O transfer size exceeds those limits, and it also prevents comparing the performance characteristics of different CDB lengths of the same command. Add a new engine option, cdb_len, that selects the SCSI CDB length used for READ/WRITE commands. Supported values are 10, 16, and 32. The 32-byte command is encoded as a variable-length CDB (opcode 0x7F) with the READ(32) and WRITE(32) service actions. The default is 0 (auto): the smallest CDB whose LBA and transfer-length fields can hold the request is chosen, escalating from READ(10)/WRITE(10) to READ(16)/WRITE(16) as the LBA or transfer length grows, mirroring the sg engine. This lets fio drive devices whose capacity or per-I/O size exceeds the 10-byte limits without requiring the user to pick a CDB length up front. The 32-byte CDB shares the same 64-bit LBA and 32-bit length field widths as the 16-byte CDB, so auto-escalation never reaches it and it remains opt-in only (for exercising the variable-length CDB path itself). When an explicit cdb_len is set, requests whose LBA or transfer length would overflow the fields of the selected CDB length are rejected with -EINVAL rather than being silently promoted to a larger CDB. This preserves the user's explicit intent when a specific CDB length is chosen for benchmarking. SYNCHRONIZE CACHE follows cdb_len as well: the 10-byte command (0x35) is issued when cdb_len=10, and the 16-byte command (0x91) is issued otherwise. SBC does not define a 32-byte SYNCHRONIZE CACHE service action, so cdb_len=32 falls back to the 16-byte command. UNMAP has no CDB length selection and keeps its fixed 10-byte CDB regardless of cdb_len. Signed-off-by: Jungwon Lee <jjung1.lee@samsung.com>

The existing write_mode=verify option is currently only handled for the nvme cmd_type. Extend it to the bsg cmd_type so that SCSI VERIFY commands can be used as the write-side operation of a workload. This is useful for measuring the cost of medium verification against the same address range that would otherwise be written. The VERIFY CDB length follows the cdb_len option: VERIFY(10)=0x2F, VERIFY(16)=0x8F, and VERIFY(32) via the variable-length CDB service action. The data comparison behavior is controlled by a new verify_bytchk option that maps to the CDB BYTCHK field: 0 - medium verification only, no host data transfer (SG_DXFER_NONE) 1 - compare the full transfer against the medium byte by byte 3 - compare a single block against every block in the range For BYTCHK 1 and 3 the command sends data to the device (SG_DXFER_TO_DEV); BYTCHK 3 transfers only a single block while the CDB still carries the full block count. verify_bytchk defaults to 0 and is rejected unless write_mode=verify. VERIFY is the first bsg command that issues SG_DXFER_NONE with a non-zero io_u->xfer_buflen (the buflen encodes the block count that the device verifies against the medium). The previous fio_bsg_uring_cmd_init() lumped SG_DXFER_NONE into the else branch that populates din_xferp/din_xfer_len, which happened to be harmless only because DDIR_SYNC's xfer_buflen was zero. Tighten the else to match SG_DXFER_FROM_DEV explicitly so SG_DXFER_NONE leaves the data transfer fields cleared. writefua is rejected when combined with write_mode=verify. The FUA bit is not defined for the VERIFY command in any of the CDB variants that we support, and silently dropping the flag would hide a user misconfiguration. Signed-off-by: Jungwon Lee <jjung1.lee@samsung.com>

Introduce a new engine option, read_mode, that selects which command variant is issued for DDIR_READ, symmetric to the existing write_mode option. Two values are supported: 'read', which preserves the current behavior, and 'prefetch', which issues SCSI PRE-FETCH commands. PRE-FETCH pulls the requested LBAs from the medium into the device read cache without transferring the data to the host. This is useful for warming the device read cache prior to a subsequent measurement, and for measuring the overhead of the prefetch path itself. The PRE-FETCH CDB length follows the cdb_len option: PRE-FETCH(10)=0x34 and PRE-FETCH(16)=0x90. SBC does not define a 32-byte PRE-FETCH service action, so cdb_len=32 is rejected with read_mode=prefetch rather than silently falling back. readfua is also rejected in this mode because the FUA bit is not defined for PRE-FETCH. read_mode is currently only meaningful for the bsg cmd_type. Passing a non-default value with cmd_type=nvme is rejected at engine init rather than silently ignored, so that users notice a misconfiguration. Signed-off-by: Jungwon Lee <jjung1.lee@samsung.com>

vincentkfu · 2026-08-10T16:41:46Z

It seems that read_mode=prefetch returns CONDITION_MET SCSI status 0x04 which means success but Fio treats this as an error return.

root@localhost:~/fio-dev/fio-bsg/fio# ./fio --name=test --ioengine=io_uring_cmd --rw=read --number_ios=1 "--filename=/dev/bsg/6\:0\:0\:0" --cmd_type=bsg --read_mode=prefetch
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=1
fio-3.42-124-g5e9e-dirty
Starting 1 process
fio: io_u error on file /dev/bsg/6:0:0:0: Device-specific error: read offset=0, buflen=4096
fio: io_uring_cmd: /dev/bsg/6:0:0:0: BSG SCSI Status: 0x04;
fio: pid=29286, err=4/file:io_u.c:2057, func=io_u error, error=Interrupted system call

test: (groupid=0, jobs=1): err= 4 (file:io_u.c:2057, func=io_u error, error=Interrupted system call): pid=29286: Mon Aug 10 16:32:02 2026
  cpu          : usr=2.08%, sys=0.00%, ctx=1, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0.00ns, window=0.00ns, percentile=100.00%, depth=1

Run status group 0 (all jobs):

root@localhost:~# sg_seek -p -l 0 -c 1 /dev/sdb
sg_seek: Condition met

Please intercept this return value (perhaps in fio_ioring_cmd_event) to avoid treating this as an error condition.