verify: add --verify_type option for crash-consistent verification #1948

minwooim · 2025-07-21T04:10:14Z

This patch introduces a new --verify_type option to support crash-consistent offline verification scenarios. The primary motivation is to handle cases where write operations are interrupted by device failures or unexpected power loss during the write phase, requiring verification to be limited to data that was properly persisted before the interruption.

Key features:

New --verify_type=flush option filters verification candidates based on fsync completion timing
Tracks fsync completion timestamps during write phase
During verification, excludes writes that completed after the last fsync
Always includes FUA (Force Unit Access) writes regardless of timing, as they bypass cache and are immediately persistent
Works with existing verify_state_save mechanism for offline verification

Use case:
This is particularly useful for testing storage durability guarantees in scenarios such as:

Simulated power failures during write workloads
Device error injection testing

and espeically with such ioengines which directly communicates with the device directly as io_uring_cmd does.

This enhancement enables more realistic testing of data durability in storage systems by ensuring only properly synchronized data is verified after simulated failures.

Examples:

write phase

[global]
ioengine=io_uring_cmd
cmd_type=nvme
filename=/dev/ng0n1
rw=write
bs=4k
verify=pattern
verify_pattern=%o

iodepth=32
size=32k
fsync=3

[test]
verify_type=flush
do_verify=0
verify_state_save=1

read phase

[global]
ioengine=io_uring_cmd
cmd_type=nvme
filename=/dev/ng0n1
rw=write
bs=4k
verify=pattern
verify_pattern=%o

iodepth=32
size=32k
fsync=3

[test]
verify_type=flush
do_verify=1
verify_only=1
verify_state_load=1

sitsofe

@minwooim:

I've very briefly looked over this and from what you're saying the purpose is to allow the verification of only those writes that happened before an fsync? How does this interact with things like trims? Do you really foresee a verify_type other than flush?

Some additions that will be needed:

You've updated the man page but you'll also need to update the HOWO documentation around roughly https://github.com/axboe/fio/blob/master/HOWTO.rst?plain=1#L3984 (yeah I know "why two places"? It's a boring story)
You'll need to bump FIO_SERVER_VER in server.h (this is mentioned in https://github.com/axboe/fio/blob/master/.github/PULL_REQUEST_TEMPLATE.md )

minwooim · 2025-07-21T09:36:04Z

I've very briefly looked over this and from what you're saying the purpose is to allow the verification of only those writes that happened before an fsync? How does this interact with things like trims? Do you really foresee a verify_type other than flush?

Not this time since we already have trim_verify_zero option for the TRIM-ed offset ranges. Rather than excluding the TRIM-ed offsets, we can verify them with zero-data.

You've updated the man page but you'll also need to update the HOWO documentation around roughly https://github.com/axboe/fio/blob/master/HOWTO.rst?plain=1#L3984 (yeah I know "why two places"? It's a boring story)

You'll need to bump FIO_SERVER_VER in server.h (this is mentioned in https://github.com/axboe/fio/blob/master/.github/PULL_REQUEST_TEMPLATE.md )

Will add these two, Thanks for catching this.

vincentkfu · 2025-07-22T01:22:09Z

io_u.c

 	if (!f->last_write_comp)
 		return;

+	now_nsec = utime_since_now(&io_u->start_time) * 1000;  /* Convert to nanoseconds */


ntime_since_now exists

vincentkfu · 2025-07-22T01:34:22Z

engines/io_uring.c

 	}

+	/* Mark FUA writes for verification state tracking */
+	if (io_u->ddir == DDIR_WRITE && (ld->cdw12_flags[DDIR_WRITE] & (1 << 30)))


I think this would be clearer if you used o->writefua for the condition

Also, the sg ioengine has a writefua flag as well.

vincentkfu · 2025-07-22T01:36:21Z

io_u.c

+		return;
+
+	/* Track the last FLUSH completion timestamp */
+	completion_time = utime_since_now(&io_u->start_time) * 1000;


ntime_since_now
Also, we already calculate a completion time elsewhere for all operations. Is there a way to use that completion time instead of calculating a second one?

vincentkfu · 2025-07-22T01:39:41Z

There is also a t/verify-state.c file that will need updating.

vincentkfu · 2025-07-28T19:23:46Z

verify.h

 	VERIFY_PATTERN,			/* verify specific patterns */
 	VERIFY_PATTERN_NO_HDR,		/* verify specific patterns, no hdr */
 	VERIFY_NULL,			/* pretend to verify */
+	VERIFY_FLUSH,			/* verify with flush timing consideration */


Why is this change needed?

Missed removing this one, Thanks for catching this.

vincentkfu · 2025-07-28T19:24:22Z

For your commit message there needs to be just one Signed-off-by line.

This patch introduces a new --verify_type option to support crash-consistent offline verification scenarios. The primary motivation is to handle cases where write operations are interrupted by device failures or unexpected power loss during the write phase, requiring verification to be limited to data that was properly persisted before the interruption. Key features: - New --verify_type=flush option filters verification candidates based on fsync completion timing - Tracks fsync completion timestamps during write phase - During verification, excludes writes that completed after the last fsync - Always includes FUA (Force Unit Access) writes regardless of timing, as they bypass cache and are immediately persistent - Works with existing verify_state_save mechanism for offline verification Use case: This is particularly useful for testing storage durability guarantees in scenarios such as: 1. Simulated power failures during write workloads 2. Device error injection testing and espeically with such ioengines which directly communicates with the device directly as io_uring_cmd does. This enhancement enables more realistic testing of data durability in storage systems by ensuring only properly synchronized data is verified after simulated failures. Examples: 1. write phase [global] ioengine=io_uring_cmd cmd_type=nvme filename=/dev/ng0n1 rw=write bs=4k verify=pattern verify_pattern=%o iodepth=32 size=32k fsync=3 [test] verify_type=flush do_verify=0 verify_state_save=1 2. read phase [global] ioengine=io_uring_cmd cmd_type=nvme filename=/dev/ng0n1 rw=write bs=4k verify=pattern verify_pattern=%o iodepth=32 size=32k fsync=3 [test] verify_type=flush do_verify=1 verify_only=1 verify_state_load=1 Signed-off-by: Minwoo Im <[email protected]>

sitsofe · 2025-07-21T20:41:05Z

thread_options.h

 	struct zone_split zone_split[DDIR_RWDIR_CNT][ZONESPLIT_MAX];
 	uint32_t zone_split_nr[DDIR_RWDIR_CNT];
-	uint32_t pad2;
+	uint64_t pad2;


Technically you can get rid of the pad as you will now have 64 byte alignment...

sitsofe · 2025-08-18T21:40:45Z

engines/io_uring.c


+	/* Mark FUA writes for verification state tracking */
+	if (io_u->ddir == DDIR_WRITE && o->writefua)
+		io_u_set(td, io_u, IO_U_F_FUA);


Should this be hidden in the DDIR_WRITE case of fio_nvme_uring_cmd_prep() in nvme.c?

sitsofe · 2025-08-18T21:41:19Z

t/verify-state.c

 	}

-	if (hdr->version == 0x04)
+	if (hdr->version == 0x05)


You're going to need to bump this up even further after rebasing...

sitsofe · 2025-08-18T21:43:54Z

fio.1

+failure should be verified.
+.P
+This option requires \fBverify_state_save\fR to be enabled and is only
+effective during offline verification (separate verify job). Default: none


I think you need to explicitly list that none as one of the options above too.

sitsofe · 2025-08-18T21:46:20Z

io_u.c


 		if (ddir == DDIR_WRITE)
-			file_log_write_comp(td, f, io_u->offset, bytes);
+			file_log_write_comp(td, f, io_u->offset, bytes, io_u);


If we're going to pass the full io_u does it make sense to pass the offset separately rather than calculate it in the function?

sitsofe · 2025-08-18T21:49:27Z

thread_options.h


 	struct zone_split zone_split[DDIR_RWDIR_CNT][ZONESPLIT_MAX];
 	uint32_t zone_split_nr[DDIR_RWDIR_CNT];
+	uint32_t pad2;


After rebasing you'll have to see if you still need the pad.

sitsofe · 2025-08-18T21:50:02Z

verify-state.h

 };

-#define VSTATE_HDR_VERSION	0x04
+#define VSTATE_HDR_VERSION	0x05  /* Incremented for FLUSH count support */


Might need bumping after rebasing.

sitsofe requested changes Jul 21, 2025

View reviewed changes

minwooim force-pushed the verify_type branch from 03c7deb to 13e7263 Compare July 21, 2025 09:39

vincentkfu reviewed Jul 22, 2025

View reviewed changes

minwooim force-pushed the verify_type branch 2 times, most recently from 603be44 to 18c23e9 Compare July 24, 2025 07:22

vincentkfu reviewed Jul 28, 2025

View reviewed changes

minwooim force-pushed the verify_type branch from 18c23e9 to 35c7ca4 Compare July 28, 2025 23:06

sitsofe requested changes Aug 18, 2025

View reviewed changes

verify: add --verify_type option for crash-consistent verification #1948

Are you sure you want to change the base?

verify: add --verify_type option for crash-consistent verification #1948

Conversation

minwooim commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sitsofe left a comment

Choose a reason for hiding this comment

Uh oh!

minwooim commented Jul 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vincentkfu commented Jul 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vincentkfu commented Jul 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

minwooim commented Jul 21, 2025 •

edited

Loading