btrfs-progs: re-enable tree search v2 ioctl

Commit d73e698 ("btrfs-progs: temporarily disable usage of v2 of
search tree ioctl") turned off TREE_SEARCH_V2 detection because it led
to an infinite loop in load_chunk_info() (reachable via 'btrfs
filesystem usage', reproducible by mkfs-tests/001): from the second
item in the returned buffer the data read back as all zeros while
nr_items was non-zero, so the search key never advanced.

The root cause is the type of the variable-sized buffer buf in struct
btrfs_ioctl_search_args_v2: it is declared as __u64[], but the buffer
holds a packed byte stream addressed by byte offsets. Adding such an
offset to a __u64 pointer in btrfs_tree_search_data() scaled it by 8,
so every item after the first landed past the valid data and read
zeros, which fed a zeroed key back into the search and spun forever.

Declare buf as __u8 in all in-tree copies of the struct so the offset
arithmetic is in bytes, and restore the tree search autodetection so
callers use the v2 ioctl when supported. The struct size is unchanged:
buf stays 8-byte aligned after the __u64 buf_size.

Considering a lot of subcommands are utilizing tree-search ioctl, hide
the v2 tree search behind experimental build first, only after we have
confirmed everything is fine move it out of experimental builds.

Suggested-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: You-Kai Zheng <ykzheng@synology.com>
[ Hide the v2 tree search behind experimental builds ]
Signed-off-by: Qu Wenruo <wqu@suse.com>

Author: ukzheng

common/fsfeatures.c

@@ -708,15 +708,11 @@ int btrfs_tree_search_ioctl(int fd, struct btrfs_tree_search_args *sa)
 {
 	/* On first use check the supported status and save it. */
 	if (!tree_search_v2_initialized) {
-#if 0
-		/*
-		 * Keep using v1 until v2 is fully tested, in some cases it
-		 * does not return properly formatted results in the buffer.
-		 */
+#ifdef EXPERIMENTAL
 		if (btrfs_tree_search2_ioctl_supported(fd) == 1)
 			tree_search_v2_supported = true;
-#endif
 		tree_search_v2_initialized = true;
+#endif
 	}
 	sa->use_v2 = tree_search_v2_supported;
 

kernel-shared/uapi/btrfs.h

@@ -626,7 +626,7 @@ struct btrfs_ioctl_search_args_v2 {
 	__u64 buf_size;		   /* in - size of buffer
 					    * out - on EOVERFLOW: needed size
 					    *       to store item */
-	__u64 buf[];                       /* out - found items */
+	__u8 buf[];                        /* out - found items */
 };
 _static_assert(sizeof(struct btrfs_ioctl_search_args_v2) == 112);
 

libbtrfs/ioctl.h

@@ -385,7 +385,7 @@ struct btrfs_ioctl_search_args_v2 {
         __u64 buf_size;			   /* in - size of buffer
                                             * out - on EOVERFLOW: needed size
                                             *       to store item */
-        __u64 buf[0];                      /* out - found items */
+        __u8 buf[0];                       /* out - found items */
 };
 
 /* With a @src_length of zero, the range from @src_offset->EOF is cloned! */

libbtrfsutil/btrfs.h

@@ -590,7 +590,7 @@ struct btrfs_ioctl_search_args_v2 {
 	__u64 buf_size;		   /* in - size of buffer
 					    * out - on EOVERFLOW: needed size
 					    *       to store item */
-	__u64 buf[0];                       /* out - found items */
+	__u8 buf[0];                        /* out - found items */
 };
 
 /* With a @src_length of zero, the range from @src_offset->EOF is cloned! */