Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot mount "Input/output error" #365

Open
corbinjurgens opened this issue Jul 28, 2022 · 6 comments
Open

Cannot mount "Input/output error" #365

corbinjurgens opened this issue Jul 28, 2022 · 6 comments

Comments

@corbinjurgens
Copy link

corbinjurgens commented Jul 28, 2022 •
edited
Loading

libfuse 3.2.6
fuse-overlayfs 1.8.2 prebuilt x86_64
amazonlinux2

I have been trying to use fuse-overlayfs to merge two goofys s3 folders.

I succeeded using migratefs (a fork of fuse-overlayfs) but had a different issue (copyup permission issue) so I am now trying fuse-overlayfs without copyup feature.

I call the following

sudo fuse-overlayfs -f -d  -o lowerdir=/mnt/html/upload_aws -o upperdir=/mnt/html/upload_wasabi -o workdir=/mnt/html/upload_wasabi /mnt/html/upload

Result

uid=unchanged
uid=unchanged
upperdir=/mnt/html/upload_wasabi
workdir=/mnt/html/upload_wasabi
lowerdir=/mnt/html/upload_aws
mountpoint=/mnt/html/upload
plugins=<none>
fuse-overlayfs: cannot read upper dir: Input/output error

On the upper directories goofys debug, the error seems to be:

2022/07/29 01:45:52.302399 s3.DEBUG DEBUG: Validate Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

Full upper goofys debug log:


2022/07/29 01:45:51.437198 fuse.DEBUG Op 0x000005b8        connection.go:408] <- GetInodeAttributes (inode 1)
2022/07/29 01:45:51.437323 fuse.DEBUG GetAttributes 1  []
2022/07/29 01:45:51.437567 fuse.DEBUG Op 0x000005b8        connection.go:491] -> OK ()
2022/07/29 01:45:51.440671 fuse.DEBUG Op 0x000005ba        connection.go:408] <- OpenDir (inode 1)
2022/07/29 01:45:51.441009 fuse.DEBUG OpenDir 1  []
2022/07/29 01:45:51.441365 fuse.DEBUG Op 0x000005ba        connection.go:491] -> OK ()
2022/07/29 01:45:51.442442 fuse.DEBUG Op 0x000005bc        connection.go:408] <- OpenDir (inode 1)
2022/07/29 01:45:51.442570 fuse.DEBUG OpenDir 1  []
2022/07/29 01:45:51.442695 fuse.DEBUG Op 0x000005bc        connection.go:491] -> OK ()
2022/07/29 01:45:51.443307 fuse.DEBUG Op 0x000005be        connection.go:408] <- GetXattr (inode 1, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:51.444496 fuse.DEBUG GetXattr 1  [trusted.overlay.origin]
2022/07/29 01:45:51.446570 fuse.DEBUG Op 0x000005be        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.447892 fuse.DEBUG GetXattr 1  [user.fuseoverlayfs.origin]
2022/07/29 01:45:51.447745 fuse.DEBUG Op 0x000005c0        connection.go:408] <- GetXattr (inode 1, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:51.448388 s3.DEBUG DEBUG: Validate Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:51.449699 s3.DEBUG DEBUG: Build Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:51.452055 s3.DEBUG DEBUG: Sign Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:51.453653 s3.ERROR code=InvalidParameter msg=1 validation error(s) found., err=InvalidParameter: 1 validation error(s) found.
caused by: ParamMinLenError: minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:51.455393 s3.ERROR code=InvalidParameter msg=1 validation error(s) found., err=InvalidParameter: 1 validation error(s) found.
caused by: ParamMinLenError: minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:51.457162 fuse.DEBUG Op 0x000005c0        connection.go:493] -> Error: "InvalidParameter: 1 validation error(s) found.\n- minimum field size of 1, HeadObjectInput.Key.\n"
2022/07/29 01:45:51.457607 fuse.ERROR *fuseops.GetXattrOp error: InvalidParameter: 1 validation error(s) found.
2022/07/29 01:45:51.458606 fuse.ERROR - minimum field size of 1, HeadObjectInput.Key.
2022/07/29 01:45:51.459532 fuse.DEBUG Op 0x000005c2        connection.go:408] <- ReleaseDirHandle
2022/07/29 01:45:51.459899 fuse.DEBUG ReleaseDirHandle 
2022/07/29 01:45:51.460563 fuse.DEBUG Op 0x000005c2        connection.go:491] -> OK ()
2022/07/29 01:45:51.464642 fuse.DEBUG Op 0x000005c4        connection.go:408] <- LookUpInode (parent 1, name ".wh.")
2022/07/29 01:45:51.464998 fuse.DEBUG Inode.LookUp 1  [.wh.]
2022/07/29 01:45:51.467044 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh. HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f55c1d0bf44c98cedb147747adcc74cb4034c772ac9cb18aa3297c533cbbfbe3
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.467211 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh./ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=58ef78e800210967e193c15bba038644c93fb644b9705e3e8a155a02ee8f01f0
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.467077 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh.%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=58b3660c2661a1aa81fd3a938d0249998512c435046588821a19ab17267539b4
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.495414 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: +XcQpncPhfHKBhFtgK81Tv4iL0yjXIQaABZJ/VE4sUYsufy7g1yzY8UZRjYojvubigQkhV9Abn6j
X-Amz-Request-Id: 3833CB436D039451


-----------------------------------------------------
2022/07/29 01:45:51.496274 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:51.497702 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: 8TKDDODdJrJJ6QiHslBt5O8TuQwFo2BtXBpmLDx7hJ+HpU3AKiTOIvjTY9N3afzn1WmUjz1dWPeE
X-Amz-Request-Id: 073B7AF662083608


-----------------------------------------------------
2022/07/29 01:45:51.497810 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 073B7AF662083608, host id: 8TKDDODdJrJJ6QiHslBt5O8TuQwFo2BtXBpmLDx7hJ+HpU3AKiTOIvjTY9N3afzn1WmUjz1dWPeE
2022/07/29 01:45:51.498048 s3.DEBUG HEAD .wh. = no such file or directory
2022/07/29 01:45:51.507085 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: BDH2th3vZStD8FzM403iDd2lRMyJ60PO2O0nL9hiFjVjzeJy/0VrZrpL4K5zHnirr3VLw6qgOzWW
X-Amz-Request-Id: D81533D470412681


-----------------------------------------------------
2022/07/29 01:45:51.507139 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: D81533D470412681, host id: BDH2th3vZStD8FzM403iDd2lRMyJ60PO2O0nL9hiFjVjzeJy/0VrZrpL4K5zHnirr3VLw6qgOzWW
2022/07/29 01:45:51.507442 s3.DEBUG HEAD .wh./ = no such file or directory
2022/07/29 01:45:51.507500 fuse.DEBUG <-- LookUpInode 1 .wh. no such file or directory
2022/07/29 01:45:51.507630 fuse.DEBUG Op 0x000005c4        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:51.508851 fuse.DEBUG Op 0x000005c6        connection.go:408] <- OpenDir (inode 1)
2022/07/29 01:45:51.509178 fuse.DEBUG OpenDir 1  []
2022/07/29 01:45:51.510257 fuse.DEBUG Op 0x000005c6        connection.go:491] -> OK ()
2022/07/29 01:45:51.510606 fuse.DEBUG Op 0x000005c8        connection.go:408] <- ReadDir (inode 1)
2022/07/29 01:45:51.511130 fuse.DEBUG ReadDir 1  [0]
2022/07/29 01:45:51.512441 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&prefix= HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=dcc9a8aa7a8c2026639c69393556d19bf03ad9badc73dc98da3691d9dc8c9cbc
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.534118 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: CNczelyjbQXMnYsF5pYy4JeZFjT9ufgQe320J2oELMhVwvUiIFVorjVXce+6Qlmq2uLLJZWrStu+
X-Amz-Request-Id: D2A924220ED0D423


-----------------------------------------------------
2022/07/29 01:45:51.536087 s3.DEBUG &{[] [.test: 4 haha: 2 s3fs_exist.txt: 0 test.txt: 6 wasabi.txt: 11] <nil> false }
2022/07/29 01:45:51.537058 fuse.DEBUG <-- ReadDir 1  [. 1]
2022/07/29 01:45:51.537148 fuse.DEBUG <-- ReadDir 1  [.. 2]
2022/07/29 01:45:51.537193 fuse.DEBUG <-- ReadDir 1  [.test 3]
2022/07/29 01:45:51.538208 fuse.DEBUG <-- ReadDir 1  [haha 4]
2022/07/29 01:45:51.539033 fuse.DEBUG <-- ReadDir 1  [s3fs_exist.txt 5]
2022/07/29 01:45:51.539277 fuse.DEBUG <-- ReadDir 1  [test.txt 6]
2022/07/29 01:45:51.540223 fuse.DEBUG <-- ReadDir 1  [wasabi.txt 7]
2022/07/29 01:45:51.541040 fuse.DEBUG Op 0x000005c8        connection.go:491] -> OK ()
2022/07/29 01:45:51.541623 fuse.DEBUG Op 0x000005ca        connection.go:408] <- LookUpInode (parent 1, name ".wh..test")
2022/07/29 01:45:51.542481 fuse.DEBUG Inode.LookUp 1  [.wh..test]
2022/07/29 01:45:51.546471 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh..test HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=89b5cc9ac157da82c1205c626fc3186d40668764a54690dc28c6bedd4d9e4085
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.547414 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh..test/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4aad81f77eb972cdbe7f496b10150d2179578837dd010191a604c041d7cd888b
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.547589 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh..test%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=15f38ac59f9c9a01945e6d958c7bbfad0fbd0e3db6076c8ba60e78ac5620155c
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.567059 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: 2o08ptqr2oPWp/XS2h4zq7QfOJiD897m4eEjAy9QAkOcG9l8duLF1KJ1NabG8DkH7RWufm/tKC0g
X-Amz-Request-Id: AC84DFB711E1CC3E


-----------------------------------------------------
2022/07/29 01:45:51.567147 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: AC84DFB711E1CC3E, host id: 2o08ptqr2oPWp/XS2h4zq7QfOJiD897m4eEjAy9QAkOcG9l8duLF1KJ1NabG8DkH7RWufm/tKC0g
2022/07/29 01:45:51.567394 s3.DEBUG HEAD .wh..test = no such file or directory
2022/07/29 01:45:51.570143 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: LpszYukDKZuAxZau8iQEgt1UhGEzXgqNYFe9UPtgTL1lWQ6/AiDsCsenl3XnsWd4JY9yd+1Gla+N
X-Amz-Request-Id: 0993DF431033138D


-----------------------------------------------------
2022/07/29 01:45:51.570250 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: J85m1O6vd75p7eY2tMILZ/hBubNp+81yYa9FC2H3gpN7AXgJ11Jgn9V1AY9nkPH47Or+PqRWRXyH
X-Amz-Request-Id: B984F90CBD68AE86


-----------------------------------------------------
2022/07/29 01:45:51.570320 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:51.570259 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 0993DF431033138D, host id: LpszYukDKZuAxZau8iQEgt1UhGEzXgqNYFe9UPtgTL1lWQ6/AiDsCsenl3XnsWd4JY9yd+1Gla+N
2022/07/29 01:45:51.570700 s3.DEBUG HEAD .wh..test/ = no such file or directory
2022/07/29 01:45:51.570961 fuse.DEBUG <-- LookUpInode 1 .wh..test no such file or directory
2022/07/29 01:45:51.571338 fuse.DEBUG Op 0x000005ca        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:51.572568 fuse.DEBUG Op 0x000005cc        connection.go:408] <- LookUpInode (parent 1, name ".test")
2022/07/29 01:45:51.573397 fuse.DEBUG Ref 4 .test [10]
2022/07/29 01:45:51.573539 fuse.DEBUG <-- LookUpInode 1 .test <nil>
2022/07/29 01:45:51.574375 fuse.DEBUG Op 0x000005cc        connection.go:491] -> OK (inode 4)
2022/07/29 01:45:51.575301 fuse.DEBUG Op 0x000005ce        connection.go:408] <- OpenFile (inode 4)
2022/07/29 01:45:51.575356 fuse.DEBUG OpenFile 4 .test []
2022/07/29 01:45:51.576730 fuse.DEBUG Op 0x000005ce        connection.go:491] -> OK ()
2022/07/29 01:45:51.578060 fuse.DEBUG Op 0x000005d0        connection.go:408] <- GetXattr (inode 4, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:51.579404 fuse.DEBUG GetXattr 4 .test [trusted.overlay.origin]
2022/07/29 01:45:51.580093 fuse.DEBUG Op 0x000005d0        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.580900 fuse.DEBUG Op 0x000005d2        connection.go:408] <- GetXattr (inode 4, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:51.581608 fuse.DEBUG GetXattr 4 .test [user.fuseoverlayfs.origin]
2022/07/29 01:45:51.581776 fuse.DEBUG Op 0x000005d2        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.582274 fuse.DEBUG Op 0x000005d4        connection.go:408] <- FlushFile (inode 4)
2022/07/29 01:45:51.584109 fuse.DEBUG FlushFile 4 .test []
2022/07/29 01:45:51.585351 fuse.DEBUG <-- FlushFile 4 .test [<nil> 106 4]
2022/07/29 01:45:51.587168 fuse.DEBUG Op 0x000005d4        connection.go:491] -> OK ()
2022/07/29 01:45:51.588513 fuse.DEBUG Op 0x000005d6        connection.go:408] <- ReleaseFileHandle
2022/07/29 01:45:51.589474 fuse.DEBUG ReleaseFileHandle .test 106 4
2022/07/29 01:45:51.590030 fuse.DEBUG Op 0x000005d6        connection.go:491] -> OK ()
2022/07/29 01:45:51.808008 fuse.DEBUG Op 0x000005d8        connection.go:408] <- LookUpInode (parent 1, name ".wh.haha")
2022/07/29 01:45:51.808099 fuse.DEBUG Inode.LookUp 1  [.wh.haha]
2022/07/29 01:45:51.811189 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.haha HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a53161a789e2a10d02bbc54a813c9892151e04d4859d2d490b009c9f7b597515
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.812090 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh.haha%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=bc79b3d45270aca40e6f2c565aceff841e1400684839961c0993119aa2b43a8c
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.814377 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.haha/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=5ffc2abc3d8ece44a4c18bb874f5efcfbfeed724b15ac6d75054d27421ea41c2
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.830757 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Id-2: zTvkZYwE2Ei02Gya6/v7MBoIv+uYXcQXypKcecdDseH7r8Eo0FTm9lVPixb4yPBGCNHC8rGHQ6RH
X-Amz-Request-Id: 9466DDFEF4181B5D


-----------------------------------------------------
2022/07/29 01:45:51.831003 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 9466DDFEF4181B5D, host id: zTvkZYwE2Ei02Gya6/v7MBoIv+uYXcQXypKcecdDseH7r8Eo0FTm9lVPixb4yPBGCNHC8rGHQ6RH
2022/07/29 01:45:51.831079 s3.DEBUG HEAD .wh.haha = no such file or directory
2022/07/29 01:45:51.840213 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: /qnDWFrhfynWMRHqSRPQ6bxRNxZbnPZ8272lu46jlbZSSCdaSgw/6OYplW2tjoQOtSnyFjiJmtz6
X-Amz-Request-Id: FD4AB376B3AF1318


-----------------------------------------------------
2022/07/29 01:45:51.840297 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: 5amz28o40UMvC7F1n21CZBNInhcrfQ6GBQ5v3iMAzgjroqOkw2ztSYb4mw/9eeZ605U8LkNu0wyS
X-Amz-Request-Id: B19CA80E804AF06B


-----------------------------------------------------
2022/07/29 01:45:51.840337 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:51.840363 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: B19CA80E804AF06B, host id: 5amz28o40UMvC7F1n21CZBNInhcrfQ6GBQ5v3iMAzgjroqOkw2ztSYb4mw/9eeZ605U8LkNu0wyS
2022/07/29 01:45:51.840399 s3.DEBUG HEAD .wh.haha/ = no such file or directory
2022/07/29 01:45:51.840523 fuse.DEBUG <-- LookUpInode 1 .wh.haha no such file or directory
2022/07/29 01:45:51.840739 fuse.DEBUG Op 0x000005d8        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:51.841878 fuse.DEBUG Op 0x000005da        connection.go:408] <- OpenFile (inode 5)
2022/07/29 01:45:51.842053 fuse.DEBUG OpenFile 5 haha []
2022/07/29 01:45:51.842712 fuse.DEBUG Op 0x000005da        connection.go:491] -> OK ()
2022/07/29 01:45:51.843597 fuse.DEBUG Op 0x000005dc        connection.go:408] <- GetXattr (inode 5, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:51.844040 fuse.DEBUG GetXattr 5 haha [trusted.overlay.origin]
2022/07/29 01:45:51.844340 fuse.DEBUG Op 0x000005dc        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.845838 fuse.DEBUG Op 0x000005de        connection.go:408] <- GetXattr (inode 5, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:51.846167 fuse.DEBUG GetXattr 5 haha [user.fuseoverlayfs.origin]
2022/07/29 01:45:51.846494 fuse.DEBUG Op 0x000005de        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.848572 fuse.DEBUG Op 0x000005e0        connection.go:408] <- FlushFile (inode 5)
2022/07/29 01:45:51.852183 fuse.DEBUG FlushFile 5 haha []
2022/07/29 01:45:51.852389 fuse.DEBUG <-- FlushFile 5 haha [<nil> 107 5]
2022/07/29 01:45:51.853049 fuse.DEBUG Op 0x000005e0        connection.go:491] -> OK ()
2022/07/29 01:45:51.859218 fuse.DEBUG Op 0x000005e2        connection.go:408] <- ReleaseFileHandle
2022/07/29 01:45:51.859769 fuse.DEBUG ReleaseFileHandle haha 107 5
2022/07/29 01:45:51.860643 fuse.DEBUG Op 0x000005e2        connection.go:491] -> OK ()
2022/07/29 01:45:51.920701 fuse.DEBUG Op 0x000005e4        connection.go:408] <- LookUpInode (parent 1, name ".wh.s3fs_exist.txt")
2022/07/29 01:45:51.921255 fuse.DEBUG Inode.LookUp 1  [.wh.s3fs_exist.txt]
2022/07/29 01:45:51.922328 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.s3fs_exist.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2f4c453b4de05443a21c9b31075f358205aa59e5c9c7c387466646e3a7a09db2
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.923417 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.s3fs_exist.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=58d11bec31bb6b766d8232728fa2f1faf58690c993af06fa18db0124f5d16e5d
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.923771 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh.s3fs_exist.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3df4bf319abf6e55b325c29689362923bbb97207bceb270feac233460af3cdd4
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164551Z


-----------------------------------------------------
2022/07/29 01:45:51.946318 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: jeaAs7HMTXglfG9pjWkom1t468q7MENevxdoPJq33suur251hV/oqdvfhE6es4QLOazdlpGgGWzU
X-Amz-Request-Id: 9CCB1755A711AF40


-----------------------------------------------------
2022/07/29 01:45:51.946397 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: 9UVjPe26Q+wISRFFEn6X7kJ709n4IOEBwBR6Q087I2irgUiDd40q290bTR3egkBUJCr3WWjnIPNH
X-Amz-Request-Id: 08749EF136495B19


-----------------------------------------------------
2022/07/29 01:45:51.946450 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 9CCB1755A711AF40, host id: jeaAs7HMTXglfG9pjWkom1t468q7MENevxdoPJq33suur251hV/oqdvfhE6es4QLOazdlpGgGWzU
2022/07/29 01:45:51.946466 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 08749EF136495B19, host id: 9UVjPe26Q+wISRFFEn6X7kJ709n4IOEBwBR6Q087I2irgUiDd40q290bTR3egkBUJCr3WWjnIPNH
2022/07/29 01:45:51.946529 s3.DEBUG HEAD .wh.s3fs_exist.txt/ = no such file or directory
2022/07/29 01:45:51.946632 s3.DEBUG HEAD .wh.s3fs_exist.txt = no such file or directory
2022/07/29 01:45:51.947630 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:51 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: ThWofjROPq6WoL8yFmEWKZ6Wcplf1XFJxfYCybRkQbV9W1a9VFDWuCH54WfmiT58B3XufOzSvUwQ
X-Amz-Request-Id: 5E26FDE30355C42C


-----------------------------------------------------
2022/07/29 01:45:51.947724 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:51.947864 fuse.DEBUG <-- LookUpInode 1 .wh.s3fs_exist.txt no such file or directory
2022/07/29 01:45:51.948078 fuse.DEBUG Op 0x000005e4        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:51.949042 fuse.DEBUG Op 0x000005e6        connection.go:408] <- OpenFile (inode 6)
2022/07/29 01:45:51.949251 fuse.DEBUG OpenFile 6 s3fs_exist.txt []
2022/07/29 01:45:51.950152 fuse.DEBUG Op 0x000005e6        connection.go:491] -> OK ()
2022/07/29 01:45:51.950514 fuse.DEBUG Op 0x000005e8        connection.go:408] <- GetXattr (inode 6, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:51.951392 fuse.DEBUG GetXattr 6 s3fs_exist.txt [trusted.overlay.origin]
2022/07/29 01:45:51.951507 fuse.DEBUG Op 0x000005e8        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.953028 fuse.DEBUG Op 0x000005ea        connection.go:408] <- GetXattr (inode 6, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:51.953194 fuse.DEBUG GetXattr 6 s3fs_exist.txt [user.fuseoverlayfs.origin]
2022/07/29 01:45:51.954655 fuse.DEBUG Op 0x000005ea        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:51.955234 fuse.DEBUG Op 0x000005ec        connection.go:408] <- FlushFile (inode 6)
2022/07/29 01:45:51.956098 fuse.DEBUG FlushFile 6 s3fs_exist.txt []
2022/07/29 01:45:51.957088 fuse.DEBUG <-- FlushFile 6 s3fs_exist.txt [<nil> 108 6]
2022/07/29 01:45:51.957627 fuse.DEBUG Op 0x000005ec        connection.go:491] -> OK ()
2022/07/29 01:45:51.958085 fuse.DEBUG Op 0x000005ee        connection.go:408] <- ReleaseFileHandle
2022/07/29 01:45:51.958241 fuse.DEBUG ReleaseFileHandle s3fs_exist.txt 108 6
2022/07/29 01:45:51.958309 fuse.DEBUG Op 0x000005ee        connection.go:491] -> OK ()
2022/07/29 01:45:52.033627 fuse.DEBUG Op 0x000005f0        connection.go:408] <- LookUpInode (parent 1, name ".wh.test.txt")
2022/07/29 01:45:52.034377 fuse.DEBUG Inode.LookUp 1  [.wh.test.txt]
2022/07/29 01:45:52.037651 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh.test.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7bd9b9d9fff3d07e79178bbd8e20b11bba02f5dba7251921283ee0d6a936c51f
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z


-----------------------------------------------------
2022/07/29 01:45:52.039428 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.test.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=b093f5b1f460dfd6e842cef244207e39dd14be5b8486e12b7560793453a1d618
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z


-----------------------------------------------------
2022/07/29 01:45:52.037775 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.test.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7d7b32cc46b5d4874722633b07ce9c8484ccab3b94a7f45487963e39cb68a137
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z


-----------------------------------------------------
2022/07/29 01:45:52.068083 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: kUHYNm5p+BbZborrPncv1anlBLu3VuzvjoVib7ZrIBe/vFxTeC4cZLNdEIgTz7OxkHn6JD1EB25S
X-Amz-Request-Id: 8A50D08FB2138805


-----------------------------------------------------
2022/07/29 01:45:52.068361 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:52.068674 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: oT2/2rcUut7XLn7Arx2l7Y7sSpsn8XTajo96yBX2Qqx3AOSK5l1TlvyU2ErxCO2WC2Y5p8NdVnSV
X-Amz-Request-Id: 0B1D33662AC1C5DE


-----------------------------------------------------
2022/07/29 01:45:52.069099 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 0B1D33662AC1C5DE, host id: oT2/2rcUut7XLn7Arx2l7Y7sSpsn8XTajo96yBX2Qqx3AOSK5l1TlvyU2ErxCO2WC2Y5p8NdVnSV
2022/07/29 01:45:52.069327 s3.DEBUG HEAD .wh.test.txt = no such file or directory
2022/07/29 01:45:52.072286 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: e1SJ9AGKrKsy3JOLPMZdZMmW8zl7ZZgENnxtgPT41YZs3Xjo3S6moKOvE/Ck/iuR6vwgwLPa+lC5
X-Amz-Request-Id: 1F1BEAE88758367B


-----------------------------------------------------
2022/07/29 01:45:52.072411 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 1F1BEAE88758367B, host id: e1SJ9AGKrKsy3JOLPMZdZMmW8zl7ZZgENnxtgPT41YZs3Xjo3S6moKOvE/Ck/iuR6vwgwLPa+lC5
2022/07/29 01:45:52.072507 s3.DEBUG HEAD .wh.test.txt/ = no such file or directory
2022/07/29 01:45:52.072547 fuse.DEBUG <-- LookUpInode 1 .wh.test.txt no such file or directory
2022/07/29 01:45:52.073505 fuse.DEBUG Op 0x000005f0        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:52.074021 fuse.DEBUG Op 0x000005f2        connection.go:408] <- LookUpInode (parent 1, name "test.txt")
2022/07/29 01:45:52.074387 fuse.DEBUG Ref 7 test.txt [10]
2022/07/29 01:45:52.075800 fuse.DEBUG <-- LookUpInode 1 test.txt <nil>
2022/07/29 01:45:52.076003 fuse.DEBUG Op 0x000005f2        connection.go:491] -> OK (inode 7)
2022/07/29 01:45:52.076649 fuse.DEBUG Op 0x000005f4        connection.go:408] <- OpenFile (inode 7)
2022/07/29 01:45:52.077008 fuse.DEBUG OpenFile 7 test.txt []
2022/07/29 01:45:52.078126 fuse.DEBUG Op 0x000005f4        connection.go:491] -> OK ()
2022/07/29 01:45:52.079043 fuse.DEBUG Op 0x000005f6        connection.go:408] <- GetXattr (inode 7, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:52.079204 fuse.DEBUG GetXattr 7 test.txt [trusted.overlay.origin]
2022/07/29 01:45:52.079292 fuse.DEBUG Op 0x000005f6        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:52.079641 fuse.DEBUG Op 0x000005f8        connection.go:408] <- GetXattr (inode 7, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:52.080601 fuse.DEBUG GetXattr 7 test.txt [user.fuseoverlayfs.origin]
2022/07/29 01:45:52.080818 fuse.DEBUG Op 0x000005f8        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:52.081528 fuse.DEBUG Op 0x000005fa        connection.go:408] <- FlushFile (inode 7)
2022/07/29 01:45:52.082721 fuse.DEBUG FlushFile 7 test.txt []
2022/07/29 01:45:52.083034 fuse.DEBUG <-- FlushFile 7 test.txt [<nil> 109 7]
2022/07/29 01:45:52.084312 fuse.DEBUG Op 0x000005fa        connection.go:491] -> OK ()
2022/07/29 01:45:52.085691 fuse.DEBUG Op 0x000005fc        connection.go:408] <- ReleaseFileHandle
2022/07/29 01:45:52.085901 fuse.DEBUG ReleaseFileHandle test.txt 109 7
2022/07/29 01:45:52.086050 fuse.DEBUG Op 0x000005fc        connection.go:491] -> OK ()
2022/07/29 01:45:52.148247 fuse.DEBUG Op 0x000005fe        connection.go:408] <- LookUpInode (parent 1, name ".wh.wasabi.txt")
2022/07/29 01:45:52.148605 fuse.DEBUG Inode.LookUp 1  [.wh.wasabi.txt]
2022/07/29 01:45:52.150579 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.wasabi.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=898dc42e06044a270000fc1ae8499bb0c12a9f42855e2cc7a7478d6e409a0703
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z


-----------------------------------------------------
2022/07/29 01:45:52.150636 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/.wh.wasabi.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=90f2caa96ba12a73751e4ecd2bf22e02aaa450f2947b9a7f46196844a8f4598a
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z


-----------------------------------------------------
2022/07/29 01:45:52.150782 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=.wh.wasabi.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220728/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=bce7abb2794d937b78750472b882d0eba2918043f61db5a731970c95d29d62de
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220728T164552Z


-----------------------------------------------------
2022/07/29 01:45:52.171246 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: o3jsQfpnL/8mRYbQ8S/6bSui8gHDEJxWpw5BxBlDAtx+UaYz3IegbOCGJdVBmYUNIfeYZ5Gd/jZy
X-Amz-Request-Id: B662050F4099F5BE


-----------------------------------------------------
2022/07/29 01:45:52.171312 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: B662050F4099F5BE, host id: o3jsQfpnL/8mRYbQ8S/6bSui8gHDEJxWpw5BxBlDAtx+UaYz3IegbOCGJdVBmYUNIfeYZ5Gd/jZy
2022/07/29 01:45:52.171409 s3.DEBUG HEAD .wh.wasabi.txt/ = no such file or directory
2022/07/29 01:45:52.175005 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head1)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: 7Ttw9LjYUGmXMm3Q2ZGowVE2qytY4eclaD6esqB+fzqMCuIEXwEoz/qWeWvR2kNrDShA4QWBt+r3
X-Amz-Request-Id: E4FD9FEB56CA7874


-----------------------------------------------------
2022/07/29 01:45:52.175562 s3.DEBUG &{[] [] <nil> false }
2022/07/29 01:45:52.211338 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 28 Jul 2022 16:45:52 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head2)
X-Amz-Id-2: l/drxGJvFfCRhzdnj7Wly8ZyQs95kl6emwu3byBGcugOoxFMk/0hTatIMPc9BvC7z+Ym913sd8Tj
X-Amz-Request-Id: 710A9CCB1C33F10C


-----------------------------------------------------
2022/07/29 01:45:52.211474 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 710A9CCB1C33F10C, host id: l/drxGJvFfCRhzdnj7Wly8ZyQs95kl6emwu3byBGcugOoxFMk/0hTatIMPc9BvC7z+Ym913sd8Tj
2022/07/29 01:45:52.211527 s3.DEBUG HEAD .wh.wasabi.txt = no such file or directory
2022/07/29 01:45:52.211559 fuse.DEBUG <-- LookUpInode 1 .wh.wasabi.txt no such file or directory
2022/07/29 01:45:52.212998 fuse.DEBUG Op 0x000005fe        connection.go:493] -> Error: "no such file or directory"
2022/07/29 01:45:52.213220 fuse.DEBUG Op 0x00000600        connection.go:408] <- OpenFile (inode 8)
2022/07/29 01:45:52.213358 fuse.DEBUG OpenFile 8 wasabi.txt []
2022/07/29 01:45:52.214050 fuse.DEBUG Op 0x00000600        connection.go:491] -> OK ()
2022/07/29 01:45:52.214475 fuse.DEBUG Op 0x00000602        connection.go:408] <- GetXattr (inode 8, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 01:45:52.215072 fuse.DEBUG GetXattr 8 wasabi.txt [trusted.overlay.origin]
2022/07/29 01:45:52.215288 fuse.DEBUG Op 0x00000602        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:52.215855 fuse.DEBUG Op 0x00000604        connection.go:408] <- GetXattr (inode 8, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 01:45:52.216184 fuse.DEBUG GetXattr 8 wasabi.txt [user.fuseoverlayfs.origin]
2022/07/29 01:45:52.217218 fuse.DEBUG Op 0x00000604        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:52.218428 fuse.DEBUG Op 0x00000606        connection.go:408] <- FlushFile (inode 8)
2022/07/29 01:45:52.219100 fuse.DEBUG FlushFile 8 wasabi.txt []
2022/07/29 01:45:52.219242 fuse.DEBUG <-- FlushFile 8 wasabi.txt [<nil> 110 8]
2022/07/29 01:45:52.219425 fuse.DEBUG Op 0x00000606        connection.go:491] -> OK ()
2022/07/29 01:45:52.221339 fuse.DEBUG Op 0x00000608        connection.go:408] <- ReleaseFileHandle
2022/07/29 01:45:52.221744 fuse.DEBUG ReleaseFileHandle wasabi.txt 110 8
2022/07/29 01:45:52.222356 fuse.DEBUG Op 0x00000608        connection.go:491] -> OK ()
2022/07/29 01:45:52.293515 fuse.DEBUG Op 0x0000060a        connection.go:408] <- ReadDir (inode 1)
2022/07/29 01:45:52.293733 fuse.DEBUG ReadDir 1  [7]
2022/07/29 01:45:52.294060 fuse.DEBUG Op 0x0000060a        connection.go:491] -> OK ()
2022/07/29 01:45:52.294882 fuse.DEBUG Op 0x0000060c        connection.go:408] <- OpenDir (inode 1)
2022/07/29 01:45:52.295223 fuse.DEBUG OpenDir 1  []
2022/07/29 01:45:52.295596 fuse.DEBUG Op 0x0000060c        connection.go:491] -> OK ()
2022/07/29 01:45:52.296401 fuse.DEBUG Op 0x0000060e        connection.go:408] <- GetXattr (inode 1, name "trusted.overlay.opaque", name trusted.overlay.opaque)
2022/07/29 01:45:52.297256 fuse.DEBUG GetXattr 1  [trusted.overlay.opaque]
2022/07/29 01:45:52.299099 fuse.DEBUG Op 0x0000060e        connection.go:493] -> Error: "no data available"
2022/07/29 01:45:52.299594 fuse.DEBUG Op 0x00000610        connection.go:408] <- ReleaseDirHandle
2022/07/29 01:45:52.299774 fuse.DEBUG Op 0x00000612        connection.go:408] <- OpenDir (inode 1)
2022/07/29 01:45:52.300041 fuse.DEBUG OpenDir 1  []
2022/07/29 01:45:52.300785 fuse.DEBUG Op 0x00000612        connection.go:491] -> OK ()
2022/07/29 01:45:52.301137 fuse.DEBUG Op 0x00000614        connection.go:408] <- GetXattr (inode 1, name "user.fuseoverlayfs.opaque", name user.fuseoverlayfs.opaque)
2022/07/29 01:45:52.301543 fuse.DEBUG GetXattr 1  [user.fuseoverlayfs.opaque]
2022/07/29 01:45:52.302702 fuse.DEBUG ReleaseDirHandle 
2022/07/29 01:45:52.302399 s3.DEBUG DEBUG: Validate Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:52.303376 s3.DEBUG DEBUG: Build Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:52.303411 s3.DEBUG DEBUG: Sign Request s3/HeadObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:52.303499 s3.ERROR code=InvalidParameter msg=1 validation error(s) found., err=InvalidParameter: 1 validation error(s) found.
caused by: ParamMinLenError: minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:52.303550 s3.ERROR code=InvalidParameter msg=1 validation error(s) found., err=InvalidParameter: 1 validation error(s) found.
caused by: ParamMinLenError: minimum field size of 1, HeadObjectInput.Key.

2022/07/29 01:45:52.305516 fuse.DEBUG Op 0x00000610        connection.go:491] -> OK ()
2022/07/29 01:45:52.306354 fuse.DEBUG Op 0x00000614        connection.go:493] -> Error: "InvalidParameter: 1 validation error(s) found.\n- minimum field size of 1, HeadObjectInput.Key.\n"
2022/07/29 01:45:52.307036 fuse.ERROR *fuseops.GetXattrOp error: InvalidParameter: 1 validation error(s) found.
2022/07/29 01:45:52.308539 fuse.ERROR - minimum field size of 1, HeadObjectInput.Key.
2022/07/29 01:45:52.308241 fuse.DEBUG Op 0x00000616        connection.go:408] <- ReleaseDirHandle
2022/07/29 01:45:52.308382 fuse.DEBUG ReleaseDirHandle 
2022/07/29 01:45:52.309270 fuse.DEBUG ReleaseDirHandle 
2022/07/29 01:45:52.309715 fuse.DEBUG Op 0x00000618        connection.go:408] <- ReleaseDirHandle
2022/07/29 01:45:52.310812 fuse.DEBUG Op 0x0000061a        connection.go:408] <- ReleaseDirHandle
2022/07/29 01:45:52.310812 fuse.DEBUG ReleaseDirHandle 
2022/07/29 01:45:52.311243 fuse.DEBUG Op 0x00000616        connection.go:491] -> OK ()
2022/07/29 01:45:52.311317 fuse.DEBUG Op 0x0000061a        connection.go:491] -> OK ()
2022/07/29 01:45:52.311676 fuse.DEBUG Op 0x00000618        connection.go:491] -> OK ()

Any ideas would be appreciated"
Thank you
@giuseppe
Copy link
Member

workdir and upperdir must be two different directories, but they must be on the same file system

@corbinjurgens
Copy link
Author

corbinjurgens commented Jul 29, 2022 •
edited
Loading

Thank you! That is one thing I was unsure about. When you say same file system, do you mean it can be a entirely different goofys s3 mount location (as long as its still a fuse s3 mount), or should be the same disk (meaning i would need to make two folders on a disk, one for data and one for workdir)?

I had tried making workdir to something like workdir=/mnt/html/upload_wasabi/overlay but that doesn't work either.

Also, in my particular use case where these same disks are mounted on many machines, I would want to make sure workdir contents are shared correct?

@corbinjurgens
Copy link
Author

corbinjurgens commented Jul 29, 2022 •
edited
Loading

I tried

sudo fuse-overlayfs -f -d  -o lowerdir=/mnt/html/upload_aws -o upperdir=/mnt/html/upload_wasabi/data -o workdir=/mnt/html/upload_wasabi/overlay /mnt/html/upload

uid=unchanged
uid=unchanged
upperdir=/mnt/html/upload_wasabi/data
workdir=/mnt/html/upload_wasabi/overlay
lowerdir=/mnt/html/upload_aws
mountpoint=/mnt/html/upload
plugins=<none>
fuse-overlayfs: cannot read upper dir: Input/output error

But still same error.

Full goofys log


2022/07/29 17:46:01.219660 fuse.DEBUG Op 0x000000d2        connection.go:408] <- GetInodeAttributes (inode 1)
2022/07/29 17:46:01.219953 fuse.DEBUG GetAttributes 1  []
2022/07/29 17:46:01.220819 fuse.DEBUG Op 0x000000d2        connection.go:491] -> OK ()
2022/07/29 17:46:01.222007 fuse.DEBUG Op 0x000000d4        connection.go:408] <- LookUpInode (parent 1, name "data")
2022/07/29 17:46:01.222556 fuse.DEBUG Ref 7 data [2]
2022/07/29 17:46:01.222773 fuse.DEBUG lookup expired 7 data []
2022/07/29 17:46:01.222806 fuse.DEBUG Inode.LookUp 1  [data]
2022/07/29 17:46:01.223650 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7be1a2a276cfcb383040d6c77546fffd62640ccba1be09bf9aeb1193d727a99d
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z


-----------------------------------------------------
2022/07/29 17:46:01.223938 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ac560fdc31d6bcecb1dcf6cf3bb47315b2e0dabdcd74b29db0a64ff55d873939
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z


-----------------------------------------------------
2022/07/29 17:46:01.225187 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7c6e50c17c0bb189b49d035e9457612ba0d5912a6efc26327c21ab7d9876c7fd
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z


-----------------------------------------------------
2022/07/29 17:46:01.936117 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: mLm3MHEtwJqQxV+dNYRqtzi56H+VLZZ80B9e8oKMPFPrURRyPPRnod6wcYMT5YwPLNSpmiLunSm4
X-Amz-Request-Id: 5D87A5C31251D28D


-----------------------------------------------------
2022/07/29 17:46:01.936191 s3.DEBUG &{[] [data/: 0] 0xc0002c0388 true }
2022/07/29 17:46:01.936531 fuse.DEBUG invalidate cache because attributes changed 7 data [{4096 2022-07-29 08:43:31 +0000 UTC} {4096 0001-01-01 00:00:00 +0000 UTC}]
2022/07/29 17:46:01.936813 fuse.DEBUG <-- LookUpInode 1 data <nil>
2022/07/29 17:46:01.937402 fuse.DEBUG Op 0x000000d4        connection.go:491] -> OK (inode 7)
2022/07/29 17:46:01.942743 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Fri, 29 Jul 2022 08:46:02 GMT
Etag: "d41d8cd98f00b204e9800998ecf8427e"
Last-Modified: Fri, 29 Jul 2022 08:43:16 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: GIyLMoxKyrMp+o2OAOxOqpPiI37UYSN895bZXACFaGBcGWCYm+/3W7iYwb1XgpVK1jz+BaAzEt9I
X-Amz-Request-Id: 9E1E57B996BDD12C
Content-Length: 0


-----------------------------------------------------
2022/07/29 17:46:01.942873 s3.DEBUG data/: 0
2022/07/29 17:46:01.943797 fuse.DEBUG Op 0x000000d6        connection.go:408] <- OpenDir (inode 7)
2022/07/29 17:46:01.944254 fuse.DEBUG OpenDir 7 data []
2022/07/29 17:46:01.947118 fuse.DEBUG Op 0x000000d6        connection.go:491] -> OK ()
2022/07/29 17:46:01.947741 fuse.DEBUG Op 0x000000d8        connection.go:408] <- OpenDir (inode 7)
2022/07/29 17:46:01.948159 fuse.DEBUG OpenDir 7 data []
2022/07/29 17:46:01.948534 fuse.DEBUG Op 0x000000d8        connection.go:491] -> OK ()
2022/07/29 17:46:01.949749 fuse.DEBUG Op 0x000000da        connection.go:408] <- GetXattr (inode 7, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 17:46:01.950377 fuse.DEBUG GetXattr 7 data [trusted.overlay.origin]
2022/07/29 17:46:01.950780 fuse.DEBUG Op 0x000000da        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:01.951910 fuse.DEBUG Op 0x000000dc        connection.go:408] <- GetXattr (inode 7, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 17:46:01.952191 fuse.DEBUG GetXattr 7 data [user.fuseoverlayfs.origin]
2022/07/29 17:46:01.952416 fuse.DEBUG Op 0x000000dc        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:01.953248 fuse.DEBUG Op 0x000000de        connection.go:408] <- ReleaseDirHandle
2022/07/29 17:46:01.953757 fuse.DEBUG ReleaseDirHandle data
2022/07/29 17:46:01.954239 fuse.DEBUG Op 0x000000de        connection.go:491] -> OK ()
2022/07/29 17:46:01.956795 fuse.DEBUG Op 0x000000e0        connection.go:408] <- LookUpInode (parent 7, name ".wh.")
2022/07/29 17:46:01.957716 fuse.DEBUG Inode.LookUp 7 data [.wh.]
2022/07/29 17:46:01.958970 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh. HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=fa8423dc364c1142e3e367f701aa4678810233dada6d427bf5a719331c448417
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z


-----------------------------------------------------
2022/07/29 17:46:01.959832 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh./ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d5c8c0ca2643f275feb27be461c9acdd0ae99f3059ca968ec33f9c2418352998
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z


-----------------------------------------------------
2022/07/29 17:46:01.960422 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Id-2: BTqVNNgbeW1rIvoHZvN6ThWu0JdBXvKWOrIYRrSXGgp+tjNuhhE4B/xwgp+aNCOyrcYLx7/qQPOV
X-Amz-Request-Id: DBBF63656B04E180


-----------------------------------------------------
2022/07/29 17:46:01.960791 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: DBBF63656B04E180, host id: BTqVNNgbeW1rIvoHZvN6ThWu0JdBXvKWOrIYRrSXGgp+tjNuhhE4B/xwgp+aNCOyrcYLx7/qQPOV
2022/07/29 17:46:01.962197 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh.%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=90285c9b0b15cdf07382b72b3ff6ced703098e92f3762dadbf5974a8b7ad1db3
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z


-----------------------------------------------------
2022/07/29 17:46:01.983259 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: RzBuNOnrbWj0d90gcWm0zQuj5QvYdaw/OCfbxmNznNWoM2K7z/DxumFvP0J9cR1T+lngozbjWp/a
X-Amz-Request-Id: 9C2CF3986FB0E938


-----------------------------------------------------
2022/07/29 17:46:01.983326 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 9C2CF3986FB0E938, host id: RzBuNOnrbWj0d90gcWm0zQuj5QvYdaw/OCfbxmNznNWoM2K7z/DxumFvP0J9cR1T+lngozbjWp/a
2022/07/29 17:46:01.983855 s3.DEBUG HEAD data/.wh. = no such file or directory
2022/07/29 17:46:01.987219 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: I19h+kTzy3WbF/3ISma+/myGtmUnP9Oj0fYQWpxr9IMQ5EXsi0vIWFwBLjJtvxHlMNiWm0bZ+GW2
X-Amz-Request-Id: 79F6CFC5796AE57B


-----------------------------------------------------
2022/07/29 17:46:01.987281 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 79F6CFC5796AE57B, host id: I19h+kTzy3WbF/3ISma+/myGtmUnP9Oj0fYQWpxr9IMQ5EXsi0vIWFwBLjJtvxHlMNiWm0bZ+GW2
2022/07/29 17:46:01.987380 s3.DEBUG HEAD data/.wh./ = no such file or directory
2022/07/29 17:46:01.987861 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: MausWABbFee4cpL/xd9AQk+D2ifnehEj+MPNfdAmcZI3Oan8g1WUAY0dQWFQsOaJ/yKh1BUnF6BB
X-Amz-Request-Id: 3F282589FCBCAA17


-----------------------------------------------------
2022/07/29 17:46:01.988769 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:01.988927 fuse.DEBUG <-- LookUpInode 7 .wh. no such file or directory
2022/07/29 17:46:01.989321 fuse.DEBUG Op 0x000000e0        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:01.990229 fuse.DEBUG Op 0x000000e2        connection.go:408] <- OpenDir (inode 7)
2022/07/29 17:46:01.991152 fuse.DEBUG OpenDir 7 data []
2022/07/29 17:46:01.991981 fuse.DEBUG Op 0x000000e2        connection.go:491] -> OK ()
2022/07/29 17:46:01.993630 fuse.DEBUG ReadDir 7 data [0]
2022/07/29 17:46:01.993354 fuse.DEBUG Op 0x000000e4        connection.go:408] <- ReadDir (inode 7)
2022/07/29 17:46:01.994741 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&prefix=data%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=951ec57ba0654355286846c1e5078022654d2729577742daafc6a26f8498feab
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084601Z


-----------------------------------------------------
2022/07/29 17:46:02.017300 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: Qf7g/B/leLEP+FfTUeQJVDP19gOkriuUVxW9MzKK89qYSjvFBB7brmkgnIzyaGFKkC4bLVR3iJZ+
X-Amz-Request-Id: 64DC7AE6FA417D90


-----------------------------------------------------
2022/07/29 17:46:02.017482 s3.DEBUG &{[] [data/: 0 data/haha: 2 data/s3fs_exist.txt: 0 data/test.txt: 6 data/wasabi.txt: 11] <nil> false }
2022/07/29 17:46:02.017844 fuse.DEBUG <-- ReadDir 7 data [. 1]
2022/07/29 17:46:02.017915 fuse.DEBUG <-- ReadDir 7 data [.. 2]
2022/07/29 17:46:02.018034 fuse.DEBUG <-- ReadDir 7 data [haha 3]
2022/07/29 17:46:02.018095 fuse.DEBUG <-- ReadDir 7 data [s3fs_exist.txt 4]
2022/07/29 17:46:02.018401 fuse.DEBUG <-- ReadDir 7 data [test.txt 5]
2022/07/29 17:46:02.018779 fuse.DEBUG <-- ReadDir 7 data [wasabi.txt 6]
2022/07/29 17:46:02.019234 fuse.DEBUG Op 0x000000e4        connection.go:491] -> OK ()
2022/07/29 17:46:02.020003 fuse.DEBUG Op 0x000000e6        connection.go:408] <- LookUpInode (parent 7, name ".wh.haha")
2022/07/29 17:46:02.021293 fuse.DEBUG Inode.LookUp 7 data [.wh.haha]
2022/07/29 17:46:02.022389 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh.haha%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=773b3485a88f4ac8ee209291a6c7b3d983b6196a761d4631bbffd77963179a51
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.022441 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.haha/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=63a302708d40c0cddf1b3501a2505f9e954f3f3adfe369b2f8100d97f096ccd8
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.022817 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.haha HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e1d15934d1df54a449231e9c2d61d8962758684f0e1b0e6c3bcf92fb611ad684
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.042206 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: nQQNcRq2HMR2yZ6MJbhRjJDxnynkSzMaDKgsGw5TzmcI2N3z+Tn4K6kblkeHJAy/e2YpmQRzRssU
X-Amz-Request-Id: 9EAC2EC7AAA296B1


-----------------------------------------------------
2022/07/29 17:46:02.042321 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:02.046026 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: Q+RsYdLHT/C7kPIR6DP6YZ6Qd+8Q6MrKLZOBM2BK6X+1QYMJ8VlVu8pP3cvXBW8oI3TC5KisagKI
X-Amz-Request-Id: E5F18A1C8B39E5E3


-----------------------------------------------------
2022/07/29 17:46:02.046118 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: E5F18A1C8B39E5E3, host id: Q+RsYdLHT/C7kPIR6DP6YZ6Qd+8Q6MrKLZOBM2BK6X+1QYMJ8VlVu8pP3cvXBW8oI3TC5KisagKI
2022/07/29 17:46:02.046296 s3.DEBUG HEAD data/.wh.haha/ = no such file or directory
2022/07/29 17:46:02.047439 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: QD5MtiUM5Jtm8uuwD1vM+HtJ/TnnEHHlIq8g5MLTj0c7RxpwWdgkNr8n5AgNbELisk6fGXbqmzYS
X-Amz-Request-Id: C939F87AE3440EB5


-----------------------------------------------------
2022/07/29 17:46:02.047741 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: C939F87AE3440EB5, host id: QD5MtiUM5Jtm8uuwD1vM+HtJ/TnnEHHlIq8g5MLTj0c7RxpwWdgkNr8n5AgNbELisk6fGXbqmzYS
2022/07/29 17:46:02.047809 s3.DEBUG HEAD data/.wh.haha = no such file or directory
2022/07/29 17:46:02.047892 fuse.DEBUG <-- LookUpInode 7 .wh.haha no such file or directory
2022/07/29 17:46:02.048262 fuse.DEBUG Op 0x000000e6        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:02.048921 fuse.DEBUG Op 0x000000e8        connection.go:408] <- LookUpInode (parent 7, name "haha")
2022/07/29 17:46:02.049199 fuse.DEBUG Ref 3 data/haha [1]
2022/07/29 17:46:02.050276 fuse.DEBUG <-- LookUpInode 7 haha <nil>
2022/07/29 17:46:02.050744 fuse.DEBUG Op 0x000000e8        connection.go:491] -> OK (inode 3)
2022/07/29 17:46:02.051780 fuse.DEBUG Op 0x000000ea        connection.go:408] <- OpenFile (inode 3)
2022/07/29 17:46:02.051931 fuse.DEBUG OpenFile 3 data/haha []
2022/07/29 17:46:02.052524 fuse.DEBUG Op 0x000000ea        connection.go:491] -> OK ()
2022/07/29 17:46:02.053639 fuse.DEBUG Op 0x000000ec        connection.go:408] <- GetXattr (inode 3, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 17:46:02.053812 fuse.DEBUG GetXattr 3 data/haha [trusted.overlay.origin]
2022/07/29 17:46:02.053969 fuse.DEBUG Op 0x000000ec        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.054509 fuse.DEBUG Op 0x000000ee        connection.go:408] <- GetXattr (inode 3, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 17:46:02.055040 fuse.DEBUG GetXattr 3 data/haha [user.fuseoverlayfs.origin]
2022/07/29 17:46:02.055163 fuse.DEBUG Op 0x000000ee        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.056082 fuse.DEBUG FlushFile 3 data/haha []
2022/07/29 17:46:02.056162 fuse.DEBUG <-- FlushFile 3 data/haha [<nil> 18 3]
2022/07/29 17:46:02.056359 fuse.DEBUG Op 0x000000f0        connection.go:408] <- FlushFile (inode 3)
2022/07/29 17:46:02.056742 fuse.DEBUG Op 0x000000f0        connection.go:491] -> OK ()
2022/07/29 17:46:02.057059 fuse.DEBUG Op 0x000000f2        connection.go:408] <- ReleaseFileHandle
2022/07/29 17:46:02.057202 fuse.DEBUG ReleaseFileHandle data/haha 18 3
2022/07/29 17:46:02.057421 fuse.DEBUG Op 0x000000f2        connection.go:491] -> OK ()
2022/07/29 17:46:02.228875 fuse.DEBUG Op 0x000000f4        connection.go:408] <- LookUpInode (parent 7, name ".wh.s3fs_exist.txt")
2022/07/29 17:46:02.229026 fuse.DEBUG Inode.LookUp 7 data [.wh.s3fs_exist.txt]
2022/07/29 17:46:02.230504 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.s3fs_exist.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ff16ef129599723d94550ae861f87f163ff8db6f1c722099db548989dc4db0f4
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.230578 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.s3fs_exist.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=dd41551993fe9d417d4c383b46edd44f218867ba74a5f1d4dd2dad7b99edac75
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.232342 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh.s3fs_exist.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=672a2076c585517a517568c7c872e3e84fcad57ef09e86cd85e287beebfcb48a
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.250949 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: gosoqlk0W6lgRpgcBcP5KHxgMyBoMjuMkJEjZgq+Dc9zv3yT0/VTN2ohvsGNVCfH87aI1qMbbRDv
X-Amz-Request-Id: A7DA0CDF774196EF


-----------------------------------------------------
2022/07/29 17:46:02.251015 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: A7DA0CDF774196EF, host id: gosoqlk0W6lgRpgcBcP5KHxgMyBoMjuMkJEjZgq+Dc9zv3yT0/VTN2ohvsGNVCfH87aI1qMbbRDv
2022/07/29 17:46:02.251190 s3.DEBUG HEAD data/.wh.s3fs_exist.txt/ = no such file or directory
2022/07/29 17:46:02.254850 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: jbfBflgsD6TG6bR8yjrUtltHxvPxtnX+2PMNWqabEk9/kvyLyPoJexH/4J5hiys4RA1uO1IZQZWN
X-Amz-Request-Id: 5182B56A26578396


-----------------------------------------------------
2022/07/29 17:46:02.254987 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 5182B56A26578396, host id: jbfBflgsD6TG6bR8yjrUtltHxvPxtnX+2PMNWqabEk9/kvyLyPoJexH/4J5hiys4RA1uO1IZQZWN
2022/07/29 17:46:02.255015 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: hf6s+XjpisCKS99E3YePvxhFUqPTCsOSprBKl8Qop3hm595WHLuE+fB9etB58YLoBAGD3gvoAxNq
X-Amz-Request-Id: 725A324681BB353D


-----------------------------------------------------
2022/07/29 17:46:02.256131 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:02.255145 s3.DEBUG HEAD data/.wh.s3fs_exist.txt = no such file or directory
2022/07/29 17:46:02.257221 fuse.DEBUG <-- LookUpInode 7 .wh.s3fs_exist.txt no such file or directory
2022/07/29 17:46:02.258195 fuse.DEBUG Op 0x000000f4        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:02.259275 fuse.DEBUG Op 0x000000f6        connection.go:408] <- LookUpInode (parent 7, name "s3fs_exist.txt")
2022/07/29 17:46:02.260405 fuse.DEBUG Ref 4 data/s3fs_exist.txt [1]
2022/07/29 17:46:02.261002 fuse.DEBUG <-- LookUpInode 7 s3fs_exist.txt <nil>
2022/07/29 17:46:02.262174 fuse.DEBUG Op 0x000000f6        connection.go:491] -> OK (inode 4)
2022/07/29 17:46:02.262933 fuse.DEBUG Op 0x000000f8        connection.go:408] <- OpenFile (inode 4)
2022/07/29 17:46:02.263181 fuse.DEBUG OpenFile 4 data/s3fs_exist.txt []
2022/07/29 17:46:02.263485 fuse.DEBUG Op 0x000000f8        connection.go:491] -> OK ()
2022/07/29 17:46:02.265339 fuse.DEBUG Op 0x000000fa        connection.go:408] <- GetXattr (inode 4, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 17:46:02.265800 fuse.DEBUG GetXattr 4 data/s3fs_exist.txt [trusted.overlay.origin]
2022/07/29 17:46:02.265932 fuse.DEBUG Op 0x000000fa        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.267369 fuse.DEBUG Op 0x000000fc        connection.go:408] <- GetXattr (inode 4, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 17:46:02.267413 fuse.DEBUG GetXattr 4 data/s3fs_exist.txt [user.fuseoverlayfs.origin]
2022/07/29 17:46:02.268331 fuse.DEBUG Op 0x000000fc        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.269584 fuse.DEBUG Op 0x000000fe        connection.go:408] <- FlushFile (inode 4)
2022/07/29 17:46:02.269999 fuse.DEBUG FlushFile 4 data/s3fs_exist.txt []
2022/07/29 17:46:02.270088 fuse.DEBUG <-- FlushFile 4 data/s3fs_exist.txt [<nil> 19 4]
2022/07/29 17:46:02.271082 fuse.DEBUG Op 0x000000fe        connection.go:491] -> OK ()
2022/07/29 17:46:02.271454 fuse.DEBUG Op 0x00000100        connection.go:408] <- ReleaseFileHandle
2022/07/29 17:46:02.271791 fuse.DEBUG ReleaseFileHandle data/s3fs_exist.txt 19 4
2022/07/29 17:46:02.272102 fuse.DEBUG Op 0x00000100        connection.go:491] -> OK ()
2022/07/29 17:46:02.349135 fuse.DEBUG Op 0x00000102        connection.go:408] <- LookUpInode (parent 7, name ".wh.test.txt")
2022/07/29 17:46:02.349368 fuse.DEBUG Inode.LookUp 7 data [.wh.test.txt]
2022/07/29 17:46:02.351002 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh.test.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0f57739327956f6b6bb6b70fd7d008a6ffd417d9f6e6465fc335e919bddaa562
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.351070 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.test.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a26b7d7dda17c5c2deffea4bd2c5dabf4458f6cb69339b0ed43be7395c00c927
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.350959 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.test.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4f56cc5b160ab936d3cea85bd1df2b575f9bb1104ef05dd0aadbae8fe988f7cc
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.370923 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: aywVI6em2Phum6noQVVETvUCN62pHaE/6BqfgL0Ive+ENrLiYrB3SCwv0uFdNU5uITbIy0DXA76w
X-Amz-Request-Id: 31395DB1FA78EBCD


-----------------------------------------------------
2022/07/29 17:46:02.370983 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:02.439007 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: Mlqwmen02/EnzQYYfLe0nWIinY+VVZJlbZoSO1o+1eHd+swKDY9aqs59Im5Ex1ZQvXbeqP9t+NuM
X-Amz-Request-Id: 6E1E6A9F4236FA7D


-----------------------------------------------------
2022/07/29 17:46:02.439165 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 6E1E6A9F4236FA7D, host id: Mlqwmen02/EnzQYYfLe0nWIinY+VVZJlbZoSO1o+1eHd+swKDY9aqs59Im5Ex1ZQvXbeqP9t+NuM
2022/07/29 17:46:02.439439 s3.DEBUG HEAD data/.wh.test.txt = no such file or directory
2022/07/29 17:46:02.440161 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: mUbhsVvg5OjqexfMlj8LnwIblEGyI6GbCZ51I+ExljwGXCeejAs7pm6NIlMfIcojQAXUIJAzCJoT
X-Amz-Request-Id: 52A6BD86437DE9F8


-----------------------------------------------------
2022/07/29 17:46:02.440290 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 52A6BD86437DE9F8, host id: mUbhsVvg5OjqexfMlj8LnwIblEGyI6GbCZ51I+ExljwGXCeejAs7pm6NIlMfIcojQAXUIJAzCJoT
2022/07/29 17:46:02.440373 s3.DEBUG HEAD data/.wh.test.txt/ = no such file or directory
2022/07/29 17:46:02.440466 fuse.DEBUG <-- LookUpInode 7 .wh.test.txt no such file or directory
2022/07/29 17:46:02.440569 fuse.DEBUG Op 0x00000102        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:02.441240 fuse.DEBUG Op 0x00000104        connection.go:408] <- LookUpInode (parent 7, name "test.txt")
2022/07/29 17:46:02.441986 fuse.DEBUG Ref 5 data/test.txt [1]
2022/07/29 17:46:02.442142 fuse.DEBUG <-- LookUpInode 7 test.txt <nil>
2022/07/29 17:46:02.443075 fuse.DEBUG Op 0x00000104        connection.go:491] -> OK (inode 5)
2022/07/29 17:46:02.444337 fuse.DEBUG Op 0x00000106        connection.go:408] <- OpenFile (inode 5)
2022/07/29 17:46:02.444800 fuse.DEBUG OpenFile 5 data/test.txt []
2022/07/29 17:46:02.445494 fuse.DEBUG Op 0x00000106        connection.go:491] -> OK ()
2022/07/29 17:46:02.446433 fuse.DEBUG Op 0x00000108        connection.go:408] <- GetXattr (inode 5, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 17:46:02.446755 fuse.DEBUG GetXattr 5 data/test.txt [trusted.overlay.origin]
2022/07/29 17:46:02.447210 fuse.DEBUG Op 0x00000108        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.448272 fuse.DEBUG Op 0x0000010a        connection.go:408] <- GetXattr (inode 5, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 17:46:02.448409 fuse.DEBUG GetXattr 5 data/test.txt [user.fuseoverlayfs.origin]
2022/07/29 17:46:02.448566 fuse.DEBUG Op 0x0000010a        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.448800 fuse.DEBUG Op 0x0000010c        connection.go:408] <- FlushFile (inode 5)
2022/07/29 17:46:02.449234 fuse.DEBUG FlushFile 5 data/test.txt []
2022/07/29 17:46:02.449840 fuse.DEBUG <-- FlushFile 5 data/test.txt [<nil> 20 5]
2022/07/29 17:46:02.449932 fuse.DEBUG Op 0x0000010c        connection.go:491] -> OK ()
2022/07/29 17:46:02.451087 fuse.DEBUG Op 0x0000010e        connection.go:408] <- ReleaseFileHandle
2022/07/29 17:46:02.451377 fuse.DEBUG ReleaseFileHandle data/test.txt 20 5
2022/07/29 17:46:02.451642 fuse.DEBUG Op 0x0000010e        connection.go:491] -> OK ()
2022/07/29 17:46:02.515944 fuse.DEBUG Op 0x00000110        connection.go:408] <- LookUpInode (parent 7, name ".wh.wasabi.txt")
2022/07/29 17:46:02.516110 fuse.DEBUG Inode.LookUp 7 data [.wh.wasabi.txt]
2022/07/29 17:46:02.519987 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.wasabi.txt HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f9a388012941337807606fb0b449063f7a307549af1b3766021ffbcef3ca532e
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.524253 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh.wasabi.txt/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=fb6d31d2d6df977ae603c498d85e9983b6857a2e1fab6788eeb0bc40bfec4d88
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.525530 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh.wasabi.txt%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=b49edf03ad429107a2fa5ed18bb054edda4b3717f893c1e4eb52be6dc89f24cc
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.540007 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: e5AQxSTMgBhP3dCdmXHOkjdIX0iaFjLv1hS1Ro4r8GZeZBxEjEISLIpWYLRFvWNovObbOkGKZyzh
X-Amz-Request-Id: A51E2DE582F141A8


-----------------------------------------------------
2022/07/29 17:46:02.541142 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: A51E2DE582F141A8, host id: e5AQxSTMgBhP3dCdmXHOkjdIX0iaFjLv1hS1Ro4r8GZeZBxEjEISLIpWYLRFvWNovObbOkGKZyzh
2022/07/29 17:46:02.541345 s3.DEBUG HEAD data/.wh.wasabi.txt = no such file or directory
2022/07/29 17:46:02.546012 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: sNE/VifE1Qra2p/sxOEuCYuWLJtq3EbftoaVcBeFn8pNJD25yvMdnwF8UwyQbmjBt1u1wJtKXIos
X-Amz-Request-Id: 3FCDB4C5F790C3C3


-----------------------------------------------------
2022/07/29 17:46:02.546339 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 3FCDB4C5F790C3C3, host id: sNE/VifE1Qra2p/sxOEuCYuWLJtq3EbftoaVcBeFn8pNJD25yvMdnwF8UwyQbmjBt1u1wJtKXIos
2022/07/29 17:46:02.548187 s3.DEBUG HEAD data/.wh.wasabi.txt/ = no such file or directory
2022/07/29 17:46:02.549309 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: cgW5kcWA5gxe1sdx2fCDjXkkaGfoGw2Bur8r/ddbKtV/7vKJ20IpuePN0NZy+6vcfCb3HaOTm0Pw
X-Amz-Request-Id: DEA8FAC2C8D6366C


-----------------------------------------------------
2022/07/29 17:46:02.549407 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:02.549508 fuse.DEBUG <-- LookUpInode 7 .wh.wasabi.txt no such file or directory
2022/07/29 17:46:02.549750 fuse.DEBUG Op 0x00000110        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:02.550265 fuse.DEBUG Op 0x00000112        connection.go:408] <- LookUpInode (parent 7, name "wasabi.txt")
2022/07/29 17:46:02.550768 fuse.DEBUG Ref 6 data/wasabi.txt [1]
2022/07/29 17:46:02.550937 fuse.DEBUG <-- LookUpInode 7 wasabi.txt <nil>
2022/07/29 17:46:02.551049 fuse.DEBUG Op 0x00000112        connection.go:491] -> OK (inode 6)
2022/07/29 17:46:02.552086 fuse.DEBUG Op 0x00000114        connection.go:408] <- OpenFile (inode 6)
2022/07/29 17:46:02.553206 fuse.DEBUG OpenFile 6 data/wasabi.txt []
2022/07/29 17:46:02.555026 fuse.DEBUG Op 0x00000114        connection.go:491] -> OK ()
2022/07/29 17:46:02.555732 fuse.DEBUG Op 0x00000116        connection.go:408] <- GetXattr (inode 6, name "trusted.overlay.origin", name trusted.overlay.origin)
2022/07/29 17:46:02.556361 fuse.DEBUG GetXattr 6 data/wasabi.txt [trusted.overlay.origin]
2022/07/29 17:46:02.556484 fuse.DEBUG Op 0x00000116        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.556862 fuse.DEBUG Op 0x00000118        connection.go:408] <- GetXattr (inode 6, name "user.fuseoverlayfs.origin", name user.fuseoverlayfs.origin)
2022/07/29 17:46:02.557220 fuse.DEBUG GetXattr 6 data/wasabi.txt [user.fuseoverlayfs.origin]
2022/07/29 17:46:02.558169 fuse.DEBUG Op 0x00000118        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.558869 fuse.DEBUG Op 0x0000011a        connection.go:408] <- FlushFile (inode 6)
2022/07/29 17:46:02.559324 fuse.DEBUG FlushFile 6 data/wasabi.txt []
2022/07/29 17:46:02.559366 fuse.DEBUG <-- FlushFile 6 data/wasabi.txt [<nil> 21 6]
2022/07/29 17:46:02.559450 fuse.DEBUG Op 0x0000011a        connection.go:491] -> OK ()
2022/07/29 17:46:02.560467 fuse.DEBUG Op 0x0000011c        connection.go:408] <- ReleaseFileHandle
2022/07/29 17:46:02.561035 fuse.DEBUG ReleaseFileHandle data/wasabi.txt 21 6
2022/07/29 17:46:02.561221 fuse.DEBUG Op 0x0000011c        connection.go:491] -> OK ()
2022/07/29 17:46:02.629178 fuse.DEBUG Op 0x0000011e        connection.go:408] <- ReadDir (inode 7)
2022/07/29 17:46:02.629709 fuse.DEBUG ReadDir 7 data [6]
2022/07/29 17:46:02.629807 fuse.DEBUG Op 0x0000011e        connection.go:491] -> OK ()
2022/07/29 17:46:02.630161 fuse.DEBUG Op 0x00000120        connection.go:408] <- OpenDir (inode 7)
2022/07/29 17:46:02.630298 fuse.DEBUG OpenDir 7 data []
2022/07/29 17:46:02.630757 fuse.DEBUG Op 0x00000120        connection.go:491] -> OK ()
2022/07/29 17:46:02.631249 fuse.DEBUG Op 0x00000122        connection.go:408] <- GetXattr (inode 7, name "trusted.overlay.opaque", name trusted.overlay.opaque)
2022/07/29 17:46:02.631501 fuse.DEBUG GetXattr 7 data [trusted.overlay.opaque]
2022/07/29 17:46:02.631787 fuse.DEBUG Op 0x00000122        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.632119 fuse.DEBUG Op 0x00000124        connection.go:408] <- ReleaseDirHandle
2022/07/29 17:46:02.632733 fuse.DEBUG Op 0x00000126        connection.go:408] <- OpenDir (inode 7)
2022/07/29 17:46:02.633164 fuse.DEBUG ReleaseDirHandle data
2022/07/29 17:46:02.633303 fuse.DEBUG Op 0x00000124        connection.go:491] -> OK ()
2022/07/29 17:46:02.633766 fuse.DEBUG OpenDir 7 data []
2022/07/29 17:46:02.634176 fuse.DEBUG Op 0x00000126        connection.go:491] -> OK ()
2022/07/29 17:46:02.634988 fuse.DEBUG Op 0x00000128        connection.go:408] <- GetXattr (inode 7, name "user.fuseoverlayfs.opaque", name user.fuseoverlayfs.opaque)
2022/07/29 17:46:02.635971 fuse.DEBUG GetXattr 7 data [user.fuseoverlayfs.opaque]
2022/07/29 17:46:02.636127 fuse.DEBUG Op 0x00000128        connection.go:493] -> Error: "no data available"
2022/07/29 17:46:02.636486 fuse.DEBUG Op 0x0000012a        connection.go:408] <- ReleaseDirHandle
2022/07/29 17:46:02.636778 fuse.DEBUG Op 0x0000012c        connection.go:408] <- LookUpInode (parent 7, name ".wh..wh..opq")
2022/07/29 17:46:02.636874 fuse.DEBUG ReleaseDirHandle data
2022/07/29 17:46:02.636947 fuse.DEBUG Op 0x0000012a        connection.go:491] -> OK ()
2022/07/29 17:46:02.637269 fuse.DEBUG Inode.LookUp 7 data [.wh..wh..opq]
2022/07/29 17:46:02.638420 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh..wh..opq/ HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7138f2ef2ccbbe01c06c591462da26400b3a1aca0ca267dfefd0cee8944fad64
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.638821 s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /staging-worldswitch-html-upload.world-switch.com/data/.wh..wh..opq HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=97d0c8d3d6a7985a5f6d73c6477e5b369795de05e989ef6f676bb90fbcf563c2
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.640115 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /staging-worldswitch-html-upload.world-switch.com?delimiter=%2F&max-keys=1&prefix=data%2F.wh..wh..opq%2F HTTP/1.1
Host: s3.ap-northeast-1.wasabisys.com
User-Agent: aws-sdk-go/1.17.13 (go1.16.15; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=E5QY0IIA1855T9HVBUAI/20220729/ap-northeast-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=5f3ea37f58f8a12be21eeffdf6ec3f543e7c713fbd483b9ae34de8cac52f2e21
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220729T084602Z


-----------------------------------------------------
2022/07/29 17:46:02.658716 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head3)
X-Amz-Id-2: VC9ZVLtmySeIBRu/nvEjLqzTHKLFcZELMIgVmtKbOmHPPToCjnEvdZn9DU9BHy2Dkt6XjJ0h+H/8
X-Amz-Request-Id: 67896E55D02482A9


-----------------------------------------------------
2022/07/29 17:46:02.658831 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 67896E55D02482A9, host id: VC9ZVLtmySeIBRu/nvEjLqzTHKLFcZELMIgVmtKbOmHPPToCjnEvdZn9DU9BHy2Dkt6XjJ0h+H/8
2022/07/29 17:46:02.658922 s3.DEBUG HEAD data/.wh..wh..opq/ = no such file or directory
2022/07/29 17:46:02.659313 s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Id-2: iHynM0NrNX3x9tl8Tkqs0sxhOAwBnzhNWvpQ4feb93Rsr7462X0UO+t3V5l5G4jAmAyHBp2lkxuV
X-Amz-Request-Id: 1FD2E5F062775ACD


-----------------------------------------------------
2022/07/29 17:46:02.659758 s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, not retrying, error NotFound: Not Found
	status code: 404, request id: 1FD2E5F062775ACD, host id: iHynM0NrNX3x9tl8Tkqs0sxhOAwBnzhNWvpQ4feb93Rsr7462X0UO+t3V5l5G4jAmAyHBp2lkxuV
2022/07/29 17:46:02.659888 s3.DEBUG HEAD data/.wh..wh..opq = no such file or directory
2022/07/29 17:46:02.661941 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 29 Jul 2022 08:46:02 GMT
Server: WasabiS3/7.5.1035-2022-06-08-c4b39686a7 (head5)
X-Amz-Bucket-Region: ap-northeast-1
X-Amz-Id-2: H0TEv5ADuOuFLKKcAO23xbZ5a+T3odScGr7PbjaHrmNYiwSgoGDyceky9c27V2gOf1p9j9mYOjh+
X-Amz-Request-Id: A72AD4F1A6548A9E


-----------------------------------------------------
2022/07/29 17:46:02.662049 s3.DEBUG &{[] [] <nil> false }
2022/07/29 17:46:02.662101 fuse.DEBUG <-- LookUpInode 7 .wh..wh..opq no such file or directory
2022/07/29 17:46:02.662204 fuse.DEBUG Op 0x0000012c        connection.go:493] -> Error: "no such file or directory"
2022/07/29 17:46:02.664324 fuse.DEBUG Op 0x0000012e        connection.go:408] <- ReleaseDirHandle
2022/07/29 17:46:02.665381 fuse.DEBUG ReleaseDirHandle data
2022/07/29 17:46:02.668281 fuse.DEBUG Op 0x0000012e        connection.go:491] -> OK ()
2022/07/29 17:46:08.052420 fuse.DEBUG Op 0x00000130        connection.go:408] <- ReleaseDirHandle
2022/07/29 17:46:08.052761 fuse.DEBUG ReleaseDirHandle data
2022/07/29 17:46:08.053438 fuse.DEBUG Op 0x00000130        connection.go:491] -> OK ()

@giuseppe
Copy link
Member

giuseppe commented Jul 29, 2022 •
edited
Loading

could you run fuse-overlayfs with the -d option?

EDIT: nevermind, sorry, I see you already did

@giuseppe
Copy link
Member

does it work if you try as root with native overlay?

@corbinjurgens
Copy link
Author

corbinjurgens commented Aug 1, 2022 •
edited
Loading

I tried native overlay but it said that this type of filesystem was not supported (in the case of goofys aws/wasabi s3) which is why I then moved onto migratefs.

So i tried something like

sudo mount -t overlay -o lowerdir=/home/ec2-user/overlay_test/lower,upperdir=/home/ec2-user/overlay_test/upper,workdir=/home/ec2-user/overlay_test/work none /home/ec2-user/overlay_test/overlay

I should mention I have been trying to use fuse-overlayfs inside a docker instance. (of amazonlinux2). I tested migratefs on a production server with same results but have not yet tried fuse-overlayfs. Could this be a cause?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@giuseppe @corbinjurgens