[Ask For Help] metadata operations works but failed to write data into storage servers.

[SimonCqk]

I have deployed 3FS in a production Kubernetes environment (RoCE RDMA container network). Components like Mgmtd and Meta appear normal, and the list-nodes command shows proper results. After mounting FUSE, metadata operations like directory listing work correctly, but data writes echo hello > /xxxx/hello.txt fail with the following errors. RDMA network connectivity between FUSE clients and Storage nodes has been verified

list node output

stderr

[2025-04-13T07:18:58.223072746+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1173 ERROR] Give up retrying 1/1 ops 0x7f1fbc792758 after #3 retries, elapsed time: 60001ms, max retry time: 60000ms, user: Uid(0)/Gid(0), usercall: #1
[2025-04-13T07:18:58.223156990+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1637 ERROR] #1/1 Failed batchRead operation 0x7f1fbc7641f0 with id RequestId(0), routing target: ChainId(900100014)@ChainVersion(2)@RoutingInfoVersion(18):TargetId(0)@NodeId(0):ChannelId(0)#ChannelSeqNum(0), chunk range: ChunkId(00000000-00000003-E0040000-00000000), user: Uid(0)/Gid(0), usercall: #1, status: StorageClient::NotAvailable(7005)
[2025-04-13T07:18:58.223172010+00:00 hf3fs_fuse_main:235453 StorageClientImpl.cc:1637 ERROR] Finally collected 1/1 failed batchRead operations 0x7f1fbc77d768 issued by user Uid(0)/Gid(0), usercall: #1
[2025-04-13T07:18:58.223277021+00:00 hf3fs_fuse_main:235453 FuseOps.cc:1544 ERROR]   hf3fs_read error, ino 0x000000000003e004, size 4096, off 0, err -7005

stdout

[SimonCqk]

New stderr from storage-server side, it seems that targets creation were not completed as expected?

[Navacore]

I don't know what they're accusing me of, please stop sending messages. El dom, 13 de abr. de 2025 3:48 a. m., Simon_CQK ***@***.***> escribió:

…

New stderr from storage-server side, it seems that targets creation were not completed as expected? image.png (view on web) <https://github.com/user-attachments/assets/f55f88c1-1dbb-44d7-bc9d-e25ac5cc3703> — Reply to this email directly, view it on GitHub <#239 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BQ43KJP6NWJUE465JT3XHBD2ZIJELAVCNFSM6AAAAAB3A2K3W6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJZHA2DKNJQHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> *SimonCqk* left a comment (deepseek-ai/3FS#239) <#239 (comment)> New stderr from storage-server side, it seems that targets creation were not completed as expected? image.png (view on web) <https://github.com/user-attachments/assets/f55f88c1-1dbb-44d7-bc9d-e25ac5cc3703> — Reply to this email directly, view it on GitHub <#239 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BQ43KJP6NWJUE465JT3XHBD2ZIJELAVCNFSM6AAAAAB3A2K3W6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJZHA2DKNJQHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>