warp causes terminal hang

[eco-minio]

Periodically during warp runs the terminal will enter a state where it cannot take any new input and scrollback is impossible. Only issuing a reset resolves it. Logs from latest failed run attached although im not sure there is anything helpful there. Latest runs were with 1.1.0-rc although this has been observed at least since 1.0.8 and probably before that as well.

warp-logs.zip

[klauspost]

Before the update???

I have no idea where to start or end with this one. I can only see this being fixed by someone that can actually repro it.

[ykargin]

seems like --influxdb also affected. Downgrade to 1.0.5 - b176f4c helps.

[eco-minio]

So I have been fairly reliably triggering this during delete when there is an error condition. Following just happened

minio@purico-5337-os:~/benchmarks$ ./delete-test 
Using flags: --objects 50000000 --noclear --list-existing
╭─────────────────────────────────╮
│ WARP S3 Benchmark Tool by MinIO │
╰─────────────────────────────────╯
                                                                                                             
Downloading Operations: Benchmark data written to "objsize-1KiB-threads-80-delete-25-02-18-08-56.json.zst"...
                                                                                                             
 λ ███░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   8%                                                                  
                                                                     

Report: DELETE. Concurrency: 320. Ran: 25s
 * Average: 410220.71 obj/s
 * Reqs: Avg: 66.9ms, 50%: 21.8ms, 90%: 219.7ms, 99%: 611.7ms, Fastest: 3.2ms, Slowest: 2329.5ms, StdDev: 124.9ms

Throughput by client:
Client 1 throughput: 118277.97 obj/s
Client 2 throughput: 117781.48 obj/s
Client 3 throughput: 117781.48 obj/s
Client 4 throughput: 117002.29 obj/s

Throughput, split into 21 x 1s:
 * Fastest: 623362.27 obj/s
 * 50% Median: 518768.30 obj/s
 * Slowest: 326910.17 obj/s


Using flags: --objects 50000000 --noclear --list-existing
╭─────────────────────────────────╮
│ WARP S3 Benchmark Tool by MinIO │
╰─────────────────────────────────╯
                                                                          
Preparing: Client  192.168.100.183:7761 : Requested stage prepare start...
                                                                       
 λ                                                                        

warp: <ERROR> \\d              Stage failed. Client 192.168.100.182 returned error. no objects found for bucket warp-bench.
                                                                                                          Using flags: --objects 50000000 --noclear --list-existing
╭─────────────────────────────────╮
│ WARP S3 Benchmark Tool by MinIO │
╰─────────────────────────────────╯
                                                                          
Preparing: Client  192.168.100.183:7761 : Requested stage prepare start...
                                                                          
 λ                                                                        

warp: <ERROR> 
              Stage failed. Client 192.168.100.181 returned error. no objects found for bucket warp-bench.
                                                                                                          minio@purico-5337-os:~/benchmarks$ ^C
minio@purico-5337-os:~/benchmarks$ 

This was from a script running delete iterating over object sizes. The first run goes fine, then the next two die i think since there are no pre-existing objects (and list-existing is being used). At this point, terminal becomes unusable until reset command is issued

[klauspost]

@eco-minio - but DELETE is deleting objects...??? If you are using --list-existing and there are none left, there isn't much more it can do?

[eco-minio]

I am saying that because there is an error condition, warp exits and appears to do so in a way that hangs the terminal. So I don't think it is warp per se, rather whatever is happening when the error is triggered.

[klauspost]

@eco-minio Ah... ok.. Yeah, that should be actionable. I will check tomorrow.

[klauspost]

Unfortunately it doesn't repro on Windows. I've tried a secondary cancelation path and locked it to the OS thread. There is a chance that can fix it.

[christoph-heiss]

Hi,

I came across what seems to be the exact same issue. Can reproduce it reliably on Linux (NixOS, foot as terminal emulator).

A bisect points to 483f844 as the bad commit. From a glance, seems plausible since it does some output-related things, especially regarding newlines etc.

[harshavardhana]

@christoph-heiss please try the fix #376