Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zeekctl netstats gets stuck sometimes if zeek worker crashed #3509

Open
6U-U9 opened this issue Dec 15, 2023 · 0 comments
Open

zeekctl netstats gets stuck sometimes if zeek worker crashed #3509

6U-U9 opened this issue Dec 15, 2023 · 0 comments

Comments

@6U-U9
Copy link

6U-U9 commented Dec 15, 2023

I was running a deamon bash script capture zeek stats, it runs zeekctl netstats every 5 seconds.
And there's another scripts that runs zeekctl deploy if it finds zeek crushed.
But I met a situation that if zeek crush at a specific time, that zeekctl netstats command will suspend and never ends, which cause zeekctl deploy waits for lock owned by zeekctl netstats and thus will not restart crushed zeek.

netstats stuck

netstats stuck2

Envronment: Ubuntu 20.04 on virtual box ( 6cores i7-12700, 8G RAM)
Version: zeek 6.0.1
Except node.cfg, nothing is modified.

[logger-1]
type=logger
host=localhost

[manager]
type=manager
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=enp0s3

I tried to reproduce this problem, and this is the script that works to trigger this problem.
By print more logs, I guess that zeekctl netstats gets stuck at /ZeekControl/events.py line 67 endpoint.peer(host, node.getPort(), 1)

#! /bin/bash
echo "bash pid: $$"
sleep_time=0.01

while true; do

    zeekctl deploy
    zeek_pid=$(zeekctl status | grep worker | awk '{print $5}')
    echo "zeek pid: $zeek_pid"

    test() {
        #sleep 10
        zeekctl netstats 
    }

    test &
    test_pid=$!
    #echo "sleep $sleep_time"
    sleep $sleep_time
    kill -9 $zeek_pid
    wait $test_pid

    echo $sleep_time
    sleep_time=$(bc <<< "$sleep_time+0.01")
done

Thanks for helping me solving this problem.

@0xxon 0xxon changed the title zeekctl netstats gets stuck sometimes if zeek worker crushed zeekctl netstats gets stuck sometimes if zeek worker crashed Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant