Sentry 24.4.2 and severe log spam from clickhouse/snuba

[plemelin]

Self-Hosted Version

24.4.2

CPU Architecture

x86_64

Docker Version

26.1.2

Docker Compose Version

2.27.0

Steps to Reproduce

Ensure you blocked HTTP and HTTPS to your sentry instance.

Simply restart everything with docker compose up -d --force-recreate

Look at the logs.

Expected Result

Logs contain valuable information

Actual Result

Logs are constantly spammed with a high volume of:

snuba-generic-metrics-sets-consumer-1           | {"timestamp":"2024-05-10T21:52:21.842453Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}
snuba-issue-occurrence-consumer-1               | {"timestamp":"2024-05-10T21:52:22.134422Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}
snuba-generic-metrics-counters-consumer-1       | {"timestamp":"2024-05-10T21:52:22.165962Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}
clickhouse-1                                    | 2024.05.10 21:52:22.166625 [ 300 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):
clickhouse-1                                    | 
clickhouse-1                                    | 0. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x13c4ee8e in /usr/bin/clickhouse
clickhouse-1                                    | 1. Poco::Net::SocketImpl::peerAddress() @ 0x13c510d6 in /usr/bin/clickhouse
clickhouse-1                                    | 2. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x110e6f0b in /usr/bin/clickhouse
clickhouse-1                                    | 3. DB::HTTPServerConnection::run() @ 0x110e5d6e in /usr/bin/clickhouse
clickhouse-1                                    | 4. Poco::Net::TCPServerConnection::start() @ 0x13c5614f in /usr/bin/clickhouse
clickhouse-1                                    | 5. Poco::Net::TCPServerDispatcher::run() @ 0x13c57bda in /usr/bin/clickhouse
clickhouse-1                                    | 6. Poco::PooledThread::run() @ 0x13d89e59 in /usr/bin/clickhouse
clickhouse-1                                    | 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x13d860ea in /usr/bin/clickhouse
clickhouse-1                                    | 8. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
clickhouse-1                                    | 9. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
clickhouse-1                                    |  (version 21.8.13.1.altinitystable (altinity build))
snuba-transactions-consumer-1                   | {"timestamp":"2024-05-10T21:52:22.204371Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}
clickhouse-1                                    | 2024.05.10 21:52:22.239660 [ 289 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):
clickhouse-1                                    | 
clickhouse-1                                    | 0. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x13c4ee8e in /usr/bin/clickhouse
clickhouse-1                                    | 1. Poco::Net::SocketImpl::peerAddress() @ 0x13c510d6 in /usr/bin/clickhouse
clickhouse-1                                    | 2. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x110e6f0b in /usr/bin/clickhouse
clickhouse-1                                    | 3. DB::HTTPServerConnection::run() @ 0x110e5d6e in /usr/bin/clickhouse
clickhouse-1                                    | 4. Poco::Net::TCPServerConnection::start() @ 0x13c5614f in /usr/bin/clickhouse
clickhouse-1                                    | 5. Poco::Net::TCPServerDispatcher::run() @ 0x13c57bda in /usr/bin/clickhouse
clickhouse-1                                    | 6. Poco::PooledThread::run() @ 0x13d89e59 in /usr/bin/clickhouse
clickhouse-1                                    | 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x13d860ea in /usr/bin/clickhouse
clickhouse-1                                    | 8. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
clickhouse-1                                    | 9. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
clickhouse-1                                    |  (version 21.8.13.1.altinitystable (altinity build))
snuba-metrics-consumer-1                        | {"timestamp":"2024-05-10T21:52:22.239374Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}

Event ID

No response

[plemelin]

I created this ticket following comments from @tm-artemb, @peektoseen and @krzysieqq here: #2963 (comment)

[aldy505]

The spam is caused by the rust consumer. Please refer to this comment #2876 (comment) or getsentry/snuba#5707 (comment)

If changing that fixed the problem, please let me know, and I'll close this issue :D

[williamdes]

If changing that fixed the problem, please let me know, and I'll close this issue :D

Reverting to Python fixes the issue, but there is a real bug to be fixed in snuba rust consumers.
Maybe move the issue on the snuba repo

[aldy505]

If changing that fixed the problem, please let me know, and I'll close this issue :D

Reverting to Python fixes the issue, but there is a real bug to be fixed in snuba rust consumers. Maybe move the issue on the snuba repo

The issue's already there, see getsentry/snuba#5707

[plemelin]

Thank you for the help here.
This can be closed and I'll monitor the one in the snuba project