-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reconnect stops working if server is down #211
Comments
Hm, if I remove the following from Once Redis server is back up and running, everything works as expected.
EDIT: Another solution that seems to work is calling |
Unfortunately I can't reproduce this issue, I get normal reconnection attempts
|
How are you shutting down the redis server? Since you are disabling health-checks if the server does not send EOF downstream the connection will never notice that redis is down and might behave as if it were connected. |
I stop it simply by running However, I observe the same behavior if I run the sample program and the redis server is never started, i.e. no redis server is running. Effectively the program is trying to connect to a inactive/unused port. Could you please also try that scenario on your end?
|
Hi,
Ok, that is also what I am doing.
Nothing changed, the reconnect loop runs non-stop until I restart the server.
Debian testing + gcc 14.2.0
That would be really helpful. Thanks. |
Created a batteries-included Dockerfile (please see attachment Dockerfile_redis_reproduction.txt) that makes a fresh executable that replicates this behavior.
I built it as you normally would, e.g. tagged with redis_reproduction
|
Many thanks. I will try it when I get some free time, that might take a week in my current schedule. |
cfg.health_check_interval = {}; //Disable PING
cfg.reconnect_wait_interval = std::chrono::seconds(3); //Increase retry time btw, what is the point of disabling health-checks but enabling reconnects? Both are coupled to each other, with disabled health-checks we can only detect clean disconnects initiated by the server. It will react to a server that freezes for example. |
Thanks, hope it is helpful and I didn't do something silly to cause this behavior. No real point to disabling the health checks to be honest, just wanted to disable the implicit health-check to keep the logger a little bit quitter. Once I observed that it does not matter, i.e. the behavior remained the same, I left it disabled to have as few moving parts as possible. |
Short feedback: I tried to reproduce this many times but still could not do it. I couldn't build the docker file too. |
Thank you for trying to reproduce. I don't know how our environments differ that you do not observe this... At a bit of a loss, especially if you can't get the supplied Dockerfile to build. I just retried a full rebuild of the docker file I attached. It took around half an hour, but it built on the first try for me :\ |
I just managed to reproduce this problem on your container and will investigate it further. |
Having a similar issue, my environment is Ubuntu, Linux 6.8.0-45-generic x86_64, boost 1.85 with boost cobalt when my app code is started with out the Redis Server running. all I tried @zlausd's solution and it did help resolve this.
source code for my calls to boost::redis: program log; health_check_id here is unnamed:
|
If you are using a local redis running on a container:
I was able to reproduce like this: #include <boost/asio.hpp>
#include <boost/redis.hpp>
#include <boost/redis/connection.hpp>
#include <boost/redis/src.hpp>
#include <iostream>
#include <semaphore>
void Test(boost::redis::connection& connection,
boost::asio::io_context& io_context,
std::counting_semaphore<1>& semaphore,
int counter) {
auto request = std::make_shared<boost::redis::request>();
request->push("SET", "key", "value");
auto response = std::make_shared<boost::redis::response<std::string>>();
io_context.post([&connection, request, response, &semaphore, counter]() {
std::cout << "post " << counter << std::endl;
connection.async_exec(
*request, *response,
[request, response, &semaphore, counter]
(const boost::system::error_code& error, std::size_t) {
if (error) {
std::cout << "Failed to execute: " << error.to_string() << std::endl;
}
std::cout << "Done for " << counter << std::endl;
semaphore.release();
});
});
}
int main() {
boost::asio::io_context io_context{1};
std::thread thread;
// Set up.
boost::redis::connection connection(io_context);
boost::redis::config config;
connection.async_run(config, {boost::redis::logger::level::info},
[](const boost::system::error_code& error) {
if (error) {
std::cout << "Redis server setup error: " << error.to_string() << std::endl;
}
});
thread = std::thread([&io_context] {
std::cout << "RedisGlobalCache running" << std::endl;
io_context.run();
std::cout << "RedisGlobalCache finished" << std::endl;
});
std::vector<std::unique_ptr<std::counting_semaphore<1>>> semaphores;
for (int it = 0; it < 3; ++it) {
semaphores.push_back(std::make_unique<std::counting_semaphore<1>>(0));
Test(connection, io_context, *semaphores[it], it);
}
// Stop.
system("docker stop server_local_redis");
std::cout << "Stopping local redis server" << std::endl;
std::this_thread::sleep_for(std::chrono::seconds(10));
// Add more work.
std::counting_semaphore<1> last_semaphore(0);
Test(connection, io_context, last_semaphore, 10);
// Start.
system("docker start server_local_redis");
std::cout << "Starting local redis server" << std::endl;
std::this_thread::sleep_for(std::chrono::seconds(1));
for (auto& semaphore : semaphores) {
semaphore->acquire();
}
last_semaphore.acquire();
io_context.post([&connection] { connection.cancel(); });
thread.join();
std::cout << std::endl << std::endl;
return 0;
} Compiled with:
Output was:
It hangs at the end as no reconnections are attempted again and there are still pending operations. Adding |
Using the
(I downloaded the include/boost and just copied it to my boost include folder) |
The connector class is not a template anymore, can you please check the connector.hpp file was replaced? See
|
My bad! I copied it to the wrong location. It compiled now, and it works with my example!
|
Related question: I have set |
I am not exactly sure what the code is doing. cancel_if_not_connected is checked only once at the time async_exec is called. that means, if a disconnect occurs after that it won't cause the request to be canceled. If you have more questions about that I suggest opening another issue. |
Thanks for responding, I think I got the gist of it. To clarify: if the connection to the server goes down, all enqueued async_exec will get cancelled if |
My environment is Boost 1.85, GCC 14.2.1 (C++23) and Redis 7.2.5.
Observing a small quirk and not sure if its an issue with my setup or something else.
If I leave the Redis server shutdown and initiate a redis::connection with async_run, then it will only attempt to reconnect two times.
After the second ' Connection refused', nothing else on the connection instance works (i.e. async_exec no longer runs or cancels).
Expected behavior when Redis is down: Perpetual reconnect loop and requests should be failing since
cancel_if_not_connected
is explicitly set to true.Boost/Redis logs with added timestamps
Reproduction source code
The text was updated successfully, but these errors were encountered: