Skip to content

Redis error during high traffic times #233

@brostwalt87

Description

@brostwalt87

We are currently running authservice using AWS Elasticache as a backing store. During high traffic hours, we often see this issue in the logs

[2023-03-02 15:14:44.482] [console] [trace] hmget: redis connection closed error, retrying
[2023-03-02 15:14:44.482] [console] [trace] hmget: redis connection closed error, retrying
[2023-03-02 15:14:44.483] [console] [trace] hmget: redis connection closed error, retrying
[2023-03-02 15:14:44.483] [console] [error] hmget: redis connection closed error, throwing error

I have looked into different ways of tweaking the Elasticache server, including upgrading the instance to cache.m5.large. So pure resource consumption does not seem to be an issue. The elasticache server is running 2 replicas - a primary and secondary. Also, authservice is running a deployment with 2 pods, but has the ability to autoscale to 3. Resources do not seem to be an issue there

Also we have the following set on redis/elasticache:
appendonly no
maxmemory-policy allkeys-lru

Would like to see if there is a way to either optimize redis or increase the number of retries via a variable that can be added in at runtime

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions