Current Item in redis cahe keeps increasing - Understanding Metadata Storage and Cache Management in Asynq with AWS Elasticache - Redis

[DipinVasuHPE]

I've built a straightforward Go application comprising producer and consumer processes that communicate via AWS Elasticache - Redis. While the server doesn't explicitly define any caching mechanisms, I've noticed that Asynq, the library I'm using for background job processing, stores metadata in the Redis cache, which appears to keep increasing over time. The current item count in the Redis cache seems to rise as task volume or traffic grows.

** Need to confirm on our assumption that Asynq uses a cache when used as job queue

I'm concerned about the scalability implications, particularly as task volume or traffic grows. Is there a TTL (time-to-live) defined for the data stored by Asynq in Redis? I'm curious about whether there's an expiry mechanism in place or any other measures to manage and reset the cache, especially under increasing load. Any insights or guidance on this matter would be greatly appreciated!

for example :

KEYS * :

1. "asynq:{default}:processed:2024-02-22"
2. "asynq:{default}:processed:2024-02-15"
3. "asynq:{default}:processed:2024-04-12"
4. "asynq:{default}:processed:2024-04-05"
5. "asynq:{default}:processed:2024-02-14"
6. "asynq:{default}:processed:2024-03-26"

CC : @hibiken

[kamikazechaser]

Is there a TTL (time-to-live) defined for the data stored by Asynq in Redis

Yes see the rention period option. Check the wiki for more details.

[DipinVasuHPE]

Correct me if I'm mistaken, but it seems that the retention you mentioned primarily addresses queue tasks rather than cache management. Do you have a direct reference or link you could share with me for further clarification?

https://pkg.go.dev/github.com/hibiken/asynq#section-sourcefiles:~:text=the%20task%20into.-,func%20Retention%20%C2%B6,-added%20in%20v0.19.0

[kamikazechaser]

As the docs say: "A (successfully) completed task will be deleted" after the specified duration elapses. That will also remove it from Redis. If your task was archived (e.g. through retry exhaustion) then it will be deleted after 90 days.

[DipinVasuHPE]

@kamikazechaser Currently, I haven't noticed any archived tasks in Redis, but I'm particularly concerned about the cached metadata by Asynq. Can you confirm if Asynq has set an expiry for the cache? Knowing the precise timing of cache clearance would be good.