Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Presto worker nodes memory consumption is gradually increasing over a period of time & eventually its killing the service #24316

Open
pavakalyankillari opened this issue Jan 2, 2025 · 0 comments

Comments

@pavakalyankillari
Copy link

Presto version: 0.278.1

Presto worker nodes memory consumption is gradually increasing over a period of time & eventually its killing the service.

Hi, we are using Presto with the below configs:

One coordinator & 3 workers.

Coordinator configs:

coordinator=true
node-scheduler.include-coordinator=true
http-server.http.port=49152
node-scheduler.max-splits-per-node=2500
query.max-run-time=5m
#task.max-worker-threads=24
query.max-memory=12GB
query.max-memory-per-node=2GB
discovery-server.enabled=true
discovery.uri=http://localhost:49152
join-distribution-type=BROADCAST
optimizer.join-reordering-strategy=NONE

worker configs:

coordinator=false
http-server.http.port=49001
node-scheduler.max-splits-per-node=2500
query.max-run-time=5m
#task.max-worker-threads=64
query.max-memory=12GB
query.max-memory-per-node=2GB
#discovery-server.enabled=true
discovery.uri=http://:49152

Looks like the cache is building up as soon as the query is executed & we get the result.
Is there a way to clear this cache & get rid of this memory build up issue?

We are trying the below thing.

Automatic Cache Reset with Presto Configuration Settings

While Presto doesn't support direct cache invalidation, you can configure some parameters to manage memory and reduce the impact of caching:
Memory Spill and Query Timeouts
If you want Presto to handle memory more dynamically and avoid excessive memory usage, enable spilling and set query timeouts:
Enable Spilling (if not already enabled):

experimental.spill-enabled=true

query.max-run-time=5m

But looks like it is not a sophisticated way.

We are currently re-launching the service whenever there is a high memory usage. Could you suggest a better approach?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant