-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
db/config.cc: increment components_memory_reclaim_threshold config default #18611
base: master
Are you sure you want to change the base?
db/config.cc: increment components_memory_reclaim_threshold config default #18611
Conversation
…fault Incremented the components_memory_reclaim_threshold config's default value to 0.8 as the previous value was too strict and caused unnecessary eviction in otherwise healthy clusters. Fixes scylladb#18607 Signed-off-by: Lakshmi Narayanan Sreethar <[email protected]>
🟢 CI State: SUCCESS✅ - Build Build Details:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@denesb No, what are you doing. 0.8 effectively disables the entire mechanism.
The system gradually adjusts memtable flushing speed to keep memtable memory usage under 0.5 (and stops accepting writes at the extreme), and doesn't even start flushing until it reaches 0.25 0.15.
Bloom filter memory usage isn't going to reach 0.8 because the system is all but guaranteed to OOM at 0.75 0.85, and that's even without counting all the non-cache, non-memtable memory usage.
If you want to prevent OOM crashes with this feature, the threshold must be way under 0.85 (with enough margin to cover all non-LSA, non-bloom memory usage). If you want to guarantee that, then it must be somewhere under (0.5 - margin).
For example, take the last case of bloom explosion from a few days ago (https://github.com/scylladb/scylla-enterprise/issues/4156). The affected node started choking on bad_allocs at bloom fraction of 0.6, and crashed (due to bad_alloc-induced flush failure) at 0.65.
Edit: I misremembered the defaults — we start flushing at 0.15, not at 0.25. Which means that 0.8 might be enough to prevent some OOMs, but that's cutting it very close.
0.6 is way too strict. In another customer case, the cluster is healthy and steady at 0.6 BF memory usage ratio and forcing it below makes IO and latencies jump and results in escalation. So we have to use the max() of all use-cases as the default and lower on deployments where the default is already problematic. |
Steady at 0.6? Which customer case was that? |
https://github.com/scylladb/scylla-enterprise/issues/4183 |
@avikivity - per our discussion on Sunday - please review. |
The goal of the feature was to prevent OOM. If we use the max() of all clusters, it will never prevent OOM since different clusters have other non-LSA components (prepared statements, cached queriers, running ops, background writes, repair with its large buffers, group 0 holding all the tablets metadata). We can increase it to 0.2, but not beyond. |
If we make the default too conservative, field is just going to disable it on all clusters and then we also aren't preventing OOMs. |
@avikivity - please help move this forward. |
@avikivity - please look at this. |
0.8 makes the protection meaningless. |
Alright, so let's make it 0.2. |
Do you need to remove your approval until the value is correctly set? |
Incremented the components_memory_reclaim_threshold config's default value to 0.8 as the previous value was too strict and caused unnecessary eviction in otherwise healthy clusters.
Fixes #18607