Determining the largest keys in etcd/clarity around versions

[ns-rlewkowicz]

I'm trying to figure out what is taking the most size in our k8 etcd databases.

I calculate by grabbing a key, calculating the length of the value portion of key and multiplying it by the version.

A key's life spans a generation, from creation to deletion. Each key may have one or multiple generations. Creating a key increments the **version** of that key, starting at 1 if the key does not exist at the current revision. Deleting a key generates a key tombstone, concluding the key’s current generation by resetting its version to 0. Each modification of a key increments its version; so, versions are monotonically increasing within a key's generation. Once a compaction happens, any generation ended before the compaction revision will be removed, and values set before the compaction revision except the latest one will be removed.

/registry/services/endpoints/foo/bar

in our cluster has 1041754 versions not revisions. There's no way they all exist.

How can I get the real space of a key?

[jmhbnz]

Hey @ns-rlewkowicz - Thanks for your question. I would start with the "what occupies the most space" section of this blog post from earlier this year: https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/#what-data-occupies-most-of-the-storage-space

[ns-rlewkowicz]

I'm still dissecting the esoteric one liner, but at a glance, I don't know that this is logically valuable?

It's just counting the number of keys in a space. Secrets alone would weight this wrong if you had a team abusing it etc.

I do goofy stuff in here, I think you can actually lock the client, I should probably using grpc options etc. blah blah blah.

import sys
import etcd3
import logging
import threading
from time import sleep

logging.basicConfig(level="INFO")

# This is my little global object paradigm.
# It's python magic but I use it to have a global object to use between threads
persistent_object = type('', (), {})()
persistent_object.sizes = {}
persistent_object.keys=[]
persistent_object.clients=[]
persistent_object.thread_index={}

threads = 10
# Max number of threads
threadLimiter = threading.BoundedSemaphore(threads)
# Probably could do this with grpc, but I didn't want to dig into that
# I create as many clients as there are threads and pass them accordingly
for i in range(1, threads):
    globals()[f"client_{i}"] = etcd3.client(ca_cert=sys.argv[1], cert_key=sys.argv[2], cert_cert=sys.argv[3])
    persistent_object.clients.append(globals()[f"client_{i}"])

# This creates a decorator that makes any method non blocking
def dont_block(f):
    def wrap(*args):
        threading.Thread(target=f, args=args, daemon=True).start()
    return wrap

def incr_last_byte(data):
    s = bytearray(data)
    s[-1] = s[-1] + 1
    return bytes(s)

def walk_the_tree(etcd=persistent_object.clients[0], range_start='\0',
                  range_end=None, keys_only=False, chunk_size=40000):

    if range_end is None:
        if range_start == '\0':
            range_end = range_start
    if range_end is None:
        range_end = incr_last_byte(
            etcd3.utils.to_bytes(range_start)
            )

    chunk_size_okay = True

    while True:
        range_request = etcd3.etcdrpc.RangeRequest()
        range_request.key = etcd3.utils.to_bytes(range_start)
        range_request.keys_only = keys_only
        range_request.range_end = etcd3.utils.to_bytes(range_end)
        range_request.sort_order = 0
        range_request.sort_target = 0
        range_request.serializable = True
        range_request.limit = chunk_size
        while True:
            try:
                range_response = etcd.kvstub.Range(
                    range_request,
                    etcd.timeout,
                    credentials=etcd.call_credentials,
                    metadata=etcd.metadata,
                    )
                break
            except Exception as e:
                chunk_size = int(chunk_size / 1.2)
                if chunk_size_okay:
                    if chunk_size < 1:
                        chunk_size = 1
                        chunk_size_okay = False
                else:
                    raise
                range_request.limit = chunk_size

        for kv in range_response.kvs:
            if keys_only == False:
                persistent_object.sizes[kv.key.decode('utf-8')] = len(kv.value)
                if kv.version > 1:
                    persistent_object.sizes[kv.key.decode('utf-8')] = persistent_object.sizes[kv.key.decode('utf-8')] * kv.version
            else:
                persistent_object.keys.append(kv.key.decode('utf-8'))
        if not range_response.more:
            break
        range_start = incr_last_byte(kv.key)

walk_the_tree(keys_only=True)

event = threading.Event()

# The client is not thread safe, and I don't want to rewrite all this
# We track thread indexs and release the client to them
lock = threading.Lock()

@dont_block
def get_sizes(current, fin):
    this_thread = None
    threadLimiter.acquire()
    while this_thread == None:
        with lock:
            for i in range(0, threads-1):
                if i not in persistent_object.thread_index:
                    this_thread = i
                    persistent_object.thread_index[i] = None
    try:
        walk_the_tree(etcd=persistent_object.clients[this_thread], range_start=current[0],
                      range_end=current[-1], keys_only=False, chunk_size=40000)
        # for (value, meta) in persistent_object.clients[this_thread].get_range(current[0], current[-1]):
        #     persistent_object.sizes[meta.key.decode('utf-8')] = len(value)
        #     if meta.version > 1:
        #         persistent_object.sizes[meta.key.decode('utf-8')] = persistent_object.sizes[meta.key.decode('utf-8')] * meta.version
    finally:
        with lock:
            del persistent_object.thread_index[this_thread]
        threadLimiter.release()
    if fin == True:
        event.set()


start = 0
end = len(persistent_object.keys)
# Keys to fetch at a time
step = 1000
fin = False

for i in range(start, end, step):
    x = i
    current=persistent_object.keys[x:x+step]
    if x+step >= end:
        fin = True
    get_sizes(current, fin)

while True:
    if not event.is_set():
        sleep(2)
    else:
        break

with open("sizes.txt", "w") as f:
    for i, (k,v) in enumerate(persistent_object.sizes.items()):
        f.write(k + "###" + str(v) + '\n')

I need a way to get the number of real versions or someway to get the current compat and cross reference that here:

if kv.version > 1:
                    persistent_object.sizes[kv.key.decode('utf-8')] = persistent_object.sizes[kv.key.decode('utf-8')] * kv.version

[jmhbnz]

Very commonly, it’s the count of total objects stored in etcd that leads to an increase in storage consumption.

The blog I referred to above gives you a way of seeing how many of what objects exist, and may give you a smoking gun to investigate further.

You can also do this from the kube api side with something like kubectl get --raw=/metrics | grep apiserver_storage_objects |awk '$2>100' |sort -g -k 2.

Other (but less common from what I have seen) reasons for database size are:

• Object revision updates, i.e. spammy updates to single resources
• Object size, i.e. pod spec with embedded certs instead of configmap or secret

Those are both more difficult to pinpoint which is why I suggest investigating total object counts first.

[ns-rlewkowicz]

@jmhbnz I think it's a little of both the less common options.
(I'm just sharing my work to help those that find this same path)

Changing kv.version > 1: to kv.version < 0: is a quick hack way to pinpoint object size in the now. Nothing too crazy on that front.

Any recommendations for targeting spammy updates? In the above I add range_request.min_mod_revision = 1370140052 and can target the larger keys that are updating frequently at least.

[jmhbnz]

Ok if we think it's the latter things get a bit tricky. I'll refer you some great advice from the AWS EKS etcd team here: https://aws.amazon.com/blogs/containers/managing-etcd-database-size-on-amazon-eks-clusters/

Long story short:

You can analyze the request volume to the Kubernetes API server over a specific period of time to identify anomalous trends in request patterns. These anomalies can potentially correlate to a surge in etcd space consumption. You can monitor the API server metric apiserver_request_total to see the total request volume.

Sometimes, the etcd database size grows because a few objects go through rapid updates. In this case, the total object count or the size per object is relatively small, but there are large number of revisions per object.

Once you have a timeframe you want to investigate you will need to dig down into kube apiserver audit logs and filter requests by type and user agent. This will help you understand more about the request and its caller.

There are EKS specific examples for doing that in the link. I'm not sure what flavour of kubernetes you are running, would suggest reaching out to your provider or referring to their docs to explore how audit logs are managed for that particular distro of k8s.

Hope that helps. If you do find the culprit please report back here and let us know! 🙏🏻

[rlewkowicz]

I’m just reviewing my old work. There was a team creating thousands of pods and instead of having a health check, they just let it die. It was creating phenomenal pod churn. We targeted the team and asked them to add health checks

PS: I need a job if anyone finds my work useful. LinkedIn in bio, reach out