etcd delete leaving stale mappings

[prashil1996]

I’ve a backend service which fetches key from etcd and once our process is done, we have to delete the mappings in the etcd.

We are deleting with key and still the mappings are present in the etcd store.
Any inputs?

If you need more inputs from me to identify the issue, please revert.

[ahrtr]

We are deleting with key and still the mappings are present in the etcd store.

Note that etcd doesn't really delete the key/value, it just marks it as deleted, which may be completely removed in following compaction operation. Please read https://etcd.io/docs/v3.5/learning/data_model/

But it's transparent to users. You should be fine as long as you can't read the key/value after you remove it.

[prashil1996]

This makes sense, so to give you an understanding on how my service work I need to tell you what we write to ETCD. As part of the service, we bring in some data from user and then depending on the input provided, we authenticate and authorise the input payload. Once that is done, we write the metadata to ETCD and the continue to process the workload provided by the customer.
Down the pipeline, there are multiple services which writes to the ETCD along with the respective metadata and once the process is completed or failed(if it fails), we unprovision/delete the work related from our space and then the meta from the etcd.

In our code, we use a custom delete method which invokes the ETCD client to delete based on the key.
client.getKVClient.delete(key).get(etcdClientDefaultTimeout, TimeUnit.SECONDS).getDeleted

@ahrtr Since you said that it only removes or marks the entry as deleted, what can I do apart from running an automated Jenkins job to delete the entries permanently? Does marking them deleted and leaving them in ETCD leads to performance issues?

What steps can I take and process I can follow to make sure that if data is being deleted from ETCD, it should be permanent?
Also for some jobe we trigger, we are not able to query them once they are deleted. It is only some jobs which even after completion is deleted from our space and the meta for that job is not deleted.

Thanks

[ahrtr]

From user perspective, you just need to perform compaction periodically, and defragmentation when needed.

Reference:

• https://etcd.io/docs/v3.5/op-guide/maintenance/
• https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/

[prashil1996]

@ahrtr Thanks for the reply, I shall do this but also, one more doubt since you have been so helpful. We make calls using gRPC and as part of the logs, I see Caused by: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 10485760: 89638766 while we call ETCD client to delete based on the key.

Can you shed some light on how is ETCD impacted on message limit through gRPC? We have a default timeout to 120 seconds for etcd operations.
Do you think that 120 seconds is less if our delete operations are high? Like to point, our stale mappings are in the ranges of 400k to 600k.

Thanks

[ahrtr]

I am not a gRPC expert, can you raise a question in the grpc-go community?

Can you shed some light on how is ETCD impacted on message limit through gRPC?

One possible reason I can think of might be related to WithPrevKV, and the previous key/value exceeded the grpc limitation?

Do you think that 120 seconds is less if our delete operations are high?

I think It depends on the network situation. But usually 120s should be a value bigger enough.

our stale mappings are in the ranges of 400k to 600k.

Sorry, I do not understand the "stale mappings" issue. Can you please provide detailed steps on how to reproduce the issue? and what's the etcd version?

[prashil1996]

Any inputs @ahrtr ?