[#25722] yb-admin command to clear rocksdb block cache #28733

kenju · 2025-09-29T06:33:15Z

Resolves #25722

Changes Made:

Implementation touches the whole request flow from CLI, client side, RPC to server backend. I added the clear_cache (NEW) command to the yb-admin and implemented the corresponding client-side logic that broadcasts cache clearing requests to all tablet servers in the cluster.

On the server side, I created a new ClearCache RPC service and implemented the core cache clearing functionality (= that traverses all tablets on each tserver to purge RocksDB block caches for both regular and intents databases).

How to Purge

The cache clearing mechanism uses SetCapacity(0) → SetCapacity(original) pattern to purge all cached blocks while preserving the original cache configuration. For each tablet server, the implementation iterates through all tablet peers, retrieves their associated tablets, and clears the block caches of both the regular_db (main data storage) and intents_db (transaction intents storage).

Testing:

✅ I validated the behaviour from the locally running multi-node servers:

Set up 2-node cluster using ./bin/yugabyted start
Inserted test data to populate block caches
Successfully executed ./build/latest/bin/yb-admin clear_cache and verified results both from tserver logs and metrics

Here is the trimmed log from the local running cluster on purging block caches:

I0929 07:20:06.320581 1865035776 tablet_service.cc:2610] Received ClearCache RPC request from 127.0.0.1:51992
I0929 07:20:06.320622 1865035776 tablet_service.cc:2614] Found 18 tablet peers on this tserver
I0929 07:20:06.320639 1865035776 tablet_service.cc:2596] Purged regular block cache for tablet c81ac388637d4c7c8b5d8886c218a661 (capacity: 9895604649 bytes)
I0929 07:20:06.320647 1865035776 tablet_service.cc:2596] Purged intents block cache for tablet c81ac388637d4c7c8b5d8886c218a661 (capacity: 9895604649 bytes)
I0929 07:20:06.320874 1865035776 tablet_service.cc:2596] Purged regular block cache for tablet 2d385b1548704559be8ee72196cdf735 (capacity: 9895604649 bytes)
I0929 07:20:06.320879 1865035776 tablet_service.cc:2596] Purged intents block cache for tablet 2d385b1548704559be8ee72196cdf735 (capacity: 9895604649 bytes)
(...)
W0929 07:20:06.320926 1865035776 tablet_service.cc:2623] Failed to get tablet 7dfe5ede85d841ffb5cd390ab6fedc43: Illegal state (yb/tablet/tablet_peer.cc:864): Tablet not running: tablet object 7dfe5ede85d841ffb5cd390ab6fedc43 has invalid state kDestroyed
W0929 07:20:06.320931 1865035776 tablet_service.cc:2623] Failed to get tablet 6611490ba739471c9e1e488bcbcb6777: Illegal state (yb/tablet/tablet_peer.cc:864): Tablet not running: tablet object 6611490ba739471c9e1e488bcbcb6777 has invalid state kDestroyed
I0929 07:20:06.320937 1865035776 tablet_service.cc:2596] Purged regular block cache for tablet 57bafd8ed1f848b29e358e857a1d6977 (capacity: 9895604649 bytes)
I0929 07:20:06.320940 1865035776 tablet_service.cc:2596] Purged intents block cache for tablet 57bafd8ed1f848b29e358e857a1d6977 (capacity: 9895604649 bytes)
I0929 07:20:06.320943 1865035776 tablet_service.cc:2649] Successfully cleared cache on 10 tablets
I0929 07:20:06.320950 1865035776 tablet_service.cc:2653] Successfully responded to ClearCache RPC

Here is the screenshot from the Grafana showing the following metrics:

block_cache_single_touch_usage{job="yugabytedb-tserver"}
block_cache_multi_touch_usage{job="yugabytedb-tserver"}

CLAassistant · 2025-09-29T06:33:23Z

All committers have signed the CLA.

kenju · 2025-09-29T06:35:36Z

src/yb/tserver/tablet_service.cc

+    // without permanently restricting its size.
+    auto original_capacity = block_cache->GetCapacity();
+    block_cache->SetCapacity(0);
+    block_cache->SetCapacity(original_capacity);


inspired by

yugabyte-db/src/yb/rocksdb/db/readahead_test.cc

Lines 228 to 234 in 915c743

void PurgeBlockCache() {

auto* block_cache = table_factory_->table_options().block_cache.get();

auto capacity = block_cache->GetCapacity();

block_cache->SetCapacity(0);

block_cache->SetCapacity(capacity);

LOG(INFO) << "Purged block cache";

}

kenju · 2025-09-29T06:35:50Z

src/yb/tserver/tablet_service.h

 #include "yb/tserver/tserver_admin.service.h"
 #include "yb/tserver/tserver_service.service.h"

+namespace rocksdb {


needs for forward declaration

kenju · 2025-09-29T06:36:10Z

src/yb/tserver/tserver_admin.proto

+
+message ClearCacheResponsePB {
+  optional TabletServerErrorPB error = 1;
+  optional uint64 cache_capacity_bytes = 2;


this field ss more for logging purpose

kenju · 2025-09-29T06:37:11Z

src/yb/tserver/tserver_admin.proto

  repeated RbsInfo rbs_infos = 2;
 }

+message ClearCacheRequestPB {}


if you want to expand this command in the future (e.g. adding flag to decide if excluding system tables, targeting specific tablets, etc.) this is a good starting point

kenju · 2025-09-29T06:37:51Z

src/yb/tools/yb-admin_cli.cc

  return Status::OK();
 }

+const auto clear_cache_args = "[<timeout_in_seconds>] (default 20)";


Followed @arybochkin's comment on #26078 (comment)

kenju · 2025-10-01T15:54:03Z

@rthallamko3 can I get a reviewer for this PR please?

rthallamko3 · 2025-10-01T19:27:08Z

@ttyusupov , Can you help review the changes?

ttyusupov · 2025-10-13T15:28:01Z

src/yb/tserver/tablet_service.cc

+    const ClearCacheRequestPB* req, ClearCacheResponsePB* resp, rpc::RpcContext context) {
+  LOG(INFO) << "Received ClearCache RPC request from " << context.requestor_string();
+
+  TabletPeers tablet_peers = server_->tablet_manager()->GetTabletPeers();


We don't have to do that for every peers because RocksDB block cache is shared across all tablets:

yugabyte-db/src/yb/tserver/ts_tablet_manager.h

Line 788 in 44cb504

tablet::TabletOptions tablet_options_;

ttyusupov · 2025-10-13T15:33:54Z

src/yb/tools/yb-admin_cli.cc

  REGISTER_COMMAND(flush_table);
  REGISTER_COMMAND(flush_table_by_id);
  REGISTER_COMMAND(flush_sys_catalog);
+  REGISTER_COMMAND(clear_cache);


clear_block_cache should be better to avoid confusion with other caches

ttyusupov · 2025-10-13T15:35:09Z

src/yb/tserver/tablet_service.cc

+    // Setting the cache capacity to 0 forces the cache to evict all stored entries, effectively clearing its contents. 
+    // Immediately restoring the capacity to its original value allows the cache to resume normal operation 
+    // without permanently restricting its size.
+    auto original_capacity = block_cache->GetCapacity();


nit: const auto

ttyusupov · 2025-10-13T15:44:38Z

src/yb/tools/yb-admin_cli.cc

 }

+const auto clear_cache_args = "[<timeout_in_seconds>] (default 20)";
+


It would be good to add automated test into yb-admin_client-test.cc that will:

create table

put some data into it

flush table to disk (so we have SST files)

run scan query other the whole table to load SST blocks into block_cache

make sure block cache usage is above expected level

clear block cache

make sure block cache usage is zero

ttyusupov · 2025-10-13T16:59:36Z

src/yb/tools/yb-admin_client.cc

+  if (servers.empty()) {
+    return STATUS(IllegalState, "No tablet servers found in cluster");
+  }
+  LOG(INFO) << "Found " << servers.size() << " tablet servers" << endl;


Discussed internally that we shouldn't be using LOG(INFO) for yb-admin tool going forward

kenju added 2 commits September 28, 2025 15:49

add clear_cache yb-admin

5c33b99

purge rocksdb block cache

6b45547

kenju mentioned this pull request Sep 29, 2025

[DocDB] yb-admin command to clear the block cache #25722

Open

1 task

kenju changed the title ~~[#25722] Implemented yb-admin command to clear the block cache~~ [#25722] yb-admin command to clear rocksdb block cache Sep 29, 2025

kenju commented Sep 29, 2025

View reviewed changes

rthallamko3 requested a review from ttyusupov October 1, 2025 19:26

kenju mentioned this pull request Oct 3, 2025

[DocDB] Add unit tests to cover yb-admin flush_table and compact_table output #28579

Open

1 task

ttyusupov reviewed Oct 13, 2025

View reviewed changes

	void PurgeBlockCache() {
	auto* block_cache = table_factory_->table_options().block_cache.get();
	auto capacity = block_cache->GetCapacity();
	block_cache->SetCapacity(0);
	block_cache->SetCapacity(capacity);
	LOG(INFO) << "Purged block cache";
	}

		}

		const auto clear_cache_args = "[<timeout_in_seconds>] (default 20)";

[#25722] yb-admin command to clear rocksdb block cache #28733

Are you sure you want to change the base?

[#25722] yb-admin command to clear rocksdb block cache #28733

Conversation

kenju commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Made:

How to Purge

Testing:

Uh oh!

CLAassistant commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kenju commented Oct 1, 2025

Uh oh!

rthallamko3 commented Oct 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kenju commented Sep 29, 2025 •

edited

Loading

CLAassistant commented Sep 29, 2025 •

edited

Loading