Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug Report: Wasteful Topo calls in loadTabletsTrigger in the healthcheck #17876

Open
GuptaManan100 opened this issue Feb 28, 2025 · 0 comments · May be fixed by #17872
Open

Bug Report: Wasteful Topo calls in loadTabletsTrigger in the healthcheck #17876

GuptaManan100 opened this issue Feb 28, 2025 · 0 comments · May be fixed by #17872

Comments

@GuptaManan100
Copy link
Member

Overview of the Issue

In #12893 we added the capability to the health check to ask the topology watcher to reload all the tablets but that is wasteful, because we find that we don't have the primary tablet for a specific shard, we load all the tablets.

For most use-cases this isn't that much of an issue, because we use a single List call to list all the tablets. So the only impact is that we're reading a lot of data that we don't need but nothing else.

Specifically for large customers for 256 shards or so, there are ~750 vttablets, and reading all of these tablet records in List can exceed the maximum data etcd (or other topo server), can send in one go. Following this we read each tablet separately. This is an issue because we end up doing a lot of topo calls, most of which aren't needed.

We should instead only reload the tablets for the keyspace shard in question that we don't have the primary for.

Reproduction Steps

Given in the description

Binary Version

main

Operating System and Environment details

-

Log Fragments

@GuptaManan100 GuptaManan100 added Needs Triage This issue needs to be correctly labelled and triaged Type: Bug Component: Cluster management and removed Needs Triage This issue needs to be correctly labelled and triaged labels Feb 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant