Stats GC: Unexpected behavior that an existing table with one record in stats_meta doesn't find its memory obj in memory cache

[winoros]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

No clear reproduce step, it's found in super big cluster testing with 1 million tables.

#57844
From the debugging image, you can see that


The log said that the table is not found in the memory cache. But we can read its table meta from infoschema and can read its stats meta from the physical table mysql.stats_meta.
And our stats initialization also reads all the things from mysql.stats_meta into memory.

So this could not find is very strange and should be treated as a bug.

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

current master

[winoros]

As you can see, it happened when we iterate rows read from mysql.stats_meta, which means that it did have a record in the kv storage. Then we got the table meta from infoschema and it succeeded.
So it did have stats.

[winoros]

In the testing env, this phenomenon also occurs when there's no foreground workload.
This makes this issue more strange :(

[winoros]

Another insight may be related to this issue.

The unneeded analyze metrics differ from the two tidb nodes.
And the most unexpected not founds are coming from the tidb node1. But tidb node0 also has the log.

So this issue seems still to be related to the table not analyzed.

[winoros]

In this pull request https://github.com/pingcap/tidb/pull/52347/files#diff-3e2f686997ae61d9cca386e3d4430ada2d09002649faa0714340f0a9cd0dcf37R96 .
We've made sure that the put into cache would succeed finally. So I still believe that the init stats should be correct. 🤔

[winoros]

Similar to the issue #57901.
The stats' GC process also uses the same assumption.

It directly uses the condition no row in mysql.stats_histograms without checking whether the table meta exists to decide the existence the of table.
So the table stats is wrongly deleted by gc process.
(And then being re-created when we dump row count changes into memory, and then loading into memory and being deleted by gc process again, hahaha...)

[Rustin170506]

@winoros If I understand correctly, this issue is dumpcate with #57983, right?