Lightning: fix performance regression introduced by behavior change of columnAPI.Cols()

[OliverS929]

What problem does this PR solve?

Issue Number: close #56705

Problem Summary:

Cached the result of columnAPI.Cols() in Encoder so that it could be repeatably used in encoding.

What changed and how does it work?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.
  • The original test cases were able to cover the change.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[tiprow]

Hi @OliverS929. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.

Please upload report for BASE (release-6.5@57b9e4b). Learn more about missing BASE report.

Additional details and impacted files

@@               Coverage Diff                @@
##             release-6.5     #59529   +/-   ##
================================================
  Coverage               ?   73.6837%           
================================================
  Files                  ?       1098           
  Lines                  ?     354162           
  Branches               ?          0           
================================================
  Hits                   ?     260960           
  Misses                 ?      76502           
  Partials               ?      16700           

[OliverS929]

@D3Hunter @lance6716 Please help take a look.

[Benjamin2037]

Please add test cases.

[D3Hunter]

maybe add some manual test about performance, to make sure it works as expected, before vs after

[D3Hunter]

/hold

you can unhold youself after test

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, lance6716

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [D3Hunter,lance6716]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-02-14 02:58:06.240014792 +0000 UTC m=+584528.636236849: ☑️ agreed by D3Hunter.
• 2025-02-14 03:23:22.575371357 +0000 UTC m=+586044.971593419: ☑️ agreed by lance6716.

[tangenta]

/unhold

[lance6716]

/hold

[OliverS929]

Manual test completed to verify the effectiveness of this commit:

• Testing cluster configuration: 1 TiDB, 1 TiKV, 1 PD. All on v6.5.11.
• Test setup: A table with 500 INT(11) columns, imported using a CSV file containing 100k rows of randomly generated integers.
• Test comparison: using tidb-lightning from the original 6.5.11 mirror (before fix) and tidb-lightning manual built with this commit.
• Test result:
  • Before fix, the encoding progress takes more than 133 seconds:
  • After fix, the encoding progress only takes 9 seconds:
    

Please help take a look @D3Hunter @tangenta @lance6716.

Since the testing result is as expected, I am going to unhold this PR for now. Please feel free to put this PR on hold if you feel more testing is needed.

[OliverS929]

/unhold

[OliverS929]

/retest

[tiprow]

@OliverS929: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[OliverS929]

/ok-to-test