Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd (ticdc): a better Kafka consumer #10990

Merged
merged 106 commits into from
Jun 19, 2024
Merged

cmd (ticdc): a better Kafka consumer #10990

merged 106 commits into from
Jun 19, 2024

Commits on Apr 26, 2024

  1. update kafka consumer

    @wk989898
    wk989898 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    80245d8 View commit details
    Browse the repository at this point in the history

  2. update kafka consumer

    @wk989898
    wk989898 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    13e4d94 View commit details
    Browse the repository at this point in the history

  3. fmt

    @wk989898
    wk989898 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    5381b0f View commit details
    Browse the repository at this point in the history

  4. lint

    @wk989898
    wk989898 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    b724425 View commit details
    Browse the repository at this point in the history

  5. fmt

    @wk989898
    wk989898 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    1034e76 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2024

  1. Merge branch 'pingcap:master' into kafka_consumer

    @wk989898
    wk989898 committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    934e6a9 View commit details
    Browse the repository at this point in the history

  2. specify getDefaultReplicaConfig

    @wk989898
    wk989898 committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    29928dd View commit details
    Browse the repository at this point in the history

  3. fmt

    @wk989898
    wk989898 committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    09d07ef View commit details
    Browse the repository at this point in the history

  4. add write lock

    @wk989898
    wk989898 committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    e3d8bb0 View commit details
    Browse the repository at this point in the history

  5. fix wrong partition num

    @wk989898
    wk989898 committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    138bfde View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. add tls

    @wk989898
    wk989898 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    2ee3430 View commit details
    Browse the repository at this point in the history

  2. fmt

    @wk989898
    wk989898 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    86fe3ba View commit details
    Browse the repository at this point in the history

  3. update tls config

    @wk989898
    wk989898 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    b4e4af8 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. chore

    @wk989898
    wk989898 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    9041033 View commit details
    Browse the repository at this point in the history

  2. update

    @wk989898
    wk989898 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    46a4cae View commit details
    Browse the repository at this point in the history

  3. fix bug

    @wk989898
    wk989898 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    afa0765 View commit details
    Browse the repository at this point in the history

  4. update saram consumer

    @wk989898
    wk989898 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    dc390ce View commit details
    Browse the repository at this point in the history

  5. update option

    @wk989898
    wk989898 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    9859bfa View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. fix saram consumer

    @wk989898
    wk989898 committed May 1, 2024
    Configuration menu
    Copy the full SHA
    fb261b1 View commit details
    Browse the repository at this point in the history

  2. add confluent consumer

    @wk989898
    wk989898 committed May 1, 2024
    Configuration menu
    Copy the full SHA
    d51a251 View commit details
    Browse the repository at this point in the history

  3. chore

    @wk989898
    wk989898 committed May 1, 2024
    Configuration menu
    Copy the full SHA
    543ff7c View commit details
    Browse the repository at this point in the history

  4. update

    @wk989898
    wk989898 committed May 1, 2024
    Configuration menu
    Copy the full SHA
    a09cdd3 View commit details
    Browse the repository at this point in the history

  5. chore

    @wk989898
    wk989898 committed May 1, 2024
    Configuration menu
    Copy the full SHA
    cfa3449 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. fix typo

    @wk989898
    wk989898 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    5a3dfc3 View commit details
    Browse the repository at this point in the history

  2. chore

    @wk989898
    wk989898 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    6128e12 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'tiflow/master' into kafka_consumer

    @wk989898
    wk989898 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    f0e05a3 View commit details
    Browse the repository at this point in the history

  4. keep consistent with the master

    @wk989898
    wk989898 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    8096c87 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. debug

    @wk989898
    wk989898 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    d27968b View commit details
    Browse the repository at this point in the history

  2. fix tidbSourceID error

    @wk989898
    wk989898 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    6ec7700 View commit details
    Browse the repository at this point in the history

  3. debug

    @wk989898
    wk989898 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    a9766c7 View commit details
    Browse the repository at this point in the history

  4. update

    @wk989898
    wk989898 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    8349e98 View commit details
    Browse the repository at this point in the history

  5. debug

    @wk989898
    wk989898 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    c91cdd3 View commit details
    Browse the repository at this point in the history

  6. fmt

    @wk989898
    wk989898 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    c9dddee View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. modify timeout and retry logic

    @wk989898
    wk989898 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    d582724 View commit details
    Browse the repository at this point in the history

  2. fmt

    @wk989898
    wk989898 committed May 10, 2024
    Configuration menu
    Copy the full SHA
    56bde2c View commit details
    Browse the repository at this point in the history

Commits on May 11, 2024

  1. update write logic

    @wk989898
    wk989898 committed May 11, 2024
    Configuration menu
    Copy the full SHA
    639312f View commit details
    Browse the repository at this point in the history

  2. fix

    @wk989898
    wk989898 committed May 11, 2024
    Configuration menu
    Copy the full SHA
    2b073a4 View commit details
    Browse the repository at this point in the history

  3. rewrite

    @wk989898
    wk989898 committed May 11, 2024
    Configuration menu
    Copy the full SHA
    d3f7c17 View commit details
    Browse the repository at this point in the history

Commits on May 12, 2024

  1. fix

    @wk989898
    wk989898 committed May 12, 2024
    Configuration menu
    Copy the full SHA
    e945a8d View commit details
    Browse the repository at this point in the history

  2. lint

    @wk989898
    wk989898 committed May 12, 2024
    Configuration menu
    Copy the full SHA
    aeb8e72 View commit details
    Browse the repository at this point in the history

  3. write in time

    @wk989898
    wk989898 committed May 12, 2024
    Configuration menu
    Copy the full SHA
    5c015dd View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. update

    @wk989898
    wk989898 committed May 14, 2024
    Configuration menu
    Copy the full SHA
    2e1c966 View commit details
    Browse the repository at this point in the history

  2. lint

    @wk989898
    wk989898 committed May 14, 2024
    Configuration menu
    Copy the full SHA
    84d35a0 View commit details
    Browse the repository at this point in the history

  3. hold on when meet error

    @wk989898
    wk989898 committed May 14, 2024
    Configuration menu
    Copy the full SHA
    74e67e0 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. update

    @wk989898
    wk989898 committed May 15, 2024
    Configuration menu
    Copy the full SHA
    dada862 View commit details
    Browse the repository at this point in the history

  2. avoid exit when the broker down

    @wk989898
    wk989898 committed May 15, 2024
    Configuration menu
    Copy the full SHA
    49aa48d View commit details
    Browse the repository at this point in the history

  3. add lock

    @wk989898
    wk989898 committed May 15, 2024
    Configuration menu
    Copy the full SHA
    59d9736 View commit details
    Browse the repository at this point in the history

  4. remove lock

    @wk989898
    wk989898 committed May 15, 2024
    Configuration menu
    Copy the full SHA
    40b0561 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. use confluent as default

    @wk989898
    wk989898 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    ab9a622 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. update

    @wk989898
    wk989898 committed May 27, 2024
    Configuration menu
    Copy the full SHA
    3e017c9 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into kafka_consumer

    @wk989898
    wk989898 committed May 27, 2024
    Configuration menu
    Copy the full SHA
    c310df0 View commit details
    Browse the repository at this point in the history

  3. chore

    @wk989898
    wk989898 committed May 27, 2024
    Configuration menu
    Copy the full SHA
    e7bef1c View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. Update cmd/kafka-consumer/confluent_consumer.go 

    Co-authored-by: CharlesCheung <[email protected]>
    @3AceShowHand @CharlesCheung96
    3AceShowHand and CharlesCheung96 committed May 28, 2024
    Configuration menu
    Copy the full SHA
    c8becd3 View commit details
    Browse the repository at this point in the history

  2. Update cmd/kafka-consumer/consumer.go 

    Co-authored-by: CharlesCheung <[email protected]>
    @3AceShowHand @CharlesCheung96
    3AceShowHand and CharlesCheung96 committed May 28, 2024
    Configuration menu
    Copy the full SHA
    dbfa549 View commit details
    Browse the repository at this point in the history

  3. rename consumerOption to the option.

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    a744c79 View commit details
    Browse the repository at this point in the history

  4. remove confluent from the consumer.

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    9bfd5ca View commit details
    Browse the repository at this point in the history

  5. adjust how to initialize the consumer

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    d5f1886 View commit details
    Browse the repository at this point in the history

  6. adjust how to initialize the writer

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    d499a6a View commit details
    Browse the repository at this point in the history

  7. read message no timeout.

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    0583721 View commit details
    Browse the repository at this point in the history

  8. fix format

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    4d4fdfa View commit details
    Browse the repository at this point in the history

  9. fix cancel

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    eccf2c9 View commit details
    Browse the repository at this point in the history

  10. adjust table id.

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    9c58cfa View commit details
    Browse the repository at this point in the history

  11. fix check

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    3e709f0 View commit details
    Browse the repository at this point in the history

  12. just panic immediately, no need to return error.

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    e112b29 View commit details
    Browse the repository at this point in the history

  13. panic if new the consumer meet error.

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    3b1f9c8 View commit details
    Browse the repository at this point in the history

  14. panic all the way.

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    6801928 View commit details
    Browse the repository at this point in the history

  15. simplify write message to the downstream

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    7249e3f View commit details
    Browse the repository at this point in the history

  16. simplify write message to the downstream

    @3AceShowHand
    3AceShowHand committed May 28, 2024
    Configuration menu
    Copy the full SHA
    dcf0e24 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. do not commit if received DDL not executed.

    @3AceShowHand
    3AceShowHand committed May 29, 2024
    Configuration menu
    Copy the full SHA
    d40840d View commit details
    Browse the repository at this point in the history

  2. simplify the code.

    @3AceShowHand
    3AceShowHand committed May 29, 2024
    Configuration menu
    Copy the full SHA
    ea5fb06 View commit details
    Browse the repository at this point in the history

  3. adjust the code.

    @3AceShowHand
    3AceShowHand committed May 29, 2024
    Configuration menu
    Copy the full SHA
    ef982f9 View commit details
    Browse the repository at this point in the history

  4. add more logs to help debug

    @3AceShowHand
    3AceShowHand committed May 29, 2024
    Configuration menu
    Copy the full SHA
    5c7f053 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. fix

    @wk989898
    wk989898 committed May 30, 2024
    Configuration menu
    Copy the full SHA
    7cbbe2e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into kafka_consumer

    @wk989898
    wk989898 committed May 30, 2024
    Configuration menu
    Copy the full SHA
    9a138b5 View commit details
    Browse the repository at this point in the history

  3. adjust the log level to debug for some verbose logs.

    @3AceShowHand
    3AceShowHand committed May 30, 2024
    Configuration menu
    Copy the full SHA
    bc025eb View commit details
    Browse the repository at this point in the history

Commits on May 31, 2024

  1. update

    @wk989898
    wk989898 committed May 31, 2024
    Configuration menu
    Copy the full SHA
    e8780b0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'kafka_consumer' of https://github.com/wk989898/tiflow i… 

    …nto kafka_consumer
    @wk989898
    wk989898 committed May 31, 2024
    Configuration menu
    Copy the full SHA
    c70aaf2 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. add more logs.

    @3AceShowHand
    3AceShowHand committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    b52aa58 View commit details
    Browse the repository at this point in the history

  2. fix build.

    @3AceShowHand
    3AceShowHand committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    88a10f1 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into kafka_consumer

    @3AceShowHand
    3AceShowHand committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    378af4f View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. add more test to cover index flag.

    @3AceShowHand
    3AceShowHand committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    d2c9df2 View commit details
    Browse the repository at this point in the history

  2. add unit test to cover index column flags.

    @3AceShowHand
    3AceShowHand committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    7dd9454 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'fix-index-flag' into kafka_consumer

    @3AceShowHand
    3AceShowHand committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    5e0c851 View commit details
    Browse the repository at this point in the history

  4. check kafka produce out-of-order IBM/sarama#2619

    @wk989898
    wk989898 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    ae93222 View commit details
    Browse the repository at this point in the history

  5. update logs.

    @3AceShowHand
    3AceShowHand committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    a22e303 View commit details
    Browse the repository at this point in the history

  6. chore(cdc,dm): add prow OWNERS files to controal approving for critia… 

    …l files (pingcap#10894)

close pingcap#10895
    @wuhuizuo @3AceShowHand
    wuhuizuo authored and 3AceShowHand committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    983bde3 View commit details
    Browse the repository at this point in the history

  7. dep: update tidb package (pingcap#11198)

    @lance6716 @3AceShowHand
    lance6716 authored and 3AceShowHand committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    6c37933 View commit details
    Browse the repository at this point in the history

  8. add more logs to the kafka consumer

    @3AceShowHand
    3AceShowHand committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    c7c9df3 View commit details
    Browse the repository at this point in the history

  9. add more logs.

    @3AceShowHand
    3AceShowHand committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    2613eec View commit details
    Browse the repository at this point in the history

  10. also print out schema, table,committs

    @3AceShowHand
    3AceShowHand committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    c26b825 View commit details
    Browse the repository at this point in the history

  11. fix the signature

    @3AceShowHand
    3AceShowHand committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    a0ddb81 View commit details
    Browse the repository at this point in the history

  12. fix the signature

    @3AceShowHand
    3AceShowHand committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    98bb9f5 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. add one more unit test.

    @3AceShowHand
    3AceShowHand committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    1399930 View commit details
    Browse the repository at this point in the history

  2. add timezone to the kafka consumer image.

    @3AceShowHand
    3AceShowHand committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    4e4e967 View commit details
    Browse the repository at this point in the history

  3. update error log

    @wk989898
    wk989898 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    a2354f1 View commit details
    Browse the repository at this point in the history

  4. update error handle

    @wk989898
    wk989898 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    294ed27 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. fix

    @wk989898
    wk989898 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    854a4ab View commit details
    Browse the repository at this point in the history

  2. retry if failed on read and commit message.

    @3AceShowHand
    3AceShowHand committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    acd59cb View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. Merge branch 'master' into kafka_consumer

    @wk989898
    wk989898 committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    c5ab229 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. set max allowed buffered message count and bytes

    @3AceShowHand
    3AceShowHand committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    ae2214c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'kafka_consumer' of https://github.com/wk989898/tiflow i… 

    …nto kafka_consumer
    @3AceShowHand
    3AceShowHand committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    a8abb93 View commit details
    Browse the repository at this point in the history

  3. remove some code.

    @3AceShowHand
    3AceShowHand committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    e1a6f7c View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. remove some debug code

    @3AceShowHand
    3AceShowHand committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    4b19081 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. fix format

    @3AceShowHand
    3AceShowHand committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    d373b0b View commit details
    Browse the repository at this point in the history

  2. add table id to some logs

    @3AceShowHand
    3AceShowHand committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    0eb1395 View commit details
    Browse the repository at this point in the history

  3. flush by the row change event.

    @3AceShowHand
    3AceShowHand committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    58f71db View commit details
    Browse the repository at this point in the history