Skip to content

Releases: Altinity/clickhouse-sink-connector

Release v2.7.1

09 Jul 22:31
Compare
Choose a tag to compare

Lightweight
Docker: altinity/clickhouse-sink-connector:2.7.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.7.1-lt/images/sha256-6aeb1dcbcd7b2f0e0f43f3f31cdc198a594977b2c2e338552465ad50eebd02ef

What's Changed

New Contributors

Full Changelog: 2.7.0...2.7.1

Release v2.7.0

30 Jun 16:35
Compare
Choose a tag to compare

Lightweight
Docker: altinity/clickhouse-sink-connector:2.7.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.7.0-lt/images/sha256-2e0f1a0487a71a11b19200320707545ec5c5992b0abc37ad09ec954c36352f37

Breaking Changes

Configuration Changes
offset.storage.jdbc.offset.table.name to offset.storage.jdbc.table.name
offset.storage.jdbc.offset.table.ddl to offset.storage.jdbc.table.ddl
offset.storage.jdbc.offset.table.delete to offset.storage.jdbc.table.delete

What's new

What's Changed

New Contributors

Full Changelog: 2.6.0...2.7.0

Release v2.6.0

14 Apr 04:05
459cd71
Compare
Choose a tag to compare

Lightweight
Docker: altinity/clickhouse-sink-connector:2.6.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.6.0-lt/images/sha256-1114ee20b6ba76f78c2d4858ee3805115753aeecd26221875d68538ba1aeb6a8

What's New

MySQL SET Data Type #881
ARM support for docker builds (Experimental) - #955
Debezium upgraded to 3.1.0.AlphaFinal - #1001
PostgreSQL time with timezone mapping added - #951

What's Changed

Full Changelog: 2.5.1...2.6.0

Release v2.5.1

26 Feb 02:26
8771fa1
Compare
Choose a tag to compare

Lightweight
Docker: altinity/clickhouse-sink-connector:2.5.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.5.1-lt/images/sha256-ac7372e68da990fc0cee91923a779a4eca31407496aa0a15b2ec4a4a52acc165

What's new:

error.max.retries: All the clickhouse database calls will be retried based on the configuration variable.
Hikari connection pool: All the clickhouse db calls will be routed through Hikari connection pool. The number of connections are configurable.https://github.com/Altinity/clickhouse-sink-connector/blob/develop/doc/connection_pool.md

What's Changed

Postgres Numeric -> BigDecimal exception
Resolved an exception occurring when mapping PostgreSQL NUMERIC data types to Java's BigDecimal, enhancing compatibility with PostgreSQL sources.
Issue #980

DateTime64 truncation limit behavior different from ClickHouse
Addressed inconsistencies in DateTime64 truncation behavior between the connector and ClickHouse, ensuring uniform data handling.
Issue #971

DDL Translation error: modify column datatype with DEFAULT NULL
Fixed errors in Data Definition Language (DDL) translation when modifying column data types with DEFAULT NULL constraints, improving schema evolution support.
Issue #965

Hikari Connection Pool
​The pull request #893 introduces the integration of the HikariCP connection pool. This enhancement aims to efficiently manage ClickHouse connections, addressing potential connection leaks identified in issue #867. By implementing HikariCP, the connector benefits from improved performance and reliability through effective connection pooling.

Full Changelog: 2.5.0...2.5.1

Release v2.5.0

11 Dec 14:35
432d73d
Compare
Choose a tag to compare

Lightweight
Docker: altinity/clickhouse-sink-connector:2.5.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.5.0-lt/images/sha256-60d5edd7d89eeee4ec64ec802c95c223096793654cce256d58c4e15219c4d0e2?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.5.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.5.0-kafka/images/sha256-4cc61265b51b4a1bdfbed244c4b074daedb852c8a665ca1b9cf19a70243eb197?context=explore

What's new:

MongoDB support: Restored mongodb config.yml and instructions to setup sink connector to replicate MongoDB databases.
Debezium upgrade to 3.0.0. - Updated Debezium to the latest stable 3.0.0.
Ignore DDL statements based on regular expressions: ignore DDL from MySQL based on regular expressions. ignore.ddl.regex
Kafka Debezium upgrade: MySQL Debezium connector was upgraded to latest stable 3.0.0.
Performance improvements: Removed additional calls to clickhouse to get table schema.
MySQL DDL Support: Alter table rename without table keyword

What's Changed

New Contributors

Full Changelog: 2.4.0...2.4.1t

Release v2.4.0

17 Oct 23:31
Compare
Choose a tag to compare

Lightweight
Docker: altinity/clickhouse-sink-connector:2.4.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.4.0-lt/images/sha256-d509cafcadd11de0fd25b935efb0ee9e49429e9150ca9950c89d194e7a0b7f77?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.4.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.4.0-kafka/images/sha256-21e31d08819c6af5c52c9cf80fefe6cba979d87da41a689c66f5cbed4f4b0e5a?context=explore

What's new:

ddl retry: When the ddl.retry: "true" variable is added to config.yml, if for some reason DDL's translated by sink connector fails to execute on ClickHouse, logic has been added to retry on an exponential backoff loop.
MySQL POINT & POLYGON data types: The Polygon and Point datatypes will be mapped to ClickHouse.
MySQL DDL Support: Added support for DROP CONSTRAINT and fixed bug with dropping NULL in ALTER TABLE.
sink-connector-client Added commands to purge schema history and offset table.
Exclude Generated/Alias and Materialized: Added logic to exclude Materialized and Alias columns.

What's Changed

New Contributors

Full Changelog: 2.3.0...2

Release v2.3.1

18 Sep 21:05
f5da93a
Compare
Choose a tag to compare

Lightweight
Docker: altinity/clickhouse-sink-connector:2.3.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.1-lt/images/sha256-4cfbc7711a0c8fd47cda0385476026ce8456fb6a0eaeac7c9455f220c044cf07?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.3.1-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.1-kafka/images/sha256-577822ac4cb14dc7e48fba32239638915742ca2a758d518e3084ef81c375892a?context=explore

What's Changed

  • Update state_storage.md by @subkanthi in #771
  • Update README.md to include state storage link by @subkanthi in #772
  • Updated offset state storage documentation. by @subkanthi in #774
  • Update state_storage.md to include schema storage. by @subkanthi in #775
  • Update state_storage.md to include postgresql offsets by @subkanthi in #776
  • Update quickstart.md to start sink connector service by @subkanthi in #785
  • Added logic to get latest release from github by @subkanthi in #786
  • Update quickstart.md with script to set environment variable. by @subkanthi in #787
  • Update quickstart_postgres.md by @subkanthi in #788
  • Added updates to script by @subkanthi in #789
  • Added single.threaded flag to Mariadb test to validate replication in… by @subkanthi in #781
  • Update production_setup.md by @subkanthi in #796
  • Changed logging level to info for STRUCT EMPTY not a valid CDC record by @subkanthi in #795
  • Update production_setup.md to include max_paritions_per_insert by @subkanthi in #804
  • Update production_setup.md , fixed broken link by @subkanthi in #809
  • Update config.ym by @subkanthi in #813
  • Update README.md to include initial data dump/load by @subkanthi in #821
  • 794 change logging level of struct empty not a valid cdc record + record to info by @subkanthi in #825
  • 801 records are not acknowledged or the offsets are not updated in singlethreaded mode by @subkanthi in #827

Full Changelog: 2.3.0...2.3.1

Release v2.3.0

27 Aug 00:17
eaba052
Compare
Choose a tag to compare

Lightweight
Docker: altinity/clickhouse-sink-connector:2.3.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.0-lt/images/sha256-0634822631c25a88a1d5fe662000dacdcf85adac6fd907765f784ac693b92df0?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.3.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.0-kafka/images/sha256-4e228117805895f78608db8c101071d8286f0e23370d677ba268d0dd1ff7f291?context=explore

What's Changed

  • Update Monitoring.md to include metrics.port by @subkanthi in #735
  • Added integration test for PostgreSQL keepermap by @subkanthi in #728
  • Enable metrics by default for MySQL and postgres by @subkanthi in #750
  • Added documentation to set the replication start position by @subkanthi in #753
  • Added documentation for using JAR file for postgres replication by @subkanthi in #754
  • Update quickstart.md by @subkanthi in #763
  • Update quickstart.md update docker tag. by @subkanthi in #764
  • Removed schema history configuration settings for postgres by @subkanthi in #755
  • 725 we cant start grafana in sink connector we have cert issue by @subkanthi in #760
  • Added JMX exporter to export JMX metrics from debezium. by @subkanthi in #757
  • Disable validation of source database by @subkanthi in #716
  • Added Integration test to validate truncate event replication in post…gresql by @subkanthi in #759
  • Fix set lsn to accept string and not long by @subkanthi in #752
  • Added logic in retrying database in case of failure by @subkanthi in #761
  • 630 postgres heartbeat setup documentation by @subkanthi in #765
  • 628 add integration test for mariadb by @subkanthi in #673
  • Added functionality to replicate in single threaded mode based on configuration without using a Queue by @subkanthi in #756
  • Convert localDateTime to String in show_replica_status API call by @subkanthi in #736

Full Changelog: 2.2.1...2.3.0

Release v2.2.1

30 Jul 05:11
9796379
Compare
Choose a tag to compare

Lightweight
Docker: altinity/clickhouse-sink-connector:2.2.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.2.1-lt/images/sha256-40c1fca45b6c7c66417e9b4010776b6092a8f2ba2cd5bb59e461c4de17b9c384?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.2.1-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.2.1-kafka/images/sha256-78571eec2d5d9103bf75a993851e051d739cf110b07160e9f294d5d3baf7f494?context=explore

What's Changed

Full Changelog: 2.2.0...2.2.1

Release v2.2.0

16 Jul 01:31
bde03a9
Compare
Choose a tag to compare

Lightweight
Docker: altinity/clickhouse-sink-connector:2.2.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.2.0-lt/images/sha256-c2e2cf6d6dc59043a0cbc097ac908381779a2eaf3508ff2a0062f3beb8f2c4cb?context=explore

Breaking Changes

The environment variable LOGGING_LEVEL is no longer used for setting logging level of the application,from this release it will be set in the Root level="info" line in log4j2.xml.

    <Root level="info" additivity="false">
            <AppenderRef ref="console" />

Examples(Debug)

    <Root level="debug" additivity="false">
            <AppenderRef ref="console" />

Examples(Error)

    <Root level="error" additivity="false">
            <AppenderRef ref="console" />

Logging is configured using https://github.com/Altinity/clickhouse-sink-connector/blob/develop/sink-connector-lightweight/docker/log4j2.xml and is mounted to the docker container in https://github.com/Altinity/clickhouse-sink-connector/blob/develop/sink-connector-lightweight/docker/clickhouse-sink-connector-lt-service.yml. Refer to https://github.com/Altinity/clickhouse-sink-connector/blob/develop/doc/logging.md on setup.

What's Changed

  • Upgraded to debezium 2.7.0 and update new function definition for MyS… by @subkanthi in #587
  • Added log.error statements for all RunTimeException by @subkanthi in #588
  • Update development.md by @subkanthi in #604
  • Add TesTflows tests support for ReplicatedReplacingMergeTree engine by @Selfeer in #610
  • Revert back debezium version to 2.5.0.Beta1 by @subkanthi in #616
  • Added support to use user provided rmt delete column by @subkanthi in #593
  • Update quickstart README by @hoanganht1k27 in #618
  • Fix runners 24 4 by @Selfeer in #621
  • Fixed renaming of is_deleted column when the source columns have backticks by @subkanthi in #620
  • Added logic to include database name in the logs by @subkanthi in #613
  • Added Integration test to validate mapping of MySQL JSON data types t… by @subkanthi in #592
  • Fix fails testflows by @Selfeer in #629
  • Fix broken CI/CD for TestFlows by @Selfeer in #633
  • 631 add database name or UUID to rrmt engine definition when tables are auto created by @subkanthi in #632
  • Resolve 2 2 0 conflicts by @Selfeer in #651
  • 631 add database name or UUID to rrmt engine definition when tables are auto created by @subkanthi in #643
  • 2.1.0 by @subkanthi in #647
  • Database name overrides by @subkanthi in #624
  • Remove clickhouse server database by @subkanthi in #614
  • Added Integration test to validate mismatch of mysql columns by @subkanthi in #627
  • Log exception cause when there is an exception starting connector by @subkanthi in #652
  • Added production setup architecture by @subkanthi in #638
  • 575 add integration test multiple schema for postgresql by @subkanthi in #579
  • Return error code when there is an error retrieving show_replica_status by @subkanthi in #658
  • Added logic to OVERRIDE CREATE DATABASE SQL by @subkanthi in #653
  • Changes to fix grafana datasource by @subkanthi in #639
  • Change MySQL docker image used in Java Integration test from latest t… by @subkanthi in #670
  • Added logic to move the CREATE VIEW operation to after the connector … by @subkanthi in #668
  • Avoid printing error logs when there are no records in the queue by @subkanthi in #674
  • Updated documentation on handling datetime/timestamp fields by @subkanthi in #662
  • Upgraded to debezium 2.7.0 and update new function definition for MyS… by @subkanthi in #590

New Contributors

Full Changelog: 2.1.0...2.2.0