Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v24.3.x] Do not enable translation stm when datalake is disabled #25421

Merged
merged 1 commit into from
Mar 19, 2025
Merged

[v24.3.x] Do not enable translation stm when datalake is disabled #25421

merged 1 commit into from
Mar 19, 2025

Conversation

mmaslankaprv
Copy link
Member

Backport of PR #25404

@mmaslankaprv mmaslankaprv added this to the v24.3.x-next milestone Mar 18, 2025
@mmaslankaprv mmaslankaprv added the kind/backport PRs targeting a stable branch label Mar 18, 2025
@mmaslankaprv mmaslankaprv marked this pull request as ready for review March 18, 2025 11:53
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Mar 18, 2025 •
edited
Loading

Retry command for Build#63296

please wait until all jobs are finished before running the slash command



/ci-repeat 1
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_recovery_mode@{"cloud_storage_type":1,"filesystem_catalog_mode":true}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_avro_schema@{"cloud_storage_type":1,"query_engine":"spark","use_serde_parquet":true}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_e2e_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":true,"query_engine":"spark"}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_topic_lifecycle@{"cloud_storage_type":1,"filesystem_catalog_mode":true}
tests/rptest/tests/polaris_catalog_smoke_test.py::PolarisCatalogSmokeTest.test_connecting_to_catalog@{"cloud_storage_type":1,"with_tls":false}
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_recovery_mode@{"cloud_storage_type":1,"filesystem_catalog_mode":false}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_avro_schema@{"cloud_storage_type":1,"query_engine":"spark","use_serde_parquet":false}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_e2e_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":false,"query_engine":"spark"}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_topic_lifecycle@{"cloud_storage_type":1,"filesystem_catalog_mode":false}
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_disabled_partitions@{"cloud_storage_type":1,"filesystem_catalog_mode":false}
tests/rptest/tests/datalake/rest_catalog_connection_test.py::RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog@{"cloud_storage_type":1,"use_serde_parquet":false}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_avro_schema@{"cloud_storage_type":1,"query_engine":"trino","use_serde_parquet":false}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_upload_after_external_update@{"cloud_storage_type":1}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_e2e_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":false,"query_engine":"trino"}
tests/rptest/tests/polaris_catalog_smoke_test.py::PolarisCatalogSmokeTest.test_connecting_to_catalog@{"cloud_storage_type":1,"with_tls":true}
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_disabled_partitions@{"cloud_storage_type":1,"filesystem_catalog_mode":true}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeMetricsTest.test_lag_metrics@{"cloud_storage_type":1}
tests/rptest/tests/datalake/coordinator_retention_test.py::CoordinatorRetentionTest.test_retention@{"cloud_storage_type":1}
tests/rptest/tests/datalake/rest_catalog_connection_test.py::RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog@{"cloud_storage_type":1,"use_serde_parquet":true}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_avro_schema@{"cloud_storage_type":1,"query_engine":"trino","use_serde_parquet":true}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_e2e_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":true,"query_engine":"trino"}
tests/rptest/tests/datalake/compaction_gaps_test.py::CompactionGapsTest.test_translation_no_gaps@{"cloud_storage_type":1}

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Mar 18, 2025 •
edited
Loading

CI test results

test results on build#63296
test_id test_kind job_url test_status passed
rptest.tests.datalake.compaction_gaps_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-427c-851d-b65a6d7e015c FAIL 0/20
rptest.tests.datalake.coordinator_retention_test.CoordinatorRetentionTest.test_retention.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf65-4122-812a-c63ec16bebab FAIL 0/20
rptest.tests.datalake.coordinator_retention_test.CoordinatorRetentionTest.test_retention.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-45a0-bb9f-972bc9295981 FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_avro_schema.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.use_serde_parquet=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf66-46fe-b7ac-770fcacd5cea FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_avro_schema.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.use_serde_parquet=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f31f-4b33-bd73-432a27a345f3 FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_avro_schema.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.use_serde_parquet=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf67-4727-9805-30396fee61da FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_avro_schema.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.use_serde_parquet=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f320-4622-a85c-db79aad1c215 FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_avro_schema.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.use_serde_parquet=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf67-4f5f-b717-f149033a054e FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_avro_schema.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.use_serde_parquet=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-427c-851d-b65a6d7e015c FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_avro_schema.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.use_serde_parquet=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf65-4122-812a-c63ec16bebab FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_avro_schema.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.use_serde_parquet=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-45a0-bb9f-972bc9295981 FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_e2e_basic.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.filesystem_catalog_mode=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf66-46fe-b7ac-770fcacd5cea FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_e2e_basic.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.filesystem_catalog_mode=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f31f-4b33-bd73-432a27a345f3 FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_e2e_basic.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.filesystem_catalog_mode=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf67-4727-9805-30396fee61da FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_e2e_basic.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.filesystem_catalog_mode=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f320-4622-a85c-db79aad1c215 FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_e2e_basic.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.filesystem_catalog_mode=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf67-4f5f-b717-f149033a054e FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_e2e_basic.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.filesystem_catalog_mode=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-427c-851d-b65a6d7e015c FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_e2e_basic.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.filesystem_catalog_mode=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf65-4122-812a-c63ec16bebab FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_e2e_basic.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.filesystem_catalog_mode=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-45a0-bb9f-972bc9295981 FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_topic_lifecycle.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf66-46fe-b7ac-770fcacd5cea FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_topic_lifecycle.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f31f-4b33-bd73-432a27a345f3 FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_topic_lifecycle.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf67-4727-9805-30396fee61da FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_topic_lifecycle.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f320-4622-a85c-db79aad1c215 FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_upload_after_external_update.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf67-4f5f-b717-f149033a054e FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_upload_after_external_update.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-427c-851d-b65a6d7e015c FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeMetricsTest.test_lag_metrics.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf65-4122-812a-c63ec16bebab FAIL 0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeMetricsTest.test_lag_metrics.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-45a0-bb9f-972bc9295981 FAIL 0/20
rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf67-4f5f-b717-f149033a054e FAIL 0/20
rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-427c-851d-b65a6d7e015c FAIL 0/20
rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf65-4122-812a-c63ec16bebab FAIL 0/20
rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-45a0-bb9f-972bc9295981 FAIL 0/20
rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf66-46fe-b7ac-770fcacd5cea FAIL 0/20
rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f31f-4b33-bd73-432a27a345f3 FAIL 0/20
rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf67-4727-9805-30396fee61da FAIL 0/20
rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f320-4622-a85c-db79aad1c215 FAIL 0/20
rptest.tests.datalake.rest_catalog_connection_test.RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog.cloud_storage_type=CloudStorageType.S3.use_serde_parquet=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf67-4f5f-b717-f149033a054e FAIL 0/20
rptest.tests.datalake.rest_catalog_connection_test.RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog.cloud_storage_type=CloudStorageType.S3.use_serde_parquet=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-427c-851d-b65a6d7e015c FAIL 0/20
rptest.tests.datalake.rest_catalog_connection_test.RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog.cloud_storage_type=CloudStorageType.S3.use_serde_parquet=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf65-4122-812a-c63ec16bebab FAIL 0/20
rptest.tests.datalake.rest_catalog_connection_test.RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog.cloud_storage_type=CloudStorageType.S3.use_serde_parquet=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f321-45a0-bb9f-972bc9295981 FAIL 0/20
rptest.tests.polaris_catalog_smoke_test.PolarisCatalogSmokeTest.test_connecting_to_catalog.cloud_storage_type=CloudStorageType.S3.with_tls=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf67-4727-9805-30396fee61da FAIL 0/20
rptest.tests.polaris_catalog_smoke_test.PolarisCatalogSmokeTest.test_connecting_to_catalog.cloud_storage_type=CloudStorageType.S3.with_tls=False ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f31f-4b33-bd73-432a27a345f3 FAIL 0/20
rptest.tests.polaris_catalog_smoke_test.PolarisCatalogSmokeTest.test_connecting_to_catalog.cloud_storage_type=CloudStorageType.S3.with_tls=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a95d-cf67-4f5f-b717-f149033a054e FAIL 0/20
rptest.tests.polaris_catalog_smoke_test.PolarisCatalogSmokeTest.test_connecting_to_catalog.cloud_storage_type=CloudStorageType.S3.with_tls=True ducktape https://buildkite.com/redpanda/redpanda/builds/63296#0195a961-f320-4622-a85c-db79aad1c215 FAIL 0/20
test results on build#63316
test_id test_kind job_url test_status passed
rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.ABS.retention_type=retention.bytes ducktape https://buildkite.com/redpanda/redpanda/builds/63316#0195ab15-b8ce-4e90-a703-f64ede1fddaf FLAKY 1/2
rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.TIRED_STORAGE.cleanup_policy=compact.delete ducktape https://buildkite.com/redpanda/redpanda/builds/63316#0195ab19-f487-4694-9292-e84fb3980a05 FLAKY 1/2
rptest.tests.partition_force_reconfiguration_test.NodeWiseRecoveryTest.test_node_wise_recovery.dead_node_count=2 ducktape https://buildkite.com/redpanda/redpanda/builds/63316#0195ab19-f487-4694-9292-e84fb3980a05 FLAKY 1/2
rptest.tests.schema_registry_test.SchemaRegistryAutoAuthTest.test_normalize.dataset_type=JSON ducktape https://buildkite.com/redpanda/redpanda/builds/63316#0195ab19-f487-4218-ba4b-2c28be27c533 FLAKY 1/2
rptest.tests.schema_registry_test.SchemaRegistryTest.test_normalize.dataset_type=JSON ducktape https://buildkite.com/redpanda/redpanda/builds/63316#0195ab15-b8cd-46cf-8e6a-140f34a2cc06 FLAKY 1/2
rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_unavailable_test.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/63316#0195ab19-f487-4694-9292-e84fb3980a05 FLAKY 1/2
storage_single_thread_rpunit.storage_single_thread_rpunit unit https://buildkite.com/redpanda/redpanda/builds/63316#0195aad2-b544-468c-91d4-bf7597e0e608 FLAKY 1/2

@mmaslankaprv mmaslankaprv modified the milestones: v24.3.x-next, v24.3.8 Mar 18, 2025
@mmaslankaprv
dl: do not enable translation stm when datalake is disabled
70d0afe 
When `iceberg_enabled` cluster configuration property is set to false we
do not have to create `datalake::translation::state_machine` for each
partition. This way when datalake capabilities are disabled in the
cluster we will not have to reply the whole log for the datalake state
machine.

Signed-off-by: Michał Maślanka <[email protected]>
(cherry picked from commit 485e9a4)
@mmaslankaprv mmaslankaprv force-pushed the manual-backport-25404-v24.3.x-126 branch from f5eb471 to 70d0afe Compare March 18, 2025 19:53
@andrwng andrwng merged commit b1dd9f5 into redpanda-data:v24.3.x Mar 19, 2025
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WillemKauf WillemKauf WillemKauf approved these changes

@andrwng andrwng andrwng approved these changes

@bharathv bharathv Awaiting requested review from bharathv

Assignees
No one assigned
Labels
area/redpanda kind/backport PRs targeting a stable branch
Projects
None yet
Milestone
v24.3.8
Development

Successfully merging this pull request may close these issues.
4 participants
@mmaslankaprv @vbotbuildovich @WillemKauf @andrwng