Apache Iceberg version
1.11.0 (latest release)
Query engine
Flink
Please describe the bug 🐞
Component: Flink / DynamicIcebergSink Versions:
- Apache Iceberg: 1.11.0 (iceberg-flink-runtime-2.1)
- Apache Flink: 2.2.1
- Flink Kubernetes Operator: 1.15.0
- Catalog: S3 Tables (AWS)
Description:
After an ungraceful restart caused by node eviction (EKS node group recycle / AMI rotation), the DynamicIcebergSink enters a state where it:
- Consumes records from Kafka at full rate (~600–2500 records/sec)
- Produces zero Iceberg commits to any table
- Logs no errors or warnings
- Completes checkpoints successfully with zero failures
- Reports zero pendingCommittables and zero successfulCommittables
The job appears completely healthy from the outside (RUNNING state, Kafka offsets advancing, lag ~0, checkpoints completing) but silently writes nothing.
The condition is sticky across internal Flink task-level failovers (restoring from checkpoint reproduces the same broken state). It is only cleared by a full job resubmission that discards all operator state (operator-managed stateless restart or full delete + redeploy).
Root Cause Hypothesis:
When the job restores from a checkpoint after an ungraceful crash, the DynamicIcebergSink's internal writer state is restored into a broken or inconsistent state. The writer silently accepts incoming DynamicRecords but never flushes them to data files, resulting in zero committables reaching the committer.
This is distinct from a committer-level failure — the committer is not stuck, it simply has nothing to commit because the writer never produces output.
Willingness to contribute
Apache Iceberg version
1.11.0 (latest release)
Query engine
Flink
Please describe the bug 🐞
Component: Flink / DynamicIcebergSink Versions:
Description:
After an ungraceful restart caused by node eviction (EKS node group recycle / AMI rotation), the DynamicIcebergSink enters a state where it:
The job appears completely healthy from the outside (RUNNING state, Kafka offsets advancing, lag ~0, checkpoints completing) but silently writes nothing.
The condition is sticky across internal Flink task-level failovers (restoring from checkpoint reproduces the same broken state). It is only cleared by a full job resubmission that discards all operator state (operator-managed stateless restart or full delete + redeploy).
Root Cause Hypothesis:
When the job restores from a checkpoint after an ungraceful crash, the DynamicIcebergSink's internal writer state is restored into a broken or inconsistent state. The writer silently accepts incoming DynamicRecords but never flushes them to data files, resulting in zero committables reaching the committer.
This is distinct from a committer-level failure — the committer is not stuck, it simply has nothing to commit because the writer never produces output.
Willingness to contribute