Optimal Strategies for Iceberg Retention and Compaction to Enhance Performance

[Shekharrajak]

Query engine

Spark, EMR

Question

Hello Team,

I’d like to open a discussion on the best methods for maintaining Iceberg tables. I’m interested in your use cases and perspectives on the following options:

• Implementing a scheduled job using Spark Iceberg table APIs to handle retention, deletion of orphan files, and compaction.
• Using a scheduled job with Java Iceberg table APIs for retention, orphan file deletion, and compaction.
• Comparing the MOR (Merge on Read) versus COW (Copy on Write) update strategies and their respective impacts.

Looking forward to your insights!