Data lost when a large number of equality delete records exist

[CodingJun]

Describe

When a table contains a large number of equality delete records, some data rows that should be returned by queries are missing. This appears to happen even when the rows are not logically deleted. When using other engines (e.g., Apache Spark) to read the same table, the results are correct and all expected rows are returned. This suggests that the table data and delete files are valid, and the problem is likely specific to this engine’s equality-delete implementation or planning logic.

Env

Duckdb version: 1.4.1 (also occurs in 1.4.3)
Iceberg version: 1.5.2
Spark version: 3.5.4

Steps to reproduce

1. Create a table and insert a dataset of size X (such as 50k).
2. Write a large number of equality delete records Y (such as 49995).
3. Run a query such as: SELECT * FROM <table>.
4. Observe that some rows that are not deleted are missing from the result.

Expected behavior

Queries should return all rows that are not logically deleted, regardless of the number of equality delete records.

Actual behavior

Some valid rows are filtered out when the number of equality delete records grows large.

Example (for fast reproduce)

1. Download the attachment equality_test.zip
2. Unzip the archive and place the content into the directory: /tmp/data/
3. Execute select * from iceberg_scan('/tmp/data/equality_test/metadata/v3.metadata.json'); in Duckdb
4. Observe that the result is empty.
5. Execute select * from iceberg_scan('/tmp/data/equality_test/metadata/v3.metadata.json') where _id = 'id_183550'; in Duckdb
6. Observe that the result is not empty.

[CodingJun]

Spark Query Result (for comparison)

Execute the same query in Spark:

SELECT * FROM equality_test;

Result:

[CodingJun]

Update

The cause of the issue appears to be:

1. Deleted-row filtering is performed inside IcebergMultiFileReader::FinalizeChunk.

   duckdb-iceberg/src/iceberg_functions/iceberg_multi_file_reader.cpp

   Lines 398 to 414 in 1c0c4c6

   	void IcebergMultiFileReader::FinalizeChunk(ClientContext &context, const MultiFileBindData &bind_data,
   	BaseFileReader &reader, const MultiFileReaderData &reader_data,
   	DataChunk &input_chunk, DataChunk &output_chunk,
   	ExpressionExecutor &executor,
   	optional_ptr<MultiFileReaderGlobalState> global_state) {
   	// Base class finalization first
   	MultiFileReader::FinalizeChunk(context, bind_data, reader, reader_data, input_chunk, output_chunk, executor,
   	global_state);
   	D_ASSERT(global_state);
   	// Get the metadata for this file
   	const auto &multi_file_list = dynamic_cast<const IcebergMultiFileList &>(*global_state->file_list);
   	auto file_id = reader.file_list_idx.GetIndex();
   	auto &data_file = multi_file_list.data_files[file_id];
   	auto &local_columns = reader.columns;
   	ApplyEqualityDeletes(context, output_chunk, multi_file_list, data_file, local_columns);
   	}

2. If all rows in the current chunk are filtered out as deleted, the method returns an empty chunk.
3. When PhysicalTableScan::GetData receives an empty chunk, it considers the scan operator to be finished and does not attempt to read the next batch of data. https://github.com/duckdb/duckdb/blob/200198b3e08c0373ca32b9496ae325b44e0d16e0/src/execution/operator/scan/physical_table_scan.cpp#L106