feat(spark): implement column pruning for incremental queries

[suryaprasanna]

Describe the issue this Pull Request addresses

This PR implements column pruning optimization for incremental queries by migrating IncrementalRelationV1 and IncrementalRelationV2 from TableScan to PrunedScan interface. Currently, incremental queries read all columns from the source files even when only a subset is needed, leading to unnecessary I/O and memory overhead.

Summary and Changelog

Users gain improved query performance for incremental reads by only reading required columns from source files instead of the full schema.

Changes:

• Migrated IncrementalRelationV1 and IncrementalRelationV2 from TableScan to PrunedScan
• Implemented buildScan(requiredColumns: Array[String]) to accept column pruning
• Added getPrunedSchema() to build schema with required columns plus mandatory fields (_hoodie_commit_time and partition columns)
• Added filterRequiredColumnsFromDF() to remove auxiliary columns from final result
• Enabled Parquet filter pushdown and record-level filters for better optimization
• Updated HoodieStreamSourceV1 and HoodieStreamSourceV2 to pass required columns

Impact

Performance improvement - Incremental queries will read fewer columns from Parquet files, reducing I/O and improving query latency, especially for wide tables when selecting few columns.

Risk Level

Low - This is an optimization that maintains backward compatibility. The changes only affect the incremental query path and fallback to reading all necessary fields if required.

Documentation Update

None - This is an internal optimization with no user-facing configuration changes or API modifications.

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[hudi-bot]

CI report:

• cb2abef Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build