Weekly Plan (Andrew Lamb) March 10, 2025

[alamb]

This is an attempt to organize myself and make what I plan to work on more visible

Weekly High Level Goals

• Complete board report: March 2025 ASF Board Report (March 12) #13713
• Make arrow release: Release arrow-rs / parquet minor version 54.3.0 (Mar 2025) arrow-rs#7107
• Complete draft blog post about the DataFusion Optimizer
• Get v1 of tree explain plans [EPIC] Complete SQL EXPLAIN Tree Rendering #14914 ready for wider review with @irenjj

(BTW if anyone wants to help with writing or releases that would be 🤗 )

Other projects I think are strategically important

These are the ones I plan to look at / review PRs for first

• Anything performance related (though really I am interested in things that help many/all DataFusion users)
• pushing expressions down into scans: Support Push down expression evaluation in TableProviders #14993
• Spark function feat: Add datafusion-spark crate #14392 (comment) (e.g. this with @shehabgamin)
• Productionizing / hardening external sorts: A complete solution for stable and safe sort with spill #14692 with @2010YOUY01

Weekly Stretch / Nice to have goals

• Figure out next steps for Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) #12393
• Figure out where next to improve our performance
• Try to move an example or two from datafusion-examples

Background

I am putting this list on github because:

1. I like how github renders checklists w/ PR titles so it is easy to track (I currently have a local text file...)
2. I thought others might be interested from seeing what I am doing / planning to do
3. It makes me feel better that I don't have time to review all the PRs 😭

The way I prioritize PRs is in the following order

1. Bug fixes
2. Documentation / UX / API improvements (things that make DataFusion easier/better to work with)
3. Performance improvements
4. New features with wide appeal
5. New functions

Note new features and functions are deliberately at the bottom

[alamb]

Review Queue

DataFusion: Bugs/UX/Performance

• perf: unwrap cast for comparing ints =/!= strings #15110
• fix: nested window function #15033
• fix: handle duplicate WindowFunction expressions in Substrait consumer #15211
• add manual trigger for extended tests in pull requests #14331
• shell script to collect Benchmarks #15144
• Fix datafusion proto crate json feature #15172
• hash join: add build-side join keys to memory accounting #14222
• Simpler to see expressions in explain tree mode #15163

DataFusion features

• feat: Add datafusion-spark crate #15168
• Support metadata columns (location, size, last_modified) in ListingTableProvider #15181
• feat: Add ConfigOptions to ScalarFunctionArgs #13527
• feat: Add config max_temp_directory_size to limit max disk usage for spilling queries #14975
• Introduce Async User Defined Functions  #14837
• feat: Fix multi-lines printing issue for datafusion-cli and add the streaming printing feature back #14954
• Add union_tag scalar function #14687

arrow-rs features

• Add Parquet Modular decryption (read) support + encryption flag arrow-rs#6637
• [ObjectStore] Add SpawnService arrow-rs#7253
• Add hooks to json encoder to override default encoding or add support for unsupported types arrow-rs#7015
• feat: impl display for DataType::List arrow-rs#7051
• Avro codec enhancements arrow-rs#6965