Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically check "invariants" #3

Open
edmondop opened this issue Jan 21, 2025 · 0 comments
Open

Automatically check "invariants" #3

edmondop opened this issue Jan 21, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@edmondop
Copy link
Owner

Is your feature request related to a problem or challenge?

Is your feature request related to a problem or challenge?
I extracted this from apache#13651 so it was more visible

Part of apache#13648
Inspired by / copied from apache#13651 from @wiedld
During upgrade, downstream systems often experience issues due to implicit changes (not explicit API changes) of LogicalPlans that DataFusion code begins relying on, and which result in unintended consequences when upgrading to a new version of DataFusion (see apache#13525).

Describe the solution you'd like
The idea is to make the current implicit assumptions ("Invariants" in more formal language)( explict and automatically check them.

Examples of implicit assumptions:

Schema column names can't be repeated (this is explicitly mentioned on apache#13525)
Inputs to UnionExec must have the same schema
...
Describe alternatives you've considered
I like the approach @wiedld took in apache#13651 :

define the invariants
check the invariants for extensible interfaces (which may be user defined)
throw the error closer to the problem (rather than weird behavior later)
Additional context
Sub tasks:

Define infrastructure to check LP invariant. PR: apache#13651

Define infrastructure to check physical plan invariants: apache#13986

Define infrastructure for user-defined invariants. See issue: apache#14029

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

@edmondop edmondop added the enhancement New feature or request label Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant