-
Notifications
You must be signed in to change notification settings - Fork 14.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mark the hardcoded hybrid executors as deprecate and multi-exec as stable #46944
base: v2-10-stable
Are you sure you want to change the base?
Mark the hardcoded hybrid executors as deprecate and multi-exec as stable #46944
Conversation
Multiple Executor Configuration (aka hybrid executors) has been released for over half a year and can be marked stable. This gives users an option to migrate to from the old hardcoded hybrid executors (e.g CeleryKubernetesExecutor)
These executors will be removed in Airflow 3.0. Mark them as deprecated so users are aware and have time to migrate. - A deprecation warning will be printed when the executors are loaded - Documentation has been updated to note that they are deprecated - A news fragment added as well.
CC: @jedcunningham @jscheffl @amoghrajesh PR to mark the static executors as deprecated in Airflow core and mark multi-exec as stable. Re: https://lists.apache.org/thread/qmdf5dx71lzcvf4fd7qyc9gq0z0j2jqr |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good in my view
Co-authored-by: Jed Cunningham <[email protected]>
Co-authored-by: Jed Cunningham <[email protected]>
Getting the tests to ignore the deprecation warning at the right time has proven very annoying/tricky. The last stubborn one is related to one instances of the warning not being thrown, but it seems that some underlying failure is actually happening, but I'm not sure why it's happening on this branch that should be otherwise stable. And I also cannot produce it locally. From the test failure logs:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
We already have coverage for this behaviour with the other executor
In preparation of deprecating support in Airflow core for the old hardcoded hybrid executors in V3, mark them as deprecated and emit warnings when they are loaded. Also mark multi exec as stable to give users somewhere to migrate towards.
To broadcast the message of their deprecation this PR:
- Adds a deprecation warning when these old hybrid executors are loaded
- Documentation has been updated to note that they are deprecated and provide steps for migration.
- A news fragment was added as well.
The warning message:

Rendered docs:

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.