Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve(xcom): validate key in XCom.set() to prevent empty strings #46929

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

kiran2706
Copy link


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Copy link

boring-cyborg bot commented Feb 20, 2025

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://s.apache.org/airflow-slack

@kiran2706
Copy link
Author

What Changed

  • Added validation to ensure key in XCom.set() is not an empty string.
  • Raises ValueError if key is empty, preventing invalid XCom entries.

Why This Change?

  • Empty key can lead to difficult-to-debug issues when reading XComs.
  • Improves input validation for more robust XCom behavior.

Related Issue

Fixes #46925

Notes

  • This is a non-breaking change; it only validates input to improve reliability.

@kiran2706 kiran2706 force-pushed the bugfix/xcom-key-validation branch from 29ad7c4 to 3a622fe Compare February 20, 2025 12:37
Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder I'd we should also do something client side in the task sdk

@kiran2706
Copy link
Author

@ashb I’ve applied the suggested change. Let me know if anything else is needed. Thanks!

@kiran2706 kiran2706 requested a review from ashb February 20, 2025 13:12
@amoghrajesh amoghrajesh requested a review from uranusjr February 20, 2025 14:03
@kiran2706 kiran2706 force-pushed the bugfix/xcom-key-validation branch from 7a622e6 to e6f2801 Compare February 20, 2025 14:43
@ashb ashb linked an issue Feb 20, 2025 that may be closed by this pull request
2 tasks
@kiran2706 kiran2706 force-pushed the bugfix/xcom-key-validation branch from e6f2801 to 3ce170d Compare February 20, 2025 16:45
@kiran2706 kiran2706 force-pushed the bugfix/xcom-key-validation branch from 3ce170d to 761da97 Compare February 20, 2025 17:20
@kiran2706
Copy link
Author

@ashb , @amoghrajesh – Based on #46417 , we were leaning towards the conclusion of not allowing None as a key. However, @uranusjr brought up a valid point, and the docstring explicitly states that users can pass None to remove the filter:

:param key: A key for the XComs. If provided, only XComs with matching keys will be returned. Pass None (default) to remove the filter.

I find it useful to allow None so that users can retrieve all XComs when needed. What are your thoughts on this?

@ashb
Copy link
Member

ashb commented Feb 21, 2025

Allowing None/not set on fetch is okay. Set def shouldn't accept None nor empty string though.

We'll need to make sure that we handle key=None specially in the TaskSDK client though -- that should evalute to _ not passing_ the key argument -- but that might need a new API end point to allow that, as the URL is currently /xcoms/{dag_id}/{run_id}/{task_id}/{key} -- and we can't pass None there, as it has to be a string.

So to do the key=None = get all xcom values properly we would need to add a list endpoint /xcoms/{dag_id}/{run_id}/{task_id}/ that returns the key names found, and then client side we iterate over each value individually (as we didn't have a multiget on purpose, deciding to do it client side instead).

But that is a separate PR, and here we're talking about mostly set path, plus ensuring min length on the XCom get makes sense too (as that API end point cant return multiple values)

@kiran2706 kiran2706 force-pushed the bugfix/xcom-key-validation branch from 761da97 to 28a2c51 Compare February 21, 2025 10:58
@kiran2706
Copy link
Author

@ashb - Thanks for sharing your thoughts and clarifying my doubts. I've used Pydantic to validate the key in get_many() while still allowing None.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Xcom pull is failing when is key is None
4 participants