Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate the native options parser into Python. #20887

Merged
merged 7 commits into from
May 10, 2024

Conversation

benjyw
Copy link
Sponsor Contributor

@benjyw benjyw commented May 8, 2024

This change:

A) Provides Python bindings to the Rust options parser.
B) Integrates the Rust options parser into the options parsing flow.

Currently, the existing legacy parser and the native parser are
both consulted, and their results compared, with discrepancies
logged. This will allow us to evaluate and fix such discrepancies
during a transition period.

@benjyw
Copy link
Sponsor Contributor Author

benjyw commented May 8, 2024

Note that this is categorized as a new feature because we do want it to be mentioned in the changelog.

rust_option_type = option_type
rust_member_type = member_type

if option_type is dict:
Copy link
Sponsor Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Below is a bunch of munging to massage the raw rust values into their expected Python types.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like there's some pretty... intricate processing here with the eval calls and the shlex.spliting.

Might we ever end up with fully Rust-side manipulation of options, or would we expecting to always go back out through Python for these sort of adjustments? If the first one, is there a path for moving more of this logic into Rust too?

Copy link
Sponsor Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this mostly emulates what happens in the native parser, which unfortunately supports all sorts of clever types, expressed as a str -> type cast.

I don't yet know which of these will make sense to represent in Rust. At a guess, I'd say shell_str and path yes, but targets no?

Anyway, I think this will be best revisited after we get rid of the legacy parser. Then we can do more chipping away at the Python side of things once a whole ton of that cruft is gone.

@benjyw benjyw requested a review from huonw May 8, 2024 04:04
Copy link
Contributor

@huonw huonw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So good, really nice to be running them in parallel 👍

src/python/pants/option/global_options.py Outdated Show resolved Hide resolved
src/python/pants/option/global_options.py Outdated Show resolved Hide resolved
src/python/pants/option/global_options.py Outdated Show resolved Hide resolved
src/python/pants/option/native_options.py Show resolved Hide resolved
rust_option_type = option_type
rust_member_type = member_type

if option_type is dict:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like there's some pretty... intricate processing here with the eval calls and the shlex.spliting.

Might we ever end up with fully Rust-side manipulation of options, or would we expecting to always go back out through Python for these sort of adjustments? If the first one, is there a path for moving more of this logic into Rust too?

src/python/pants/option/options.py Outdated Show resolved Hide resolved
src/python/pants/option/options.py Show resolved Hide resolved
@huonw
Copy link
Contributor

huonw commented May 8, 2024

Also, when you have a chance, please merge main (or rebase onto it) and add some release notes to docs/notes/2.22.x.md. See #20888 for more info.

Copy link
Contributor

@huonw huonw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, other than the notes movement & text addition!

docs/notes/2.22.x.md Outdated Show resolved Hide resolved
@benjyw benjyw merged commit a3eaf70 into pantsbuild:main May 10, 2024
25 checks passed
@benjyw benjyw deleted the native_option_parser branch May 10, 2024 01:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants