Skip to content

feat: support lists in 'resolve_keyed_by' #608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

ahal
Copy link
Collaborator

@ahal ahal commented Nov 21, 2024

I tested this in Gecko and it resulted in an identical full graph

@ahal ahal self-assigned this Nov 21, 2024
@ahal ahal requested review from a team and bhearsum November 21, 2024 18:49
@ahal ahal force-pushed the push-zptlxmssuqnn branch 2 times, most recently from 5daf787 to 8c4f563 Compare November 21, 2024 20:23
Copy link
Contributor

@jcristau jcristau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this breaks on python 3.8?

@bhearsum
Copy link
Contributor

Moving this back to draft to get it out of review queues.

@bhearsum bhearsum marked this pull request as draft December 23, 2024 13:55
@bhearsum bhearsum removed their request for review December 23, 2024 13:55
@ahal
Copy link
Collaborator Author

ahal commented Jan 9, 2025

Oops, I forgot about this PR.. I'll get it into shape soon

@ahal ahal force-pushed the push-zptlxmssuqnn branch from 8c4f563 to 4ed10f0 Compare January 9, 2025 21:59
@ahal ahal marked this pull request as ready for review January 10, 2025 14:30
@ahal ahal requested review from jcristau and hneiva January 10, 2025 14:31
Copy link
Contributor

@jcristau jcristau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this worth an addition to the "Keyed By" section of docs/concepts/transforms.rst, or an example somewhere?

@ahal ahal force-pushed the push-zptlxmssuqnn branch from 4ed10f0 to 89a1539 Compare January 17, 2025 21:42
@ahal ahal requested a review from jcristau January 17, 2025 21:43
Copy link
Contributor

@bhearsum bhearsum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc improvements here are fantastic!

Comment on lines +23 to +24
),
),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some tests for error / corner cases would be welcome. Eg:
container: {"a": [{"b": 1}, {"c": 2}]}
subfield: a.b

@transforms.add
def resolve_artifact_paths(config, tasks):
for task in tasks:
resolve_keyed_by(task, "worker.artifacts.path", task["label"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to bikeshed on this too much, so this is certainly not a blocker, but I have to admit that I find it a bit unintuitive that .path accesses multiple list items (in addition to access plain dict keys). I wonder if jq-style syntax would be preferable here? Eg: worker.artifacts[].path?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you. I'll mark this as draft until I get a chance to implement a [] syntax.

task["label"],
platform=platform,
)
yield task
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
yield task
yield new_task

@ahal
Copy link
Collaborator Author

ahal commented Mar 28, 2025

I didn't end up needing this for the thing I was working on at the time, so I've neglected it again.. but I'll try and get this landed at some point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants