Reproducible and faster generation - #403
Open
gergondet-woven wants to merge 2 commits into
Open
Conversation
This patch tries to improve reproducibility. The original code uses set in a couple of places but the output of set is not guaranteed to stay the same even if the inputs are the same. We work around this in two ways: - where possible, replace set with an ordered_set version - in some places, sort the deps/data list given to Bazel
- in build_dependency_graph don't visit the same packages multiple times - make some changes to avoid a lot of duplication in the output dependencies - add an unbounded lru_cache for most library information extraction
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi there, thanks for merging our changes earlier this year and for the significant overhaul in #396. We have rebased our own patches on top of that and found some that we hope can be useful to everyone 🙏
This PR proposes changes to bazel_ros2_rules that make the initial BUILD.bazel generation:
The first patch replaces set usage (which is non deterministic) with an ordered_set and sorting (when the order shouldn't matter).
Before this patch, we would get confusingly bad cache hits as the inputs and outputs of the actions would be the same but the linker script would have the libraries in different order (leading to unnecessary extra builds and tests actions being run).
The second one fixes an issue with the scrapping code that can make it revisit the same packages many times and add some caching to the system lookups which also happen many times during scrapping. Overall this made the repository generation twice as fast in our CI environment.
This change is