Consolidate contingent paths in prune_binaries.py and switch to the upstream's lite source tarball #3202
Conversation
|
Right... without |
|
Better, I think, to just elaborate this PR. All it seems you're missing here (quickly eyeballing) is the change to |
Ahrotahn
changed the title
utils/prune_binaries.py
Outdated
| 'chrome/test/data/', | ||
| 'components/test/data/', | ||
| # prune the base path instead of individual ones since clones skip them | ||
| 'content/test/data/', |
There was a problem hiding this comment.
Note that some material under content/test/data/ is needed when building content_shell, see https://chromium-review.googlesource.com/c/chromium/tools/build/+/6175706
| 'third_party/llvm-build-tools/', | ||
| 'third_party/llvm-build/', | ||
| 'third_party/llvm/', | ||
| 'third_party/node/linux/', | ||
| 'third_party/rust-src/', | ||
| 'third_party/rust-toolchain/', | ||
| 'third_party/webgl/', | ||
| ) |
There was a problem hiding this comment.
I'd suggest keeping the lite/tarball sections of this list as near-copies of the upstream lists, just to make cross-checking and future merges easier to deal with. That's why I moved the VK-GL-CTL entry, etc.
|
Sure thing, I've added the specific test data paths and re-ordered everything. The only exception are the sysroots since the clone script is able to obtain ones that aren't available in the tarballs. |
|
Thanks, this LGTM. I see the new entries in |
|
My only concern is whether lite tarball is usable for macOS build. @Cubik65536 have you tried building from lite tarball by any chance? |
I am super busy this week so the earliest time I can confirm this is next week, maybe weekend. I will try this as soon as I got time to do so. |
|
@Ahrotahn If you're ready we can merge now and deal with macOS later. |
Handles the differences between full tarball, lite tarball, and cloned sources. Removes the skip_unused and sysroot flags from _extraction.py and downloads.py.
|
I'd give it another day or so to see if that causes problems with anyone's setup. I've been thinking about any possible impact these changes might have. I've just made a small change that adds the sysroots.json as an exclusion to domain substitution. This is because the lite tarball doesn't contain any sysroots. So now anyone that relied on that would need to run install-sysroot.py to obtain them and the domain substitution would previously cause that to fail. I've also added a few more contingent paths to handle possible differences between clones and tarballs. |
|
portable linux build works for me |
|
On NixOS as well, with minor modifications |
Much obliged, please let us know what you find. Note that the lite tarball has these directories removed, so what I need to know is which (if any) of those needs to be restored for a successful macOS build. If you find that files are missing, you can just copy in the appropriate directory from the regular/big tarball, and try again. (In the lite tarball, those directories aren't fully gone, they just have most of the file content removed. So you'll need to replace the appropriate directory in the lite tarball with one from the full tarball.) |
|
Windows build works as well. |
In the past we have only needed to deal with the differences between the full tarball and cloned sources. The lite tarballs are significantly smaller now with the changes @iskunk has made, so now we need to be able to reconcile the differences between all three sources.
This PR moves the contingent paths logic from the cloning script into the pruning script so all three sources are treated equally.
This also removes the
--skip-unusedflag fromdownloads.pyand_extraction.py. That flag skipped extraction of the contingent paths in order to save space, but there are still some files in those paths that are necessary for building and the lite tarball can now be used instead.The
--sysrootflag was removed fromdownloads.pyand_extraction.pyas well since it was only used to exclude the sysroot when--skip-unusedis present. The flag is still present and functions normally in the other scripts.If either of those flags are passed to
downloads.pya warning will be logged noting that they are non-functional and will be fully removed in the future. This should hopefully allow leeway for anyone that might still have the flags set in their builds.These changes will allow us to proceed with using the lite tarball (#3197) and should help with the possibility of moving the contingent paths into their own list file in the future.