Work list for R 4.4/Bioconductor 3.19 builds #49778

mfansler · 2024-08-03T10:16:11Z

Preparation for R 4.4/Bioconductor 3.19

It looks like we'll cross the r-essentials milestone¹ for building R 4.4 on Conda Forge this weekend. This seems like a good point to start thinking about the transition of Bioconductor packages here to v3.19. To expedite this, it would be useful to identify required dependencies that need work to migrate. This helps those of us working on Conda Forge to focus our efforts.

This list should now be comprehensive thanks to a helpful script from @aliciaaevans. However, feel free comment or check things off if you see they have completed. Updated info is available on the Conda Forge Status page.

Problematic R 4.4 Migrations

📢 Updated 2 September 2024 using missingCranPackages.py script.

Awaiting migration of 62 packages

In PR (20 packages)

NB: This is where the most help is needed!

Bot Error (4 packages)

Licenses have been packaged for these as of 9 Sept. and should start migration within ~24 hours.

r-rebus.unicode: bot error (bot CI job): main: Error running migrate-feedstock in container - JSON could not parse stdout:
r-rebus.datetimes: bot error (bot CI job): main: Error running migrate-feedstock in container - JSON could not parse stdout:
r-rebus.numbers: bot error (bot CI job): main: Error running migrate-feedstock in container - JSON could not parse stdout:
r-logr: bot error (bot CI job): main: Error running migrate-feedstock in container - JSON could not parse stdout:

Not Solvable (7 packages)

r-spdep: not solvable (bot CI job) @ main r-sf * cannot be installed because there are no viable options
r-magick: not solvable (bot CI job) @ main No candidates were found for imagemagick *.
r-gstat: not solvable (bot CI job) @ main r-sf >=0.7_2 cannot be installed because there are no viable options
r-rmpi: not solvable (bot CI job) @ main No candidates were found for openmpi 5.*.
r-transformr: not solvable (bot CI job) @ main r-sf * cannot be installed because there are no viable options
r-protolite: not solvable (bot CI job) @ main r-sf * cannot be installed because there are no viable options
r-tkrplot: not solvable (bot CI job) @ main No candidates were found for tcl *.

Awaiting Parents (31 packages)

"How can I help?"

If you would like to help, you can either send PRs with fixes or make comments on the Conda Forge feedstocks about possible solutions.

Generally the three status categories correspond to different types of troubleshooting.

"In PR" - feedstocks with this are usually having compilation problems (mostly C++); typically the issue is not about compilation per se but rather getting linking working.
"Not solvable" - feedstocks with this usually indicate something about the packages they depend on is wrong; identifying those specific packages is the first step and then figuring out how to fix those upstream
"Bot error" - feedstocks with this usually mean something is substandard about the recipe itself and the migrator is crashing on it; anecdotally, this frequently involves license issues, such as not including an explicit license_file: entry in the meta.yaml

The "In PR" is often the most difficult and requires some knowledge of how code compilation is done in R; the "Not solvable" involves sleuthing around; and "Bot error" is more like review work - read the recipe and compare it one that works.

_{[1] The Conda Forge r-essentials package is a metapackage that represents common workflows used in R, such as Shiny, tidyverse, and Jupyter kernel support.}

The text was updated successfully, but these errors were encountered:

mfansler · 2024-08-03T10:18:46Z

CC: @daler just a heads up

aliciaaevans · 2024-08-14T20:53:26Z

conda-forge/r-clusterr-feedstock#25 seems like maybe the Travis build just needs to be re-triggered.

mfansler · 2024-08-16T07:21:24Z

conda-forge/r-clusterr-feedstock#25 seems like maybe the Travis build just needs to be re-triggered.

Travis CI is failing for all linux-ppc64le jobs. There are some notes in the Status Issue about how to convert to either cross-compiling or emulation on Azure to get around this.

danielnachun · 2024-09-03T23:43:28Z

I took a pass through all the "In PR" failures (rerendering a handful whose logs had expired) and posted a comment for each one either extracting the error, or when possible proposing a fix. I will try to push fixes for the ones where I had a possible solution.

@mfansler One quick question - what do you think about just adding -pthread to PKG_LIBS for the handful of packages failing due to this? I think there only 3 affected by this in the above list:
conda-forge/r-pdftools-feedstock#48
conda-forge/r-strawr-feedstock#4
conda-forge/r-rmixmod-feedstock#17

I agree with the your suggestion in conda-forge/r-base-feedstock#325 to add -pthread to the Makeconf for Windows so that we don't have to keep adding this manually, but I'm not sure we need to wait for that be done for just 3 packages.

mfansler · 2024-09-04T07:29:05Z

@danielnachun that's wonderful - thanks for spending time with this! ❤️

Unfortunately, I'm short on bandwidth today, but will try having a look as soon as I can.

RE: -pthread - yes, we can just manually override to expedite getting to Bioconductor 3.19.

danielnachun · 2024-09-06T17:46:53Z

The discussion in conda-forge/r-rcppalgos-feedstock#16 prompted me to check something I should have looked into sooner - among the packages that are only failing on Windows, which ones have never had a successful Windows build? It turns out these 6 packages have not, and they are some of the weirder or more difficult failures:

So as not to block the R 4.4 migration, I think we should consider skipping the Windows builds on these for now as they don't have obvious solutions and Windows users were never able to use them anyway.

danielnachun · 2024-09-06T18:06:04Z

@mfansler for packages that don't have a PR because of bot errors or unsolvable dependencies, is it sufficient just to rerender the feedstock with the necessary manual fixes? Or does the migrator make other manual changes (aside from re-enabling Windows if it was skipped)?

Also does the migrator automatically open PRs for packages that were previously blocked by a missing parent? Or does a PR need to be made to https://github.com/conda-forge/conda-forge-pinning-feedstock?

mfansler · 2024-09-09T07:43:14Z

@danielnachun so far, I've adopted the practice that it is better, when possible, to fix the recipe without manually forcing a R 4.4/UCRT migration, so that the bot can start working normally. Specifically, to your question: Yes, the migrator will eventually notice fixed recipes and (AFAIK) will retry solving failed recipes every ~24 h.

When we manually migrate it, the problem often doesn't go away (e.g., downstream dependencies may encounter same issue) and technical debt accrues. However, there are exceptions that are unavoidable, so here are the different cases I see:

bot error - all of these I've fixed to now have been driven by substandard recipes, typically involving the lack of a proper license file. Rerendering the recipe without adding R 4.4 migration, but fixing the substandard aspect will unlock the recipe and should make it so that recipe just works in future migrations.
not solvable + CI error - some recipes come out as not solvable because one of their requirements made it through PR, but had a CI error that resulted in not uploading the build artifact. This is a matter of tracking down the problematic dependency.
not solvable + missing platform - another scenario is that the recipe's dependency couldn't build a particular platform. Sometimes this can be an error (e.g., there was a selector on skip: true that the bot didn't recognize) and we should try to fix it; in other cases, it can really be intrinsic, and I think this case is what you're interested.

Manual Migration

For the case 3, manual migration is done by adding in the migration file (see example commit) and making some changes to the meta.yaml:

delete definition and uses of {{ native }} Jinja variable
bump build number
delete merge_build_host: true
remove any skips
add any stdlib if missing
if pkg-config is used, remove the {{ posix }} prefix from it

Then a rerender needs to be done.

I believe the bot will recognize that the package is now available, and downstream dependencies should start flowing. However, that will not be the case if you are still skipping platforms - the downstream dependencies will be not solvable.

mfansler · 2024-09-10T10:21:37Z

@danielnachun FYI, any new PRs for R packages appear blocked at the moment: conda-forge/conda-forge-pinning-feedstock#6401

danielnachun · 2024-09-10T23:45:53Z

I was going to ask why things weren't restarting but that makes sense. Hopefully this gets figured out soon!

When we manually migrate it, the problem often doesn't go away (e.g., downstream dependencies may encounter same issue) and technical debt accrues.

This an important point for skipping Windows - if nothing depends on the package, then it doesn't propagate elsewhere. So we should put the most effort on fixing packages with dependents. Of the 6 I posted above, only r-magick and r-ttfpt1 seem to have a significant number of dependents, and we've now fixed r-ttfpt1 on Windows anyway. Getting imagemagick built on Windows may be a challenge but seems like it would resolve some issues

One other question - if a package is noarch: generic, does it still get count as being blocked by missing parents if one its dependencies is available for Linux but not Windows? I ask because noarch: generic packages are only built on Linux. Of course the package wouldn't be installable on Windows but that ideally shouldn't prevent its migration to a newer R.

mfansler · 2024-09-11T04:40:41Z

@danielnachun correct, noarch builds won't be blocked by a parent that fails on win-64. The solver will only try a linux-64 solve.

For the 6 feedstocks you highlighted, I've fixed all but r-rcppalgos (merged with win-64 skipped) and r-primme (just haven't gotten to yet, but seems doable). The r-magick in particular I was able to use the vendored imagemagick rather than trying to get Conda Forge building that for win-64. For r-rmpi, I've addressed the not solvable aspect, but we will still need to adjust further once the migrator actually sends a PR.

Thanks for all the new reviews! 🤩 I may not have time today, but I will eventually get to them.

mfansler added the help wanted label Aug 3, 2024

aliciaaevans mentioned this issue Aug 20, 2024

feat: Prereqs for Bioconductor script #50209

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Work list for R 4.4/Bioconductor 3.19 builds #49778

Work list for R 4.4/Bioconductor 3.19 builds #49778

mfansler commented Aug 3, 2024 •

edited

Loading

mfansler commented Aug 3, 2024

aliciaaevans commented Aug 14, 2024

mfansler commented Aug 16, 2024

danielnachun commented Sep 3, 2024

mfansler commented Sep 4, 2024

danielnachun commented Sep 6, 2024

danielnachun commented Sep 6, 2024

mfansler commented Sep 9, 2024 •

edited

Loading

mfansler commented Sep 10, 2024

danielnachun commented Sep 10, 2024

mfansler commented Sep 11, 2024

Work list for R 4.4/Bioconductor 3.19 builds #49778

Work list for R 4.4/Bioconductor 3.19 builds #49778

Comments

mfansler commented Aug 3, 2024 • edited Loading

Preparation for R 4.4/Bioconductor 3.19

Problematic R 4.4 Migrations

Awaiting migration of 62 packages

In PR (20 packages)

Bot Error (4 packages)

Not Solvable (7 packages)

Awaiting Parents (31 packages)

"How can I help?"

mfansler commented Aug 3, 2024

aliciaaevans commented Aug 14, 2024

mfansler commented Aug 16, 2024

danielnachun commented Sep 3, 2024

mfansler commented Sep 4, 2024

danielnachun commented Sep 6, 2024

danielnachun commented Sep 6, 2024

mfansler commented Sep 9, 2024 • edited Loading

Manual Migration

mfansler commented Sep 10, 2024

danielnachun commented Sep 10, 2024

mfansler commented Sep 11, 2024

mfansler commented Aug 3, 2024 •

edited

Loading

mfansler commented Sep 9, 2024 •

edited

Loading