Start conversion to Conda Build 3

[Lnaden]

This PR converts the conda-build-all script to be compatible with
Conda Build 3 and Conda 4.4, using the API as much as I can. Its
imperfect due to the imperfect API, but I think I managed to get what
we need without high-maintenance work arounds.

In theory, this should make the conda-build-all script much more stable
and we can switch back to using the conda-forge docker image as a
base after some iterations. I've basically rebuilt the script from scratch, salvaging what I could.

We've kicked this can down the road about as far as it can go at this
point so I think its time to implement a proper fix.

All existing functionality should remain unchanged from how the script
is used and invoked.

Complete list of everything I tweaked:

• Simplify imports to minimal set
• Import only from the Conda(-build) API when possible
• Added docstrings everywhere
• Use metapackage functions directly instead of reading from various metapackage dicts
• Reduce calls to the CLI directly where possible, replacing with API
• Convert previous multi-python multi-numpy build-mixing to Conda Build 3 Variant system
  • Variants compiled from command line, still supports --python and --numpy
• Build enumeration automatically checks for duplications
• Still uses old package string styles instead of hashed dependencies for checking if package already on Anaconda (Will need to convert at some point)
• Adds ability to check if package is already built locally, and can be overridden with --rebuild
• Preserved checking against specific channels, both at command line and meta.yaml level
• Package building handled by API instead of CLI
• Functions more generalized to allow importing the conda-build-all script if need be
• Switched all print statements to the Python 3 .format() mini-language over % formatting
• Minimized amount of excess data passed between functions, handled mostly by Metadata now
• Added ability for built and uploaded tarballs to be removed to save build space (e.g. Travis-CI) through the -c or --clean command line arg. This is False by default (related: [fahmunge] 0.3.0 conda-recipes#877)
• Unpinned the conda and conda-build versions from the scripts. Will need to make changes to the Docker Image though to really fix it.

Related issues: omnia-md/conda-recipes#799
Potentially resolved issues from these changes:

• Skipping Python-3.6 with old Numpy combination conda-recipes#712
• conda-build-all conda-recipes#403 (not likely, but something to consider)

[Lnaden]

I have also tried to add comments everywhere I can so other devs can follow what the script is doing and its easier to maintain later. If something is unclear, please comment on it so I can clean that up!

[jchodera]

@andrizzi and @pgrinaway: Can you help with reviewing this too?

[jchodera]

@Lnaden: Thanks so much for digging in and tackling this!

[Lnaden]

As part of the re-work of the docker images. Do we want to tag the ones which are Conda-forge based cf-texliveXX-cuda-YY or something of the like, or just replace the existing texliveXX-cudaYY ones?

I'm also working on a possible solution to the travis proliferation. For the conda-dev-recipes version we would have to keep the pyXX+cudaYY permutations (e.g. py35+cuda90, py35+cuda95, etc.) For the conda-recipes I might be able to make a single docker image which is conda-forge based, has all the cuda versions, and isn't large.

Idea for the how this would work:

• Base conda-forge + TexliveXX docker image
• Add CUDA versions to image how we used to
  • Download tarball, untar, move the RPMs we need, install the RPMs we need, remove files
  • This is different from how its currently done: Download full driver, install full driver
• Extend the conda-build-all script to accept variants for CUDA_SHORT_VERSION
• On the dev-recipes, we can build all the combinations
• On the release recipes, we split the builds by Python Version, and use all the CUDA_SHORT_VERSION variants.

For example, if I am building yank and openmm with the following variant scheme:

base_variants = {
    'python': ['2.7', '3.5', '3.6', '3.7'], 
    'numpy': ['1.09'], 
    'CUDA_SHORT_VERSION': ['75', '90']
}

The full variant proliferation would result in 16 builds (2 packages * 2 CUDA versions * 4 pythons = 16),
however, because of the de-duplication I have in conda-build-all based on package strings, we instead only get 12 builds:

• OpenMM: 4 Python * 2 cuda = 8
• Yank: 4 python = 4 . (Yank does not build on Python 2.7, but assuming it does for this example)

We will very easily run out of time on the Travis builds, but we can just re-trigger them over and over and get the builds done, but that way we will only have 3-4 builds instead of >10 if we have one build for each cuda and python, with many possible duplications if builds which dont depend on cuda get built on multiple cuda travis builds for the same python version.

Thoughts?

[jchodera]

As part of the re-work of the docker images. Do we want to tag the ones which are Conda-forge based cf-texliveXX-cuda-YY or something of the like, or just replace the existing texliveXX-cudaYY ones?

Let's NOT replace the existing ones! They're working now, and if we start replacing them and end up with a broken build system, we're screwed.

[Lnaden]

Let's NOT replace the existing ones!

That is what I figured. So I can make new tags if my testing bears out. Does the cf-.... prefix work as a tag indicating of the new Conda-forge derivative builds?

[jchodera]

@Lnaden : It would be great if we could use the conda-forge docker image directly so we don't have to worry about getting out of sync.

Since we last checked, there are now more cudatoolkit packages available on the anaconda channel which might also be of use to us, though this still seems incomplete. If we only need to support the latest versions (9.2 and 10.0, which sounds eminently reasonable!) we can go with these packages that are complete, though the PPC version is missing (important for Summit), though we could help contribute here.

[Lnaden]

It would be great if we could use the conda-forge docker image directly so we don't have to worry about getting out of sync.

I had been working on that first also, see: omnia-md/omnia-linux-anvil#12.
I need to circle back to that to verify I only install the minimal amount of items to keep the image size small. I was having issues with Docker Hub doing the auto-docker images (nothing happened when I clicked the "new image from github" button, so I started building the base on docker cloud (https://cloud.docker.com/u/omniamd/repository/docker/omniamd/omnia-linux-anvil).

[jchodera]

This might suddenly be becoming much more urgent given the upcoming conda-build migration:
https://twitter.com/condaforge/status/1083379357840744448?s=19

[jchodera]

@Lnaden : Just checking in on this again!

[Lnaden]

@jchodera Okay... I think I got everything added finally and this is ready to see if it works.

• The docker images are updates and based on Conda-Forge's base image. (https://cloud.docker.com/u/omniamd/repository/docker/omniamd/omnia-linux-anvil/tags)
• I have updated the conda-build-all script to allow newer versions of Conda and Conda Build
• I have preserved the functionality of the the old script while improving the following:
  • Uses the API as much as possible
  • Does not try and manually infer things from the outputs
  • Fully documented so the next person can follow along easier
  • Added the --cuda option to allow specifying a manual CUDA build
  • Does not fail on build fails
  • Uses the New-ish Conda Variant system
  • Preserves the naming system from our old builds to ensure we dont double our builds, as opposed to the new hashed tarball strings Conda moved to (something we will need to re-evaluate in the future)
• Updated the Xcode image on travis to 7.3 (cc Update xcode from 6.4 to 7.3.1 conda-recipes#717)

I imagine there will be some failures here, but I think this is almost ready finally. If this works here, we c

cc @peastman and @andrrizzi

[andrrizzi]

I went through it quickly, but it looks good to me! It looks like there's a problem with the build in Travis though. Not sure if it's expected.

[jchodera]

@Lnaden : It looks like the openmm recipe is failing to build:

CMake Error: The source directory "/opt/conda/conda-bld/openmm_1547581391466/work" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.

The error seems to suggest that the way that the github package is unpacked is different with this newer version of conda-build---perhaps the github repo source directory is not already entered on build?

I'll add some debug info to see if we can figure out what is going on.

[jchodera]

@Lnaden : Whoops, you're not working in a branch on this repo, so I can't debug your PR. Can you poke around a bit, or upload your changes as a branch in this repo if you'd like some help debugging?

[Lnaden]

I'm looking into it

[Lnaden]

The issue comes with the work directory getting shuffled in the process since I invoke conda build twice through the API, and me not quite correctly handling all the variants. The block of code controlling this behavior is here, and I can almost work around it though the dirty flag, but it only works if only one variant is specified.

I'm still digging into it.

[jchodera]

Thanks for tackling this, @Lnaden!

[jchodera]

@Lnaden : Looks like we'll need to bite the bullet and make this transition soon. Any thoughts on what we can do to move forward here?

[Lnaden]

My local tests have about 1 last issue where the variants either all build and fail together for a package (e.g. py 27,36,37 ), or their work directories clash/don't exist if I try to build them separate.

Give me some time over the next couple days, I'll let you know by Thursday mid-day where things stand

[jchodera]

I haven't yet had a chance to delve too deeply into conda-build-all, but some thoughts:

• Is conda-build-all still being actively developed here?
• Is it possible that the features we need are now in conda-build or the referenced conda-concourse-ci tools?

[Lnaden]

Is conda-build-all still being actively developed here?

No, that fell by the wayside with Conda Build 3 being released

features we need are now in conda-build

That is what I am trying to do, use conda-build itself as much as possible. And the problem I am running into is using the mechanic means either all of the variants for a package pass, or they all fail.

or the referenced conda-concourse-ci

I have not looked too much into this tool to get a good sense of if it would be helpful.

[Lnaden]

I think I finally have all the features in, replicating most of the existing behavior and our build style while using as much of the API as I can. Lets see if the tests pass

[jchodera]

Looks like the docker image is either missing CUDA or the CUDA libraries are in a different location and we have to update the paths:
https://travis-ci.org/omnia-md/conda-dev-recipes/jobs/489744345#L3424

[Lnaden]

I had to tweak the images and the CUDA lib pointer, but its building locally

[Lnaden]

The Linux builds are passing, brew is giving me issues, but progress!

[Lnaden]

I think I'll need to do some fiddling with the build and host sections of the meta.yaml file, but for the purposes of getting the conversion to conda build 3 going, that can be handled in a separate pr

[Lnaden]

This passes linux tests for the most part. There are some changes which would likely need to be made with respect to the CI configs to be more optimal, but I think this will get us onto conda build 3 finally.

The Travis OSX image is having issues, and the AppVeyor images are having argument issues, but the conda-build items appear to be resolved

[jchodera]

@Lnaden: Thanks so much for this!

If you want to open a PR into a branch of omnia-md/conda-dev-recipes instead of master, I can merge this in and take over debugging/fixing.

[Lnaden]

This is about to be replaced so I am closing this.