[question] Building using Conan CMakePresets via Jenkins jobs in parallel

[amachado-pie]

What is your question?

I'm having a problem with building multiple targets with Jenkins jobs in parallel using CMakePresets.
The source code is checkout once, and a build folder for each target job is created.
I had to create a CONAN_HOME folder per job since the Conan cache failed when trying multiple jobs using the same folder. But that's okay.

The issue I'm having is with the root file CMakePresets.json which Conan adds as the preset's include path in this file.
As there are several jobs, each one compiling a target, each one edits the CMakePresets.json adding the preset path.
But when the job ends and the build folder is deleted, CMakePresets.json becomes with a dangling path that the other build will give an error when invoking cmake --preset android-release saying that another preset --linux-release no longer exists.

The pipeline is something like this:

stage('Build Setup Packages') {
    steps {
        script {
        def buildMatrix = [:]
        def failedBuilds = []

        def versions = ['light', 'pro']
        def targets = ['linux-x86', 'windows', 'android']

        for (def version in versions) {
          for (def target in targets) {
            def jobName = "${target}-${version}"
            def buildDir = "build_${target}_${version}"
            def thisTarget = target
            def thisVersion = version

            buildMatrix[jobName] = {
            stage("Build ${jobName}") {
              // Run make_setup.sh with BUILD_FOLDER set to job-specific directory a local Conan cache
              sh """#!/bin/bash
                cd /workdir
                mkdir -p ${buildDir}                                
                export BUILD_FOLDER="${buildDir}"
                export CONAN_HOME="/workdir/.conan_${jobName}"

                mkdir -p "\${CONAN_HOME}/profiles"
                cp -r /home/builder/.conan2/profiles/* "\${CONAN_HOME}/profiles"

                conan remote disable conancenter 1> /dev/null
                conan remote add artifactory-packages "${artifactory_pkg_url}" 1> /dev/null

                echo "Building ${jobName} with \$(nproc) parallel jobs"
                scripts/make_setup.sh "${thisTarget}" "${thisVersion}"
              """
            }
          }
        }
        def results = parallel(buildMatrix)
        ...
  }

The script make_setup.sh basically does something like this:

eval "conan install .  -pr:h ./conan_profiles/{arch} -pr:b default -s build_type=${BUILD_TYPE^} -b missing -b outdated -c tools.cmake.cmake_layout:build_folder_vars='[\"const.${BUILD_PRESET}\"]'"
cmake --preset conan-${BUILD_PRESET}-${BUILD_TYPE,,}
cmake --build --preset conan-${BUILD_PRESET}-${BUILD_TYPE,,} -j$(nproc)

Would it be possible for Conan to edit and CMake to use CMakePresets.json defined in a build directory instead of the source directory to avoid collision? I wanted to avoid having to do code checkouts or source copy to be able to launch compilation jobs for each target.

I was able to reduce the occurrence of dangling paths in CMakePresets.json by deleting the file before invoking conan install ... & cmake --preset conan-xxx.
But even so, I've had problems with a job failing because it couldn't find the preset because it was deleted by another one:

16:05:44  conanfile.py (project/0.1): Generating aggregated env files
16:05:44  conanfile.py (project/0.1): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
16:05:44  Install finished successfully
16:05:44  CMake Error: Could not read presets from /workdir:
16:05:44  File not found: /workdir/CMakePresets.json

I know it's a bit confusing, but any suggestions on how to implement parallel CI/CD pipeline Jenkins jobs with Conan and CMake Presets?

Thanks!

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @amachado-pie

Thanks for your question

Let me try to see if I understood correctly:

• Several parallel jobs are building something
• The jobs use different Conan caches, because they parameterize the CONAN_HOME using the job number
• But the jobs seems to use the same buildDir, so basically there is just one single common source checkout?
• The Conan creates presets, Conan might create different subfolders, with different names for different configurations
• But the root CMakeUserPresets.json is including() those presets
• The job is removing the different subfolders so the inclusion from the CMakeUserPresets.json is broken

Did I get the issue correctly?

If this is the case, some quick questions/possible approaches:

• Why not using different buildDirs or different source checkouts for the parallel jobs? In general every parallel job in jenkins should have its independent and isolated workspace with its own checkout?
• Why removing the build subfolders in the first place, why not leaving them alone, they will typically be garbage collected later?

[amachado-pie]

Hi @memsharded thanks for your feedback!

Did I get the issue correctly?

Exactly!

Why not using different buildDirs or different source checkouts for the parallel jobs? In general every parallel job in jenkins should have its independent and isolated workspace with its own checkout?

The source code checkout is done in the Jenkins pipeline workspace, a Docker container with the necessary toolchains is executed by mounting the Jenkins workspace as a /workdir volume and within the container the Jenkins stages and parallel jobs are executed.

If we had to create a clean checkout in a workspace per target then we would have to launch as many Docker containers as targets, or serialize the builds. Or doing source checkouts inside the container instead of in the Jenkins workspace would imply that the container has credentials to access the repos, etc.

We found this approach to be more appropriate because in practice the source code directory (source_dir) should be immutable across builds. What changes are the files in the build directories (build_dir), so having parallel compilation jobs using a single source_dir workspace was expected.
But what deviates from this assumption is that CMakeUserPresets.json is actually being modified at build-time.

But I understand that Conan does it this way, and I understand that it is the only way because unfortunately, from what I have read, CMake does not allow you to indicate the path of the CMakeUserPresets.json file to use and then generate it in the root of the build_dir of the target in question.

Why removing the build subfolders in the first place, why not leaving them alone, they will typically be garbage collected later?

Good question :)
The target does conan install . to install and build the main project but at the end it does conan install --requires pkg/1.x. to copy other artifacts from this dependency and the latter is deleting the build folder so as not to have files from the main project build.
But we will have to see here, not to delete the build and create a sub-folder for the dependencies and not to use the main one to keep the CMakePreset path that was added to the root's CMakeUserPresets.json.

[memsharded]

But I understand that Conan does it this way, and I understand that it is the only way because unfortunately, from what I have read, CMake does not allow you to indicate the path of the CMakeUserPresets.json file to use and then generate it in the root of the build_dir of the target in question.

Yes, exactly, CMake mandates that the presets file is only besides the root CMakeLists.txt, it cannot be parameterized to be elsewhere.

The target does conan install . to install and build the main project but at the end it does conan install --requires pkg/1.x. to copy other artifacts from this dependency and the latter is deleting the build folder so as not to have files from the main project build.

With a deployer or something like that?

Yes, maybe everything, the buildDir, the deployer folder, etc can also be parameterized or use the build number

From the Conan side what could be done and I think it is feasible and might make sense is that for every conan install, it could check the paths included in the CMakeUserPresets.json, and if they are not existing, they could be removed. What do you think, would that alleviate the issue?

[amachado-pie]

From the Conan side what could be done and I think it is feasible and might make sense is that for every conan install, it could check the paths included in the CMakeUserPresets.json, and if they are not existing, they could be removed. What do you think, would that alleviate the issue?

May be an idea to explore, but how you know if that include is "managed" by Conan or not to delete it if the path doesn't exist?
Something like creating a CMakeConanPresets.json that is managed by conan?

Current: CMakeUserPresets.json:

{
    "version": 4,
    "vendor": {
        "conan": {}
    },
    "include": [
        "build_android_light/android-light/Release/generators/CMakePresets.json",
        "build_android_pro/android-pro/Release/generators/CMakePresets.json"
    ]
}

Proposal: CMakeUserPresets.json:

{
    "version": 4,
    "vendor": {
        "conan": {}
    },
    "include": [
        "CMakeConanPresets.json"
    ]
}

CMakeConanPresets.json

{
    "version": 4,
    "vendor": {
        "conan": {}
    },
    "include": [
        "build_android_light/android-light/Release/generators/CMakePresets.json",
        "build_android_pro/android-pro/Release/generators/CMakePresets.json"
    ]
}

Then for every conan install, it could check whether or not CMakeUserPresets.json already has CMakeConanPresets.json and add it if necessary, and then validate the "managed CMakeConanPresets.json, check the paths included in and if they are not existing, they could be removed.

[memsharded]

The CMakeUserPresets.json and the CMakePresets.json generated by Conan contain:

"vendor": {
        "conan": {}
    },

So we know it is managed by Conan, so we are free to modify, remove or whatever is necessary in that file. That is exactly the goal of that field, it shouldn't be necessary to redirect to other file.