Which CI environment integration should we cover next?

[gerhard]

We started writing a new doc page - From local dev to CI environment #1678 - and would like to know which CI/CD platform you would find most useful for us to document next.

Since everyone that is using Dagger today already has a GitHub account (one of the early access requirements), starting with GitHub Actions seemed like a most reasonable first step.

Drop a comment below with the CI/CD platform that you use. We intend to document the one with the most mentions next.

[gerhard]

CircleCI: https://github.com/thechangelog/changelog.com/blob/a8f472230fe8cfc2e599523b4a6340f384607d16/.circleci/config.yml

[maert]

Dagger orb (CircleCI's lego brick) here: https://github.com/logicalhq/dagger-orb
Not yet listed as a public orb in the Orb Registry: https://circleci.com/developer/orbs

[jpadams]

@gcoguiec It looks like your dagger-orb repo isn't up anymore. Would you be willing to donate the code to Dagger to maintain?

[jpadams]

Have a CircleCI example going up in PR right now. With more complex example (caching) to follow. #2842

[helderco]

I use GitLab. I have private repos on GitLab, with dagger configs. Didn't have time to make dagger work with it, but I'm able to test it.

[maudeFW]

@nicolas-goudry Did you try it already? If yes, what have been your experiences?
I'm evaluating different CI/CD Frameworks to run on GitLab CI Kubernetes Executor so input would be helpful, thanks.

[nicolas-goudry]

@maudeFW Sorry for replying so late, I didn’t have time to look into it yet…

[jpadams]

GitLab example getting better and better here: https://docs.dagger.io/1201/ci-environment

[Aboitier]

Any updated doc about integrating/migrating from gitlab-CI to dagger ?

[marcosnils]

hey @Aboitier! Dagger won't replace your CI runner, but we have several people in our Discord using both gitlab hosted and self-hosted runners with success. If you have some questions about it, would love to chat about it more in our Discord server 🙏

[TomChv]

At Epitech, there was courses on Jenkins, I'm not sure it's relevant but some infrastructures are still using it so maybe we can do something with Dagger

[solarsnake]

I'll third for jenkins

[kenyee]

4th for jenkins

[itzikiusa]

Me to

[bobhenkel]

Me too

[jpadams]

Jenkins example and video here: https://docs.dagger.io/1201/ci-environment

[maert]

GitHub knows which are the big CI tools on their platform (or competitive) and has transition guides from each one to GitHub Actions. So CircleCI, TravisCI, Jenkins, and GitLab CI/CD all make sense. Azure Pipelines (from Azure DevOps) could also make sense. Long term vision of MSFT seems to be GitHub & Actions, but a long tail of ADO users on prem and in Azure.

https://docs.github.com/en/actions/migrating-to-github-actions

Also makes sense to support the cloud's tools:

• AWS - Code Pipeline/Code Build
• Azure - Azure Pipelines
• GCP - Cloud Build
• Alibaba Cloud - Cloud DevOps Pipeline (Flow)
• Tencent Cloud - CODING
• Digital Ocean - doesn't seem to have own tool. Lots of articles from folks using above tools to build for and deploy to DO.

[DavidNavalho]

Chiming in for ADO as well, as I've seen more and more clients using those.

[tadavid-cae]

Please make Azure Devops a reality!!!

[Pre-Z]

This would be a game changer for Azure DevOps Pipelines.

[tadavid-cae]

@gerhard can you tell us the probability that you will integrate dagger with azure devops? :)

[jpadams]

@tadavid-cae @Pre-Z What would ADO support look like?

Would there need to be an extension or just an Azure Pipelines YAML example?

[gerhard]

Tekton: #1655

[maert]

And by extension, Jenkins X that uses Tekton inside.

[jpadams]

We have Tekton here :) https://docs.dagger.io/1201/ci-environment

[maert]

TeamCity by JetBrains. Could create a Kotlin script wrapper for Dagger.

[darkn3rd]

This one is important for my organization. Also, TeamCity supports inheritance, while Jenkins and others do not.

Example: root project (with secrets to access git and triggers for pull request) has a child of build artifacts, which builds binaries on various platforms, and this has test children pipelines for integration tests and unit tests that consume the artifacts, so root > build > {unit_tests, integration_tests}.

, so for example, integration test pipeline and unit test pipeline could be dependent on a parent build pipeline that is executed prior and stores executable artifacts, and this in term is inherited from a parent that fetches the code and has secrets used to access the gitrepo, as well as triggers for pull requests, and so on.

[jpadams]

@darkn3rd if you have something that could help start a simple Team City example (or links), that would be great. Also PRs accepted ;)

Inspiration: https://docs.dagger.io/1201/ci-environment

[eddumelendez]

ArgoCD

[marcosnils]

Argo is mostly a CD platform, isn't it? It already relies on pre-existing CI solutions like GA, Circle, Tekton, etc for the CI part. Not sure what could be done in Dagger as an Argo direct integration 🤔

[demasir]

@marcosnils I'm not sure if that's what @eddumelendez meant, but we use Argo Workflows to create our CI pipelines. It could be useful to have Dagger integrated with it so we could test them locally. Argo Workflows + Argo Events + Argo CD is basically a CI/CD platform by itself. For reference: https://argoproj.github.io/argo-workflows/

[marcosnils]

got it, thx! Wasn't aware about workflows. It looks like a potential opportunity for dagger as well.

[hihellobolke]

Jenkins - most enterprises still use it.. multi man-decades of tooling are around it.

[jpadams]

Jenkins example and video here: https://docs.dagger.io/1201/ci-environment

[eddumelendez]

Concoruse CI

[CorentinDy]

Concourse CI

[radiosilence]

Concourse would be amazing, seeing as it's modern, built for CD, able to handle extreme complexity and is not tied to a provider.

[manjunathb4461]

Are there any plans to implement the same for bitbucket pipelines?

[dancordi]

same to me

[manjunathb4461]

@gerhard I created a simple dagger pipe for bitbucket pipelines just like dagger-for-github
But ended up with following error

{"level":"error","error":"exit status 125","output":"docker: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed.\nSee 'docker run --help'.\n","time":"2022-04-03T07:20:49Z","time":"2022-04-03T07:20:49Z","caller":"/home/runner/work/dagger/dagger/util/buildkitd/buildkitd.go:210","message":"unable to start buildkitd"}

since bitbucket pipelines doesn't allow container(pipe in this case) to be run in --privileged mode, which is a requirement for buildkit. Github actions also doesn't allow privileged mode by default, so wondering how it was solved for github actions. I would be happy to receive your feedback regarding the same.

[joachim-isaksson]

Ended up with the same error that I couldn't get around. I may be a bit off, but I think it may be possible to get it running once this is fixed.

https://jira.atlassian.com/browse/BCLOUD-17590

[dnalbach]

Bitbucket Pipeline docs say they do not allow privileged mode.
https://support.atlassian.com/bitbucket-cloud/docs/run-docker-commands-in-bitbucket-pipelines/

A potential workaround would be to use Bitbucket Runners and self-host the build machine, at which point you have total control over it, though you do lose the ephemeral aspect. I'm going to try that for some of our projects.

[jpadams]

Seems to still not be possible with Atlassian-hosted runners at the moment, even though BuildKit support is available in preview, since --privileged=true is still not supported
https://jira.atlassian.com/browse/BCLOUD-13420
and we currently run BuildKit as --privileged in Dagger.
https://github.com/dagger/dagger/blob/main/util/buildkitd/buildkitd.go#L238

Seems like self-hosted runners with dind or a DOCKER_HOST over ssh would be needed.

image: atlassian/default-image:3

pipelines:
  default:
     - step:
         name: 'Install and run Dagger'
         script:
           - cd /usr/local
           - wget -O - https://dl.dagger.io/dagger/install.sh | sh
           - cd -
           - dagger version
           - dagger project update
           - DOCKER_BUILDKIT=1 dagger do hello --log-format plain
         services:
          - docker

[marcosnils]

My personal experience tells me to take a look at Jenkins / Jenkins X (and Tekton in consequence). Most of the times I've seen Jenkins pipeline using docker plugins, they were super difficult to follow and configure. I have a feeling that there's a big opportunity there to showcase a simpler and more efficient solution that integrates with those tools like Dagger.

[jpadams]

Jenkins and Tekton here :)
https://docs.dagger.io/1201/ci-environment

[ejunker]

AWS CodeBuild

[Jojo134]

A basic AWS Codebuild was possible using a build task configured with the aws/codebuild/standard:5.0 image and the following Buildspec, pwd, ls are still in there to see the folders/structure in the logs:

version: 0.2

phases:
  pre_build:
    commands:
      - pwd
      - curl -L https://dl.dagger.io/dagger/install.sh | sh
      - ls
      - ./bin/dagger version
  build:
    commands:
       - ./bin/dagger project init
       - ./bin/dagger project update 
       - ./bin/dagger do build

[KKcorps]

Jenkins

[cuken]

ADOC

[dieend]

Buildkite

[junkpiano]

Bitrise

[sierisimo]

Yes please, their CLI is cool but the integration would be really helpful

[dnalbach]

Bitbucket Pipelines - I have lots of clients using this

[jpadams]

#1677 (reply in thread)

[dnalbach]

AWS CodeBuild/CodePipeline

[guoyu07]

Drone

[mudassir-ngineous]

Circleci

[jpadams]

#1677 (reply in thread)

[borisovano]

Argo stack (workflows, events and pipelines)

[alberto-cerato]

Zuul CI
https://zuul-ci.org/

[talk2drys]

Bitbucket Pipeline

[jpadams]

#1677 (reply in thread)

[Pandry]

woodpecker

[scheung38]

Devtron CI/CD

[darkn3rd]

I am interested in Spinnaker CD pipelines, which are reflected in JSON, and can be downloaded. They can also be uploaded with changes to create new pipelines. There's a CLI tool that I use to manage the pipelines https://spinnaker.io/docs/guides/spin/pipeline/

[jpadams]

I am interested in Spinnaker CD pipelines, which are reflected in JSON, and can be downloaded. They can also be uploaded with changes to create new pipelines. There's a CLI tool that I use to manage the pipelines https://spinnaker.io/docs/guides/spin/pipeline/

Sounds like an ideal use for Dagger. You could consume/generate the Spinnaker JSON and invoke the CLI. Let me know if you'd like to set up a POC/pilot to get that going :)

[dezren39]

nomad jobs please :-)

[Harrolee]

GCP Cloud Build 😁

[pfcarrier]

I second that, having one more example with GCP Cloud Build and AWS Code Build would complete the collection of all usual CI plus all the hyperscaller's. one. This would create a nice "Wow" factor for any MSP that are revewing dagger since it would demonstrate all the CI of all their clients can leverage dagger and further demonstrate the joy dagger brings to the table.

[cgetzen]

I'd love to see CodeFresh support! I'm also curious if there's any documentation around how to contribute CI adapters ourselves.

[edromero]

It's not hard getting it working in Codefresh but it requires granting root privileges to the job. Currently only the hybrid option where you run the Codefresh workload on your own k8s cluster will allow this.