-
Notifications
You must be signed in to change notification settings - Fork 26
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
.github: Create CI for live pipelines docker images (#282)
* .github: Create initial CI pipeline for live images * Dummy change to rebuild * .github: Add cleanup before pipeline images * .github: Rename noop job for consistency * Add fail-fast: false confyui still needs ton of disk * .github: Add some conditions to image build * docker/liveportrait: Print more friendly wget output * .github: Skip only the docker build step for base * .github: Add condition for noop image too * .github: Fix title * .github: Cleanup hosted runner deeper * docker/liveportrait: Dummy change on dockerfile to see if it builds * .github: Debugging base pipeline rebuild check * .github: Add proper permissions to jobs * docker: Try changing streamdiffusion/noop dockerfiles * .github: Skip base image cleanup step if no build * .github: Use custom plugin to detect changed files * .github: Increase fetch depth * .github: Add conditions for noop job as well * Revert "docker: Try changing streamdiffusion/noop dockerfiles" This reverts commit 603ae19. * .github: Fix debug steps * .github: Remove debug step. It works! * .github: Rename to base_dockerfile for clarity * runner/run: Allow run-lv2v to be used with noop image
- Loading branch information
Showing
6 changed files
with
285 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| name: 'Cleanup Runner' | ||
| description: 'Cleans up GitHub-hosted runner to free up disk space for large builds' | ||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Cleanup hosted runner | ||
| shell: bash | ||
| run: | | ||
| # Remove unnecessary packages | ||
| sudo apt purge -yqq dotnet-* mono-* llvm-* libllvm* powershell* openjdk-* \ | ||
| temurin-* mongodb-* firefox mysql-* \ | ||
| hhvm google-chrome-stable \ | ||
| libgl1-mesa-dri microsoft-edge-stable azure-cli || true | ||
| # Clean apt cache | ||
| sudo apt clean | ||
| sudo apt autoremove -y | ||
| # Remove large directories | ||
| sudo rm -rf /usr/share/dotnet /usr/local/lib/android | ||
| sudo rm -rf /usr/local/share/chromium /usr/local/share/chrome | ||
| sudo rm -rf /usr/local/.ghcup /usr/local/share/powershell | ||
| sudo rm -rf /opt/hostedtoolcache/* /usr/local/lib/node_modules | ||
| # Clean npm and yarn caches | ||
| npm cache clean --force || true | ||
| yarn cache clean || true | ||
| # Remove Docker images and build cache | ||
| docker system prune -af | ||
| # Show available space after cleanup | ||
| df -h |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,241 @@ | ||
| name: Build ai-runner live pipeline Docker images | ||
|
|
||
| on: | ||
| pull_request: | ||
| paths: | ||
| - "runner/docker/Dockerfile.live-*" | ||
| - "runner/app/**" | ||
| - "runner/images/**" | ||
| push: | ||
| branches: | ||
| - main | ||
| tags: | ||
| - '*' | ||
| paths: | ||
| - "runner/docker/Dockerfile.live-*" | ||
| - "runner/app/**" | ||
| - "runner/images/**" | ||
| workflow_dispatch: | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| build-common-base: | ||
| name: Build common live base image | ||
| permissions: | ||
| pull-requests: read | ||
| runs-on: ubuntu-20.04 | ||
| steps: | ||
| - name: Check out code | ||
| uses: actions/[email protected] | ||
| with: | ||
| fetch-depth: 50 | ||
| ref: ${{ github.event.pull_request.head.sha }} | ||
|
|
||
| - name: Get changed files | ||
| id: changed-files | ||
| uses: tj-actions/changed-files@v45 | ||
| with: | ||
| files: | | ||
| runner/docker/Dockerfile.live-base | ||
| - name: Check if build needed | ||
| id: check_build | ||
| if: | | ||
| github.event_name == 'workflow_dispatch' || | ||
| github.event_name == 'push' || | ||
| ( | ||
| github.event_name == 'pull_request' && | ||
| github.event.pull_request.head.repo.full_name == github.repository && | ||
| steps.changed-files.outputs.any_changed == 'true' | ||
| ) | ||
| run: echo "should_build=true" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: Cleanup hosted runner | ||
| uses: ./.github/actions/cleanup-runner | ||
| if: steps.check_build.outputs.should_build == 'true' | ||
|
|
||
| - name: Set up Docker Buildx | ||
| uses: docker/setup-buildx-action@v3 | ||
|
|
||
| - name: Login to DockerHub | ||
| uses: docker/login-action@v3 | ||
| with: | ||
| username: ${{ secrets.CI_DOCKERHUB_USERNAME }} | ||
| password: ${{ secrets.CI_DOCKERHUB_TOKEN }} | ||
|
|
||
| - name: Build and push live-base image | ||
| uses: docker/build-push-action@v5 | ||
| if: steps.check_build.outputs.should_build == 'true' | ||
| with: | ||
| context: "{{defaultContext}}:runner" | ||
| platforms: linux/amd64 | ||
| push: ${{ github.event_name != 'pull_request' }} | ||
| tags: livepeer/ai-runner:live-base | ||
| file: docker/Dockerfile.live-base | ||
| cache-from: type=registry,ref=livepeerci/build:cache | ||
| cache-to: type=registry,ref=livepeerci/build:cache,mode=max | ||
|
|
||
| build-pipeline-images: | ||
| name: Build pipeline images | ||
| needs: build-common-base | ||
| runs-on: ubuntu-20.04 | ||
| permissions: | ||
| pull-requests: read | ||
| strategy: | ||
| matrix: | ||
| pipeline: [streamdiffusion, comfyui, liveportrait] | ||
| fail-fast: false | ||
| steps: | ||
| - name: Check out code | ||
| uses: actions/[email protected] | ||
| with: | ||
| fetch-depth: 50 | ||
| ref: ${{ github.event.pull_request.head.sha }} | ||
|
|
||
| - name: Get changed files | ||
| id: changed-files | ||
| uses: tj-actions/changed-files@v45 | ||
| with: | ||
| files_yaml: | | ||
| base_dockerfile: | ||
| - runner/docker/Dockerfile.live-base-${{ matrix.pipeline }} | ||
| liveportrait: | ||
| - runner/images/** | ||
| - runner/requirements-liveportrait.txt | ||
| - name: Cleanup hosted runner | ||
| uses: ./.github/actions/cleanup-runner | ||
|
|
||
| - name: Set up Docker Buildx | ||
| uses: docker/setup-buildx-action@v3 | ||
|
|
||
| - name: Login to DockerHub | ||
| uses: docker/login-action@v3 | ||
| with: | ||
| username: ${{ secrets.CI_DOCKERHUB_USERNAME }} | ||
| password: ${{ secrets.CI_DOCKERHUB_TOKEN }} | ||
|
|
||
| - name: Build and push pipeline base image | ||
| uses: docker/build-push-action@v5 | ||
| if: | | ||
| github.event_name == 'workflow_dispatch' || | ||
| github.event_name == 'push' || | ||
| ( | ||
| github.event_name == 'pull_request' && | ||
| github.event.pull_request.head.repo.full_name == github.repository && | ||
| ( | ||
| steps.changed-files.outputs.base_dockerfile_any_changed == 'true' || | ||
| ( | ||
| matrix.pipeline == 'liveportrait' && | ||
| steps.changed-files.outputs.liveportrait_any_changed == 'true' | ||
| ) | ||
| ) | ||
| ) | ||
| with: | ||
| context: "{{defaultContext}}:runner" | ||
| platforms: linux/amd64 | ||
| push: ${{ github.event_name != 'pull_request' }} | ||
| tags: livepeer/ai-runner:live-base-${{ matrix.pipeline }} | ||
| file: docker/Dockerfile.live-base-${{ matrix.pipeline }} | ||
| build-args: | | ||
| PIPELINE=${{ matrix.pipeline }} | ||
| cache-from: type=registry,ref=livepeerci/build:cache | ||
| cache-to: type=registry,ref=livepeerci/build:cache,mode=max | ||
|
|
||
| - name: Extract metadata for app image | ||
| id: meta | ||
| uses: docker/metadata-action@v5 | ||
| with: | ||
| images: livepeer/ai-runner | ||
| tags: | | ||
| type=raw,value=live-app-${{ matrix.pipeline }} | ||
| type=sha,prefix=live-app-${{ matrix.pipeline }}- | ||
| type=ref,event=pr,prefix=live-app-${{ matrix.pipeline }}- | ||
| type=ref,event=tag,prefix=live-app-${{ matrix.pipeline }}- | ||
| type=raw,value=latest,enable={{is_default_branch}},prefix=live-app-${{ matrix.pipeline }}- | ||
| - name: Build and push pipeline app image | ||
| uses: docker/build-push-action@v5 | ||
| with: | ||
| context: "{{defaultContext}}:runner" | ||
| platforms: linux/amd64 | ||
| push: ${{ github.event_name != 'pull_request' }} | ||
| tags: ${{ steps.meta.outputs.tags }} | ||
| file: docker/Dockerfile.live-app__PIPELINE__ | ||
| build-args: | | ||
| PIPELINE=${{ matrix.pipeline }} | ||
| cache-from: type=registry,ref=livepeerci/build:cache | ||
| cache-to: type=registry,ref=livepeerci/build:cache,mode=max | ||
|
|
||
| build-noop: | ||
| name: Build pipeline image (noop) | ||
| needs: build-common-base | ||
| runs-on: ubuntu-20.04 | ||
| permissions: | ||
| pull-requests: read | ||
| steps: | ||
| - name: Check out code | ||
| uses: actions/[email protected] | ||
| with: | ||
| fetch-depth: 50 | ||
| ref: ${{ github.event.pull_request.head.sha }} | ||
|
|
||
| - name: Get changed files | ||
| id: changed-files | ||
| uses: tj-actions/changed-files@v45 | ||
| with: | ||
| files: | | ||
| runner/docker/Dockerfile.live-app-noop | ||
| runner/app/** | ||
| - name: Check if build needed | ||
| id: check_build | ||
| if: | | ||
| github.event_name == 'workflow_dispatch' || | ||
| github.event_name == 'push' || | ||
| ( | ||
| github.event_name == 'pull_request' && | ||
| github.event.pull_request.head.repo.full_name == github.repository && | ||
| steps.changed-files.outputs.any_changed == 'true' | ||
| ) | ||
| run: echo "should_build=true" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: Cleanup hosted runner | ||
| uses: ./.github/actions/cleanup-runner | ||
| if: steps.check_build.outputs.should_build == 'true' | ||
|
|
||
| - name: Set up Docker Buildx | ||
| uses: docker/setup-buildx-action@v3 | ||
|
|
||
| - name: Login to DockerHub | ||
| uses: docker/login-action@v3 | ||
| with: | ||
| username: ${{ secrets.CI_DOCKERHUB_USERNAME }} | ||
| password: ${{ secrets.CI_DOCKERHUB_TOKEN }} | ||
|
|
||
| - name: Extract metadata for Docker | ||
| id: meta | ||
| uses: docker/metadata-action@v5 | ||
| with: | ||
| images: livepeer/ai-runner | ||
| tags: | | ||
| type=raw,value=live-app-noop | ||
| type=sha,prefix=live-app-noop- | ||
| type=ref,event=pr,prefix=live-app-noop- | ||
| type=ref,event=tag,prefix=live-app-noop- | ||
| type=raw,value=latest,enable={{is_default_branch}},prefix=live-app-noop- | ||
| - name: Build and push noop image | ||
| uses: docker/build-push-action@v5 | ||
| if: steps.check_build.outputs.should_build == 'true' | ||
| with: | ||
| context: "{{defaultContext}}:runner" | ||
| platforms: linux/amd64 | ||
| push: ${{ github.event_name != 'pull_request' }} | ||
| tags: ${{ steps.meta.outputs.tags }} | ||
| file: docker/Dockerfile.live-app-noop | ||
| cache-from: type=registry,ref=livepeerci/build:cache | ||
| cache-to: type=registry,ref=livepeerci/build:cache,mode=max |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters