-
Notifications
You must be signed in to change notification settings - Fork 2
ci(e2e): add DVP-over-DVP matrix testing with parallel execution #1577
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
asyachmenevflant
wants to merge
163
commits into
main
Choose a base branch
from
feat/ci-e2e-matrix
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 74 commits
Commits
Show all changes
163 commits
Select commit
Hold shift + click to select a range
df15f9f
ci(e2e): add DVP-over-DVP matrix testing with parallel execution
asyachmenevflant 7c7689c
ci: drop artifacts/binaries from VCS; update .gitignore
asyachmenevflant 395d32e
ci(e2e): restore workflow, Taskfiles, charts, scripts and docs
asyachmenevflant 09542e3
dvcr: make templates nil-safe; remove temporary root values.yaml
asyachmenevflant 3e056d0
ci: format YAML to satisfy prettier; drop temporary values.yaml
asyachmenevflant 8d97988
revert(templates/dvcr): drop E2E-related edits; keep tests via Module…
asyachmenevflant 46d40ec
ci(e2e): remove secrets usage in if; check webhook in step script
asyachmenevflant 8dc2a87
ci(e2e): write kubeconfig via printf to avoid heredoc EOF issues
asyachmenevflant 873905d
ci(e2e): resolve merge with main; keep full E2E workflow
asyachmenevflant bfa9e9e
ci(e2e): fix duplicate workflow_dispatch; prettier passes
asyachmenevflant 24a2a0e
ci(e2e): use env.E2E_K8S_URL in kubeconfig steps; drop report kubecon…
asyachmenevflant 12a1a5c
ci(e2e): build e2e-runner wrapper via Taskfile (tests/e2e task run:ci)
asyachmenevflant 739c81e
ci(e2e): run tests via tests/e2e Taskfile (run:ci) with TIMEOUT
asyachmenevflant 39d19e5
ci(e2e): pass TEST_TIMEOUT and E2E_PREFIX to tests/e2e Taskfile run
asyachmenevflant bfe5d25
ci(e2e): pass TIMEOUT (not TEST_TIMEOUT) to tests/e2e Taskfile
asyachmenevflant 007c90e
ci(e2e): write kubeconfig token without base64 decoding to avoid UTF-…
asyachmenevflant 71d6e8c
ci(e2e): quote token in kubeconfig; set LABELS/FOCUS_JSON defaults fo…
asyachmenevflant 348c7cd
ci(e2e): read JWT from secrets.K8S_CLUSTER_SECRET index 8 (kubeconfig)
asyachmenevflant 1591401
ci(e2e): use azure/k8s-set-context@v4 for kubeconfig setup
asyachmenevflant bd2543e
ci(e2e): remove KUBECONFIG env var to use default ~/.kube/config
asyachmenevflant c7f3c59
ci(e2e): implement complete artifact collection and Loop reporting pi…
asyachmenevflant 5a3733a
feat(ci): replace simple E2E tests with full DVP-over-DVP bootstrap
asyachmenevflant 66f2bad
fix(ci): enable cancel-in-progress for concurrency
asyachmenevflant 7a3aa6f
ci(e2e): restore ci/dvp-e2e/Taskfile.vm.yaml (VM helper tasks)
asyachmenevflant ca1d101
ci(e2e): use existing kubeconfig from GA; skip ingress/RBAC during en…
asyachmenevflant f4baf76
ci(e2e): simplify kubeconfig provisioning; use E2E_NESTED_SA_SECRET o…
asyachmenevflant c2a95ba
ci(e2e): tidy headings and finalize simplified flow (no validate, doc…
asyachmenevflant 07460c0
ci(e2e): pass registryDockerCfg via YAML overlay and use it in nested…
asyachmenevflant 02afa95
ci(e2e): ensure REGISTRY_DOCKER_CFG overlay is present and non-empty
asyachmenevflant 20a104e
ci(e2e): merge REGISTRY_DOCKER_CFG into base values via yq; pass as V…
asyachmenevflant 06066eb
ci(e2e): inject REGISTRY_DOCKER_CFG into generated values for InitCon…
asyachmenevflant 8b5a2ae
ci(e2e): map SCs per profile (sds→linstor-thin-r2; cephrbd→ceph) for …
asyachmenevflant f282f21
ci(e2e): fix registry auth and SC mapping in matrix workflow
asyachmenevflant 5de0213
ci(e2e): configure registry auth via REGISTRY_DOCKER_CFG in matrix
asyachmenevflant fca74e0
ci(e2e): wait for base VirtualImage Ready to prevent bootstrap hangs
asyachmenevflant 3ee8367
ci(e2e): propagate TARGET_STORAGE_CLASS to infra; wait VirtualImage R…
asyachmenevflant ebf44aa
revert: rollback recent infra SC and bootstrap waits to previous state
asyachmenevflant f83ee78
ci(e2e): ensure default StorageClass per profile before E2E run
asyachmenevflant de7a65b
ci(e2e): set infra storageClass from TARGET_STORAGE_CLASS to avoid pa…
asyachmenevflant 581f6f6
ci(e2e): remove parent default SC step; rely on explicit infra storag…
asyachmenevflant 27f61a4
ci(e2e): cleanup uses privileged SA context; remove kubeconfig copy a…
asyachmenevflant 2d7d250
ci(e2e): correct SDS image/snapshot SC mapping (IMG_SC=linstor-thin-r…
asyachmenevflant e95f9e2
ci(e2e): add -vv --github-output to ginkgo; use dvp-e2e- prefix for R…
asyachmenevflant 89648d5
ci(e2e): fix doubled namespace issue; cleanup filters dvp-e2e-; rever…
asyachmenevflant 78265c6
ci(e2e): prune debug/local tasks; keep only CI-needed targets; purge …
asyachmenevflant 29ceeec
ci(e2e): add JUnit failure summary step and cleanup filter dvp-e2e-ni…
asyachmenevflant 37e7038
ci(e2e): fix YAML inline python indentation for JUnit summary step
asyachmenevflant 10c35e2
ci(e2e): switch JUnit summary to python -c to fix YAML parser error
asyachmenevflant 18694a3
ci(e2e): switch to 'go tool ginkgo' (was go run) for test runner
asyachmenevflant 18aec45
ci(e2e): avoid YAML parse issues by removing emojis from echo lines
asyachmenevflant 417d85d
ci(e2e): wait for validating-webhook-handler and retry ModuleConfig a…
asyachmenevflant a1ecbb3
ci(e2e): set defaultClusterStorageClass in nested before running tests
asyachmenevflant ef22b40
ci(e2e): align SDS IMAGE_STORAGE_CLASS to linstor-thin-r2; use ginkgo…
asyachmenevflant 790a2db
ci(e2e): align with reference - ModuleConfig version 1, SDS StorageCl…
asyachmenevflant 071edec
ci(e2e): fix kubeconfig logic - remove SKIP_INGRESS_CHECK, simplify S…
asyachmenevflant 66c3929
ci(e2e): ensure tools install on cache-hit; bump cache key to v3
asyachmenevflant 1fcdf72
ci(e2e): force kubectl into ~/.local/bin, ensure d8 in PATH; add deta…
asyachmenevflant 1e95914
ci(e2e): align nested API warm-up with reference (20s readyz, 10x nod…
asyachmenevflant 1118285
ci(e2e): align Ceph SC to ceph-pool-r2-csi-rbd-immediate and provisio…
asyachmenevflant c159079
refactor(ci): remove unused setup-e2e-tools action and improve E2E wo…
b130f45
fix(ci): remove extra profiles from profiles.json
2eea9c4
refactor(ci): simplify cleanup and fix env variable issues
4ab5c67
fix(ci): improve profile config parsing to avoid env errors
7c76c8f
fix(ci): use correct ServiceAccount and improve cleanup error handling
057d72f
ci(e2e): build kubeconfig from E2E_NESTED_SA_SECRET without TOK var; …
7b3c1ab
ci(e2e): call correct cleanup task (cleanup:cleanup:namespaces)
8a8b10b
ci(e2e): drop NFS server from infra and tasks; gate NFS config by flag
e984693
ci(e2e): docker login dev-registry before pulling install image
e96aada
ci(e2e): use REGISTRY_DOCKER_CFG for registry auth; drop explicit doc…
854186c
ci(e2e): use deckhouse/modules-actions/setup for registry auth only
4782c25
ci(e2e): auth via modules-actions; add GHCR login; setup-task with token
90d4d4e
ci(e2e): add explicit registry pull check before bootstrap
54d79dc
ci(e2e): replace Taskfile with minimal GH-oriented version; drop unus…
c93dc39
ci(e2e): fix cleanup task; docker login to DEV_REGISTRY; setup-task i…
5670ca4
ci(e2e): remove GHCR login, move registry verification after docker l…
664ba44
ci(e2e): use direct docker login instead of docker/login-action
53fe4d3
ci(e2e): use CE installer image instead of dev-registry for verification
f9c6d01
ci(e2e): use CE installer image in dhctl-bootstrap task
ec69256
ci(e2e): fix cleanup namespaces array init and reorder workflow steps…
9fbe00a
ci(e2e): fix workflow architecture - add proper SSH key generation an…
9ebf9a1
ci(e2e): rely on render-infra ssh-gen dependency; remove explicit ssh…
80468bd
ci(e2e): run values prepare before infra; remove intermediate base va…
1854ba5
fix(ci): align Taskfile dependencies with backup branch logic
4568632
fix(ci): add sshPublicKey to cluster-config generation
059c944
fix(ci): ensure deckhouse.registryDockerCfg is always a string in clu…
3538e38
fix(ci): remove quote from registryDockerCfg in cluster-config template
a687aa9
fix(ci): remove registryDockerCfg field from cluster-config template
b5fb01e
fix(ci): restore registryDockerCfg config and fix cleanup namespaces
0b41113
fix(ci): use VALUES_TEMPLATE_FILE instead of explicit VALUES_FILE in …
a83ac96
fix(ci): use run:values:prepare to create proper values.yaml with dyn…
f68f775
fix(ci): use VALUES_TEMPLATE_FILE consistently and remove undeploy-fi…
0314920
debug(ci): add debugging info for REGISTRY_DOCKER_CFG secret
f0d0baf
fix(ci): properly pass REGISTRY_DOCKER_CFG secret to workflow
2f57b48
fix(ci): pass VALUES_FILE to dhctl-bootstrap task
5466145
fix(ci): use dynamic namespace consistently across all steps
d655d1f
cleanup: remove accidentally added submodules and docs
bd4c386
fix(ci): synchronize registry for CE and fix submodule error
1c00bcd
fix(ci): remove registryDockerCfg for CE registry
896ccb1
fix(ci): correct registry configuration for OSS cluster with CE dhctl
87080e8
fix(ci): disable upmeter module and fix virtualization tag
7422358
fix(ci): remove duplicate upmeter ModuleConfig
380542e
fix(ci): consolidate disabled modules and remove submodule
7b23775
fix(ci): completely remove virtualization-rnd directory
d4ccaa1
fix(ci): disable checkout caching in GitHub Actions
39dc7f3
fix(ci): skip checksum verification in Deckhouse CLI installation
5f49224
fix(ci): restore disabled-modules.yaml and fix duplication
9398651
fix(ci): restore configuration to match reference version
6f184bf
fix(ci): switch from CE to OSS registry with new credentials
d23dc1b
fix(ci): restore token creation logic for nested kubeconfig
d108102
feat(ci): add SDS module configuration for nested cluster
40bc0d5
feat(ci): add disk debugging and Ceph cluster configuration
dc5d511
refactor(ci): remove old Ceph bootstrap approach
bf39163
fix(ci): optimize Git checkout to resolve cache issues
16ddbef
fix(ci): install d8 via direct tarball to avoid checksum mismatch
57857fe
fix(ci): install d8 via installer with checksum disabled
1ab3a4d
feat(ci): conditionally render Ceph/SDS by storageProfile
018310a
refactor(ci): move storage CRs post-bootstrap and remove empty templates
3b86437
fix(ci): move disk debug before storage apply; add retries and --vali…
bf9d9fa
fix(ci): move disk debug step before storage configuration
b080244
fix(ci): use d8 v ssh for reliable VM disk debugging with retries
775e7aa
fix(ci): correct Ceph deviceFilter and add proper operator readiness …
9239193
fix(ci): make GA YAML valid; move operator-ceph ModuleConfig to file\…
42783ad
chore(ci): include Loop report scripts; add run link in profile cell\…
5685940
chore(ci): pass GitHub run URL to Loop matrix summary and link profil…
7aa9af0
fix(storage): enable SDS modules and wait for CRDs before applying re…
3fa0a23
fix(ci): resolve YAML syntax error in workflow SDS module config\n\n-…
54bfd79
fix(storage): use kubectl patch mc global for default StorageClass\n\…
b72888b
fix(sds): add retry logic and remove version from ModuleConfig\n\n- A…
7a02184
feat(ci): improve SDS CRD waiting and Ceph operator readiness checks\…
c2d36c0
fix(storage): correct disk devices and simplify SDS logic
04c9d13
fix(scripts): fix jq syntax error in get_profile_config.sh
012f203
fix(taskfile): fix heredoc syntax in infra:attach-worker-disks task
5db0383
fix(taskfile): avoid 'workers: unbound variable' by safe array build
57782b6
fix(taskfile): add quotes around SDS_SC_NAME in storageclass check
47d06b7
fix(ci): enable JUnit XML generation and reporting in E2E matrix work…
2ab725a
fix(taskfile): correct worker VM selector in infra:attach-worker-disks
9d6bacb
fix: apply critical storage and disk attachment fixes
ec262ad
hotfix: fix Taskfile YAML syntax and VirtualDisk API structure
da72af0
feat: move disk attachment before bootstrap and use base SC
d0730a0
fix: use VM name selector instead of node-group label for worker dete…
a9a4403
fix: move disk attachment after bootstrap to ensure VMs exist
8ed622e
fix(ci): resolve SDS and Ceph storage configuration issues
2fbd995
fix(ci): add SCSI bus rescan to activate hotplugged storage disks
9a1b99c
fix(ci): add sudo to SCSI rescan commands to fix permission denied er…
0e05f87
feat(ci): implement blockDeviceRefs approach for VM disk attachment
17a74a6
fix(ci): improve blockDeviceRefs task reliability
58f8e2d
fix(ci): use correct VM selector for worker nodes
a2fc6e8
fix(ci): add debug info and validation for blockDeviceRefs patch
a8cbe11
fix(ci): use inline patch instead of file-based patch for blockDevice…
70280b3
fix(ci): restore correct namespace naming for cleanup
a964de5
fix(ci): add VM startup verification after blockDeviceRefs modification
0290deb
revert(ci): switch back to hotplug for disk attachment
c3b7089
fix(ci): fix E2E test issues with bootstrap and disk attachment
8860294
fix(ci): remove forbidden dataDisk property from DVPClusterConfiguration
a8528df
fix: remove nodeGroups from DVPClusterConfiguration
0941430
docs: add comment to cluster-config template
8ce81c1
fix(ci): add push trigger to E2E Matrix Tests workflow
b1278d3
fix(ci): limit push trigger to feat/ci-e2e-matrix branch only
1cce800
refactor(ci): simplify storage configuration logic
eac5e61
fix(ci): update storage configuration to use Ceph as default
7efb344
debug(ci): improve disk attachment debugging and reduce retry time
39df983
debug(ci): improve PVC waiting logic and add debug environment variable
85b750a
debug(ci): optimize logging and add fallback for disk attachment
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| name: Setup E2E Tools | ||
| description: Install kubectl, helm, d8, task, yq with caching | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Cache binaries | ||
| uses: actions/cache@v4 | ||
| id: cache | ||
| with: | ||
| path: ~/.local/bin | ||
| key: e2e-tools-${{ runner.os }}-v3 | ||
|
|
||
| - name: Ensure tools installed (install if missing or cache empty) | ||
| shell: bash | ||
| run: | | ||
asyachmenevflant marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| set -euo pipefail | ||
| mkdir -p ~/.local/bin | ||
|
|
||
| need_install=false | ||
| for bin in helm yq task; do | ||
| if ! command -v "$bin" >/dev/null 2>&1; then | ||
| echo "missing: $bin"; need_install=true | ||
| fi | ||
| done | ||
| # Always (re)install kubectl into ~/.local/bin to shadow system kubectl | ||
| need_install=true | ||
| if [ "${need_install}" != "true" ] && [ "${{ steps.cache.outputs.cache-hit }}" = "true" ]; then | ||
| echo "All tools present from cache, skipping install." | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Install system dependencies | ||
| sudo apt-get update | ||
| sudo apt-get install -y jq apache2-utils curl bash ca-certificates | ||
|
|
||
| # Install kubectl (always to ~/.local/bin) | ||
| KUBECTL_VERSION=$(curl -Ls https://dl.k8s.io/release/stable.txt) | ||
| curl -fsSL -o kubectl "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" | ||
| chmod +x kubectl && mv kubectl ~/.local/bin/kubectl | ||
|
|
||
| # Install helm | ||
| curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash | ||
| if [ -x /usr/local/bin/helm ]; then mv /usr/local/bin/helm ~/.local/bin/; fi | ||
|
|
||
| # Install d8 | ||
| curl -fsSL -o d8-install.sh https://raw.githubusercontent.com/deckhouse/deckhouse-cli/main/d8-install.sh | ||
| bash d8-install.sh || true | ||
| if [ -x /usr/local/bin/d8 ]; then cp -f /usr/local/bin/d8 ~/.local/bin/d8; fi | ||
|
|
||
| # Install yq | ||
| curl -L -o ~/.local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.44.1/yq_linux_amd64 | ||
| chmod +x ~/.local/bin/yq | ||
|
|
||
| # Install task | ||
| sh -c "$(curl -fsSL https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin | ||
|
|
||
| # Cleanup | ||
| rm -f d8-install.sh | ||
|
|
||
| - name: Add to PATH | ||
| shell: bash | ||
| run: | | ||
| echo "$HOME/.local/bin" >> $GITHUB_PATH | ||
| echo "/usr/local/bin" >> $GITHUB_PATH | ||
|
|
||
| - name: Verify installation | ||
| shell: bash | ||
| run: | | ||
| echo "✅ Installed tools:" | ||
| which kubectl || true | ||
| kubectl version --client=true --output=yaml || kubectl version || true | ||
| which helm || true | ||
| helm version --short || true | ||
| which d8 || true | ||
| d8 version || true | ||
| which yq || true | ||
| yq --version || true | ||
| which task || true | ||
| task --version || true | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need to create your own action to setup kubectl, helm and so on. It already exists. Find it, please