Skip to content

Commit

Permalink
Introducing new go module for API, remove g/g dependency (#988)
Browse files Browse the repository at this point in the history
* removed dependency on vgopath and k8s.io/apiserver
* removed ginkgo unit test and added golang native unit test for secret reconciler
* removed g/g dependency
* added generated client, moved code-gen scripts to api module, updated k8s dependencies
* fixed unit tests and created a new Makefile target for unit tests in api go module
* fixed issues reported by gosec
* fixed ci check script
* added clean-generated-code script
* Fixed IT tests and modified .ci/check, removed GOTESTFMT tool
* adapted api package change
* removed check-apidiff and check-generate scripts - moved to api/hack and fixed skaffold, no need to have gcloud auth login
* removed config, moved samples, generating pki resources, copying crds for helm, corrected helm charts, created client as go module
* fixed crd paths to refer to api/core/crds in IT tests
* added go.work
* adopted go version check for multi-module go project
* removed -mod=mod as it does not work with go workspaces
* changed skaffold, now using latest version
* e2e test script now call prepare-local-deploy
* added helm chart usage doc, added new make target, corrected paths of pki resources and fixed skaffold debug profile config
* added git_tags for client and api go modules ensuring proper tagging upon release
* added allow dependency-type: all in dependabot, updated docs with links
* using directories to specify multiple manifests for dependabot
* upgraded helm to v3.16.4
* renamed chart template files and added NOTES.txt
* added icon url to chart.yaml
* now using make --directory to run API make targets from CI check script
* added pipeline definition to push helm charts as OCI artifact
* configured publish helmcharts for jobs
* renamed samples to examples, refactored, added client example
* removed go mod files from examples/client
* added etcd-druid client dependency to parent go.mod
* regenerated code with license header without YEAR for generated code
* added new lines to helm template yaml files
-----------
Co-authored-by: Saketh Kalaga <[email protected]>
Co-authored-by: Sonu Kumar Singh <[email protected]>
  • Loading branch information
unmarshall authored Feb 26, 2025
1 parent 0c24c05 commit a33217c
Show file tree
Hide file tree
Showing 275 changed files with 7,553 additions and 4,697 deletions.
10 changes: 7 additions & 3 deletions .ci/check
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,15 @@ cd /go/src/github.com/gardener/etcd-druid
echo "> Running check..."
make check

echo "> Running check-generate..."
make check-generate

# Run Static Application Security Testing (SAST) using gosec
echo "> Running SAST checks..."
make sast-report

echo "Running API Make targets..."

make --directory=api \
check \
tidy \
check-generate

echo -e "\n> All checks successful"
27 changes: 27 additions & 0 deletions .ci/pipeline_definitions
Original file line number Diff line number Diff line change
@@ -1,4 +1,16 @@
etcd-druid:
templates:
helmcharts:
- &etcd-druid-chart
name: etcd-druid
dir: charts
registry: europe-docker.pkg.dev/gardener-project/snapshots/charts/gardener/etcd-druid
mappings:
- ref: ocm-resource:etcd-druid.repository
attribute: image.repository
- ref: ocm-resource:etcd-druid.tag
attribute: image.tag

base_definition:
repo:
source_labels:
Expand Down Expand Up @@ -58,9 +70,19 @@ etcd-druid:
component_descriptor:
ocm_repository_mappings:
- repository: europe-docker.pkg.dev/gardener-project/releases
options:
public_build_logs: true
publish:
helmcharts:
- *etcd-druid-chart
pull-request:
traits:
pull-request: ~
options:
public_build_logs: true
publish:
helmcharts:
- *etcd-druid-chart
release:
traits:
version:
Expand All @@ -72,6 +94,9 @@ etcd-druid:
etcd-druid:
image: europe-docker.pkg.dev/gardener-project/releases/gardener/etcd-druid
tag_as_latest: True
helmcharts:
- <<: *etcd-druid-chart
registry: europe-docker.pkg.dev/gardener-project/releases/charts/gardener/etcd-druid
release:
nextversion: 'bump_minor'
assets:
Expand All @@ -86,6 +111,8 @@ etcd-druid:
see: https://github.com/securego/gosec
git_tags:
- ref_template: 'refs/tags/{VERSION}'
- ref_template: 'refs/tags/api/{VERSION}'
- ref_template: 'refs/tags/client/{VERSION}'
release_callback: .ci/bump_tag
slack:
default_channel: 'internal_scp_workspace'
Expand Down
5 changes: 4 additions & 1 deletion .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,10 @@
version: 2
updates:
- package-ecosystem: gomod
directory: "/"
directories:
- "/"
- "/api"
- "/client"
schedule:
interval: monthly
open-pull-requests-limit: 5
Expand Down
29 changes: 24 additions & 5 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
/args
/bin
/hack/tools/bin
/hack/kind/*
/.kube-secrets
/tmp/*
/dev
Expand All @@ -14,9 +12,6 @@ default.bkp*
/cover.out
*coverprofile.out
*.html
.vscode
.idea
.DS_Store
.ignore
*~

Expand All @@ -41,5 +36,29 @@ hack/e2e-test/infrastructure/kind/kubeconfig
.cache_ggshield
.gitguardian.yaml

# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib

# gosec
gosec-report.sarif

# env file
.env

# code editor ignores
.vscode
.idea
.DS_Store

# hack tools binaries and generated configurations
/hack/tools/bin/*
/hack/kind/*
api/hack/tools/bin/*

# generated/copied chart resources
charts/pki-resources/*
charts/crds/*
104 changes: 45 additions & 59 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

REPO_ROOT := $(shell dirname "$(realpath $(lastword $(MAKEFILE_LIST)))")
HACK_DIR := $(REPO_ROOT)/hack
API_HACK_DIR := $(REPO_ROOT)/api/hack
VERSION := $(shell $(HACK_DIR)/get-version.sh)
GIT_SHA := $(shell git rev-parse --short HEAD || echo "GitNotFound")
REGISTRY_ROOT := europe-docker.pkg.dev/gardener-project
Expand All @@ -24,39 +25,16 @@ KUBECONFIG_PATH := $(HACK_DIR)/kind/kubeconfig
TOOLS_DIR := $(HACK_DIR)/tools
include $(HACK_DIR)/tools.mk

# Rules for generation (code and manifests)
# -------------------------------------------------------------------------
.PHONY: check-generate
check-generate:
@$(HACK_DIR)/check-generate.sh "$(REPO_ROOT)"

# Generate manifests e.g. CRD, RBAC etc.
.PHONY: manifests
manifests: $(VGOPATH) $(CONTROLLER_GEN)
@HACK_DIR=$(HACK_DIR) VGOPATH=$(VGOPATH) go generate ./config/crd/bases
@find "$(REPO_ROOT)/config/crd/bases" -name "*.yaml" -exec cp '{}' "$(REPO_ROOT)/charts/druid/charts/crds/templates/" \;
@controller-gen rbac:roleName=manager-role paths="./internal/controller/..."

.PHONY: generate-api-docs
generate-api-docs: $(CRD_REF_DOCS)
@crd-ref-docs --source-path "$(REPO_ROOT)/api" --config "$(HACK_DIR)/api-reference/config.yaml" --output-path "$(REPO_ROOT)/docs/api-reference/etcd-druid-api.md" --renderer markdown

# Generate code
.PHONY: generate
generate: manifests generate-api-docs $(CONTROLLER_GEN) $(GOIMPORTS) $(MOCKGEN)
@go generate "$(REPO_ROOT)/internal/..."
@"$(HACK_DIR)/update-codegen.sh"
ifndef CERT_EXPIRY_DAYS
override CERT_EXPIRY_DAYS = 365
endif

# Rules for verification, formatting, linting and cleaning
# -------------------------------------------------------------------------
.PHONY: tidy
tidy:
@env GO111MODULE=on go mod tidy

.PHONY: clean
clean:
@$(HACK_DIR)/clean.sh ./api/... ./internal/...

# Clean go mod cache
.PHONY: clean-mod-cache
clean-mod-cache:
Expand All @@ -74,16 +52,12 @@ add-license-headers: $(GO_ADD_LICENSE)
# Format code and arrange imports.
.PHONY: format
format: $(GOIMPORTS_REVISER)
@$(HACK_DIR)/format.sh ./api/ ./internal/ ./test/
@$(HACK_DIR)/format.sh ./internal/ ./test/ ./examples/

# Check packages
.PHONY: check
check: $(GOLANGCI_LINT) $(GOIMPORTS) format manifests
@$(HACK_DIR)/check.sh --golangci-lint-config=./.golangci.yaml ./api/... ./internal/...

.PHONY: check-apidiff
check-apidiff: $(GO_APIDIFF)
@$(HACK_DIR)/check-apidiff.sh
check: $(GOLANGCI_LINT) $(GOIMPORTS) format
@$(HACK_DIR)/check.sh --golangci-lint-config=./.golangci.yaml ./internal/...

.PHONY: sast
sast: $(GOSEC)
Expand All @@ -97,17 +71,19 @@ sast-report: $(GOSEC)
# -------------------------------------------------------------------------
# Run tests
.PHONY: test-unit
test-unit: $(GINKGO) $(GOTESTFMT)
test-unit: $(GINKGO)
# run ginkgo unit tests. These will be ported to golang native tests over a period of time.
@TEST_COVER=$(TEST_COVER) "$(HACK_DIR)/test.sh" ./internal/controller/etcdcopybackupstask/... \
./internal/controller/secret/... \
./internal/controller/utils/... \
./internal/mapper/... \
./internal/metrics/... \
./internal/health/...
./internal/health/... \
./internal/utils/imagevector/...

# run the golang native unit tests.
@TEST_COVER=$(TEST_COVER) "$(HACK_DIR)/test-go.sh" ./api/... \
@TEST_COVER=$(TEST_COVER) "$(HACK_DIR)/test-go.sh" \
./internal/controller/etcd/... \
./internal/controller/secret/... \
./internal/controller/compaction/... \
./internal/component/... \
./internal/errors/... \
Expand All @@ -116,7 +92,7 @@ test-unit: $(GINKGO) $(GOTESTFMT)
./internal/webhook/...

.PHONY: test-integration
test-integration: $(GINKGO) $(SETUP_ENVTEST) $(GOTESTFMT)
test-integration: $(GINKGO) $(SETUP_ENVTEST)
@SETUP_ENVTEST="true" "$(HACK_DIR)/test.sh" ./test/integration/...
@SETUP_ENVTEST="true" "$(HACK_DIR)/test-go.sh" ./test/it/...

Expand All @@ -131,6 +107,7 @@ test-cov-clean:

.PHONY: test-e2e
test-e2e: $(KUBECTL) $(HELM) $(SKAFFOLD) $(KUSTOMIZE) $(GINKGO)
@$(HACK_DIR)/prepare-chart-resources.sh $(BUCKET_NAME) $(CERT_EXPIRY_DAYS)
@VERSION=$(VERSION) GIT_SHA=$(GIT_SHA) $(HACK_DIR)/e2e-test/run-e2e-test.sh $(PROVIDERS)

.PHONY: ci-e2e-kind
Expand Down Expand Up @@ -178,43 +155,47 @@ docker-clean:
# -------------------------------------------------------------------------
kind-up kind-down ci-e2e-kind ci-e2e-kind-azure ci-e2e-kind-gcs deploy-localstack deploy-fakegcs deploy-azurite test-e2e deploy deploy-dev deploy-debug undeploy: export KUBECONFIG = $(KUBECONFIG_PATH)

ifndef CLUSTER_NAME
override CLUSTER_NAME = etcd-druid-e2e
endif

.PHONY: kind-up
kind-up: $(KIND)
@$(HACK_DIR)/kind-up.sh
@$(HACK_DIR)/kind-up.sh --cluster-name $(CLUSTER_NAME)
@printf "\n\033[0;33m📌 NOTE: To target the newly created KinD cluster, please run the following command:\n\n export KUBECONFIG=$(KUBECONFIG_PATH)\n\033[0m\n"

.PHONY: kind-down
kind-down: $(KIND)
@$(HACK_DIR)/kind-down.sh

# Install CRDs into a cluster
.PHONY: install
install: manifests
kubectl apply -f config/crd/bases

# Deploy controller in the configured Kubernetes cluster in ~/.kube/config
.PHONY: deploy-via-kustomize
deploy-via-kustomize: manifests $(KUSTOMIZE)
kubectl apply -f config/crd/bases
kustomize build config/default | kubectl apply -f -
@$(HACK_DIR)/kind-down.sh --cluster-name $(CLUSTER_NAME)

# Make targets to deploy etcd-druid operator using skaffold
# --------------------------------------------------------------------------------------------------------
# Deploy controller to the Kubernetes cluster specified in the environment variable KUBECONFIG
# Modify the Helm template located at charts/druid/templates if any changes are required

ifndef NAMESPACE
override NAMESPACE = default
endif

.PHONY: prepare-helm-charts
prepare-helm-charts:
@$(HACK_DIR)/prepare-chart-resources.sh $(NAMESPACE) $(CERT_EXPIRY_DAYS)

.PHONY: deploy
deploy: $(SKAFFOLD) $(HELM)
@VERSION=$(VERSION) GIT_SHA=$(GIT_SHA) $(SKAFFOLD) run -m etcd-druid
deploy: $(SKAFFOLD) $(HELM) prepare-helm-charts
@VERSION=$(VERSION) GIT_SHA=$(GIT_SHA) $(SKAFFOLD) run -m etcd-druid -n $(NAMESPACE)

.PHONY: deploy-dev
deploy-dev: $(SKAFFOLD) $(HELM)
@VERSION=$(VERSION) GIT_SHA=$(GIT_SHA) $(SKAFFOLD) dev --cleanup=false -m etcd-druid --trigger='manual'
deploy-dev: $(SKAFFOLD) $(HELM) prepare-helm-charts
@VERSION=$(VERSION) GIT_SHA=$(GIT_SHA) $(SKAFFOLD) dev --cleanup=false -m etcd-druid --trigger='manual' -n $(NAMESPACE)

.PHONY: deploy-debug
deploy-debug: $(SKAFFOLD) $(HELM)
@VERSION=$(VERSION) GIT_SHA=$(GIT_SHA) $(SKAFFOLD) debug --cleanup=false -m etcd-druid -p debug
deploy-debug: $(SKAFFOLD) $(HELM) prepare-helm-charts
@VERSION=$(VERSION) GIT_SHA=$(GIT_SHA) $(SKAFFOLD) debug --cleanup=false -m etcd-druid -p debug -n $(NAMESPACE)

.PHONY: undeploy
undeploy: $(SKAFFOLD) $(HELM)
$(SKAFFOLD) delete -m etcd-druid
$(SKAFFOLD) delete -m etcd-druid -n $(NAMESPACE)

.PHONY: deploy-localstack
deploy-localstack: $(KUBECTL)
Expand All @@ -226,4 +207,9 @@ deploy-azurite: $(KUBECTL)

.PHONY: deploy-fakegcs
deploy-fakegcs: $(KUBECTL)
@$(HACK_DIR)/deploy-fakegcs.sh
@$(HACK_DIR)/deploy-fakegcs.sh

.PHONY: clean-chart-resources
clean-chart-resources:
@rm -f $(REPO_ROOT)/charts/crds/*.yaml
@rm -rf $(REPO_ROOT)/charts/pki-resources/*
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

[![REUSE status](https://api.reuse.software/badge/github.com/gardener/etcd-druid)](https://api.reuse.software/info/github.com/gardener/etcd-druid) [![CI Build status](https://concourse.ci.gardener.cloud/api/v1/teams/gardener/pipelines/etcd-druid-master/jobs/master-head-update-job/badge)](https://concourse.ci.gardener.cloud/teams/gardener/pipelines/etcd-druid-master/jobs/master-head-update-job) [![Go Report Card](https://goreportcard.com/badge/github.com/gardener/etcd-druid)](https://goreportcard.com/report/github.com/gardener/etcd-druid) [![License: Apache-2.0](https://img.shields.io/badge/License-Apache--2.0-blue.svg)](LICENSE) [![Release](https://img.shields.io/github/v/release/gardener/etcd-druid.svg?style=flat)](https://github.com/gardener/etcd-druid) [![Go Reference](https://pkg.go.dev/badge/github.com/gardener/etcd-druid.svg)](https://pkg.go.dev/github.com/gardener/etcd-druid) [![Docs](https://img.shields.io/badge/Docs-reference-orange)](https://gardener.github.io/etcd-druid/index.html)

`etcd-druid` is an [etcd](https://github.com/etcd-io/etcd) [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) which makes it easy to configure, provision, reconcile, monitor and delete etcd clusters. It enables management of etcd clusters through [declarative Kubernetes API model](config/crd/bases/crd-druid.gardener.cloud_etcds.yaml).
`etcd-druid` is an [etcd](https://github.com/etcd-io/etcd) [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) which makes it easy to configure, provision, reconcile, monitor and delete etcd clusters. It enables management of etcd clusters through [declarative Kubernetes API model](api/core/crds/druid.gardener.cloud_etcds.yaml).

In every etcd cluster managed by `etcd-druid`, each etcd member is a two container `Pod` which consists of:

Expand Down
47 changes: 47 additions & 0 deletions api/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# SPDX-FileCopyrightText: 2025 SAP SE or an SAP affiliate company and Gardener contributors
#
# SPDX-License-Identifier: Apache-2.0

API_MODULE_ROOT := $(shell dirname "$(realpath $(lastword $(MAKEFILE_LIST)))")
REPO_ROOT := $(shell dirname "$(API_MODULE_ROOT)")
HACK_DIR := $(API_MODULE_ROOT)/hack
API_MODULE_TOOLS_DIR := $(HACK_DIR)/tools
REPO_TOOLS_DIR := $(REPO_ROOT)/hack/tools

include $(HACK_DIR)/tools.mk

.PHONY: check-generate
check-generate:
@$(HACK_DIR)/check-generate.sh "$(API_MODULE_ROOT)"

.PHONY: generate-api-docs
generate-api-docs: $(CRD_REF_DOCS)
@crd-ref-docs --source-path "$(REPO_ROOT)/api" --config "$(HACK_DIR)/api-reference/config.yaml" --output-path "$(REPO_ROOT)/docs/api-reference/etcd-druid-api.md" --renderer markdown

.PHONY: check-apidiff
check-apidiff: $(GO_APIDIFF)
@$(HACK_DIR)/check-apidiff.sh

.PHONY: generate
generate: $(CONTROLLER_GEN) tidy generate-api-docs
@$(HACK_DIR)/generate.sh

.PHONY: clean-generated-code
clean-generated-code:
@$(HACK_DIR)/clean-generated-code.sh

.PHONY: tidy
tidy:
@env GO111MODULE=on go mod tidy

.PHONY: test-unit
test-unit:
@go test ./...

.PHONY: format
format: $(GOIMPORTS_REVISER)
@$(REPO_ROOT)/hack/format.sh ./core ./validation

.PHONY: check
check: $(GOLANGCI_LINT) $(GOIMPORTS) format
@$(REPO_ROOT)/hack/check.sh --golangci-lint-config=../.golangci.yaml ./...
Loading

0 comments on commit a33217c

Please sign in to comment.