Configuring zot on Azure (AKS) with multiple replicas

[kevin-bates]

Hello,

We support both AWS (EKS) and Azure (AKS) k8s providers. On AWS we configure zot with remote storage using s3 and it just works! On Azure, because we can't use s3, we must configure local filesystem storage and have run into different issues depending on the kind of storage class we configure. In all cases, the issues arise when multiple replicas come into play, and, in most cases, are the result of a panic from boltdb.

So I guess my first question is: Is local filesystem storage supported with mulitiple replicas (zot instances)?

Based on this portion of the docs, we have tried storage classes using either nfs or fuse with wildly different results. As a result, we're forced to pin the replicas to 1 and essentially disable HPA (i.e., max replicas is also set to 1).

Is this something to which an Issue should be opened, or are we missing something in the configuration. Here's a description of the kinds of issues we've encountered when attempting to configure the local filesystem storage with multiple replicas.

NFS (Provisioner: blob.csi.azure.com):

When configuring a storage class of azureblob-nfs-premium, multiple replicas work initially, until the deployment is restarted or HPA scales up the replica count. Once those conditions occur, the newly started replica fails with:

On restart:

{"level":"info","identity":"probe","goroutine":5642,"caller":"zotregistry.dev/zot/pkg/api/authn.go:143","time":"2025-02-06T00:00:44.426504524Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"probe","component":"session","clientIP":"10.244.1.1:37626","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Authorization":["******"],"User-Agent":["kube-probe/1.29"]},"goroutine":5642,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-02-06T00:00:44.426608377Z","message":"HTTP API"}
{"level":"warn","error":"blob not found","digest":"sha256:f7c70656c2cf7c1be3c177f8b7a9993a15a57ee6394f50d6c0fd6c04977a3268","goroutine":4739,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1249","time":"2025-02-06T00:00:44.536960583Z","message":"not found in cache"}
{"level":"info","identity":"probe","goroutine":5706,"caller":"zotregistry.dev/zot/pkg/api/authn.go:143","time":"2025-02-06T00:00:44.540032981Z","message":"user profile successfully set"}
{"level":"info","module":"http","username":"probe","component":"session","clientIP":"10.244.1.1:37630","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Authorization":["******"],"User-Agent":["kube-probe/1.29"]},"goroutine":5706,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-02-06T00:00:44.540108543Z","message":"HTTP API"}
{"level":"warn","error":"blob not found","digest":"sha256:77729c4790feab9fbe88ab43242acdfb9a66285e690427420282637ce44c9553","goroutine":4739,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1249","time":"2025-02-06T00:00:44.685039384Z","message":"not found in cache"}
{"level":"debug","blob path":"/var/lib/registry/my-image-registry.com/853c7efb-b2d5-4f3c-bee1-76cee2a11d86/blobs/sha256/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1","goroutine":4739,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1233","time":"2025-02-06T00:00:44.804655559Z","message":"blob path found"}
{"level":"warn","error":"blob not found","digest":"sha256:3ae315aa7d91d685871f919dd8faf2a1ce74a7d81663600adf90b9b714cb083d","goroutine":4739,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1249","time":"2025-02-06T00:00:44.808842309Z","message":"not found in cache"}
panic: assertion failed: Page expected to be: 19, but self identifies as 7022288508948476517
goroutine 4739 [running]:
go.etcd.io/bbolt._assert(...)
    go.etcd.io/[email protected]/db.go:1389
go.etcd.io/bbolt.(*page).fastCheck(0x7f5fe5faa000, 0x13)
    go.etcd.io/[email protected]/page.go:57 +0x1d9
go.etcd.io/bbolt.(*Tx).page(0xc00169e620?, 0x13?)
    go.etcd.io/[email protected]/tx.go:541 +0x7b
go.etcd.io/bbolt.(*Bucket).pageNode(0xc00295c0c0?, 0x13?)
    go.etcd.io/[email protected]/bucket.go:748 +0x85
go.etcd.io/bbolt.(*Cursor).search(0xc00214a588, {0xc0015380a0, 0x47, 0x47}, 0x18?)
    go.etcd.io/[email protected]/cursor.go:281 +0x48
go.etcd.io/bbolt.(*Cursor).searchPage(0xc00214a588, {0xc0015380a0, 0x47, 0x47}, 0x5650523a30e0?)
    go.etcd.io/[email protected]/cursor.go:341 +0x165
go.etcd.io/bbolt.(*Cursor).search(0xc00214a588, {0xc0015380a0, 0x47, 0x47}, 0xc0039c0cc0?)
    go.etcd.io/[email protected]/cursor.go:298 +0x1c5
go.etcd.io/bbolt.(*Cursor).seek(0xc00214a588, {0xc0015380a0?, 0x56504f0a4000?, 0xc000942800?})
    go.etcd.io/[email protected]/cursor.go:159 +0x2e
go.etcd.io/bbolt.(*Bucket).Put(0xc00295c0c0, {0xc000ff5090, 0x47, 0x47?}, {0xc000942800, 0x464c, 0x464c})
    go.etcd.io/[email protected]/bucket.go:301 +0x125
zotregistry.dev/zot/pkg/meta/boltdb.(*BoltDB).SetRepoReference.func1(0xc0028cba40?)
    zotregistry.dev/zot/pkg/meta/boltdb/boltdb.go:199 +0x213
go.etcd.io/bbolt.(*DB).Update(0x5650528cbed8?, 0xc00214a858)
    go.etcd.io/[email protected]/db.go:886 +0x75
zotregistry.dev/zot/pkg/meta/boltdb.(*BoltDB).SetRepoReference(0xc0027345a0, {0x5650528cbed8?, 0x5650554692e0?}, {0xc002b545a9, 0x40}, {0xc002b545f4, 0x32}, {{0x56504f38c469, 0x2a}, {0xc000ff5090, ...}, ...})
    zotregistry.dev/zot/pkg/meta/boltdb/boltdb.go:184 +0x1a5
zotregistry.dev/zot/pkg/meta.SetImageMetaFromInput({0x5650528cbed8, 0x5650554692e0}, {0xc002b545a9, 0x40}, {0xc002b545f4, 0x32}, {0x56504f38c469?, _}, {0xc000ff5090, 0x47}, ...)
    zotregistry.dev/zot/pkg/meta/parse.go:383 +0xd51
zotregistry.dev/zot/pkg/extensions/sync.(*DestinationRegistry).copyManifest(0xc002c1e370, {0xc002b545a9, 0x40}, {0xc000a3c000, 0x1372, 0x1500}, {0xc002b545f4, 0x32}, {0x56505292d0c8, 0xc001eda2a0})
    zotregistry.dev/zot/pkg/extensions/sync/destination.go:238 +0x613
zotregistry.dev/zot/pkg/extensions/sync.(*DestinationRegistry).CommitImage(0xc002c1e370, {0x5650528e4100, 0xc000fbcc00}, {0xc002b545a9, 0x40}, {0xc002b545f4, 0x32})
    zotregistry.dev/zot/pkg/extensions/sync/destination.go:116 +0x386
zotregistry.dev/zot/pkg/extensions/sync.(*BaseService).syncTag(0xc003a82008, {0x5650528cbff0, 0xc0029cfd10}, {0xc002b545a9, 0x40}, {0xc002b545c5, 0x24}, {0xc002b545f4, 0x32})
    zotregistry.dev/zot/pkg/extensions/sync/service.go:483 +0x12cf
zotregistry.dev/zot/pkg/extensions/sync.(*BaseService).SyncImage(0xc003a82008, {0x5650528cbff0, 0xc0029cfd10}, {0xc002b545a9, 0x40}, {0xc002b545f4, 0x32})
    zotregistry.dev/zot/pkg/extensions/sync/service.go:312 +0x24c
zotregistry.dev/zot/pkg/extensions/sync.(*BaseOnDemand).syncImage(0xc00547aa20, {0x5650528cbff0, 0xc0029cfd10}, {0xc002b545a9, 0x40}, {0xc002b545f4, 0x32}, 0xc002551740)
    zotregistry.dev/zot/pkg/extensions/sync/on_demand.go:122 +0x18e
created by zotregistry.dev/zot/pkg/extensions/sync.(*BaseOnDemand).SyncImage in goroutine 4753
    zotregistry.dev/zot/pkg/extensions/sync/on_demand.go:76 +0x3bc

When hpa kicks in:

{"level":"info","params":{"distSpecVersion":"1.1.0","GoVersion":"go1.22.8","Commit":"v2.1.2-rc2-513f2a0dc94c5f36d9d6e1455b562c192664a4c2","ReleaseTag":"v2.1.2-rc2","BinaryType":"-imagetrust-lint-metrics-mgmt-profile-scrub-search-sync-ui-userprefs","Storage":{"RootDirectory":"/var/lib/registry","Dedupe":false,"RemoteCache":false,"GC":true,"Commit":true,"GCDelay":3600000000000,"GCInterval":3600000000000,"Retention":{"DryRun":false,"Delay":0,"Policies":null},"StorageDriver":null,"CacheDriver":null,"SubPaths":null},"HTTP":{"Address":"0.0.0.0","ExternalURL":"","Port":"5000","AllowOrigin":"","TLS":{"Cert":"/secrets/tls/tls.crt","Key":"/secrets/tls/tls.key","CACert":""},"Auth":{"FailDelay":0,"HTPasswd":{"Path":"/secret/htpasswd"},"LDAP":null,"Bearer":null,"OpenID":null,"APIKey":false,"SessionKeysFile":""},"AccessControl":{"repositories":{"**":{"Policies":[{"Users":["user","metrics","probe"],"Actions":["read"],"Groups":null}],"DefaultPolicy":[],"AnonymousPolicy":null}},"AdminPolicy":{"Users":["admin"],"Actions":["read","create","update","delete"],"Groups":null},"Groups":null,"Metrics":{"Users":["metrics"]}},"Realm":"","Ratelimit":null},"Log":{"Level":"debug","Output":"","Audit":""},"Extensions":{"Search":{"Enable":true,"CVE":null},"Sync":{"Enable":true,"CredentialsFile":"/secrets/reg-central-key/config.json","DownloadDir":"/tmp/sync","Registries":[{"URLs":["https://index.docker.io"],"PollInterval":0,"Content":[{"Prefix":"**","Tags":null,"Destination":"/docker.io","StripPrefix":false}],"TLSVerify":true,"OnDemand":true,"CertDir":"","MaxRetries":null,"RetryDelay":null,"OnlySigned":null},{"URLs":["https://registry.gitlab.com"],"PollInterval":0,"Content":[{"Prefix":"**","Tags":null,"Destination":"/registry.gitlab.com","StripPrefix":false}],"TLSVerify":true,"OnDemand":true,"CertDir":"","MaxRetries":null,"RetryDelay":null,"OnlySigned":null},{"URLs":["https://ghcr.io"],"PollInterval":0,"Content":[{"Prefix":"**","Tags":null,"Destination":"/ghcr.io","StripPrefix":false}],"TLSVerify":true,"OnDemand":true,"CertDir":"","MaxRetries":null,"RetryDelay":null,"OnlySigned":null},{"URLs":["https://quay.io"],"PollInterval":0,"Content":[{"Prefix":"**","Tags":null,"Destination":"/quay.io","StripPrefix":false}],"TLSVerify":true,"OnDemand":true,"CertDir":"","MaxRetries":null,"RetryDelay":null,"OnlySigned":null},{"URLs":["https://gcr.io"],"PollInterval":0,"Content":[{"Prefix":"**","Tags":null,"Destination":"/gcr.io","StripPrefix":false}],"TLSVerify":true,"OnDemand":true,"CertDir":"","MaxRetries":null,"RetryDelay":null,"OnlySigned":null},{"URLs":["https://registry.k8s.io"],"PollInterval":0,"Content":[{"Prefix":"**","Tags":null,"Destination":"/registry.k8s.io","StripPrefix":false}],"TLSVerify":true,"OnDemand":true,"CertDir":"","MaxRetries":null,"RetryDelay":null,"OnlySigned":null},{"URLs":["https://my-image-registry.com"],"PollInterval":0,"Content":[{"Prefix":"**","Tags":null,"Destination":"/my-image-registry.com","StripPrefix":false}],"TLSVerify":true,"OnDemand":true,"CertDir":"","MaxRetries":null,"RetryDelay":null,"OnlySigned":null}]},"Metrics":{"Enable":true,"Prometheus":{"Path":"/metrics"}},"Scrub":null,"Lint":null,"UI":{"Enable":true},"Mgmt":null,"APIKey":null,"Trust":null},"scheduler":null,"cluster":null},"goroutine":1,"caller":"zotregistry.dev/zot/pkg/api/controller.go:261","time":"2025-02-06T23:49:33.652885893Z","message":"configuration settings"}
{"level":"info","cpus":4,"max. open files":1048576,"listen backlog":"4096","max. inotify watches":"1048576","goroutine":1,"caller":"zotregistry.dev/zot/pkg/api/controller.go:130","time":"2025-02-06T23:49:33.653002055Z","message":"runtime params"}
panic: invalid page type: 5: 10
goroutine 1 [running]:
go.etcd.io/bbolt.(*Cursor).search(0xc00408f278, {0xc003b021e0, 0xf, 0xf}, 0x18?)
    go.etcd.io/[email protected]/cursor.go:283 +0x274
go.etcd.io/bbolt.(*Cursor).searchPage(0xc00408f278, {0xc003b021e0, 0xf, 0xf}, 0x55fc85bfd0e0?)
    go.etcd.io/[email protected]/cursor.go:341 +0x165
go.etcd.io/bbolt.(*Cursor).search(0xc00408f278, {0xc003b021e0, 0xf, 0xf}, 0xc003afeed0?)
    go.etcd.io/[email protected]/cursor.go:298 +0x1c5
go.etcd.io/bbolt.(*Cursor).seek(0xc00408f278, {0xc003b021e0?, 0x7fc160076800?, 0x10?})
    go.etcd.io/[email protected]/cursor.go:159 +0x2e
go.etcd.io/bbolt.(*Bucket).CreateBucket(0xc0039d7500, {0xc00408f329, 0xf, 0xd?})
    go.etcd.io/[email protected]/bucket.go:172 +0xf5
go.etcd.io/bbolt.(*Bucket).CreateBucketIfNotExists(0xc0039d7500, {0xc00408f329, 0xf, 0xf})
    go.etcd.io/[email protected]/bucket.go:204 +0x2b
zotregistry.dev/zot/pkg/meta/boltdb.New.func1(0xc00179aa00?)
    zotregistry.dev/zot/pkg/meta/boltdb/boltdb.go:73 +0x1e8
go.etcd.io/bbolt.(*DB).Update(0xc003afebd0?, 0x55fc860c8838)
    go.etcd.io/[email protected]/db.go:886 +0x75
zotregistry.dev/zot/pkg/meta/boltdb.New(0xc003af58c8, {{{0x55fc860f89a8, 0xc0020d9710}, 0xff, {0x0, 0x0}, {0xc002299000, 0x1, 0x1f4}, {0xc003a0d890, ...}, ...}})
    zotregistry.dev/zot/pkg/meta/boltdb/boltdb.go:37 +0x26
zotregistry.dev/zot/pkg/meta.Create({0x55fc82aa2f1d?, 0xc003a0d890?}, {0x55fc8601b2e0, 0xc003af58c8}, {0x55fc85a4f2a0?, 0xc0037f5810?}, {{{0x55fc860f89a8, 0xc0020d9710}, 0xff, {0x0, ...}, ...}})
    zotregistry.dev/zot/pkg/meta/meta.go:52 +0x99
zotregistry.dev/zot/pkg/meta.New({{0xc0021bd518, 0x11}, 0x0, 0x0, 0x1, 0x1, 0x34630b8a000, 0x34630b8a000, {0x0, 0x0, ...}, ...}, ...)
    zotregistry.dev/zot/pkg/meta/meta.go:35 +0x1e5
zotregistry.dev/zot/pkg/api.(*Controller).InitMetaDB(0xc003a17200)
    zotregistry.dev/zot/pkg/api/controller.go:334 +0x13d
zotregistry.dev/zot/pkg/api.(*Controller).Init(0xc003a17200)
    zotregistry.dev/zot/pkg/api/controller.go:280 +0x20a
zotregistry.dev/zot/pkg/cli/server.NewServerRootCmd.newServeCmd.func2(0xc003a0a100?, {0xc004bb6990, 0x1, 0x55fc82a9b7b4?})
    zotregistry.dev/zot/pkg/cli/server/root.go:75 +0xef
github.com/spf13/cobra.(*Command).execute(0xc00066f508, {0xc004bb6960, 0x1, 0x1})
    github.com/spf13/[email protected]/command.go:985 +0xaca
github.com/spf13/cobra.(*Command).ExecuteC(0xc00066e908)
    github.com/spf13/[email protected]/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(0xc0000061c0?)
    github.com/spf13/[email protected]/command.go:1041 +0x13
main.main()
    zotregistry.dev/zot/cmd/zot/main.go:10 +0x18

Fuse (Provisioner: blob.csi.azure.com):

When configuring a storage class of azureblob-fuse-premium, the stability of the replicas appears fine, but the actual processing within the zot pods encounter frequent image pull failures. It seems the layers are misordered since both the expected and unexpected values appear to be valid layer names. This is true even with a single replica.

Failed to pull image "localhost:30500/my-image-registry.com/8bdb0e3b-ff28-4f6e-a3ba-887bd06e6440:69baf48d-b9ac-4d26-a121-4dd17182c222": rpc error: code = FailedPrecondition desc = failed to pull and unpack image "localhost:30500/my-image-registry.com/8bdb0e3b-ff28-4f6e-a3ba-887bd06e6440:69baf48d-b9ac-4d26-a121-4dd17182c222": failed commit on ref "layer-sha256:2ab09b027e7f3a0c2e8bb1944ac46de38cebab7145f0bd6effebfe5492c818b6": unexpected commit digest sha256:805d09e4cfce34478b5d162441d3dee25c4972898994c4cde43a62466e2b813f, expected sha256:2ab09b027e7f3a0c2e8bb1944ac46de38cebab7145f0bd6effebfe5492c818b6: failed precondition

Default Storage (Provisioner: disk.csi.azure.com)

Lastly, if we attempt to use our default storage class, the second replica cannot start since the the boltdb is already in use:

Error: operation timeout: boltdb file is already in use, path '/var/lib/registry/cache.db'
{"level":"error","error":"operation timeout: boltdb file is already in use, path '/var/lib/registry/cache.db'","dbPath":"/var/lib/registry/cache.db","goroutine":1,"caller":"zotregistry.dev/zot/pkg/storage/cache/boltdb.go:58","time":"2025-02-06T22:22:24.664884051Z","message":"failed to create cache db"}
{"level":"error","error":"operation timeout: boltdb file is already in use, path '/var/lib/registry/cache.db'","goroutine":1,"caller":"zotregistry.dev/zot/pkg/cli/server/root.go:76","time":"2025-02-06T22:22:24.664926976Z","message":"failed to init controller"}

Thank you for your help.

[andaaron]

Local file system using boltdb doesn't work for multiple zot instances using the same file system and/or same boltdb file. It could work if they have separate storage and each is responsible for a subset of images.

There was some work done for azure support in #2221 / #2223 but it was not complete, and is far from finished.

[kevin-bates]

Thank you @andaaron. I will monitor that issue (sorry I hadn't seen it prior to my inquiry). It might be helpful to amend the docs for local filesystem to indicate that multiple instances/replcas using the same boltdb is not supported. I have gone ahead and opened project-zot/project-zot.github.io#189 and will provide a corresponding pull request, if that proves helpful.