neonvm: continuous synchronisation of files in pod/vm

[conradludgate]

For #1213 I need the renewed certificate secret to be updated within the vm.

It's ok if there's some lag on this synchronisation as certificates are renewed with generous amount of time left until expiry.

Design:
neonvm-runner will subscribe to inotify events for a specific list of secrets. If an event is detected, the secret contents are sent in a http request to neonvm-daemon.

The files will be owned by the daemon user, and read-only by postgres.

If there's an error when uploading, then there's a secondary loop every 5 minutes to catch up, using a checksum to compare.

We mirror kubernetes and use atomicwriter to atomically update multiple files in the secret.

Useful links:
https://ahmet.im/blog/kubernetes-secret-volumes-delay/
https://ahmet.im/blog/kubernetes-inotify/

Some discussion here:
https://neondb.slack.com/archives/C03TN5G758R/p1737723555448339
https://neondb.slack.com/archives/C03TN5G758R/p1738669103059979

[github-actions]

No changes to the coverage.

HTML Report

Click to open

[conradludgate]

open questions:

• how to we know which secrets should be synchronised (currently /var/sync)
• should we support configmap too
• is relying on ..data from atomicwriter relying on an unstable implementation detail?

[mikhail-sakhnov]

@conradludgate @sharnoff Have you considered instead of relying on inotify and push model to modify neonvm-daemon to subscribe for secret changes via kube-API to reduce the number of moving parts?
It feels more natural for the k8s ecosystem.

• add serviceaccount to the pod (or modify default service account to allow fetching secrets, which is bad idea from the security perspective)
• call kubernetes.default.svc from the VM.
• [probably, not sure] modify iptables rules to allow calling k8s api from VM

[conradludgate]

I think this inevitably has the same issue. The service account token updates in the pod, and the VM needs to notice that change

[conradludgate]

I've looked into service accounts. They also use the same form of atomic directory updates.

Currently there's no projected volume disk source for VM objects, but this is the kinda thing that could be hard-coded in the neomvm-runner code. Add /var/run/secrets/kubernetes.io/serviceaccount/..data to the secrets map and it should just work:tm:

[edude03]

High level, feels like we need a shared file system primitive for the VMs, I keep seeing items that this would potentially solve.

[conradludgate]

High level, feels like we need a shared file system primitive for the VMs

Not sure exactly what this means

[sharnoff]

Broadly looking good -- left a thorough review; I expect 1, maybe 2, much smaller rounds remaining

Also, could you update the PR title to have the structure component: Title ? (ordinarily the PR title format checker would catch that, but feat: is a sufficient prefix to trick the regex 😄)
In this case, s/feat/neonvm/ would suffice, but it's up to you

[sharnoff]

LGTM, pending resolution of remaining comments -- thanks!