Collecting some research notes on for implementation direction:

Cf-deployment specifies update_mode for at least some variables: https://github.com/cloudfoundry/cf-deployment/blob/main/cf-deployment.yml#L2070

The bosh director has support for forwarding this information to credhub: https://www.pivotaltracker.com/n/projects/956238/stories/164818604
And is used here: https://github.com/cloudfoundry/bosh/blob/e0ac496c97eff51c3c7071928f2b96e4b347847c/src/bosh-director/lib/bosh/director/config_server/client.rb#L89-L116

Credhub api docs are currently down but below a screenshot from a local copy:

Credhub cli go client supports converge: cloudfoundry/credhub-cli@37f2848

Credhub cli generate command has --no-overwrite flag, no --mode or similar to set converge, regenerate has nothing like that. But if we already have all generation options from the manifest we should be able to use generate.

https://bosh.io/docs/manifest-v2/#features

Warning
The converge_variables feature can result in unintended side effects. It will regenerate all variables associated with a deployment, even those that are specified in a runtime config. For this reason we strongly recommend that you use the update_mode configuration on the variable that you wish to converge.

From the above it seems like the following requirements could be handled by BOSH using converge_variables and update_mode:

• 1. using variables from the rendered manifest instead of kit.yml to define the secrets behaviour by setting update_mode to no-overwrite for "fixed" secrets and CA's and converge on all others
• 2. check script will only error on missing credhub values if they are not specified by variables or bosh_variables sections this should not be needed anymore since changes to variable options should trigger the generation of new credential version when update_mode is set to converge
• 3.1 rotate secrets will use regenerate
• 3.2 but will have to handle changes to the manifest (ie if I change the password length or certificate properties, how does that get sent to credhub prior to regeneration without deploying the manifest.) This is exactly what update_mode converge was designed for

3.1 could be implemented using the bosh variables cli command in combination with bosh manifest (to get the variable update_mode, so we can skip no-overwrite and isCa) and bosh config (to get runtime-config variable definitions).

The above should work just fine for the simple cases, but does not fix the 3 phase deploy required for rotating ca's.
Or the cc db encryption key, which requires an errand. Or the application ca used by diego to create certs on the fly.