Secret values stored in Vault are not automatically updated by ArgoCD when updated in Vault

[cdchris12]

We've noticed that any values we update in Vault are not synced by ArgoCD until after a new catalog push. As far as we can tell, there seems to be nothing changing in the actual catalog when a Vault secret is updated (as evidenced by Commodore's refusal to push a no-op catalog update).

Steps to Reproduce the Problem

1. Start with a synfected cluster, with the backup-k8up component configured
2. Update any backup-k8up secret value stored in Vault
3. Notice that ArgoCD, even with manually forced sync actions, does not pick up the change
4. Notice that Commodore sees no changes will refuse to push a catalog update with no changes
5. Make a nonsensical change to the cluster catalog (adding a useless env var, for example), which will force ArgoCD to update the backup-k8up component
6. After pushing the change, notice that ArgoCD has picked up the changed values from Vault and updated the syn-backup component deployment accordingly

Actual Behavior

When values are updated in Vault, these updates are not applied until a new catalog is pushed via Commodore. This can be particularly detrimental for services which rely on things like access keys (as the syn-backup component does) which need to be rotated regularly.

Expected Behavior

We would expect ArgoCD to talk directly to Vault to pick up and deploy any changed secret values as soon as they are changed in Vault. The need for a Commodore catalog update before ArgoCD will process these updates hinders actions like easily rotating access keys and API keys.

[simu]

Hi @cdchris12

We've noticed that any values we update in Vault are not synced by ArgoCD until after a new catalog push. As far as we can tell, there seems to be nothing changing in the actual catalog when a Vault secret is updated (as evidenced by Commodore's refusal to push a no-op catalog update).

The secret reference in the cluster catalog doesn't change when you change the secret value in Vault, since it's only a reference to the key in Vault's KV backend which holds the secret. In fact, Commodore doesn't need to access Vault at all in the current secret management architecture.

The secret references in the catalog are Kapitan secret references. These references are processed and replaced by the actual secrets by ArgoCD by running kapitan refs --reveal as an ArgoCD plugin for each ArgoCD application. ArgoCD only runs the plugin when there's a change in the application's manifests (as you've observed) or on a "Hard Refresh", unfortunately a regular "Refresh" doesn't run the plugin.

To trigger a "Hard Refresh", click the triangle on the "Refresh" button and click "Hard Refresh" in the pop-up menu:

We would expect ArgoCD to talk directly to Vault to pick up and deploy any changed secret values as soon as they are changed in Vault. The need for a Commodore catalog update before ArgoCD will process these updates hinders actions like easily rotating access keys and API keys.

As noted above, you don't need to generate a Commodore catalog update, but can instead trigger a "Hard Refresh" of the application for which you've rotated a secret.

So far, having to manually trigger a "Hard Refresh" after updating a secret of an application has been sufficient for us, so we've never investigated other options to integrate Vault with ArgoCD. Please note that it seems that other ArgoCD-Vault integrations suffer from the same problem, see e.g. https://argocd-vault-plugin.readthedocs.io/en/stable/usage/#refreshing-values-from-secrets-managers.

On a larger scale, a potential alternative to the current Project Syn secret management architecture would be to use something like External Secrets Operator, which reconciles secrets directly on Kubernetes based on their values in Vault.

Note that External Secrets Operator (ESO) is not a drop-in replacement for the current Project Syn secret management mechanism. Support for ESO would require that the component (or Commodore, in case we would want to support this style of secret management more broadly) has an option to generate ExternalSecret resources suitable for ESO instead of directly managing secrets which contain Kapitan Vault references.