Secrets Management with External Secrets, Argo CD and GitOps

Introduction

My previous tutorial discussed the benefits of managing secrets via GitOps using Argo CD and the Argo CD Vault Plugin. This guide explores using the alternative secrets management tool, External Secrets Operator¹ (ESO). Although the implementation of these tools differ, both essential solve the same problem. How to manage secrets via GitOps?

How does External Secrets Operator differ from Argo CD Vault Plugin?

Prerequisites

For this guide, you’ll need the following:

• A secret management system such as HashiCorp Vault (used in this guide), AWS Secrets Manager or Google Cloud Secret Manager (ESO supports many more systems)
• Beginner to intermediate knowledge of Argo CD, Kubernetes & Git/GitHub
• Terraform ≥ v0.15

How External Secrets Operator (ESO) Works

Simply put, the External Secrets Operator uses a controller to fetch secrets from an external secrets manager and create a Kubernetes secret inside a cluster.

ESO relies on a few Custom Resources to perform this function:

SecretStore - This resource determines how the secret manager of your choice is accessed. It’s namespaced and multiple SecretStore’s can be configured to access the same secret manager with differing credentials or other secret managers entirely.

ClusterSecretStore - Acts like the SecretStore resource except it’s cluster scoped and can be accessed by ExternalSecrets in any namespace.

ExternalSecret - This resource functions as a template for creating a secret. It references the SecretStore for access details and declares what secret to fetch from the external secrets manager.

ClusterExternalSecret - Like the ExternalSecret resource except it’s cluster scoped and can create secrets in multiple specified namespaces.

Here’s a high-level overview of what this guide aims to achieve; ESO working with Argo CD to automate the creation and updating of secrets in a Kubernetes cluster via GitOps:

1. Argo CD syncs the ExternalSecret manifest from a git repo to create an ExternalSecret resource inside the Kubernetes cluster.
2. The ESO Controller inspects the configuration of the synced ExternalSecret to determine which SecretStore to use and the path in the external API (Secrets Manager e.g. Vault) the secret it located on.
3. The Controller then queries the SecretStore for the credentials required to access the external API.
4. ESO fetches the secret(s) from the secrets manager at the path declared in the ExternalSecret.
5. Finally, ESO creates/syncs an actual Kubernetes secret resource inside the cluster.

Install External Secrets Operator on a Kubernetes Cluster

Install the External Secrets Operator either via Helm (using the helm command) or via Argo CD as an App:

Helm

Add the External Secretes repo:

helm repo add external-secrets https://charts.external-secrets.io

Install the External Secrets Operator:

helm install external-secrets \
   external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  --set installCRDs=true

Argo CD (my preference)

If you haven’t already, download and install the Argo CD CLI² then login to your Argo CD installation using the argocd command:

argocd login argocd.local
Username: admin
Password:
'admin:login' logged in successfully
Context 'argocd.local' updated

Create the external-secrets app using the following argocd command:

argocd app create external-secrets \
 --repo https://charts.external-secrets.io \
 --helm-chart external-secrets \
 --revision 0.5.9 \
 --dest-namespace external-secrets \
 --sync-option CreateNamespace=true \
 --dest-server https://kubernetes.default.svc

And then sync it to complete the installation on the cluster:

argocd app sync external-secrets

Create a secret containing your Secret Manager’s credentials

Before creating a SecretStore, you’ll need to create a secret containing your external secret manager’s credentials. The SecretStore references this secret. In turn, ESO uses the specified credentials from the SecretStore to authenticate with your external secrets manager. HashiCorp’s Vault with token-based authentication is the secrets manager used in this guide.

Run the following command to store your Vault token in a file named vault_token.txt:

echo -n 'VAULT_TOKEN' > ./vault_token.txt

Then use kubectl to create a secret named vault-token containing the token stored in the vault_token.txt file:

kubectl -n external-secrets create secret generic vault-token \
 --from-file=token=./vault_token.txt

Create a ClusterSecretStore

As previously mentioned, ESO offers two types of resources for defining how secret managers are accessed, SecretStore and ClusterSecretStore.

For this guide we’ll create a CluserSecretStore:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://vault.local:8200"
      path: "secret"
      version: "v2"
      auth:
        # points to a secret that contains a vault token
        # https://www.vaultproject.io/docs/auth/token
        tokenSecretRef:
          name: "vault-token"
          key: "token"
          namespace: external-secrets

The above ClusterSeceretStore resource defines the URL of the Vault server (line 8). It also references the name (vault-token, line 15) and namespace (external-secrets, line 17) of the secret containing the static token required to authenticate against Vault.

Create a Secret in Vault

Create an example secret in Vault using the following command:

vault kv put secret/demo-secrets tokenA=too_many_secrets tokenB=not_enough_secrets

Key                Value
---                -----
created_time       2022-08-18T23:48:59.493381453Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Create an ExternalSecret (using Git and Argo CD)

Create a git repository containing the following example ExternalSecret manifest (here’s my example repo):

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: demo-secret
  namespace: default
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: demo-secret
  data:
  - secretKey: token
    remoteRef:
      key: secret/demo-secrets
      property: tokenA

The ESO documentation provides a fully annotated example of an ExternalSecret resource, but the important lines from above are:

• 5. Namespace to create the secret resource in (default).
• 9. Name of the `ClusterSecretStore` created earlier (vault-backend).
• 12. Name of created secret.
• 14. Key name of the secret value for the created Kubernetes secret.
• 16. Path of the secret in the Vault server.
• 17. Key of the secret value to be retrieved from the Vault server.

Sync this ExternalSecret from the git repository to the cluster by creating an app named demo-secret in Argo CD:

argocd app create demo-secret --repo https://github.com/colinwilson/argocd-micro-example-apps.git --path external-secrets/ExternalSecret --dest-namespace default --dest-server https://kubernetes.default.svc --sync-policy automated

A breakdown of the flags specified in the above command:

• argocd app create demo-secret creates an Argo CD application named ‘demo-secret’
• --repo https://github.com/colinwilson/argocd-micro-example-apps.git specifies the URL of your git repository
• --path external-secrets/ExternalSecret specifies the path within the git repo to use as the app directory (my example)
• --dest-namespace default the namespace to deploy the app to (default)
• --dest-server https://kubernetes.default.svc deploy the app internally i.e. the same Kubernetes cluster Argo CD is installed on
• --sync-policy automated set the sync policy to automated (optional)

If successful you can view the demo-secret app in the Argo CD UI:

Click on app to see further details:

You’ll see from the UI that the secret was successfully created. You can confirm the generated Kubernetes secret value matches the value in Vault by running the kubectl command below:

kubectl -n default get secret demo-secret -o jsonpath="{.data.token}" | base64 -d; echo

too_many_secrets

Automatic Updates

Argo CD

Now the demo-secret app has been set up and configured with the automated sync policy enabled, any changes to the ExternalSecret in your git repository will be picked up by Argo CD’s polling and reflected in the cluster. You can create new Kubernetes secrets in the cluster simply by pushing additional ExternalSecret manifests to the same (Argo CD monitored) repository path.

External Secrets Operator

The ESO controller can monitor your Secret Manager(s) at regular intervals for changes in secret values and update the corresponding cluster secret values accordingly. This interval is set by spec.refreshInterval in the ExternalSecret. In the case of this guide, the interval for demo-secret is set to 15 seconds.

Summary

Diagram of how ESO resources relate to each other

Once Argo CD, ESO and an external secrets manager are setup, the workflow for managing secrets becomes streamlined.

• Updates to existing ExternalSecret’s in your git repository result in synchronisation of those changes on the cluster.
• Pushing a new ExternalSecret manifest to your Argo CD monitored repository creates a corresponding Kubernetes secret on the cluster.
• Updates to secrets in your secret manager are automatically synced to the Kubernetes secrets on the cluster.

This is GitOps! Syncing configurations in your Git repository with your infrastructure. All without any secrets stored in the repository itself.

This guide covered the basic features of ESO. Check out both the External Secrets and Argo CD documentation for more information on other features.

References

• External Secrets Operator Documentation
• Argo CD Documentation

Footnotes