Verifying Signed Helm Charts

Introduction

The previous guide covered how Helm’s Chart Releaser Action can be configured to automate the signing of Helm charts. This brief guide describes how to verify signed Helm charts against their provenance file (*.prov) via the Helm CLI command or Terraform.

Provenance File

Signed Helm charts are usually hosted alongside an automatically generated provenance file. Using the Helm command helm verify (or Terraform) the integrity and origin of a chart can be verified against the public PGP key of the chart publisher. The contents of a typical provenance file consist of:

The Helm chart’s Chart.yaml file contents
A SHA256 hash of the chart package
The PGP signature block

See the example below:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

apiVersion: v2
appVersion: 1.16.0
description: A Helm chart for Kubernetes
name: mychart
type: application
version: 0.1.0

...
files:
  mychart-0.1.0.tgz: sha256:690fbec58c4b1b5a039594041bd35022aec5a59dc1c0325a88606ee44a80eb71
-----BEGIN PGP SIGNATURE-----

wsFcBAEBCgAQBQJh8wouCRBtNFtJj6zhpgAA8YAQAJGaj2pVqCGjZzspVC7vdqIQ
f0P/ZkA4MnFRltzh5+vUo7F7Hk2b4j+dIK1Jlq6pIiyQ39I8UMATUIG5f+JKyA2G
HBt8X5E61MbXtLen1hsiqKiTkemXqZAuYrTMQNXY/+mpTnj0lqMg77JD/m9YwZkY
UP8Omzdvv6rAOAKRhcXADatj3Zl3IoRN+n4+RoGZJ8SYGeVyWJurmoOqZYtqBlkS
tah/ZJKE4SqCNtk9SSboDpyeq0LpPRvguKqxiUT8dGjs0rujaYFAXRquIN4+I2ev
XotQxHsa4bOKhgXC3GYc6pMMX8EZPGsHgGehkAD8C5S6npQw5w+pvpfBV3rXmEn9
KzlLEWUuDGpEtUqdTNE3yypJxP+B60V8rw0lBUiQbuVqRmwsMNEgZVjAO2aue55+
kVijkT3QsV5MudopTf9OMJy69Y12iR6u2sqYpbu7Co45zSWhBm+ExOHwGfifV6AQ
EOImJgmEPd0E1pRgDNVfrTz/Aq8UtToAj04WLPiOTDrPsVh5QbhuXk64INBWhFAJ
5AlQT+OIdW07WDHjUuEH2J4H42NkjPNV5ODpCWosquczZiQWdXXM/oGgPZ7U5a37
9g9qKuJsM4R+eJm0d3J6VV3zLJ5RTr5DL48lHaDFqmnn1s7B0HUyaamglyhvL5JG
HdATzoPnf9+J0pLXy117
=iqso
-----END PGP SIGNATURE-----

Step 1 - Import Publisher’s Public PGP Key

Before a signed chart can be verified, the publisher’s/signer’s public key must be present in the local keyring. By default, GnuPG version 2 or greater stores keyrings in the keybox format¹ (.kbx), which is not currently supported by Helm when verifying signed charts. So, it’s necessary to convert/export the keyring to GnuPG’s legacy format .gpg e.g. pubring.gpg.

Since most public keys are hosted in the OpenPGP ASCII armor format (*.asc), the following command can be used to download and convert such a public key to the .gpg format in a single step:

curl https://keybase.io/hashicorp/pgp_keys.asc | gpg --dearmor > /home/colin/.gnupg/pubring.gpg

curl outputs the hosted key, gpg --dearmor converts that output to gpg’s binary format², which is then saved to the user’s GnuPG home directory as pubring.gpg.

Alternatively, the publisher’s key can also be imported normally and then exported to the .gpg format:

Import Key using Curl

curl https://keybase.io/hashicorp/pgp_keys.asc | gpg --import

Import Key from a Keyserver via its Fingerprint

gpg --keyserver pgp.mit.edu --recv-keys 72D7468F

Import Key using Keybase Username via the Keybase App

keybase pgp pull hashicorp

Once the public key has been imported to your local keyring using one of the above methods, you can export it to the .gpg format using the command below:

Export Public Keyring in GPG format

gpg --output pubring.gpg --export 72D7468F

Step 2 - Verify Signed Helm Chart

Now that the publisher’s public key is part of your local keyring you can use Helm or Terraform to verify charts signed by that publisher.

Verify Signed Helm Charts using Helm CLI

First add the desired helm chart repo e.g. HashiCorp:

helm repo add hashicorp https://helm.releases.hashicorp.com

Then use helm fetch --verify to verify the signed chart:

helm fetch --verify hashicorp/vault --version 0.19.0 --keyring pubring.gpg
Signed by: HashiCorp Security (hashicorp.com/security) <security@hashicorp.com>
Using Key With Fingerprint: C874011F0AB405110D02105534365D9472D7468F
Chart Hash Verified: sha256:fb7a99fc3d29ccb387fcf741a304124ae0c665a1d3a238024b4a54593f1fa464

If the keyring containing the public key associated with the signed chart is not in the default location, you may need to point to the keyring with --keyring PATH³

If the integrity of the chart has been compromised in any way, the helm verify command with throw an error citing a mismatch in the expected hash of the chart package:

helm fetch --verify hashicorp/vault --version 0.19.0 --keyring pubring.gpg
Error: sha256 sum does not match for vault-0.19.0.tgz:
"sha256:fb7a99fc3d29ccb387fcf741a304124ae0c665a1d3a238024b4a54593f1fa464" !=
"sha256:690fbec58c4b1b5a039594041bd35022aec5a59dc1c0325a88606ee44a80eb71"

If the requested chart is not signed, the helm fetch command throws an error:

Error: Failed to fetch provenance "https://helm.releases.hashicorp.com/vault-0.19.0.tgz.prov"

Verify Signed Helm Charts in Terraform

The helm_release resource of the helm Terraform provider can verify charts before installation by setting the verify key value to true:

resource "helm_release" "vault" {
  name             = "vault"
  namespace        = "vault"
  create_namespace = true
  repository       = "https://helm.releases.hashicorp.com"
  chart            = "vault"
  version          = "0.19.0"
  verify           = true
  keyring          = "hashicorp_pubring.gpg"
}

If the chart is successfully verified when terraform apply is run, the chart is installed without incident. However, if the chart has been tampered with in any way you’ll receive an error and the chart deployment will fail:

helm_release.vault: Creating...
╷
│ Error: openpgp: invalid signature: RSA verification failure
│
│   with helm_release.vault,
│   on main.tf line 16, in resource "helm_release" "vault":
│   16: resource "helm_release" "vault" {
│
╵

Summary

That’s it! You can now verify charts using helm verify or Terraform. These methods can be integrated into GitOps workflows in many ways to ensure charts are always verified before installation.