With the quick adoption of Kubernetes as a go-to container orchestration platform, ensuring your Kubernetes infrastructure is secure and efficient has never been more important. This guide will provide you with key insights into Helm, its role in managing Kubernetes deployments, and the security measures necessary to protect your applications.

Why Use Helm?

Helm is an open-source project under the Cloud Native Computing Foundation (CNCF) that simplifies the management of an application's lifecycle within Kubernetes deployment. Deploying an application typically requires steps such as setting up a Deployment, adjusting the application configuration via modifications performed to a ConfigMap or Secret, and deploying Custom Resource Definitions (CRDs) if needed, among other tasks.

‍

While DIY solutions, like creating your deployment files or using 'envsubst' for runtime substitution, are possible, they are often not the best solutions. Others, like Kustomize and Jsonnet, aid this process but come with limitations.

‍

Despite security not being the main focus of Helm, its features can contribute significantly towards securing Kubernetes deployments. The Helm project provides a simple command interface, a templating engine (based on the Go template language) with over 60 functions, and a repository for charts, known as the Artifact Hub, which hosts over 9000 available charts.

‍

These allow for the packaging of complex applications by specifying certain parameters, making the applications easily deployable. For instance, you can use the "architecture=replication parameter" to deploy a MySQL cluster with replication.

‍

Helm also offers advanced features like hooks, which let specific tasks run at particular points during the deployment process.

‍

Furthermore, integration with GitOps tools such as ArgoCD and Flux are possible, as is the use of library charts or named templates. These all contribute to leveraging Helm for secure Kubernetes deployments.

How to Secure Helm

Despite the wide variety of benefits Helm brings to Kubernetes application lifecycle management, it’s important to acknowledge that most Helm charts do not come with secure default settings.

‍

Securing Helm can be approached from different perspectives, depending on the user's focus, whether it involves consuming existing Helm charts, handling the Kubernetes objects generated by the charts, or creating custom Helm charts. Each of these situations demands tailored security practices to enhance the overall security of Helm.

Custom Helm Charts

Creating custom Helm charts involves several general and security-focused recommendations.

Git repository

Storing charts in a Git repository is beneficial for tracking changes and conveniently rolling back when necessary.

Proper repository for Helm charts

Despite HTTP serving as a viable option, HTTPS is the preferred method, in line with security considerations.

Using linters

Tools like "helm lint" or any other linter can verify the correctness of Helm charts, preventing issues in production environments due to minor errors. For instance, in a basic Helm chart without a proper "version", the lint will complain about it: 

<pre class="codeWrap"><code>apiVersion: v2
name: hello-world
description: A Helm chart for Kubernetes
type: application
appVersion: "0.0.1"
$ helm lint --strict
==&gt Linting .
[ERROR] Chart.yaml: version is required
[INFO] Chart.yaml: icon is recommended
[ERROR] templates/: validation: chart.metadata.version is required
[ERROR] : unable to load chart
validation: chart.metadata.version is required
Error: 1 chart(s) linted, 1 chart(s) failed
</code></pre>

Consistent versioning

Helm recommends following the SemVer2 standard for versioning charts. This practice ensures reproducibility and quicker response times when updates are necessary due to vulnerability discoveries. Two versions can be managed: the chart "version"(specified in the "Chart.yaml") and the application version ("appVersion").

<pre class="codeWrap"><code>$ helm show chart falcosecurity/falco &#124; grep -E
'^version&#124;^appVersion'
appVersion: 0.33.0
version: 2.2.0
</code></pre>

Creating test scenarios

Validate Helm deployment success by creating Kubernetes objects, providing a practical way to test the deployed chart using helm test &lt;RELEASE_NAME&gt;. This test can be something simple like a simple pod running on the same namespace quering your application API.

<pre class="codeWrap"><code>apiVersion: v1
kind: Pod
metadata:
 …
 annotations:
   "helm.sh/hook": test
spec:
 containers:
   - name: wget
     image: busybox
     command: ['wget']
     args: ['&#123;&#123; .Values.service.name &#125;&#125;:&#123;&#123; .Values.service.port &#125;&#125;']
 restartPolicy: Never
</code></pre>

Typically, these tests are saved in the "templates/tests/" folder and come with a "helm.sh/hook: test"annotation to signify their purpose.

<pre class="codeWrap"><code>$ helm test hello-world
NAME: hello-world
...
Phase:          Succeeded
$ kubectl get po -n hello-world
NAME                           READY   STATUS      RESTARTS   AGE
hello-world-78b98b4c85-kbt58   1/1     Running     0          91s
hello-world-test               0/1     Completed   0          67s
</code></pre>

Sign your charts and verify them with "helm install --verify"

Securing the software supply chain often involves ensuring the integrity of the software components. Helm facilitates this by allowing charts to be signed using "helm package --sign" for the creation of PGP-based digital signatures and storing them in provenance files.

Example:

<pre class="codeWrap"><code>$ helm package --sign --key 'Eduardo Minguez' hello-world --keyring ~/.gnupg/secring.gpg
Password for key "Eduardo Minguez (gpg key) <edu@example.com>" >
Successfully packaged chart and saved it to: /home/edu/git/my-awesome-stuff/hello-world-0.0.1.tgz</code></pre>

The provenance file will look like this: 

<pre class="codeWrap"><code>$ cat hello-world-0.0.1.tgz.prov
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
...
name: hello-world
...
files:
 hello-world-0.0.1.tgz:
sha256:b3f75d753ffdd7133765c9a26e15b1fa89784e18d9dbd8c0c51037395eeb332e
-----BEGIN PGP SIGNATURE-----
wsFcB…
-----END PGP SIGNATURE-----%
</code></pre>

In the case of the signature not matching the helm will complain: 

<pre class="codeWrap"><code>$ helm verify hello-world-0.0.1.tgz
Error: openpgp: invalid signature: hash tag doesn't match
</code></pre>

Charts can then be verified using "helm install --verify":

<pre class="codeWrap"><code>$ helm install --verify myrepo/mychart-1.2.3
</code></pre>

You can also just pull the chart to verify it:

<pre class="codeWrap"><code>$ helm pull --verify myrepo/mychart-1.2.3
</code></pre>

Note that the public key must be trusted and publicly available for this verification process to proceed.

You have to make it publicly available or the process will fail:

<pre class="codeWrap"><code>$ helm pull --verify myrepo/mychart-1.2.3
Error: openpgp: signature made by unknown entity
$ cat security/pubkey.gpg &#124; gpg --import --batch
$ helm pull --verify myrepo/mychart-1.2.3
Signed by:..
</code></pre>

Automating all previous processes

Automating steps such as testing, versioning, signing, and releasing through a CI/CD pipeline maintains consistency with best practices and mitigates potential problems stemming from manual changes. The helm/charts-repo-actions-demo can serve as a reference point for establishing a GitHub actions workflow to test and release a chart.

<pre class="codeWrap"><code>- name: Run chart-releaser
       uses: helm/chart-releaser-action@v1.4.0
       with:
         charts_dir: charts
         config: cr.yaml
       env:
         CR_TOKEN: "$&#123;&#123; secrets.GITHUB_TOKEN &#125;&#125;"
</code></pre>

These practices contribute to the more secure and efficient management of custom Helm charts.

Kubernetes Objects

Kubernetes objects, such as those created via templates, do not have inherent security measures, leaving users free to employ any practices they choose. Therefore, it's crucial to consider and incorporate several security-focused recommendations in dealing with Kubernetes objects.

Employ Role-based access control (RBAC)

RBAC restricts an object's permissions, which helps curtail risks that can arise from giving blanket permissions to all objects—for instance, using "cluster-admin" control for everything. The falcosidekick Helm chart provides a useful example of this, creating a Role, ServiceAccount, and RoleBinding to limit permissions in K8s Deployment.

<pre class="codeWrap"><code>---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: &#123;&#123; include "falcosidekick.fullname" . &#125;&#125;
…
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: &#123;&#123; include "falcosidekick.fullname" . &#125;&#125;
…
rules:
- apiGroups:
   - ""
 resources:
   - endpoints
 verbs:
   - get
‍…
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
…
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: &#123;&#123; include "falcosidekick.fullname" . &#125;&#125;
subjects:
- kind: ServiceAccount
 name: &#123;&#123; include "falcosidekick.fullname" . &#125;&#125;
…
</code></pre>

Sane defaults

Always ensure the defaults don't create easy openings for potential security breaches. For instance, if your chart involves a MySQL pod, avoid using a default password; instead, either generate one randomly or require users to specify their own. However, there are considerations for handling upgrades, which are discussed in a relevant GitHub issue and blog post. Functions like 'lookup' and resource policy annotation can prevent overwriting during upgrades.

<pre class="codeWrap"><code>&#123;&#123;- if not (lookup "v1" "Secret" .Release.Namespace "hello-world") &#125;&#125;
apiVersion: v1
kind: Secret
metadata:
 name: mysecret
 annotations:
   "helm.sh/resource-policy": "keep"
type: Opaque
stringData:
 password: &#123;&#123; randAlphaNum 24 &#125;&#125;
&#123;&#123;- end &#125;&#125;
</code></pre>

Reduce the attack surface

This can be achieved by keeping deployments compact and activating features/components as needed through flags or values. For example, the Falco Helm chart does not inherently deploy falcosidekick; however, it provides an option to enable it easily.

<pre class="codeWrap"><code>falcosidekick:
 # -- Enable falcosidekick deployment.
 enabled: false
</code></pre>

Then it is used in the Chart:

<pre class="codeWrap"><code>dependencies:
 - name: falcosidekick
   condition: falcosidekick.enabled
</code></pre>

In addition to the above, all relevant Kubernetes recommendations should also be applied, including the CIS benchmarks for this instance. For optimum results, integrate a routine scan of your Kubernetes object definitions to ascertain adherence to best practices.

If your preferred tool isn't compatible with Helm charts, the Helm template command can be used to render Kubernetes objects with the before conducting a verification.
<pre class="codeWrap"><code>$ helm template falco falcosecurity/falco --namespace falco --create-namespace --set driver.kind=ebpf > all.yaml</code></pre>

Then you can verify them:

<pre class="codeWrap"><code>$ myawesometool --verify all.yaml</code></pre>

Using Helm Charts

Using Helm charts offers a streamlined approach to managing Kubernetes applications, but it requires careful navigation.

Here are some practices to follow:

Validate through Helm template command: When working with Helm charts, especially those from third-party sources, don't place implicit trust in them. As we've observed, the "helm template" command is beneficial as it renders and outputs the Kubernetes objects that the Helm charts create. Prior to implementing these in your Kubernetes cluster, it's prudent to review the rendered results. By doing this, you're ensuring a safer and more reliable rollout of the Helm charts in your Kubernetes cluster.

Verify chart signatures: Ensure the authenticity of the charts you use by verifying digital signatures with the "helm verify" command.

Uninstall unused releases: To reduce potential attack surfaces, uninstall any Helm release that isn't in use anymore.

Stay updated: Keep your Helm Charts, "helm binary", and plugins up-to-date. You should use the latest versions, which usually come with the latest bug fixes and enhancements. If the Helm chart deploys a container image found to be vulnerable, updating it is crucial. Similarly, updates should extend to subcharts as well. To verify changes an upgrade will introduce, you can use the helm diff plugin.

<pre class="codeWrap"><code>$ helm diff --install foo --set image.tag=1.14.0 .</code></pre> command.

Or you can render manifests using <pre class="codeWrap"><code>helm template</code></pre> and use <pre class="codeWrap"><code>kubectl</code></pre> to make the diffs.

<pre class="codeWrap"><code>$ helm template --is-upgrade --no-hooks --skip-crds foo --set image.tag=1.14.0 . &#124; kubectl
diff --server-side=false -f -
</code></pre>

Encrypt kubernetes secrets

Contrary to their name, Kubernetes secrets are not secret, and Base64 is not an encryption algorithm but an encoding one. Some prefer to store encrypted secrets alongside the code, while others prefer separate storage. Consider available alternatives such as the helm-secrets plugin, Hashicorp Vault, Bitnami's Sealed secrets, Mozilla's sops, and other DIY solutions. Let us show you an example of helm secrets with AWS SSM via vals.

Creating an AWS SSM SecureString object:

<pre class="codeWrap"><code>$ aws ssm put-parameter --name mysecret --value "secret0" --type SecureString</code></pre>

Checking the helm parameter required: ‍

<pre class="codeWrap"><code>$ cat hello-world/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
 name: mysecret
type: Opaque
stringData:
 password: &#123;&#123; .Values.secretdata &#125;&#125;
</code></pre>

Verifying it: 

<pre class="codeWrap"><code>$ helm secrets --backend vals template hello-world -s templates/secret.yaml --set
‍ secretdata="ref+awsssm://mysecret"
---
# Source: hello-world/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
 name: mysecret
type: Opaque
stringData:
 password: secret0
</code></pre>

After rendering the chart, you can run nearly any command, like a script used to apply environment variables via kustomize, verifying a specific parameter, or executing Windows batch scripts. Your means to adapt and manipulate are extensive.

Example:

<pre class="codeWrap"><code>$ helm install mychart my-chart --post-renderer my-script.sh</code></pre>

The "my-script.sh" can be mostly anything including the examples we already listed.

Helm Security: Conclusion

Helm is an invaluable tool for orchestrating the lifecycle of Kubernetes applications. Despite its lack of inbuilt security measures, through adherence to recommended practices, users can significantly enhance the security profiles of their Helm chart deployments. This article provided invaluable insights into these practices and other security recommendations for both using and creating Helm charts.