Managing Helm CRDs with Terraform

Helm is a remarkable piece of technology to manage your Kubernetes deployments, and used with Terraform is perfect for deploying following the GitOps strategy.

However, Helm has a limitation: it doesn’t manage the lifecycle of Custom Resource Definitions (CRDs), meaning it will only install the CRD during the first installation of a chart. Subsequent chart upgrades will not add or remove CRDs, even if the CRDs have changed.

This can be a huge problem for a GitOps approach: having to update CRDs manually isn’t a great strategy, and makes it very hard to keep in sync with deployments and rollbacks.

For this very reason, I created a small Terraform module that will read from some online manifests of CRDs, and apply them. When parametrizing the version of the chart, it is simple to keep Helm Charts and CRDs in sync, without having to do anything manually.

Example

Let’s use Karpenter as an example of how to use the module. We want to deploy the chart with the Helm provider, and we use this new Terraform module to manage its CRDs as well:

 1resource "helm_release" "karpenter" {
 2  name            = "karpenter"
 3  namespace       = "karpenter"
 4  repository      = "https://charts.karpenter.sh"
 5  chart           = "karpenter"
 6  version         = var.chart_version
 7
 8  // ... All the other parameters necessary, skipping them here ...
 9}
10
11module "karpenter-crds" {
12  source  = "rpadovani/helm-crds/kubectl"
13  version = "0.1.0"
14  
15  crds_urls = [
16    "https://raw.githubusercontent.com/aws/karpenter/v${var.chart_version}/charts/karpenter/crds/karpenter.sh_provisioners.yaml",
17    "https://raw.githubusercontent.com/aws/karpenter/v${var.chart_version}/charts/karpenter/crds/karpenter.k8s.aws_awsnodetemplates.yaml"
18  ]
19}

As you can see, we parametrize the version of the chart, so we can be sure to have the same version for CRDs as the Helm chart. Behind the curtains, Terraform will download the raw file, and apply it with kubectl. Of course, the operator running Terraform needs to have enough permissions to launch such scripts, so you need to configure the kubectl provider.

The URLs must point to just the Kubernetes manifests, and this is why we use the raw version of the GitHub URL.

The source code of the module is available on GitHub, so you are welcome to chime in and open any issue: I will do my best to address problems and implement suggestions.

Conclusion

I use this module in production, and I am very satisfied with it: it brings under GitOps the last part I missed: the CRDs. Now, my only task when I install a new chart is finding all the CRDs, and build a URL that contains the chart version. Terraform will take care of the rest.

I hope this module can be useful to you as it is to me. If you have any question, or feedback, or if you would like some help, please leave a comment below, or write me an email at [email protected].

Ciao,
R.