Managing Helm CRDs with Terraform

Helm is a remarkable piece of technology to manage your Kubernetes deployments, and used along Terraform is perfect for deploying following the GitOps strategy.

Author's pictureAug 27, 2022 | Riccardo Padovani | [email protected]

Terraform Helm CRDs manager

Illustration by unDraw+.

What’s GitOps? Great question! As this helpful, introductory article summarize, it is Infrastructure as Code, plus Merge Requests, plus Continuous Integration. Follow the link to explore further the concept.

However, Helm has a limitation: it doesn’t manage the lifecycle of Custom Resource Definitions (CRDs), meaning it will only install the CRD during the first installation of a chart. Subsequent chart upgrades will not add or remove CRDs, even if the CRDs have changed.

This can be a huge problem for a GitOps approach: having to update CRDs manually isn’t a great strategy, and makes it very hard to keep in sync with deployments and rollbacks.

For this very reason, I created a small Terraform module that will read from some online manifests of CRDs, and apply them. When parametrizing the version of the chart, it is simple to keep Helm Charts and CRDs in sync, without having to do anything manually.

Example

Karpenter is an incredible open-source Kubernetes node provisioner built by AWS. If you haven’t tried it yet, take some minutes to read about it.

Let’s use Karpenter as an example on how to use the module. We want to deploy the chart with the Helm provider, and we use this new Terraform module to manage its CRDs as well:

resource "helm_release" "karpenter" {
  name            = "karpenter"
  namespace       = "karpenter"
  repository      = "https://charts.karpenter.sh"
  chart           = "karpenter"
  version         = var.chart_version

  // ... All the other parameters necessary, skipping them here ...
}

module "karpenter-crds" {
  source  = "rpadovani/helm-crds/kubectl"
  version = "0.1.0"
  
  crds_urls = [
    "https://raw.githubusercontent.com/aws/karpenter/v${var.chart_version}/charts/karpenter/crds/karpenter.sh_provisioners.yaml",
    "https://raw.githubusercontent.com/aws/karpenter/v${var.chart_version}/charts/karpenter/crds/karpenter.k8s.aws_awsnodetemplates.yaml"
  ]
}

As you can see, we parametrize the version of the chart, so we can be sure to have the same version for CRDs as the Helm chart. Behind the curtains, Terraform will download the raw file, and apply it with kubectl. Of course, the operator running Terraform needs to have enough permissions to launch such scripts, so you need to configure the kubectl provider.

The URLs must point to just the Kubernetes manifests, and this is why we use the raw version of the GitHub URL.

The source code of the module is available on GitHub, so you are welcome to chime in and open any issue: I will do my best to address problems and implement suggestions.

Conclusion

I use this module in production, and I am very satisfied with it: it brings under GitOps the last part I missed: the CRDs. Now, my only task when I install a new chart is finding all the CRDs, and build a URL that contains the chart version. Terraform will take care of the rest.

I hope this module can be useful to you as it is to me. If you have any question, or feedback, or if you would like some help, please leave a comment below, tweet me @rpadovani93 or write me an email at [email protected].

Ciao,
R.

Uh-oh, there should be a comment box here! Please double-check your browser is not blocking my comment system!