Helm Post Renderers: Because Sometimes You Need to Ship Fast (and Fix It Later)

When you work at a startup, a lot of times you need to “move fast and break stuff”. Unfortunately, a lot of the times the “break stuff” part quickly catches up to you and you’re sitting there with months-worth of refactoring work that needs to be done. Be it code, hardware design, or DevOps, it’s the universal truth for engineers far and wide.

In this article, instead of showing you how to properly plan, execute, and iterate on refactoring efforts, I will introduce you to an incredible way to put a giant bandage all over your Helm charts so that instead of being a competent DevOps Engineer, you can play along with the enshitification of your codebase, like me!

It all started when my manager came to me with a very bizarre task - make the entire product run on a laptop (mind you, it’s got ✨AI✨ in it).

I took a long, hard look at our helm charts - hardcoded values, inconsistent resources requests/limits, Bitnami’s image nuking was approaching fast, and the only storage class I could use was local-path (hooray, k3s!).

In a perfect world, I would’ve rolled up my sleeves and refactored these charts until they’d begged for me to stop, but alas, with limited time (and sanity) on my hands, I needed to find another solution, which is exactly when God has bestowed onto me the wonderful gift of the Helm post renderer, a feature which lets you change your charts on the fly, without changing their source “code”.

What’s Helm post renderer?

At its core, Helm is a fancy templating engine. It mashes together templates with corresponding user-supplied values, and bam - shiny new kubernetes manifests are deployed. However, helm lets you provide “last-minute” changes on these manifests - after they’re templated, but before they’re applied.
When running a helm command, you can use the --post-renderer flag and supply it with a path to a script. Helm will pass the templated manifests through the script and only then will it apply them.

For example, I wrote a bash script, utilizing yq, to perform a couple key changes on our manifests:

#!/usr/bin/env bash

set -euo pipefail

yq eval '
  (
    select(.kind == "Deployment" or .kind == "StatefulSet" or .kind == "DaemonSet")
    |
    (
      .spec.template.spec.containers[]? |= (
        .resources = {} |
        .image = (
          .image
          | sub("^docker\\.io/bitnami/"; "docker.io/bitnamilegacy/")
          | sub("^bitnami/"; "bitnamilegacy/")
        ) |
        .imagePullPolicy = "IfNotPresent"
      )
      |
      .spec.template.spec.initContainers[]? |= (
        .resources = {} |
        .image = (
          .image
          | sub("^docker\\.io/bitnami/"; "docker.io/bitnamilegacy/")
          | sub("^bitnami/"; "bitnamilegacy/")
        ) |
        .imagePullPolicy = "IfNotPresent"
      )
      |
      del(.spec.template.spec.affinity)
      |
      del(.spec.template.spec.nodeSelector)
      |
      (
	.spec.volumeClaimTemplates[]?.spec.storageClassName = "local-path"
      )
    )
  )
  //
  (
    select(.kind == "PersistentVolumeClaim")
    |
    .spec.storageClassName = "local-path"
  )
  //
  (
    select(.kind == "ServiceAccount")
    |
    .imagePullSecrets = [{"name": "ecr-creds"}]
  )
  //
  .
' -

Yes, this is quite a lot, but it’s actually a pretty clever way to manipulate the manifests using yq instead of having all my hair turn white from 0.2 seconds of interacting with awk.

This script essentially takes helm’s output, then firstly checks if the “kind” field in the manifest is either a Deployment, StatefulSet, or a DaemonSet. If so, it will remove all resources requests and limits (yalla balagan!), make sure the imagePullPolicy is IfNotPresent (instead of, for example, Always), substitute the bitnami images for bitnamilegacy images, remove all node selectors and affinities, and make all PVCs’s storage classes be local-path, as well as patching all service accounts to use the secrets for pulling images from ECR; you get the gist.

Essentially, my script forces specific fields to be equal specific values, across all identical resource types.

I would like to close on a bit of a more serious tone - this is definitely not something that I would advise anyone to use in production. Working with a helm post renderer creates more clutter and is overall a bad practice, since it enables you to basically ignore values, resulting in your source-of-truth being not quite so truth-y.

Alas, this is a nifty little trick that saved my ass in a pinch, enabling me to ship this PoC a lot sooner than I could have without it.

If you liked this post, you may want to check out my other DevOps-related posts!