Growing with Kubernetes: Implementing Continuous Deployment with ArgoCD

In our previous post, we talked about why we chose Kubernetes at Thyme Care to deploy and scale our applications. We emphasized the importance of managing the complexity of Kubernetes. In this post, we hope to share just one aspect of how we’re doing that, through the automation of our deploy process.

At Thyme Care, we’ve adopted GitOps. A single git repository serves as the source of truth for what’s deployed to our cluster at any given time, across all our environments. This infrastructure monorepo contains all our infrastructure-as-code, which includes Terraform in addition to our Kubernetes manifests. ArgoCD implements continuous deployment of our manifests, keeping our clusters in sync with the repository. All changes are done through pull requests, which enables us to enforce best practices and implement security controls through required approvals.

Managing deployments using Helm

The deployment of each of our applications on Kubernetes is described in a collection of manifests. We’ve chosen to package the manifests for each application as a Helm chart. This allows us to reason about the infrastructure for an application as a single unit. It also provides us with a templating language on top of the standard Kubernetes YAML that helps keep our manifests DRY.

reconciliationCronJob:
  enabled: true
  schedule: "0 6 * * *"

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetRequestsPerSecond: 10

Some sample configuration from one of our application’s Helm charts.

As a principle, we want to empower application developers to make changes to the deployments of their own applications. To accomplish this, the Helm charts for each service live in the application repos, alongside the code. As we grow, this will help keep ownership of a service’s infrastructure with the team that’s developing it. The templating language built into Helm charts also makes it easy to expose a set of high-level configuration that can be used to make common changes without needing to dive into the Kubernetes manifests themselves. This improves accessibility for those just learning the Kubernetes ecosystem.

Configuring continuous deployments with ArgoCD

Packaging each application as a Helm chart makes it easy to run helm install to deploy it to our cluster. In order to truly meet our needs, we need a tool to help us deploy continuously as we develop our applications. To accomplish this, we installed ArgoCD.

Each Application configured through the Argo UI monitors a repository for changes, and keeps your Kubernetes cluster in sync with the manifests in that repository. Argo understands a wide array of formats for Kubernetes manifests, including kustomize, jsonnet, plain YAML, and importantly, Helm.

The fastest approach for us to configure CD would be to add each of our repositories as an Application inside Argo, instructing it to install the Helm chart in that repo. The main disadvantage of this approach is that each of those applications would need to be managed individually. It would not be an easy process to add a new application, or to restore all of our applications in the event of a failure.

Instead, we implemented Argo’s “App of Apps” pattern. In addition to configuring Argo applications through the UI, it’s also possible to describe an Application as a Kubernetes manifest. To implement App of Apps, we created a Helm chart in our infrastructure repository. This chart contains only Argo Applications, each of which with instructions to install a Helm chart from one of our other repositories. When configuring Argo to manage just the single Helm chart in the infrastructure repository, it automatically detects the additional Applications and fans out to manage all of our services.

Our “App of Apps” in Argo.

This pattern allows us to follow the principle we outlined above of a single source of truth for all of our deployments. It also allows us to leverage Helm’s templating to abstract commonalities. After several months of iteration, we’ve parameterized the Application manifests such that our chart now contains just a single parameter listing the services to deploy:

applications:
  - name: thymebox-api
    group: applications
    repo: thymecare/thymeboxserver

    ingress:
      subdomain: api
      productionUrl: api.thymecare.com

    useIrsa: true

    additionalValues: |
      replicaCount: 2
      thyme:
        env: {{ .env }}

A sample service installed through our “App of Apps” chart.

Deploying a new application is now as easy as adding a new item to the list. While there’s a lot of power in this, we’ve also started to feel its limitations. Helm’s templating language is not a true coding language, and as we’ve built out a highly tailored API with a lot of logic, we’ve noticed that the templates are awkward, and difficult to test. We’ve started to explore technologies like cdk8s which would allow us to keep Helm charts as the foundation of our Kubernetes infrastructure while at the same time leveraging a language like Python to improve developer experience.

{{- /*
  The following enables ALB readiness gate injection when required.
    * if there is no ingress, does not add label
    * if there is an ingress, checks allowAlbReadinessGates, defaults to true

  `and` terms are eagerly evaluated. `(index (index ...))` handles a nil `.ingress`
*/ -}}
{{- if (and .ingress (index (index . "ingress" | default dict) "allowAlbReadinessGates" | default true)) }}

An example of some of the awkward logic in our app-of-apps chart. We’ve frequently turned to inline comments to help future developers understand what’s happening.

Next steps

ArgoCD has proven to be an effective tool for managing the services we deploy to Kubernetes in one place. Despite this, what we’ve outlined in this post is missing one key component that we need for a truly successful deploy process: a way to manage the version of the service deployed to each environment. For effective CI/CD, as we merge to each service’s main branch, we want to deploy the new version of that container to our staging environment. We also need a way to manually release changes to production after they’ve been thoroughly tested. In our next post, we’ll talk about how we’ve leveraged GitHub Actions to close the loop between our services repos and ArgoCD.