Skip to main content
Mandacode mandacode
Building a Home Cluster 2: Automating Cluster Setup with Terraform
·
Kubernetes Proxmox VE Talos Linux Terraform Terragrunt

Building a Home Cluster 2: Automating Cluster Setup with Terraform

From Creating VMs to Cluster Setup on Proxmox VE Using Terraform

Following up on the hardware selection and configuration article for home clusters, this time we'll cover the process of automating cluster setup on Proxmox VE nodes using Terraform.

What is Terraform?

Terraform is an IaC (Infrastructure as Code) tool that allows infrastructure to be managed as code.

I have previous experience managing infrastructure directly while running an AWS environment or a home lab. To operate a server, I manually accessed the shell to configure necessary dependencies and infrastructure, spending significant time on manual tasks such as adjusting instance specifications via the AWS console.

After discovering Terraform, I concluded that it is a tool that maximizes automation and convenience. Particularly when managing cloud infrastructure like Kubernetes clusters or hypervisors like Proxmox VE, using code to manage infrastructure proved much more advantageous in terms of scalability than directly controlling via a dashboard. It also offered benefits like tracking changes and avoiding repetitive tasks when building large or complex infrastructures in the future.

intro-terraform-workflow

(Terraform workflow)

Terraform consists of three main steps.

  1. Write: Define infrastructure components with configuration files.

  2. Plan: Compare with the current infrastructure setup to check for changes.

  3. Apply: Implement changes in the infrastructure and update the state file.

Infrastructure is defined and managed declaratively, allowing for consistent and convenient setups without repetitive tasks, thanks to its modular structure.

$ tree minimal-module/
.
├── README.md
├── main.tf
├── variables.tf
├── outputs.tf

$ tree complete-module/
.
├── README.md
├── main.tf
├── variables.tf
├── outputs.tf
├── ...
├── modules/
│   ├── nestedA/
│   │   ├── README.md
│   │   ├── variables.tf
│   │   ├── main.tf
│   │   ├── outputs.tf
│   ├── nestedB/
│   ├── .../
├── examples/
│   ├── exampleA/
│   │   ├── main.tf
│   ├── exampleB/
│   ├── .../

(Terraform Module Structure)

What is Terragrunt?

Terraform is an excellent tool for conveniently setting up infrastructure on its own. However, you have to repeatedly write code for different deployment environments, and since it's a declarative tool, complexities can arise if there are dependencies among components. Terragrunt is a wrapper tool for Terraform that solves these issues.

Unit Structure

The unit refers to terragrunt.hcl directory and is the smallest deployable unit in Terragrunt.

terraform {
  # Deploy version v0.0.3 in stage
  source = "git::git@github.com:foo/modules.git//app?ref=v0.0.3"
}

inputs = {
  instance_count = 3
  instance_type  = "t4g.micro"
}

Configured as declared above, allowing for input values to pre-defined modules.

Includes

A method for configuring provider settings once for multiple environments or commonly, instead of repeatedly for each environment.

Typically, at the root, a root.hcl is created for global settings.

# root.hcl
remote_state {
  backend = "s3"
  config = {
    bucket         = "my-tofu-state"
    key            = "${path_relative_to_include()}/tofu.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "my-lock-table"
  }
}

generate "provider" {
  path = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents = <<EOF
provider "aws" {
  assume_role {
    role_arn = "arn:aws:iam::0123456789:role/terragrunt"
  }
}
EOF
}

# app/terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
}

Besides the above structure, it's also possible to declare files like root.hcl per environment (prod/env) for environment-specific settings.

include "root" {
  path = find_in_parent_folders("root.hcl")
}

include "k8s-providers" {
  path = find_in_parent_folders("k8s-providers.hcl")
}

# Storage needs cert-manager for CSI webhook validation
dependency "security" {
  config_path  = "../05_security"
  skip_outputs = true
}

terraform {
  source = "${get_repo_root()}/modules/rook-ceph"
}

inputs = {
  kubeconfig_path = "${get_repo_root()}/.cache/production/kubeconfig"
}

State Backend

In Terraform, state is managed locally, which can be unsuitable for work environments or teams. By using a State Backend, it allows for remote management of state by integrating with external storage like S3, supporting variables, functions, and expressions for stable and consistent state management.

remote_state {
  backend = "s3"
  config = {
    bucket       = "my-tofu-state"
    key          = "${path_relative_to_include()}/tofu.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

Applying Infrastructure Configuration

The current home cluster being set up involves creating several virtual machines on Proxmox and configuring a Kubernetes cluster using Talos Linux.

Because multiple virtual machines are used, automation and consistency were necessary, which were addressed by using such IaC tools.

At this point, it was planned for services to be deployed via ArgoCD, and since it was intended to be composed as a separate project, the infrastructure and service domains were decided as follows.

  • Infrastructure Domain - Necessary setup for cluster operation

  • Service Domain - Services running on the configured cluster

Module Structure

Considered elements necessary for running services on the cluster— such as VMs, CNI, Storage (Ceph), Secret Manager, ArgoCD—as part of the infrastructure domain and set up the following Terraform modules.

> tree -L 1 modules/
modules/
├── bootstrap
├── cluster
├── gitops
├── network
├── secrets
└── storage

> cat modules/network/main.tf                                                                                                                                                                    
resource "helm_release" "cilium" {
  name       = "cilium"
  repository = "https://helm.cilium.io/"
  chart      = "cilium"
  version    = var.cilium_version
  namespace  = "kube-system"

  wait          = true
  wait_for_jobs = true
  timeout       = 600

  set = [
    {
      name  = "kubeProxyReplacement"
      value = "true"
    },
    {
      name  = "k8sServiceHost"
      value = "localhost"
    },
    {
      name  = "k8sServicePort"
      value = "7445"
    },
    {
      name  = "ipam.mode"
      value = "kubernetes"
    },
    {
      name  = "cgroup.autoMount.enabled"
      value = "false"
    },
    {
      name  = "cgroup.hostRoot"
      value = "/sys/fs/cgroup"
    },
...

> cat modules/network/outputs.tf                                                                                                                                                                 
output "cilium_status" {
  description = "Status metadata for the Cilium Helm release"
  value = {
    release   = helm_release.cilium.status
    version   = helm_release.cilium.version
    namespace = helm_release.cilium.namespace
  }
}

output "cilium_lb_pool_name" {
  description = "Name of the Cilium LoadBalancer IPPool"
  value       = kubectl_manifest.cilium_loadbalancer_ip_pool.name
}

output "cilium_l2_policy_name" {
  description = "Name of the Cilium L2 Announcement Policy"
  value       = kubectl_manifest.cilium_l2_announcement_policy.name
}

Each module was configured by responsibility, as shown above, and outputs.tf was created so that results can be used in subsequent modules.

Terragrunt Hierarchy

Terragrunt is needed to pass results from each module to the next, and to specify configurations overall or by environment.

Additionally, in such infrastructure projects, each code must be intuitive and predictable in operation. Accordingly, the folder structure and dependencies were set to apply sequentially as follows.

> tree -L 2 environments/                                                                                                                                                                        
environments/
└── production
    ├── 01_bootstrap
    ├── 02_cluster
    ├── 03_network
    ├── 04_storage
    ├── 05_secrets
    ├── 06_gitops
    ├── env.hcl
    └── k8s-providers.hcl

> cat environments/production/03_network/terragrunt.hcl                                                                                                                                          
include "root" {
  path = find_in_parent_folders("root.hcl")
}

include "k8s-providers" {
  path = find_in_parent_folders("k8s-providers.hcl")
}

# CNI must deploy after cluster is bootstrapped
dependency "cluster" {
  config_path = "../02_cluster"
  skip_outputs = true
}

terraform {
  source = "${get_repo_root()}/modules/network"
}

inputs = {
  kubeconfig_path = "${get_repo_root()}/.cache/production/kubeconfig"

  cilium_version = "1.19.4"
  cilium_lb_pool = "192.168.0.80-192.168.0.99"
}

Conclusion

In this article, we introduced the method of automating cluster configuration using Terraform and Terragrunt. In the next article, we will cover deployment strategies using ArgoCD.

Additionally, we will further explain network configuration using Cilium CNI and how to integrate AWS OIDC.

Thank you :)

Other posts in this folder