Build & Test a Kubernetes Operator with Go and CircleCI

When companies migrate to Kubernetes on public clouds (AWS EKS, GKE), they often fall into a massive financial trap: relying entirely on expensive "Managed Services" (like AWS RDS for PostgreSQL or MSK for Kafka) simply because stateful applications are notoriously difficult to manage manually.

To escape this cloud tax, elite SRE teams build Kubernetes Operators.

By writing a custom Operator in Golang, you programmatically encode human operational knowledge—backup routines, self-healing mechanics, automated failover, and scaling policies—directly into the cluster control plane. This enables hosting complex stateful workloads on highly cost-effective Dedicated Bare Metal Servers, effectively building your own "Private Managed Cloud" at flat hardware pricing.

In this tutorial, we will decode the Kubernetes Controller vs. Operator paradigm, demonstrate how to scaffold an API with Kubebuilder, fix common production runtime pitfalls like the Server-Side Apply TypeMeta crash, test the logic in CircleCI with envtest, and explain why Bare Metal NVMe storage is critical for Operator performance.

Step 1: CRDs and Scaffolding with Kubebuilder

To create Custom Resource Definitions (CRDs) cleanly, we use Kubebuilder, the official framework maintained by the Kubernetes SIGs to bootstrap Go-based Kubernetes APIs.

What is the difference between a Kubernetes Controller and an Operator?

Every Operator is a Controller, but not every Controller is an Operator. A standard Controller watches built-in Kubernetes resources (such as ReplicaSets or Deployments). An Operator introduces a Custom Resource Definition (CRD)—such as a DatabaseCluster or custom WebApp—and injects domain-specific intelligence to manage its entire lifecycle.

Initialize your Operator project using the Go toolchain and Kubebuilder:

# Initialize the Go module
mkdir my-first-operator && cd my-first-operator
go mod init [github.com/yourname/my-first-operator](https://github.com/yourname/my-first-operator)

# Scaffold the Kubebuilder project
kubebuilder init --domain irexta.com --repo [github.com/yourname/my-first-operator](https://github.com/yourname/my-first-operator)

# Create the Custom Resource Definition (CRD) and Controller
kubebuilder create api --group apps --version v1alpha1 --kind WebApp

This generates the schema in api/v1alpha1/webapp_types.go. Modify the Spec struct to define your desired operational state, then generate your CRD manifests:

make generate
make manifests

Step 2: Writing the Brain — The Reconciliation Loop

The core execution engine of an Operator is its Reconciliation Loop. It continuously compares the Actual State of the cluster against the Desired State declared in the CRD, executing state changes until both align.

The SRE Masterclass: Preventing API Spam & TypeMeta SSA Crashes

1. Preventing API Server Spam: The reconciliation loop triggers continuously on watch events. If you issue an unconditional r.Create(), your Operator will flood the API server with redundant POST requests. You must inspect the cluster with r.Get() before attempting object creation.
2. The TypeMeta SSA Bug: Modern operators use Server-Side Apply (SSA) (client.Apply) to bypass HTTP 409 concurrency conflicts. However, controller-runtime's r.Get() strips the APIVersion and Kind (TypeMeta) when loading objects into memory. If you submit an SSA patch without manually restoring these fields, the Operator crashes with a "missing apiVersion or kind" error.

Here is the production-ready Go implementation inside internal/controller/webapp_controller.go:

package controller

import (
    "context"
    "time"

    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/types"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"

    appsv1alpha1 "[github.com/yourname/my-first-operator/api/v1alpha1](https://github.com/yourname/my-first-operator/api/v1alpha1)"
)

// +kubebuilder:rbac:groups=apps.irexta.com,resources=webapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps.irexta.com,resources=webapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch;create;update;patch;delete

func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    var webApp appsv1alpha1.WebApp
    if err := r.Get(ctx, req.NamespacedName, &webApp); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Ensure proper Garbage Collection via Owner Reference
    pod := buildPodForWebApp(&webApp)
    if err := ctrl.SetControllerReference(&webApp, pod, r.Scheme); err != nil {
        return ctrl.Result{}, err
    }

    // Check if Pod already exists to prevent API Server flood
    foundPod := &corev1.Pod{}
    err := r.Get(ctx, types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, foundPod)
    if err != nil && errors.IsNotFound(err) {
        logger.Info("Creating a new Pod", "Pod.Namespace", pod.Namespace, "Pod.Name", pod.Name)
        if err := r.Create(ctx, pod); err != nil {
            logger.Error(err, "Failed to create new Pod")
            return ctrl.Result{}, err
        }
    } else if err != nil {
        logger.Error(err, "Failed to inspect existing Pod")
        return ctrl.Result{}, err
    }

    // Apply Server-Side Apply (SSA) for status updates
    webApp.Status.Phase = "Running"

    // [CRITICAL FIX]: Re-inject TypeMeta stripped by r.Get()
    webApp.APIVersion = "[apps.irexta.com/v1alpha1](https://apps.irexta.com/v1alpha1)"
    webApp.Kind = "WebApp"

    // Force ownership to resolve field conflicts at the API layer
    if err := r.Status().Patch(ctx, &webApp, client.Apply, client.FieldOwner("webapp-operator"), client.ForceOwnership); err != nil {
        logger.Error(err, "Failed to patch status via SSA")
        return ctrl.Result{RequeueAfter: time.Second * 5}, err
    }

    return ctrl.Result{}, nil
}

Step 3: How do you test a Kubernetes Operator in CircleCI?

Operators run with root-adjacent privileges against the Kubernetes API. A flawed reconciliation loop can destabilize cluster state or drop resources. Integration testing in CI is mandatory before deployment.

Rather than provisioning a full, multi-node cloud cluster for unit and integration suites, Kubebuilder provides envtest. This utility launches standalone binaries for etcd and kube-apiserver directly inside the CI test runner.

Here is the hardened .circleci/config.yml pipeline configured with dynamic binary resolution:

version: 2.1

executors:
  go-executor:
    docker:
      - image: cimg/go:1.24.0

jobs:
  lint-and-envtest:
    executor: go-executor
    steps:
      - checkout
      - run:
          name: Run Go Vet & Static Analysis
          command: go vet ./...
      - run:
          name: Install EnvTest & Control Plane Binaries
          command: |
            go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
            setup-envtest use 1.30.0 --os linux --arch amd64 --bin-dir ./testbin
      - run:
          name: Execute Reconciliation Integration Tests
          command: |
            # Dynamically export path to prevent rigid path failures in CI runners
            export KUBEBUILDER_ASSETS=$(setup-envtest use 1.30.0 --os linux --arch amd64 --bin-dir ./testbin -p path)
            make test

workflows:
  operator-pipeline:
    jobs:
      - lint-and-envtest

Step 4: Security Hardening (Never Run as Root)

Kubernetes Operators possess extensive cluster-wide RBAC permissions. If an Operator pod running as root is compromised, an attacker inherits cluster administrator capabilities.

Always restrict your Operator pod's deployment manifest with a strict securityContext:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 65532
    runAsGroup: 65532
    fsGroup: 65532
  containers:
    - name: manager
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL

🚀 The SRE Solution: Why Operators Demand Bare Metal

Once your Operator is tested and hardened, control plane performance dictates runtime stability.

┌────────────────────────────────────────────────────────┐
│ Cloud VM Control Plane: Throttled EBS IOPS → etcd Lag  │
├────────────────────────────────────────────────────────┤
│ Bare Metal: Direct PCIe NVMe → Sub-Millisecond fsync   │
└────────────────────────────────────────────────────────┘

Why does etcd disk I/O bottleneck Operator reconciliation?

Kubebuilder operators read from an in-memory client-side cache (Informer Cache), which minimizes API server reads. However, every state change—such as updating Pod statuses, patching Custom Resources, or managing replicas—must be written directly to kube-apiserver.

Every write requires a synchronous fsync to the underlying etcd key-value store.

On public cloud virtual machines (e.g., AWS EC2 with EBS), etcd runs on virtualized, network-attached block storage. These volumes suffer from strict IOPS burst throttling, noisy neighbors, and hypervisor virtualization overhead. When an Operator manages hundreds or thousands of stateful CRDs, this I/O bottleneck induces severe write-queue latency in etcd. The reconciliation loop begins to lag, cascading into delayed pod provisioning and timing-out reconciliations.

Deploying Kubernetes control planes on iRexta Dedicated Bare Metal Servers removes the hypervisor layer entirely. By pairing direct-attached, local enterprise PCIe NVMe storage with dedicated compute cores, etcd writes execute with sub-millisecond disk commit latencies. This architecture enables your custom Go Operators to reconcile thousands of operational state changes per second without throttling.

Kubernetes Operator Technical FAQ

What is the difference between a Kubernetes Controller and an Operator?

Every Operator is a Controller, but not every Controller is an Operator. A standard Controller monitors and adjusts built-in Kubernetes objects (like Deployments or Pods). An Operator extends cluster capability by pairing a Custom Resource Definition (CRD) with custom Go logic to automate the deployment, backup, and recovery of complex, domain-specific applications.

How do you handle HTTP 409 errors in a Kubernetes Operator?

HTTP 409 (StatusConflict) happens when concurrent processes write to the same resource version. Instead of optimistic locking and manual retries, use Server-Side Apply (SSA) via client.Apply and client.FieldOwner. Note that before issuing an SSA patch, you must re-inject TypeMeta (APIVersion and Kind) back into the Go struct, as r.Get() strips these values during deserialization.

How do you test a Kubernetes Operator in a CircleCI pipeline?

Install the setup-envtest binary inside your CircleCI container. It pulls lightweight etcd and kube-apiserver binaries to execute real integration tests locally without requiring access to an external, running Kubernetes cluster.

Why is Bare Metal better for running Kubernetes Operators?

Operators write cluster mutations directly to kube-apiserver, requiring sub-millisecond disk fsync operations inside etcd. Virtualized cloud storage (like AWS EBS) bottlenecks under heavy write traffic due to network IOPS throttling. Dedicated Bare Metal servers provide unthrottled local NVMe access, ensuring immediate write commits and consistent reconciliation cycles.

👉 Run high-throughput Kubernetes Operators with direct NVMe performance:
Deploy Bare Metal Kubernetes on iRexta