Showing posts with label Kubernetes. Show all posts
Showing posts with label Kubernetes. Show all posts

Tuesday, October 27, 2026

Blue-Green vs Canary Deployments on Kubernetes

A plain Kubernetes rolling update is fine for most changes, but it has one weakness: once the rollout finishes, every user is on the new version, and rolling back means another rollout. For risky releases, two strategies give you more control: blue-green and canary.

Blue-green: switch all traffic at once

Run the new version (green) next to the current one (blue), test it, then move the Service selector.

# two Deployments with different version labels
#   account-search-blue   labels: app=account-search, version=blue
#   account-search-green  labels: app=account-search, version=green

apiVersion: v1
kind: Service
metadata:
  name: account-search
spec:
  selector:
    app: account-search
    version: blue      # change to green to cut over
  ports:
  - port: 80
    targetPort: 8080
# cut over
kubectl patch service account-search \
  -p '{"spec":{"selector":{"app":"account-search","version":"green"}}}'

# instant rollback
kubectl patch service account-search \
  -p '{"spec":{"selector":{"app":"account-search","version":"blue"}}}'

Good for: releases you can fully test before exposure, and instant rollback. Cost: double capacity during the switch, and database changes must work for both versions.

Canary: shift traffic gradually

Send a small share of real traffic to the new version, watch metrics, then increase. Argo Rollouts makes this declarative:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: account-search
spec:
  replicas: 10
  selector:
    matchLabels: { app: account-search }
  template:
    # same pod template as your Deployment
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: { duration: 10m }
      - setWeight: 50
      - pause: { duration: 10m }
      - setWeight: 100

Without a service mesh or ingress integration, the weight is approximated by replica counts. With NGINX Ingress, Istio or a Gateway API controller, Rollouts can split traffic precisely. Add an AnalysisTemplate that queries Prometheus (error rate, p95 latency) and the rollout aborts and rolls back by itself when the canary looks worse than stable.

Good for: changes whose real-world behaviour you can't fully test. Cost: you need good metrics, and two versions serve users at the same time.

Which one should you use?

  • Rolling update – the default for routine changes.
  • Blue-green – big-bang releases, strict cut-over windows, or when you need instant rollback.
  • Canary – high-traffic services where a small blast radius and metric-driven promotion matter.

Whatever you pick, make database migrations backward compatible (expand, then contract). That one discipline makes every strategy here safe to roll back.

Kubernetes Requests and Limits for Java Apps: Avoiding OOMKilled and CPU Throttling

“The pod keeps getting OOMKilled but the heap is only 512 MB.” “Latency doubles every few seconds and the CPU graph looks fine.” Both of these come from the same root cause: Kubernetes resource settings that don't match how the JVM actually uses memory and CPU.

Memory: the heap is not the whole process

A Java process uses heap plus metaspace, thread stacks, the code cache, GC structures and direct buffers. If you set -Xmx512m and a 512Mi container limit, the kernel will kill the container long before Java ever throws an OutOfMemoryError.

Let the JVM size itself from the container limit instead:

env:
- name: JAVA_TOOL_OPTIONS
  value: "-XX:MaxRAMPercentage=75 -XX:+ExitOnOutOfMemoryError"
resources:
  requests:
    memory: "1Gi"
    cpu: "500m"
  limits:
    memory: "1Gi"
  • 75% leaves room for non-heap memory. For apps with many threads or heavy off-heap use (Netty, large caches) go to 60–65%.
  • Memory request = memory limit. Memory is not compressible; a pod that bursts above its request is the first to be evicted when the node is under pressure.
  • ExitOnOutOfMemoryError makes a heap OOM a clean restart instead of a half-dead JVM.

CPU: limits cause throttling

CPU limits are enforced in 100 ms windows. A JVM with several GC and JIT threads can use its whole quota in the first 20 ms and then sit throttled for 80 ms. That shows up as random latency spikes even though average CPU looks low.

  • Set a realistic CPU request so the scheduler places pods sensibly.
  • Either don't set a CPU limit, or set it well above the request (2–4×) for latency-sensitive services.
  • Watch container_cpu_cfs_throttled_periods_total in Prometheus; if it climbs, the limit is too tight.
  • The JVM sizes thread pools from the CPU count it sees. If needed, pin it with -XX:ActiveProcessorCount=2.

Finding the right numbers

  1. Run under realistic load and watch kubectl top pod plus JVM metrics from Actuator/Micrometer.
  2. Install the Vertical Pod Autoscaler in recommendation-only mode and read its suggestions for a week.
  3. Set requests around normal peak usage, then revisit after big releases.

Quick diagnosis

  • Reason: OOMKilled, exit code 137 – container limit hit; lower MaxRAMPercentage or raise the limit.
  • java.lang.OutOfMemoryError in logs – heap is too small for the workload, or there is a leak.
  • Latency spikes, CPU looks low – check throttling before you blame the code.

Injecting HashiCorp Vault Secrets into Kubernetes Pods

Kubernetes Secrets are only base64-encoded, they end up in Git or Helm values far too easily, and rotating a database password means redeploying everything. With HashiCorp Vault and the Vault Agent Injector, pods authenticate with their Kubernetes service account and receive secrets as files, without the application knowing Vault exists.

1. Enable Kubernetes auth in Vault

vault auth enable kubernetes

vault write auth/kubernetes/config \
    kubernetes_host="https://kubernetes.default.svc:443"

When Vault runs inside the same cluster, it can use its own service account token and CA to validate other pods' tokens.

2. Policy and role

# account-search.hcl
path "secret/data/account-search/*" {
  capabilities = ["read"]
}
vault policy write account-search account-search.hcl

vault write auth/kubernetes/role/account-search \
    bound_service_account_names=account-search \
    bound_service_account_namespaces=apps \
    policies=account-search \
    ttl=1h

Only pods running as the account-search service account in the apps namespace can log in with this role, and they can only read their own path.

3. Annotate the Deployment

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "account-search"
        vault.hashicorp.com/agent-inject-secret-db.properties: "secret/data/account-search/db"
        vault.hashicorp.com/agent-inject-template-db.properties: |
          {{- with secret "secret/data/account-search/db" -}}
          spring.datasource.username={{ .Data.data.username }}
          spring.datasource.password={{ .Data.data.password }}
          {{- end }}
    spec:
      serviceAccountName: account-search

The injector adds an init container and a sidecar. The rendered file appears at /vault/secrets/db.properties and the sidecar keeps it fresh when the secret changes or the lease renews.

4. Let Spring Boot read it

# application.yml
spring:
  config:
    import: "optional:file:/vault/secrets/db.properties"

No Vault SDK, no code change. Locally, developers use their own properties file at the same path.

Things to watch

  • The template path uses secret/data/... for KV version 2. Forgetting /data/ is the most common “permission denied”.
  • Rotation is not reload. The file updates, but a running connection pool keeps the old password until restart unless your app re-reads it. Pair rotation with a rolling restart or use Vault dynamic database credentials.
  • Alternatives: the Vault Secrets Operator syncs into native Kubernetes Secrets, and the Secrets Store CSI driver mounts secrets as volumes. Pick one approach per cluster and stick to it.
  • Keep Vault patched and restrict who can restore snapshots; Vault is now the most valuable target in the cluster.

GitOps with Argo CD: The App-of-Apps Pattern

Once you have more than a handful of applications in Argo CD, clicking “New App” in the UI stops scaling. Nobody remembers which settings were used, and rebuilding a cluster means recreating every Application by hand. The app-of-apps pattern puts the Argo CD Applications themselves in Git, managed by one root Application.

Repository layout

gitops/
├── root-app.yaml            # applied once, by hand
├── apps/                    # one Application per service
│   ├── account-search.yaml
│   ├── payments-api.yaml
│   └── ingress-nginx.yaml
└── charts/ or manifests/    # the actual workloads

The root Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/platform/gitops.git
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply it once with kubectl apply -n argocd -f root-app.yaml. From then on, adding a service is a pull request that drops a new file into apps/.

A child Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: account-search
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "10"
spec:
  project: apps
  source:
    repoURL: https://git.example.com/platform/gitops.git
    targetRevision: main
    path: charts/account-search
    helm:
      valueFiles: [values-prod.yaml]
  destination:
    server: https://kubernetes.default.svc
    namespace: account-search
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]

Tips from running it

  • Use sync waves for ordering. Give platform pieces (ingress controller, cert-manager, operators) a lower sync-wave than business apps so CRDs exist before anything uses them.
  • Use AppProjects as guard rails. Restrict which repos and namespaces the apps project can deploy to, so a typo cannot deploy into kube-system.
  • Be careful with prune on the root. Deleting a file in apps/ deletes the Application and, with the default finalizer, its workloads. That is usually what you want, but review those PRs carefully.
  • Consider ApplicationSets once the child files all look the same. A Git directory generator can create one Application per folder automatically.
  • Keep Argo CD patched. The repo-server holds your Git and Helm credentials; treat its upgrades like any other internet-facing component.

The payoff: a new cluster becomes “install Argo CD, apply one file, wait”, and the state of every environment is reviewable in Git.

Kubernetes Startup, Liveness and Readiness Probes for Spring Boot

Most Spring Boot services I see on Kubernetes have exactly one probe: a liveness check on /actuator/health. It works until the database has a hiccup, the health endpoint goes DOWN, and Kubernetes restarts every pod at the same moment. The fix is to use the three probe types for what they are actually for.

What each probe means

  • startupProbe – “Has the app finished booting?” Liveness and readiness are not checked until this passes. Perfect for slow JVM starts.
  • livenessProbe – “Is the process stuck beyond repair?” If it fails, the kubelet restarts the container.
  • readinessProbe – “Should this pod receive traffic right now?” If it fails, the pod is removed from the Service endpoints but not restarted.

Turn on the Spring Boot probe groups

# application.yml
management:
  endpoint:
    health:
      probes:
        enabled: true
  health:
    livenessstate:
      enabled: true
    readinessstate:
      enabled: true

This exposes /actuator/health/liveness and /actuator/health/readiness. Liveness only reflects the application's internal state, so a slow database will not restart your pods. If you want a dependency to affect traffic, add it to the readiness group only:

management:
  endpoint:
    health:
      group:
        readiness:
          include: readinessState, db

The Deployment

containers:
- name: account-search
  image: nexus.example.com/apps/account-search:1.8.2
  ports:
  - containerPort: 8080
  startupProbe:
    httpGet: { path: /actuator/health/liveness, port: 8080 }
    periodSeconds: 5
    failureThreshold: 30        # up to 150s to boot
  livenessProbe:
    httpGet: { path: /actuator/health/liveness, port: 8080 }
    periodSeconds: 10
    failureThreshold: 3
  readinessProbe:
    httpGet: { path: /actuator/health/readiness, port: 8080 }
    periodSeconds: 5
    failureThreshold: 3

Rules I follow

  1. Never check external dependencies in liveness. Restarting your pod will not fix the database.
  2. Use startupProbe instead of a huge initialDelaySeconds. It protects slow starts without delaying failure detection later.
  3. Expose actuator on a separate management port (management.server.port: 8081) if the app port is public, and point probes there.
  4. Enable graceful shutdown (server.shutdown: graceful) so readiness flips to refusing traffic before the pod stops.

With this split, a dependency outage drains traffic instead of triggering a restart storm, and a genuinely hung JVM still gets recycled.