Wednesday, October 28, 2026

LinkedIn DevOps Resources — October 28, 2026

LinkedIn DevOps resources — Wednesday, October 28

Thin day for fresh LinkedIn posts — only one falls inside the last 72 hours, so the rest are dated honestly rather than padded.

Tools & resources

Study materials

Posted by at No comments:

Daily DevOps & Cloud Briefing — October 28, 2026

Daily DevOps & Cloud Briefing — Wed, Oct 28, 2026

1. DevOps architecture

2. Cloud

3. Enterprise technology planning

Posted by at No comments:

Tuesday, October 27, 2026

Shift-Left Security in Jenkins: Secrets, SAST, Dependency and Image Scanning

Security reviews at the end of a release find problems when they are most expensive to fix. “Shift left” simply means running the same checks inside every pipeline run, so a leaked key or a vulnerable dependency fails the build the day it is introduced. Here is the set of stages I add to Java pipelines.

The stages

pipeline {
  agent { label 'linux' }
  environment {
    IMAGE = "nexus.example.com:8443/apps/account-search:${BUILD_NUMBER}"
  }
  stages {
    stage('Secrets scan') {
      steps { sh 'gitleaks detect --source . --redact --exit-code 1' }
    }
    stage('Build & unit tests') {
      steps { sh 'mvn -B clean verify' }
    }
    stage('SAST + quality gate') {
      steps {
        withSonarQubeEnv('sonar') { sh 'mvn -B sonar:sonar' }
        timeout(time: 10, unit: 'MINUTES') {
          waitForQualityGate abortPipeline: true
        }
      }
    }
    stage('Dependency check') {
      steps {
        sh 'mvn -B org.owasp:dependency-check-maven:check -DfailBuildOnCVSS=7'
      }
    }
    stage('Image build + scan') {
      steps {
        sh 'docker build -t $IMAGE .'
        sh 'trivy image --exit-code 1 --severity HIGH,CRITICAL --ignore-unfixed $IMAGE'
        sh 'trivy image --format cyclonedx --output sbom.json $IMAGE'
        archiveArtifacts artifacts: 'sbom.json'
      }
    }
  }
}

What each stage catches

  • Secrets scan (gitleaks) – API keys, passwords and tokens committed to the repo. Run it first; it takes seconds.
  • SAST with a quality gate (SonarQube) – injection risks, weak crypto and bugs in your code. waitForQualityGate makes the gate a hard stop instead of a dashboard nobody reads.
  • Dependency check (OWASP) – known CVEs in third-party libraries. Start with a CVSS threshold of 7 and tighten later.
  • Image scan (Trivy) – vulnerable OS packages and runtimes in the final image. --ignore-unfixed keeps the build focused on things you can actually fix.
  • SBOM – a CycloneDX list of everything inside the image. When the next big library vulnerability is announced, you can answer “are we affected?” in minutes.

Making it stick

  1. Put these stages in the shared library, not in each Jenkinsfile, so every team gets them and updates roll out in one place.
  2. Pin scanner and tool versions and pull them through your Nexus proxy. Recent npm and PyPI compromises showed that floating tags like latest are a supply-chain risk in themselves.
  3. Have a documented exception process (suppression files with an owner and an expiry date). Otherwise people disable the stage the first time it blocks a release.
  4. Start in warning mode for a week, fix the backlog, then switch to failing builds.

The goal is not zero findings; it is that every new risk is visible, owned and decided on before it reaches production.

Posted by at No comments:
Labels: , ,

Blue-Green vs Canary Deployments on Kubernetes

A plain Kubernetes rolling update is fine for most changes, but it has one weakness: once the rollout finishes, every user is on the new version, and rolling back means another rollout. For risky releases, two strategies give you more control: blue-green and canary.

Blue-green: switch all traffic at once

Run the new version (green) next to the current one (blue), test it, then move the Service selector.

# two Deployments with different version labels
#   account-search-blue   labels: app=account-search, version=blue
#   account-search-green  labels: app=account-search, version=green

apiVersion: v1
kind: Service
metadata:
  name: account-search
spec:
  selector:
    app: account-search
    version: blue      # change to green to cut over
  ports:
  - port: 80
    targetPort: 8080
# cut over
kubectl patch service account-search \
  -p '{"spec":{"selector":{"app":"account-search","version":"green"}}}'

# instant rollback
kubectl patch service account-search \
  -p '{"spec":{"selector":{"app":"account-search","version":"blue"}}}'

Good for: releases you can fully test before exposure, and instant rollback. Cost: double capacity during the switch, and database changes must work for both versions.

Canary: shift traffic gradually

Send a small share of real traffic to the new version, watch metrics, then increase. Argo Rollouts makes this declarative:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: account-search
spec:
  replicas: 10
  selector:
    matchLabels: { app: account-search }
  template:
    # same pod template as your Deployment
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: { duration: 10m }
      - setWeight: 50
      - pause: { duration: 10m }
      - setWeight: 100

Without a service mesh or ingress integration, the weight is approximated by replica counts. With NGINX Ingress, Istio or a Gateway API controller, Rollouts can split traffic precisely. Add an AnalysisTemplate that queries Prometheus (error rate, p95 latency) and the rollout aborts and rolls back by itself when the canary looks worse than stable.

Good for: changes whose real-world behaviour you can't fully test. Cost: you need good metrics, and two versions serve users at the same time.

Which one should you use?

  • Rolling update – the default for routine changes.
  • Blue-green – big-bang releases, strict cut-over windows, or when you need instant rollback.
  • Canary – high-traffic services where a small blast radius and metric-driven promotion matter.

Whatever you pick, make database migrations backward compatible (expand, then contract). That one discipline makes every strategy here safe to roll back.

Posted by at No comments:
Labels: ,

Kubernetes Requests and Limits for Java Apps: Avoiding OOMKilled and CPU Throttling

“The pod keeps getting OOMKilled but the heap is only 512 MB.” “Latency doubles every few seconds and the CPU graph looks fine.” Both of these come from the same root cause: Kubernetes resource settings that don't match how the JVM actually uses memory and CPU.

Memory: the heap is not the whole process

A Java process uses heap plus metaspace, thread stacks, the code cache, GC structures and direct buffers. If you set -Xmx512m and a 512Mi container limit, the kernel will kill the container long before Java ever throws an OutOfMemoryError.

Let the JVM size itself from the container limit instead:

env:
- name: JAVA_TOOL_OPTIONS
  value: "-XX:MaxRAMPercentage=75 -XX:+ExitOnOutOfMemoryError"
resources:
  requests:
    memory: "1Gi"
    cpu: "500m"
  limits:
    memory: "1Gi"
  • 75% leaves room for non-heap memory. For apps with many threads or heavy off-heap use (Netty, large caches) go to 60–65%.
  • Memory request = memory limit. Memory is not compressible; a pod that bursts above its request is the first to be evicted when the node is under pressure.
  • ExitOnOutOfMemoryError makes a heap OOM a clean restart instead of a half-dead JVM.

CPU: limits cause throttling

CPU limits are enforced in 100 ms windows. A JVM with several GC and JIT threads can use its whole quota in the first 20 ms and then sit throttled for 80 ms. That shows up as random latency spikes even though average CPU looks low.

  • Set a realistic CPU request so the scheduler places pods sensibly.
  • Either don't set a CPU limit, or set it well above the request (2–4×) for latency-sensitive services.
  • Watch container_cpu_cfs_throttled_periods_total in Prometheus; if it climbs, the limit is too tight.
  • The JVM sizes thread pools from the CPU count it sees. If needed, pin it with -XX:ActiveProcessorCount=2.

Finding the right numbers

  1. Run under realistic load and watch kubectl top pod plus JVM metrics from Actuator/Micrometer.
  2. Install the Vertical Pod Autoscaler in recommendation-only mode and read its suggestions for a week.
  3. Set requests around normal peak usage, then revisit after big releases.

Quick diagnosis

  • Reason: OOMKilled, exit code 137 – container limit hit; lower MaxRAMPercentage or raise the limit.
  • java.lang.OutOfMemoryError in logs – heap is too small for the workload, or there is a leak.
  • Latency spikes, CPU looks low – check throttling before you blame the code.
Posted by at No comments:
Labels: ,

Injecting HashiCorp Vault Secrets into Kubernetes Pods

Kubernetes Secrets are only base64-encoded, they end up in Git or Helm values far too easily, and rotating a database password means redeploying everything. With HashiCorp Vault and the Vault Agent Injector, pods authenticate with their Kubernetes service account and receive secrets as files, without the application knowing Vault exists.

1. Enable Kubernetes auth in Vault

vault auth enable kubernetes

vault write auth/kubernetes/config \
    kubernetes_host="https://kubernetes.default.svc:443"

When Vault runs inside the same cluster, it can use its own service account token and CA to validate other pods' tokens.

2. Policy and role

# account-search.hcl
path "secret/data/account-search/*" {
  capabilities = ["read"]
}
vault policy write account-search account-search.hcl

vault write auth/kubernetes/role/account-search \
    bound_service_account_names=account-search \
    bound_service_account_namespaces=apps \
    policies=account-search \
    ttl=1h

Only pods running as the account-search service account in the apps namespace can log in with this role, and they can only read their own path.

3. Annotate the Deployment

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "account-search"
        vault.hashicorp.com/agent-inject-secret-db.properties: "secret/data/account-search/db"
        vault.hashicorp.com/agent-inject-template-db.properties: |
          {{- with secret "secret/data/account-search/db" -}}
          spring.datasource.username={{ .Data.data.username }}
          spring.datasource.password={{ .Data.data.password }}
          {{- end }}
    spec:
      serviceAccountName: account-search

The injector adds an init container and a sidecar. The rendered file appears at /vault/secrets/db.properties and the sidecar keeps it fresh when the secret changes or the lease renews.

4. Let Spring Boot read it

# application.yml
spring:
  config:
    import: "optional:file:/vault/secrets/db.properties"

No Vault SDK, no code change. Locally, developers use their own properties file at the same path.

Things to watch

  • The template path uses secret/data/... for KV version 2. Forgetting /data/ is the most common “permission denied”.
  • Rotation is not reload. The file updates, but a running connection pool keeps the old password until restart unless your app re-reads it. Pair rotation with a rolling restart or use Vault dynamic database credentials.
  • Alternatives: the Vault Secrets Operator syncs into native Kubernetes Secrets, and the Secrets Store CSI driver mounts secrets as volumes. Pick one approach per cluster and stick to it.
  • Keep Vault patched and restrict who can restore snapshots; Vault is now the most valuable target in the cluster.
Posted by at No comments:
Labels: , ,

Terraform Remote State on S3 with Locking: A Practical Setup

The fastest way to break Terraform for a whole team is to keep terraform.tfstate on someone's laptop. Two people apply at the same time, one state file wins, and the other half of your infrastructure is now “unknown” to Terraform. A remote backend with locking fixes this.

1. The state bucket

Create the bucket once (by hand or in a tiny bootstrap Terraform project with local state):

  • Versioning on – every state write is recoverable.
  • Block all public access and default encryption (SSE-S3 or SSE-KMS).
  • A bucket policy that only allows your CI role and a small admin group.

2. The backend block

terraform {
  required_version = ">= 1.10"

  backend "s3" {
    bucket       = "shailapps-tfstate"
    key          = "envs/dev/network/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

use_lockfile = true uses S3's native conditional writes to create a .tflock object next to the state, available from Terraform 1.10. On older versions you need a DynamoDB table with a LockID string key and dynamodb_table = "..." in the backend instead; that option is now deprecated, so plan the upgrade.

3. Move existing state

terraform init -migrate-state

Terraform copies the local state into S3. Check the object exists, then delete the local file so nobody applies from it by accident.

4. Split state by blast radius

envs/
  dev/
    network/   -> key = envs/dev/network/terraform.tfstate
    eks/       -> key = envs/dev/eks/terraform.tfstate
    apps/
  prod/
    network/
    eks/

One giant state means every plan touches everything and a mistake can destroy unrelated resources. Separate states per environment and per layer keep plans fast and failures small. Read outputs across layers with the terraform_remote_state data source or, better, by looking resources up with data sources.

5. Running it from CI

  • Authenticate with an IAM role (OIDC from your CI system), never long-lived keys in variables.
  • Run terraform plan -out=tfplan on the pull request and terraform apply tfplan only after approval, so what you reviewed is exactly what gets applied.
  • If a run dies and leaves a lock behind, use terraform force-unlock LOCK_ID only after you are sure no other apply is running.

With this in place, concurrent applies fail fast with a clear lock error instead of silently corrupting state.

Posted by at No comments:
Labels: ,
Subscribe to: Posts (Atom)