Problem

Kubernetes clusters generate a constant stream of security events — admission violations, runtime anomalies, syscall alerts — that require expert triage, manual response, and accurate severity judgment. Without automation, security teams are buried in noise, slow to respond to real threats, and unable to scale coverage.

• Runtime security events require immediate triage, but manual review is slow and error-prone at volume.
• Admission control and runtime detection are often siloed with no unified response layer connecting them.
• Low-severity incidents that could be auto-remediated still consume engineer time, reducing capacity for real threats.

Solution

A Python FastAPI webhook server that sits between Kubernetes security tooling and on-call engineers. Warden receives Falco and OPA alerts, classifies severity with Claude Sonnet 4.6, and acts — auto-patching low-severity incidents immediately and surfacing AI-drafted runbooks for high-severity threats that require a human decision.

• Layer 1 — OPA Gatekeeper: Rego policies block non-compliant workloads at admission — no privileged containers, no unverified registries, no non-root violations.
• Layer 2 — Falco: eBPF kernel probes detect threats at runtime — shell spawns, privilege escalation, and suspicious syscall patterns.
• Claude Sonnet 4.6 triage: structured severity classification — low severity triggers auto-patch, high severity generates a runbook for human review.
• Azure Key Vault for secret management: the Claude API key is injected at runtime, never stored in source control or container images.

OPA Gatekeeper blocking a privileged container at admission — Layer 1 enforcement working on AKS.

Security Design

• Secrets management: Claude API key stored in Azure Key Vault, injected at runtime via managed identity — never in source control, never in container images, never in environment variables passed at build time.
• Least-privilege RBAC: Warden agent has only the Kubernetes permissions it needs — patch pods in the warden-system namespace, nothing else. Defined explicitly in a ClusterRole manifest.
• Zero-trust between components: Falcosidekick communicates with Warden via in-cluster HTTP only — no external exposure, no auth tokens in transit.
• Audit trail by design: every Claude triage decision is logged with user context, original payload, severity classification, and action taken — not as an afterthought but as a required output of the triage function.
• Namespace isolation: Falco runs in its own namespace with elevated permissions; the BlockPrivilegedContainers constraint explicitly excludes the falco namespace to prevent policy self-conflict.

Observability & Operations

• Prometheus metrics exposed at /metrics: alert_count by severity label, auto_patch_total (success/failure), claude_api_latency_seconds (histogram), warden_webhook_requests_total.
• What each metric tells you operationally: alert_count spike = active threat or noisy rule; auto_patch failure rate climbing = Kubernetes API permissions issue or cluster instability; Claude API latency p95 > 2s = triage pipeline degrading, consider fallback severity classification.
• Grafana dashboard: three panels — alerts by severity over time (bar chart), auto-patch success rate (stat panel), Claude API latency p50/p95 (time series). Dashboard JSON exported to repo at /docs/grafana-dashboard.json.
• Alerting intent: in a production deployment, alert on auto_patch_failure_total > 0 (immediate page), claude_api_latency_seconds p95 > 3s (warning), and alert_count rate > 10/minute sustained (potential attack or rule misconfiguration).

Outcome

End-to-end Kubernetes security automation proven on a live AKS cluster: Falco detected a shell spawn, Falcosidekick forwarded the alert to Warden, Claude triaged it as low severity, Warden auto-patched and returned HTTP 200 — all within 3 seconds. Total cloud spend: ~$2.

• Full AKS pipeline proven: shell spawn → Falcosidekick → Claude triage (severity=low) → auto-patch → HTTP 200.
• Two-layer security coverage: OPA admission control and Falco runtime detection operating independently on the same cluster.
• Total cloud spend across all sessions: ~$2 — production-grade Kubernetes security at near-zero cost.
• All AI triage decisions logged with full reasoning chain, original alert payload, and action taken — complete audit trail.

Falco detecting a shell spawn inside a running container — Layer 2 runtime detection triggering the full triage pipeline.