The SRE Copilot investigate view on a desktop monitor
All work

AI/ML · Web · Hackathon · Open source

SRE Copilot

NightShift: an incident war room with an AI assistant that explains why a service is failing.

Role
Design and engineering
Year
2026
Status
Hackathon prototype
Stack
  • Next.js
  • FastAPI
  • LangGraph
  • Kubernetes
  • Docker
  • OpenTelemetry
  • Prometheus
  • Loki
  • Tempo
  • Grafana
  • Postgres
  • Redis
  • 5

    Telemetry sources

  • 4

    Incident scenarios

(01)

Problem

At 3 AM, an on-call engineer should be fixing the incident, not joining six dashboards into a theory. Most observability tools answer by showing more data.

(02)

Approach

NightShift is an evidence-first incident assistant for Kubernetes and containerised systems. Ask a question in plain English, such as "Why is checkout failing?", and it gathers telemetry, connects the clues, ranks competing root causes and proposes a fix that still needs a human's approval.

(03)

Why it isn't just a chatbot

  • It correlates. Cluster state, metrics, logs and traces are read together, not summarised one paste at a time.
  • It ranks. Several competing causes come back, each with a confidence level and the evidence behind it.
  • Rules come first. Deterministic signal rules set the quality floor. The model can improve the wording and refine a real incident, but it can't invent an outage on a healthy cluster.
  • Every claim is cited. Each observation becomes an evidence object with its own ID, and any ID the model makes up is discarded before it reaches the screen.
(04)

The war room

  • System health. Service readiness, replicas, resources, errors and latency.
  • Incident feed. Problems detected before anyone files a ticket.
  • Investigation. Streamed triage, collection, correlation and response steps.
  • Hypotheses, timeline and evidence. The theories, the order things failed in, and the facts behind each claim.
  • Dependency graph. The service topology with the failing chain highlighted.
  • Fixes and audit. Impact prediction, approval, rollback and a durable record of who decided what.
(05)

Safety

Every proposed action stays "proposed" until an approver changes it, and the executor refuses anything that isn't approved. Dry-run is the default, so execution records the command without touching infrastructure. Each action carries its risk, expected outcome, rollback plan and blast radius, with predicted downtime shown before approval.

(06)

Under the hood

A FastAPI backend runs a LangGraph investigation over a telemetry gateway that reads from Kubernetes, Docker Compose, Prometheus, Loki and Tempo. Each source is queried independently and falls back on its own, and if the language model fails, the rule engine still returns a usable investigation. A scenario simulator generates coherent incidents (an out-of-memory crash loop, a slow dependency, a bad image rollout, and a healthy cluster) so the demo is repeatable without a cluster or an API key.

The SRE Copilot overview panel with pod, service and anomaly health
The SRE Copilot investigate view on a tablet

Next project

© 2026

The SIRI online store home page on a laptop, showing a model in a magenta suit

SIRI