.png&w=3840&q=75)
SRE Copilot
NightShift: an incident war room with an AI assistant that explains why a service is failing.
5
Telemetry sources
4
Incident scenarios
Problem
At 3 AM, an on-call engineer should be fixing the incident, not joining six dashboards into a theory. Most observability tools answer by showing more data.
Approach
NightShift is an evidence-first incident assistant for Kubernetes and containerised systems. Ask a question in plain English, such as "Why is checkout failing?", and it gathers telemetry, connects the clues, ranks competing root causes and proposes a fix that still needs a human's approval.
Why it isn't just a chatbot
- It correlates. Cluster state, metrics, logs and traces are read together, not summarised one paste at a time.
- It ranks. Several competing causes come back, each with a confidence level and the evidence behind it.
- Rules come first. Deterministic signal rules set the quality floor. The model can improve the wording and refine a real incident, but it can't invent an outage on a healthy cluster.
- Every claim is cited. Each observation becomes an evidence object with its own ID, and any ID the model makes up is discarded before it reaches the screen.
The war room
- System health. Service readiness, replicas, resources, errors and latency.
- Incident feed. Problems detected before anyone files a ticket.
- Investigation. Streamed triage, collection, correlation and response steps.
- Hypotheses, timeline and evidence. The theories, the order things failed in, and the facts behind each claim.
- Dependency graph. The service topology with the failing chain highlighted.
- Fixes and audit. Impact prediction, approval, rollback and a durable record of who decided what.
Safety
Every proposed action stays "proposed" until an approver changes it, and the executor refuses anything that isn't approved. Dry-run is the default, so execution records the command without touching infrastructure. Each action carries its risk, expected outcome, rollback plan and blast radius, with predicted downtime shown before approval.
Under the hood
A FastAPI backend runs a LangGraph investigation over a telemetry gateway that reads from Kubernetes, Docker Compose, Prometheus, Loki and Tempo. Each source is queried independently and falls back on its own, and if the language model fails, the rule engine still returns a usable investigation. A scenario simulator generates coherent incidents (an out-of-memory crash loop, a slow dependency, a bad image rollout, and a healthy cluster) so the demo is repeatable without a cluster or an API key.
.png&w=3840&q=75)
.png&w=3840&q=75)
