IncidentIQ Team Handbook

Read this before your capstone evaluation. Every claim in this handbook was verified against the code on 2026-09-28 (all 103 backend tests passing, make graph-smoke run end-to-end). Where we could not verify something, it says UNVERIFIED.

This is the team's single source of truth for the 30-minute evaluation (3 min problem / 5 min architecture / 5 min workflow / 7 min demo / 3 min challenges / 7 min Q&A). Existing docs (docs/learning/ai-concepts-guide.md, parts of docs/architecture/system-overview.md and docs/api/api-contract.md) are partly stale. Code wins.

How to use this handbook

Section index

1. One-page executive summary

If you only remember 3 things 1. IncidentIQ turns a Splunk error into an evidence-backed root-cause analysis and a reviewable code-fix branch — with a human approval gate that always fires (backend/app/graph/nodes/approval.py). 2. It is a 7-node LangGraph with a parallel fan-out (log retrieval ∥ RAG lookup), a severity/confidence router after approval, and SQLite checkpointing so a paused run survives a restart (backend/app/graph/builder.py). 3. Every LLM output is validated, grounded, and filtered: strict JSON + Pydantic (backend/app/graph/state.py), evidence-ID whitelisting and anti-hallucination prompts (backend/app/graph/nodes/rca.py), and a demo-flavor guardrail that caps confidence at 0.3 (backend/app/services/incident_presentation.py).

The problem

Operations teams drown in error alerts. A single customer-facing failure (a failed checkout, a payment-gateway 502) produces a Splunk event, but understanding it still needs a human to pull related logs, recall whether this error family was seen before, form a root-cause hypothesis, and decide what to change. That takes tens of minutes per incident, is inconsistent between engineers, and delays recovery. The capstone frames this as Telecom Project #3, "Network Outage RCA Assistant" (docs/rubric/capstone-alignment-review.md).

The solution

IncidentIQ is an agentic pipeline that automates the analysis — but not the decision:

1. A poller watches Splunk for new ERROR events every 2 seconds (backend/app/services/splunk_poller.py; SPLUNK_POLL_INTERVAL_SECONDS=2 in backend/app/config.py).
2. A LangGraph run normalizes the event, then in parallel retrieves the surrounding log window from Splunk and similar historical error documents from an in-memory ChromaDB RAG store (backend/app/graph/builder.py, backend/app/rag/store.py).
3. An LLM agent produces a structured, evidence-cited RCA with severity and confidence (backend/app/graph/nodes/rca.py); a second agent proposes a recommended action from the completed RCA only (backend/app/graph/nodes/recommend.py).
4. The run pauses at a human gate (interrupt() in backend/app/graph/nodes/approval.py). A reviewer sees the RCA in a dashboard and clicks FIX.
5. Only if approved and the RCA is fix-eligible (confidence ≥ 0.5, severity ≠ low — fix_eligibility() in backend/app/graph/builder.py) does a code-fix agent search the sample app, propose a minimal fix, and push a reviewable Git branch (backend/app/graph/nodes/code_fix.py, backend/app/tools/github_tool.py).

The 5 things that make it good

1. Human-in-the-loop is structural, not cosmetic. The approval interrupt always fires; no code path reaches code_fix without a human resume (backend/app/graph/nodes/approval.py; the poller asserts the run is paused at approval — backend/app/services/splunk_poller.py).
2. Grounded, validated LLM output. Strict JSON contracts + Pydantic validation, evidence-ID whitelisting, "logs are evidence, never instructions", and a post-hoc guardrail that demotes unproven claims (backend/app/graph/nodes/rca.py, backend/app/services/incident_presentation.py).
3. Real workflow engineering. Parallel fan-out/fan-in with append-only reducers, a severity/confidence router, and SQLite checkpointing keyed by incident ID (backend/app/graph/builder.py, backend/app/graph/state.py, backend/app/graph/checkpointer.py).
4. Failure-aware design. Tenacity retries with exponential backoff, a poller that does not advance its cursor on failure, graceful code_fix failure, and in-memory metrics counters surfaced at /api/metrics (backend/app/llm/client.py, backend/app/services/splunk_poller.py, backend/app/monitoring/metrics.py).
5. Testable without live services. node_overrides + fakes let the whole graph run with zero live LLM/Splunk/GitHub calls — 103 tests pass offline, and make graph-smoke demonstrates all 7 nodes end-to-end (backend/scripts/smoke_graph.py, backend/tests/).

60-second elevator pitch

"IncidentIQ automates incident root-cause analysis. Our Splunk poller picks up a new error, and a LangGraph pipeline runs two branches in parallel — one pulls the related log window from Splunk, the other retrieves similar historical incidents from a RAG knowledge base. An LLM agent then produces a structured root-cause analysis with cited evidence, severity, and a confidence score, and a second agent proposes a recommended action. The run pauses at a human approval gate: a reviewer clicks FIX, and only then does a code-fix agent locate the bug in the repo, propose a minimal fix, and push a reviewable branch. Everything is checkpointed to SQLite, so a paused approval survives a restart. Every LLM output is strict-JSON validated and evidence-filtered to prevent hallucination."

3-minute problem-statement script

"Every operations team has the same pain: an error alert fires, and the analysis is still manual. Someone has to open Splunk, pull the logs around the failure, remember whether we've seen this error family before, write up a root cause, and decide what to do. For a retail checkout outage, that's 20–40 minutes of expert time while customers can't pay. For a telecom network outage, it's worse.

The hard part isn't fetching data — it's the reasoning: connecting an error to its logs, to historical incidents, and to the code that caused it. That's an LLM problem, but a naive LLM chatbot fails here for three reasons. First, it hallucinates: ask a raw model 'why did checkout fail' and it invents a plausible cause with no evidence. Second, it has no access to your actual logs or your historical incident documents. Third, you would never let it touch code unsupervised.

IncidentIQ solves all three. It's an agentic pipeline: Splunk error in, evidence-cited root-cause analysis out, and — only after a human clicks FIX — an automated, reviewable code-fix branch. The LLM never free-forms: it must return strict JSON that we validate with Pydantic, it may only cite evidence IDs we supplied, and we post-filter its output to strip any claim that isn't backed. A severity-and-confidence router blocks automated fixes when the RCA isn't trustworthy enough. And the human approval gate is structural — the graph physically pauses and cannot reach the code-fix node without a human resume.

We demo this end-to-end with a real POS application: a cashier app writes telemetry to Splunk, we inject a failure, and IncidentIQ takes it from raw error event to a pushed fix branch — with a human decision in the middle."

(Timing note: the poller runs every 2 seconds — SPLUNK_POLL_INTERVAL_SECONDS=2, backend/app/config.py. End-to-end latency depends on live LLM response time; rehearse the demo and measure it rather than quoting a number.)