2. System architecture

If you only remember 3 things 1. Seven nodes, one shared state: intake → {retrieval ∥ rag} → rca → recommend → approval → (router) → code_fix → END (backend/app/graph/builder.py). 2. Parallel branches merge safely because errors and events are append-only (operator.add reducers in backend/app/graph/state.py) and the two branches write different fields. 3. There are two SQLite databases with separate jobs: checkpoints.sqlite (LangGraph resume state) and app.sqlite3 (queryable incident/RCA/fix records) (backend/app/graph/checkpointer.py, backend/app/db/repository.py).

Terms, defined once (plain English)

• LangGraph: a library for building LLM workflows as a graph of nodes. Each node is a Python function that reads a shared state and returns a partial update. Edges define execution order. The framework handles parallelism, merging, and persistence.
• Node: one step in the graph — a plain function def node(state) -> dict.
• State: a single Pydantic object (IncidentState) shared by all nodes in a run.
• Reducer: a rule for merging two writes to the same state field. Default is "last write wins"; operator.add means "append" — used for errors and events.
• Checkpointer: storage that saves the state after every node so a run can pause (at the human gate) and resume later, even after a process restart.
• Interrupt: LangGraph's pause mechanism — the graph stops, saves state, and waits for a Command(resume=...) call to continue.

Components

The graph, as it exists in code today

Verified against backend/app/graph/builder.py (NODE_NAMES and the add_edge calls):

graph TD
    S([START]) --> intake
    intake -->|fan-out| retrieval
    intake -->|fan-out| rag
    retrieval -->|fan-in| rca
    rag -->|fan-in| rca
    rca --> recommend
    recommend --> approval
    approval -->|interrupt: human gate| G{route_after_approval}
    G -->|approved AND fix_eligible| code_fix
    G -->|rejected, or approved but not eligible| E([END])
    code_fix --> E

• Fan-out: intake has edges to both retrieval and rag. LangGraph runs them concurrently (verified by a test that spies on both executing — backend/tests/unit/graph/test_builder.py).
• Fan-in: both retrieval and rag have edges to rca, which waits for both.
• Router: route_after_approval is a conditional edge after approval (builder.py). It calls fix_eligibility(rca):
• no RCA → not eligible
• confidence < CODE_FIX_MIN_CONFIDENCE (default 0.5, backend/app/config.py) → not eligible
• severity == "low" → not eligible
• otherwise eligible → code_fix
• Interrupt: approval calls interrupt() before the router runs, so the human gate always fires; the router only decides what happens after the decision (backend/app/graph/nodes/approval.py).

Shared state: IncidentState

backend/app/graph/state.py — a Pydantic model. Nodes return partial updates (plain dicts); LangGraph merges them.

Reducers and why parallel branches merge safely

In state.py:

errors: Annotated[list[str], operator.add] = []
events: Annotated[list[NodeEvent], operator.add] = []

operator.add tells LangGraph: when two parallel nodes both write this field, append both instead of one overwriting the other. That is the only field the two parallel branches could race on — retrieval writes retrieved_logs, rag writes similar_error_docs, and they never touch each other's fields. So the fan-in at rca is deterministic: it always sees both the log window and the RAG docs, plus the union of any events both branches appended. (Tested in backend/tests/unit/graph/test_builder.py — a parallel-execution test proves both branches run and merge.)

Why this design? The alternative — one sequential node doing Splunk retrieval then RAG — would work but doubles latency for no benefit: the two lookups are independent (neither needs the other's output; both only need error_event). A Send()-based map-reduce was considered and rejected in the git history: "No Send() needed since it's a static fan-out, not per-item map-reduce" (commit 9f65393).

Runtime lifecycle (what happens when the server starts)

backend/app/main.py lifespan, in order:

1. init_db() — create app-DB tables (backend/app/db/repository.py).
2. ingest_error_docs() — chunk + embed data/knowledge/error_docs.txt into the in-memory Chroma collection. Raises RuntimeError if 0 chunks — the app refuses to start with an empty knowledge base.
3. get_checkpointer() — open SQLite checkpointer (backend/checkpoints.sqlite).
4. build_graph(checkpointer, ...) — wire the 7 nodes.
5. Start the Splunk poller as an asyncio task (run_splunk_poller).
6. On shutdown: set the stop event, wait up to 2s, cancel, close the checkpointer connection.

GET /api/health reports database/graph/rag_documents/splunk_poller status (backend/app/main.py).

Data flow, end to end

flowchart LR
    POS[Keele POS app] -->|HEC| Splunk[(Splunk index incidentiq)]
    Splunk -->|REST Search API, poll every 2s| Poller[splunk_poller]
    Poller -->|normalize + correlate| Graph[LangGraph run, thread_id = incident_id]
    Graph -->|fetch_log_window| Splunk
    Graph -->|similarity query| Chroma[(in-memory ChromaDB)]
    Graph -->|invoke| LLM[(OpenAI-compatible LLM, gpt-5-codex)]
    Graph -->|save_rca / save_fix_result| AppDB[(app.sqlite3)]
    Graph -->|checkpoint every node| Ckpt[(checkpoints.sqlite)]
    Dashboard[Incidents.jsx /incidents] -->|GET /api/incidents| API[FastAPI]
    Dashboard -->|POST /api/incidents/id/fix = Command resume| Graph

Key point for the demo: the FIX button is not a separate code path — it is a graph resume. POST /api/incidents/{id}/fix checks the snapshot's next node is approval, then calls graph.invoke(Command(resume=ApprovalDecision(...))) (backend/app/api/routes/incidents.py).

The two SQLite databases (do not confuse them)

Why separate? The checkpointer's schema belongs to LangGraph (opaque serialized state); the app DB is ours to query and render. Mixing them would couple dashboard queries to framework internals. The checkpointer uses JsonPlusSerializer with an explicit allowlist of our state classes (state_serde() in backend/app/graph/checkpointer.py).

Known weaknesses (honest)

• The RAG collection is in-memory (chromadb.EphemeralClient, backend/app/rag/store.py) — rebuilt on every start, one collection per process.
• Metrics are in-process counters — reset on restart, no latency histograms (backend/app/monitoring/metrics.py).
• The corpus is 10 curated entries (data/knowledge/error_docs.txt — verified count). See 05-rag-pipeline.md.
• No LLM output-quality evals exist (see 10-testing.md).