GraphRAG Retrieval Failure Modes
What failure modes appear in GraphRAG retrieval when the underlying graph has noise?
Motivation
After deploying GraphRAG on the organizational knowledge graph, several query types returned wrong or incomplete answers. The failure patterns were non-obvious.
Hypothesis
Most GraphRAG failures at our graph scale are caused by entity resolution errors, not by the retrieval algorithm itself.
Approach
Labeled 80 query/response pairs as correct, partially correct, or incorrect. Manually traced each failure to its root cause in the graph.
Observations (in progress)
Preliminary breakdown of 27 failures so far:
| Failure type | Count |
|---|---|
| Incorrectly merged entities | 11 |
| Missing relationships (extraction gap) | 8 |
| Hallucinated path traversal | 5 |
| Correct graph, wrong LLM generation | 3 |
Status
Still collecting labeled examples. Will update once n=80 is reached.