GraphRAG vs vector RAG: when a knowledge graph earns its cost

Vector RAG retrieves passages by embedding similarity and is enough for most single-passage questions. GraphRAG adds extracted entities and relations on top of those passages, so multi-hop, aggregate and entity-centric questions that span several documents can be answered with structure instead of luck. A knowledge graph earns its cost only when your questions need that structure, because extraction, graph maintenance and evaluation are real, ongoing costs that a plain vector index never asks you to pay. This guide is the decision procedure I use before I recommend one or the other, written from running a Neo4j GraphRAG pipeline in production.

What is the difference between GraphRAG and vector RAG?

Vector RAG has one retrieval primitive: split documents into chunks, embed each chunk, embed the question, return the top-k chunks by similarity, and hand them to the model. LangChain's retrieval concept page describes the same five building blocks (loaders, splitters, embedding models, vector stores, retrievers) and frames retrieval as fetching relevant external knowledge at query time, which is the whole model of the world a vector pipeline has (LangChain, Retrieval). Nothing in that pipeline knows that two chunks mention the same person, that a contract clause refers to a definition three pages earlier, or that a lecture in week 9 depends on a theorem from week 3.

GraphRAG keeps the chunks and the vector index but adds a second layer: an LLM extracts entities and relations from the text, the result is written to a graph database, and retrieval can traverse that graph rather than only comparing embeddings. The Neo4j GraphRAG package for Python calls the chunk layer the lexical graph (Document and Chunk nodes joined by FROM_DOCUMENT and NEXT_CHUNK relationships) and builds the entity layer on top of it with a pipeline of loading, splitting, embedding, schema building, extraction, graph writing and entity resolution (Neo4j, Knowledge graph builder guide). Microsoft's GraphRAG project goes further and runs community detection over the entity graph, then pre-writes a summary report per community so that corpus-level questions can be answered by map-reducing over those summaries (Microsoft GraphRAG documentation).

So the honest one-line difference is this: vector RAG answers "which passages look like this question", and GraphRAG can additionally answer "what is connected to the things this question is about". Everything else in the debate follows from whether your users ask the second kind of question often enough to pay for it.

When is vector RAG enough?

Vector RAG is enough when the answer to a typical question lives inside one or two passages and the passage can be found by meaning. Product FAQs, support articles, policy documents, a single codebase's README set, and most "what does section X say" questions are in this class. The retrieval failure modes here are chunking and embedding quality, not missing structure, and they are cheap to fix: better splitting, a hybrid of keyword and vector search, a reranker, and an evaluation set. I cover that stack in RAG development, and it is where I start with most clients because it is the cheapest system that can be correct.

Two signals tell me the graph is not needed yet. First, when I read fifty real user questions and the retrieval misses are all "the right chunk existed but did not rank", a graph will not help; a reranker and a fulltext index will. Second, when the corpus is small enough that raising top-k and letting the model read more context fixes the misses, structure is being substituted with tokens, which is often the right trade at that scale. Before adding a graph I want to see failures that a similarity search cannot fix in principle, and I want them measured, not felt.

When does a knowledge graph win?

Four question shapes recur in the systems I have audited or built, and each of them breaks similarity search for a structural reason rather than a tuning reason.

Multi-hop questions

"Which supplier's contract inherits the liability cap defined in the master agreement" needs the master agreement, the definition, and the supplier contract that references it. No single chunk is similar to the whole question, so the embedding of the question lands between three clusters and retrieves none of them well. A graph stores the reference as an edge and a two-hop traversal returns all three passages in one query. This is the "local search" case in Microsoft's framing: reasoning about specific entities by fanning out to their neighbors and associated concepts.

Aggregate and corpus-level questions

"What are the main themes across this quarter's customer interviews" is not a retrieval problem at all; it is a summarization problem over the whole corpus. The GraphRAG paper by Edge et al. names this the global question and shows that conventional RAG fails on it because it returns isolated passages instead of synthesizing corpus-wide patterns (Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization). Community summaries built at indexing time are the mechanism that makes these questions answerable at query time without reading everything.

Entity-centric questions

When users ask about a named thing (a person, a product SKU, a court case, a theorem) and the thing appears under several surface forms across documents, a resolved entity node with all of its mentions attached is a better retrieval unit than any single chunk. Entity resolution is the step that pays here, and it is also the step that goes wrong most often; more on that below.

Cross-document consistency

Cited, structured output that must not contradict itself across sections is a graph problem in disguise. If a study document states a definition in chapter 2 and uses it in chapter 7, the generator needs to see both with their provenance, and a traversal from the definition node to every chunk that mentions it is the cheapest way to assemble that context. This is the case that made the graph worth it in Jozveh-AI, where the output is a long cited document rather than a chat answer.

How is a production GraphRAG pipeline built?

The pipeline I build has six stages, and the diagram below is the whole thing. The important design property is that the vector index and the graph are two views of the same chunks, so a hit from either side can be joined back to source text and cited.

Extraction and schema

Extraction is an LLM reading each chunk and emitting entities and relations. The single most important decision is whether to constrain it with a schema. Neo4j's builder supports an extracted mode where the model infers the schema, a free mode, and a custom mode where you supply node types, relationship types and allowed patterns; the docs are explicit that the schema guides extraction rather than enforcing it, so the model may still produce elements outside it. In production I always supply a schema. An unconstrained extractor produces a long tail of relation types that nobody will ever traverse, and every one of them is a cost at write time and noise at query time.

Storage in Neo4j

Chunks become nodes with a text property and an embedding property, entities become labeled nodes with a mention edge back to the chunks they came from, and a vector index sits on the chunk embeddings. Neo4j's vector index is created with CREATE VECTOR INDEX over a label and property, configured with a dimension count and a similarity function, and queried through db.index.vector.queryNodes(indexName, k, queryVector), which yields a node and a score ordered by similarity (Neo4j Cypher manual, Vector indexes). Keeping the vectors inside the same database as the graph is what makes the join in the next section a single query rather than a two-system reconciliation.

Retrieval: traverse from a vector seed

The retriever I reach for first is the pattern Neo4j ships as VectorCypherRetriever: seed with a vector search, then run a Cypher retrieval query that has the matched node and its score in scope and can walk outward from there (Neo4j, RAG user guide). Written out by hand, a two-hop retrieval looks like this. The relationship names are illustrative; your schema will name them differently.

// Seed with the k most similar chunks, then expand two hops through the entity graph
// and return the chunks that ground the neighbors, with document provenance for citation.
CALL db.index.vector.queryNodes('chunk_embeddings', 8, $queryVector)
YIELD node AS seed, score
MATCH (seed)-[:MENTIONS]->(e:Entity)
MATCH p = (e)-[:RELATES_TO]->{1,2}(neighbor:Entity)
MATCH (neighbor)<-[:MENTIONS]-(evidence:Chunk)-[:FROM_DOCUMENT]->(doc:Document)
RETURN seed.text AS seed_text,
       score,
       [n IN nodes(p) | n.name] AS path,
       collect(DISTINCT {text: evidence.text, source: doc.path})[0..5] AS evidence
ORDER BY score DESC

The ->{1,2} quantifier is Cypher's syntax for a relationship repeated between one and two times, and nodes(p) reads the node list off a bound path, both of which are documented in the manual's variable-length patterns page (Neo4j Cypher manual, Variable-length patterns). Two hops is the practical ceiling; three hops in a dense graph returns most of the corpus.

Cited generation

Every retrieved item carries its source chunk and document, and the generator is prompted to cite by chunk identifier rather than by paraphrased title. After generation, a grounding filter checks each claim against the cited chunk and drops or flags claims that the chunk does not support. This last step is not optional in a graph system, because traversal retrieves context that is related to the question rather than similar to it, and related context is exactly what tempts a model into confident, unsupported synthesis.

Hybrid retrieval and reranking

In practice I never ship graph-only retrieval. Graph traversal is high recall for structure and blind to wording; vector search is the opposite; a fulltext index catches exact names and codes that embeddings blur. Neo4j's HybridRetriever combines a vector index and a fulltext index, and HybridCypherRetriever adds the traversal step on top of both. My own pipelines do the same thing in application code, because I want to control how the candidate sets merge and to log which retriever contributed each chunk. The sketch below is generic and framework-free.

def hybrid_retrieve(question: str, k_vec: int = 8, k_graph: int = 12, k_final: int = 6):
    q_vec = embed(question)

    # 1. Similarity: top chunks by embedding distance.
    vector_hits = vector_search(q_vec, k=k_vec)            # [(chunk, score)]

    # 2. Structure: entities mentioned in those chunks, then their 1-2 hop neighborhood.
    seeds = entities_in([chunk for chunk, _ in vector_hits])
    graph_hits = neighborhood_chunks(seeds, hops=2, limit=k_graph)  # [(chunk, path)]

    # 3. Union by chunk id, keep every origin so the trace shows who found what.
    candidates: dict[str, dict] = {}
    for chunk, _ in vector_hits:
        candidates.setdefault(chunk.id, {"chunk": chunk, "origins": set()})["origins"].add("vector")
    for chunk, path in graph_hits:
        entry = candidates.setdefault(chunk.id, {"chunk": chunk, "origins": set()})
        entry["origins"].add("graph")
        entry["path"] = path

    # 4. Rerank the union against the question; the reranker, not the retriever, decides order.
    ranked = rerank(question, [c["chunk"] for c in candidates.values()])
    return [candidates[c.id] for c in ranked[:k_final]]

The reranker is the component that makes hybrid retrieval safe. Without it the two candidate sets have incomparable scores (a cosine similarity and a path length are not on the same scale), and any hand-written merge rule is a guess. A cross-encoder that scores each candidate against the question gives one ordering, and it also demotes the graph hits that are structurally related but irrelevant to this particular question, which is the most common way a graph pipeline degrades answer quality.

How do you evaluate GraphRAG?

The same way as any RAG system, with one addition. Faithfulness, answer relevance, context precision and context recall are the baseline metrics, and I walk through how I run them with RAGAS and a claim-level LLM judge in RAG evaluation with RAGAS and an LLM judge. The addition is that a GraphRAG evaluation set must contain the question shapes that justified the graph. If your test set is single-passage questions, vector RAG will match or beat the graph and you will conclude the graph was wasted, which is the correct conclusion for that test set and the wrong one for your users.

I build the set in three slices: single-passage questions (the graph must not make these worse), multi-hop questions with a labeled gold path through the graph (retrieval recall is measured on the path, not only on the chunks), and aggregate questions scored by a judge on coverage of a human-written theme list. The graph pays for itself when the second and third slices improve and the first does not regress. Measuring extraction quality separately, on a sample of chunks with hand-labeled entities and relations, is what tells you whether a retrieval failure is a graph failure or an extractor failure. Building that harness is the core of my LLM evaluation work.

What does GraphRAG cost to run and maintain?

I will not put numbers here because they depend entirely on corpus size, chunk count, model and how often the corpus changes. The shape of the cost is what matters for the decision.

• Extraction is an LLM call per chunk, at index time. Vector RAG pays one embedding call per chunk; GraphRAG pays that plus a generation call with a long structured output, and usually a second pass for entity resolution. On a corpus that is re-indexed often this dominates.
• The graph drifts. New documents introduce new surface forms for existing entities, extraction prompts get revised, schemas grow. Each of these needs a migration or a partial re-extraction, and someone must own that. A vector index has no equivalent.
• Evaluation is broader. As above, you now test extraction quality, path recall and aggregate coverage in addition to the usual RAG metrics, and you maintain gold paths as the schema changes.
• Operations gain a database. Neo4j is another stateful service with its own backups, memory sizing, index rebuilds and upgrade path. If the team already runs Postgres and a vector extension, that is a real increment.
• Latency has a traversal term. Vector search is one index probe; hybrid retrieval is a probe plus a traversal plus a reranker call. It is bounded and predictable if hops are capped, and it is unbounded if they are not.

The way I frame it to a CTO: the graph is worth it when the questions it answers are ones you would otherwise have to answer by hand, or not at all. If the alternative is "the vector pipeline gets it right after some tuning", the tuning is cheaper. Scoping that decision, with a measured baseline on your own questions, is what my GraphRAG development engagement starts with.

What fails first, in my experience

1. Entity resolution. The same entity appears as three nodes because the extractor spelled it three ways, and the traversal from any one of them sees a third of the evidence. Neo4j's builder offers exact, fuzzy and semantic resolvers; whichever you pick, run a duplicate report after every index and read it.
2. Schema sprawl. Unconstrained extraction produces hundreds of relation types, and queries written against last month's schema silently return less. Constrain the schema and version it with the code.
3. Traversal explosion. A hub entity (the company name, the course title) connects to everything, and a two-hop query through it returns the corpus. Cap hops, cap neighbors per node, and exclude hub labels from expansion.
4. Related but irrelevant context. The graph retrieves what is connected, not what is asked, and the model synthesizes confidently from it. The reranker and the post-generation grounding filter exist for this.
5. Stale graph, fresh vectors. Someone re-embeds after a chunking change but does not re-extract, and the MENTIONS edges now point at chunk ids that no longer exist. Index the two layers in one transaction or one job, never separately.

What I observed in Jozveh-AI

Decision table

This is the table I fill in with a client during the first call, one row per real question type from their logs. If most rows land in the first column, they do not need a graph yet.

If you are unsure which rows describe your users, that is itself the answer: instrument the vector pipeline, log the misses for a few weeks, and classify them. The classification is the decision. The agent side of this, where a graph becomes tool-accessible state for a LangGraph supervisor, is in LangGraph production architecture.

Further reading

• Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization (arXiv), the paper that defined the global-question case and community summaries.
• Microsoft GraphRAG documentation, indexing with entity extraction, Leiden community detection and community reports; local versus global search.
• Neo4j GraphRAG for Python, RAG user guide, VectorRetriever, VectorCypherRetriever, HybridRetriever, HybridCypherRetriever and Text2CypherRetriever.
• Neo4j GraphRAG for Python, Knowledge graph builder guide, the extraction pipeline, schema modes, lexical graph and entity resolution.
• Neo4j Cypher manual, Vector indexes and Variable-length patterns, the syntax used in the query above.
• LangChain, Retrieval, the building blocks of a vector pipeline and the 2-step, agentic and hybrid RAG shapes.