Skip to content

Route by Intent Before Reaching for a Bigger Model

Evergreen Implementation experience Derived from one enterprise engagement. Last verified 2026-09-09.

The problem

The default architecture for enterprise question answering is one path: embed the corpus, embed the question, retrieve top-k, stuff the context window, generate. When quality disappoints, the standard escalation is a bigger model, a longer context, or a reranker.

That escalation is often solving the wrong problem. A large share of real enterprise questions are not retrieval questions at all:

  • "What is the formula for this measure?" — a lookup with one correct answer sitting in a structured table.
  • "What changed in this definition between last quarter and this one?" — an aggregation across documents.
  • "What were units by market last month?" — an analytics query against a governed dataset.
  • "Why do these two markets report this differently?" — a genuine retrieval-and-synthesis question.

Routing all four through vector search means the one with an exact answer gets a paraphrase of the right table, more slowly and more expensively than a SELECT would have.

The pattern

Classify intent first, then dispatch to the path that suits it. The four routes that earned their place:

Route For Why not retrieval
Deterministic lookup Questions with one authoritative answer in structured form Faster, cheaper, and exactly right rather than approximately right
Hybrid retrieval Open questions needing synthesis across documents This is the case retrieval is actually for
Hierarchical summarisation Questions spanning more documents than fit in a context window Top-k retrieval silently truncates the evidence base
Natural language to SQL Analytics questions over governed datasets The answer is a computation, not a passage

A supervisor agent handles dispatch and tool-calling, and every route returns through the same response contract: grounded answer, citations, conversation continuity. The routes differ; the contract does not.

The failure mode it introduces

Routing is not free. A classification error becomes a routing error, and routing errors are harder to diagnose than retrieval errors.

A bad retrieval produces a visibly weak answer with visibly weak citations. A misrouted question produces a confident answer built by entirely the wrong machinery — an analytics question answered from prose documentation, for example — and it looks structurally identical to a good answer.

Mitigations that worked:

  • Track route distribution as a health metric. A sudden shift in the mix is usually a classifier problem, not a change in user behaviour.
  • Evaluate per route, never in aggregate. An aggregate quality score hides one broken route behind three healthy ones.
  • Make the chosen route visible in the response. Users detect misrouting faster than any metric, if you let them see it.

When this does not apply

  • Small or homogeneous corpora, where one path genuinely covers the question space. The complexity is not worth it.
  • Before you know the question distribution. Routes should be built against observed questions, not guessed ones. Getting the question distribution wrong means building machinery for traffic that never arrives.
  • Where no structured layer exists. Deterministic lookup and text-to-SQL both need governed structure underneath. Without it, you are back to one path whether you like it or not.