Skip to content

Competency Questions as an Evaluation Gate

Evergreen Implementation experience Derived from one enterprise engagement. Last verified 2026-09-09.

The problem

Without measurement, quality conversations about an AI system are settled by whoever demos most persuasively. That is fine until the system is asked to carry weight, at which point nobody can say whether it got better or worse last sprint — and nobody can say what "good enough to launch" means.

Borrowed from ontology engineering, a competency question is a question the system must be able to answer to be considered fit for purpose. It is a specification written as a question. A knowledge platform's competency questions are the enterprise's actual questions, and they should exist before the pipeline that answers them.

The pattern

1. Collect real questions, from the business. Not questions the platform team can imagine, and not questions the platform happens to answer well. Every question authored by the build team is a question you already know the answer to.

2. Cover every route. Where a system routes by intent, the question set must exercise each path. An aggregate quality score across a mixed set hides one broken route behind three healthy ones.

3. Score with an LLM judge, calibrated against human scoring. LLM-as-judge is cheap enough to run on every change, which is its whole value. It is also biased toward fluent answers, so calibrate a sample against human judgement before trusting the trend.

4. Track more than answer quality. The dimensions that mattered: answer quality, route distribution, context leakage, and latency. Three of those catch failure modes that quality scoring alone misses.

5. Promote the set into a release gate. A question set is an artefact until a regression on it blocks a release. Then it is a control.

What it produces

In the engagement this came from: 34 competency questions across 4 retrieval routes, LLM-judge scored, tracking quality, route mix, leakage and latency. That harness changed the conversation with the business from impressions of a demo to measured quality on their own questions — which is the entire return on building it.

It also produced the number that most constrained the next phase, 36.7 seconds average latency, which no demo would have surfaced because a demo is watched patiently.

What I would do differently

34 questions authored largely from the platform side is a thin baseline. It was enough to establish a trend and not enough to be a specification. Had those questions come from the business first, they would have defined what to build rather than tested what was built.

The sequence that would have been better: collect competency questions, get them owned by named business people, build against them, and only then automate the scoring. Same artefact, different position in the project, considerably more leverage.

When this does not apply

  • Before there is anything to answer. During discovery, competency questions are a specification exercise, not an evaluation one — valuable, but a different activity.
  • Where the business will not supply questions. If nobody will author or own the golden set, that is diagnostic. Build the smallest possible thing and use it to provoke the questions, rather than inventing them yourself and calling the result a baseline.
  • Consumer-scale systems with real user traffic, where online signals beat a curated set. This pattern is for enterprise systems with few users and high stakes per answer.