Published benchmark
Docs that can’t lie.
Here’s the measurement.
Every AI docs product claims accuracy. We measure it, publish it, and ship the harness so you can reproduce it on your own repo. To our knowledge, no other documentation or support-AI vendor publishes groundedness metrics at all.
Deterministic docs grounded by construction Reference pages are a pure function of the parsed AST — no LLM in the rendering path, nothing to hallucinate. | 100% |
Chat citation validity Every citation in AI answers resolves to a real, retrievable page; invented references are stripped by validation before the answer is returned. | 100% (8/8 sampled) |
Fabricated citations across the benchmark run 13 questions (10 in-scope, 3 adversarial out-of-scope) against a 4,385-page corpus. | 0 |
Out-of-scope refusal Asked things the docs can’t answer, the AI says "I don’t know" instead of inventing. | 3/3 |
Why we can make this claim
- Input is the code, not prose. Pages are generated from a parsed entity graph (classes, functions, endpoints, types) — every signature on a page exists in the source.
- Citations are validated, not trusted. The AI must cite retrieved page ids; any citation that doesn’t match a retrieved source is dropped before you see the answer.
- Refusal over invention. When retrieval finds nothing, the model isn’t called with an empty stage to improvise on — it declines.
Methodology
Benchmark run live against the deployed platform over Atloria’s own monorepo (4,385 generated pages, 29k+ parsed entities): 10 in-scope questions about real components and 3 adversarial out-of-scope questions. Citations were re-fetched and checked to resolve (HTTP 200) against the published corpus. The harness is a ~150-line script — reproducible against any project on the platform.