Every AI response is scored on four dimensions, each on a 0–3 scale. Scores are derived from atomized claims, not from a single end-of-response scoring pass. A response can score independently on each dimension — high accuracy + low completeness is a real outcome, as is high accuracy + low citation authority. The four dimensions are intentionally not collapsed into a composite.
The model that generates a response must not be the model that evaluates it. Self-evaluation is too generous: the same model that wrote a confident-sounding answer is unlikely to flag its own confident-sounding errors.
correctness tag, a verified_truth grounded in its own lookup, plus citation, topic, specificity, actionability, and importance.The evaluator model is configurable — default is Claude (best agentic lookup), but the cross-evaluation matrix (Claude judging GPT, GPT judging Claude) can be demonstrated on demand. The evaluator-model used is recorded in each atoms file so the detail page can show "verified by: <model>".
let A = atoms attributed to this variant let weight(atom) = atom.importance // 1–5 let credit(atom) = 1.0 if correctness ∈ {correct, correct-vague, correct-action, correct-judg, correct-range, hedge-correct} 0.6 if correctness ∈ {vague-correct, vague-action} 0.0 if correctness ∈ {WRONG, PARTIAL-WRONG} skip if correctness == admitted-gap // admitted gaps don't penalize accuracy let raw = sum(weight·credit) / sum(weight) score = floor(raw × 3 + 0.5) // 0..3, with 0.83+ → 3
admitted-gap atoms (where the AI honestly said "I don't know") do not penalize accuracy.
They penalize completeness instead.
Each scenario in data/scenarios.json carries a must_cover array — the topics
a complete answer must hit (e.g. for tiny-house: ["zoning", "size", "setbacks", "permits", "costs", "sequence"]).
Topics use the same vocabulary as atom topic fields.
let covered = distinct topics across this variant's atoms ∩ must_cover score = round(3 × |covered| / |must_cover|)
correctness == admitted-gap
on a topic in {address, zoning}, completeness is capped at 2. Admitting the AI couldn't reach
property data is honesty about the data gap — but it still means the resident didn't get
parcel-specific guidance.
This is not the same as "is the citation specific?" A vague reference to portland.gov
is more authoritative than a precise-looking citation to a Redfin listing or a third-party blog.
The hackathon brief said authoritative, not specific; this dimension reflects that.
PCC 33.205.040, Title 11, portland.gov/..., PortlandMaps, BDS Code Guide, Oregon Revised Statutesnulllet A = atoms attributed to this variant where atom carries any factual claim let raw = sum(credit(atom.citation)) / |A| score = floor(raw × 3 + 0.5)
This is the city's stated audience: "People who live and work in the city of Portland have questions about what they can do on their property." It is not a contractor, not a planner, not a code-fluent reader.
let A = atoms attributed to this variant
let raw = mean(actionability(atom)) for atom ∈ A
let jargon_penalty = 0.10 × (count of unexplained jargon tokens / 100 words) // cap at 0.30
score = floor((raw − jargon_penalty) × 3 + 0.5)
Unexplained-jargon detection looks for un-glossed acronyms (ADU, FAR, SDC, DAR, BDS) and code-citation shorthand without a plain-language gloss. The penalty is bounded to keep the dimension stable.