Evaluation
Model scorecard
Rules decide; models read. Every appetite disposition in Astra Risk comes from a deterministic engine that applies the carrier's thresholds, so the same evidence always gives the same answer and every factor cites its source. Language models do the reading: they pull the construction year and the three-year loss count out of broker prose. This page measures each reader on the same 31 broker notes.
Invented values are the failure that matters most: a year or a loss count that the note never stated. Missed values cost a broker question; wrong values cost a wrong check.
This run hit API refusals, so the model rows understate what the model can read. The refusal counts sit under each reader; rerun
npm run eval:scorecard once the quota resets.| Reader | Accuracy | Invented | Missed | Wrong | p50 | p95 | Cost per 1,000 notes |
|---|---|---|---|---|---|---|---|
| Deterministic parserregex parser | 100%31 of 31 | 0 | 0 | 0 | 0 ms | 1 ms | $0 |
| Gemini aloneno model call completed31 of 31 notes got no model result (31 × HTTP 429) | 0%0 of 31 | 0 | 0 | 0 | 1.1 s | 1.2 s | — |
| OpenAI alone | Skipped: OPENAI_API_KEY unset on the machine that ran this. | ||||||
| Models and parser, resolved by agreementno model call completed; the parser supplied every value0 of 31 notes got no model result (31 × HTTP 429) | 100%31 of 31 | 0 | 0 | 0 | 1.1 s | 1.1 s | — |
Generated 9/20/2026, 5:14:12 AM with 1000 ms between model calls. Accuracy counts a note as correct only when both fields match what a careful underwriter would take from it. Cost uses published list prices per million tokens and the tokens each call reported; the parser runs in-process. Reproduce with npm run eval:scorecard; the cases live in evals/suites/extraction.ts.
What each reader got wrong
Deterministic parser: 0 of 31
Every note read correctly.
Gemini alone: 31 of 31
- plain construction sentence with no losses: expected year 2012, losses 0; got no result.
- labelled year built and claims count: expected year 1998, losses 1; got no result.
- loss count written as a word: expected year 2003, losses 2; got no result.
- renovation years are not the construction year: expected year 1988, losses 0; got no result. Roof and sprinkler dates must not overwrite the original construction year.
- dollar loss total is not a claim count: expected year 2001, losses none; got no result. A dollar total says nothing about how many claims occurred in three years.
- count with a dollar detail alongside: expected year 2010, losses 3; got no result.
- broker update supersedes the original note: expected year 2004, losses 1; got no result. The singular 'one loss' in the correction must supersede the earlier count of 4.
- singular claim with a digit: expected year 2015, losses 1; got no result. Singular 'claim' and 'loss' are common in broker notes.
- losses none: expected year 2007, losses 0; got no result.
- loss-free phrasing: expected year 2011, losses 0; got no result.
- nothing supplied yet: expected year none, losses none; got no result.
- decade only is not a year: expected year none, losses 0; got no result. A decade is not a construction year; the guideline needs the exact year.
- retrofit code year is not the construction year: expected year 1996, losses 0; got no result.
- five-year count does not establish the three-year count: expected year 2005, losses none; got no result. The demo rule is a three-year count; a five-year count is only an upper bound, so the broker must be asked.
- claims history label and year of construction label: expected year 1994, losses 2; got no result.
- building count is not a claim count: expected year 2016, losses 0; got no result.
- loss count before the year: expected year 2009, losses 0; got no result.
- several buildings use the oldest construction year: expected year 1975, losses 0; got no result. Guidelines assess the oldest building; taking the last year mentioned hides a pre-1980 exposure.
- correction inside a single note: expected year 1998, losses 0; got no result.
- loss run summary with a total: expected year 2002, losses 2; got no result.
- zero written as a word: expected year 2019, losses 0; got no result.
- street number is not a year: expected year 1965, losses 1; got no result.
- hyphenated labels: expected year 2008, losses 1; got no result.
- empty note: expected year none, losses none; got no result.
- future year is not a construction year: expected year none, losses 0; got no result. A building that does not exist yet has no construction year to check.
- deductible per claim is not a count: expected year 2013, losses 0; got no result.
- one-year count does not establish the three-year count: expected year 2006, losses none; got no result. A twelve-month count is only a lower bound for three years.
- zero over a longer window is still zero: expected year 2000, losses 0; got no result. Zero claims in five years implies zero in three.
- count of carriers is not a count of claims: expected year 2016, losses 0; got no result.
- updated systems do not correct the construction year: expected year 1965, losses 0; got no result.
- claim reference number is not a count: expected year 2010, losses none; got no result. One claim is mentioned but its window is not stated, so the count stays unknown.
Models and parser, resolved by agreement: 0 of 31
Every note read correctly.
The appetite engine itself is pinned by 189 offline cases that run in npm test; see Models and rules.