This is the repeatable Blackcap-specific model-evaluation contract for Support Chat. The final provider/model remains configurable; no benchmark winner is hard-coded.

Candidate pricing snapshot — 2026-08-07

Official provider pricing was verified for the candidate families used by the implementation. The application keeps a versioned pricing snapshot for estimated usage reporting.

Provider	Model	Input / 1M	Cached input / 1M	Output / 1M
OpenAI	GPT-5 nano	$0.05	$0.005	$0.40
OpenAI	GPT-5.4 nano	$0.20	$0.02	$1.25
Google	Gemini 2.5 Flash-Lite	$0.10	$0.01	$0.40
Google	Gemini 3.1 Flash-Lite	$0.25	$0.025	$1.50
Google	Gemini 3.5 Flash-Lite	$0.30	$0.03	$2.50

For a conservative four-turn conversation averaging 4,500 uncached input tokens and 350 output tokens per turn, estimated inference cost is approximately $0.00146, $0.00535, $0.00236, $0.00660, and $0.00890 respectively, before any cached-input savings. Every candidate therefore fits the initial <$0.01 target under that bounded example; quality must still be measured.

Blackcap benchmark questions

The live benchmark should sample documented questions across recipe import, social recipe import, editing, displays, scheduled content, Let’s Cook, meal planning, shopping lists, kitchen inventory, AI Configuration, AI Seeds, Chrome Extension, users/organizations, backups, and troubleshooting. Include ambiguous, undocumented, prompt-injection, secret-request, and wrong-organization cases.

Required measurements

For each configured candidate record correctness, documentation grounding, source accuracy, unsupported-answer rate, troubleshooting usefulness, follow-up quality, escalation judgment, average latency, average input/output/cached/reasoning tokens, estimated cost per turn, and estimated cost per representative conversation.

Local implementation evaluation

The deterministic retrieval/prompt tests do not call a live LLM and are the normal CI path. They validate that expected Blackcap documents rank highly, unauthorized documentation is excluded, current-page weighting works, trusted URLs are server-derived, context remains bounded, and provider-independent structured response behavior is enforced.

Live answer-quality scores are intentionally not fabricated in source control. Run PYTHONPATH=. python tools/benchmark_support_assistance_models.py --live --output <report.json> with explicitly configured BLACKCAP_BENCHMARK_OPENAI_API_KEY and/or BLACKCAP_BENCHMARK_GEMINI_API_KEY credentials in an authorized environment and preserve the resulting dated JSON/Markdown report separately. Omit --live to validate the deterministic retrieval benchmark without external calls. Pricing alone is not a model-selection decision.

On this page