What this comparison provides
Section titled “What this comparison provides”This is an evaluation method, not a measured benchmark or a ranking of models. No performance winner is asserted. Choose candidate public routes from the current catalog, confirm the required protocol and capabilities, and set an approved experiment budget before making generation calls.
A useful comparison starts with your workload: extraction, support replies, code changes and long-document analysis have different acceptance criteria. Public benchmark scores cannot replace tests on representative, permissioned data.
Build a small, reviewable dataset
Section titled “Build a small, reviewable dataset”Collect examples from the task you actually need to solve, remove sensitive data and record permitted use. Include typical inputs, difficult edge cases, ambiguous requests and tasks that should be refused or escalated. Keep a held-out set separate from prompt development so repeated tuning does not turn the test into training material.
Define the rubric before seeing model answers. For extraction, score required-field correctness and invalid output separately. For support, score factual grounding and policy compliance. For code, use applicable tests and review whether the patch changes unrelated behavior. A fluent answer can still fail the task.
Keep runs comparable
Section titled “Keep runs comparable”| Record | Why it matters |
|---|---|
| Public ID, protocol, date and declared capabilities | Model names and availability can change |
| Prompt version, dataset version and output limits | Prevent hidden input differences |
| SDK version, retry policy, concurrency and timeout | Client behavior affects observed results |
| Time to first text and total wall time | Streaming responsiveness and completion time differ |
| Native usage, account price basis and final charge | Tokens alone do not establish total cost |
| Complete, failed, truncated or interrupted status | Failures must remain in the denominator |
Use equivalent tasks rather than blindly copying unsupported parameters between protocols. Fix the settings that can be aligned and disclose differences that cannot. Repeat a subset to understand variability, but do not report small samples as precise population estimates.
Score and decide
Section titled “Score and decide”Blind human reviewers to model identity where practical. Record pass/fail and rubric scores before revealing cost. Keep model-judged scores separate from human or deterministic checks; a judge may share the same biases as a candidate.
Report sample size, failed attempts, evaluation date, configuration and uncertainty alongside results. Compare cost per accepted task, not only cost per returned answer. If two candidates trade quality against latency, document that tradeoff instead of claiming a universal winner.
Start a deployment decision with a bounded trial and explicit rollback criteria. This guide does not route traffic or run benchmarks for you. Continue with protocol migration, cost and usage and errors.