The Inferock Standard
What your AI provider should make good, quantified.
This is the public methodology behind Inferock receipts, reports, and bench results: what counts as bill-bounded money loss, how evidence is graded, where current provider policy stops, why invoice-check exposure stays separate, and how money loss and time lost stay separate.
Stone Three ledger model
The headline is spent, money loss, time lost, and invoice-check exposure; arithmetic stays separate.
Provider refund policy is one ledger. Bill-bounded customer money loss is another. Time loss is its own ledger, and invoice-check exposure is visible without being summed into money loss, recognition gap, or time loss.
-
Headline
Spent
Provider spend observed by the run for priced calls it saw. This anchors the money headline.
-
Current policy
Money loss
Bill-bounded loss under The Inferock Standard, with provider-recognized recoverable and recognition gap shown inside the same money unit.
-
The point
Time loss and exposure
Time loss remains primary time. Invoice-check exposure, such as cache discount at risk, is labeled "verify your invoice" and never summed into money loss or recognition gap.
Failure taxonomy
The measures, in plain English.
Every row is constrained by evidence. If ordinary traffic does not contain the fields needed to exercise a measure, the report says that directly instead of pretending the result is clean.
| Measure | Plain-language test | Default reporting posture |
|---|---|---|
| Output integrity | Did the response come back empty, truncated, invalid JSON, or invalid against a declared schema? | Provider-recognized candidate when priced and evidenced; otherwise pricing-unknown or triage. |
| Refusal and content-filter billing | Did a provider-native refusal or content filter produce no usable completion while charge evidence says the call was billed? | Candidate only with provider-native evidence and the required charge evidence. |
| Billing integrity | Do observable usage, token recount, cache fields, duplicate identifiers, or charge observations disagree with the expected ledger? | Bill-bounded candidate only when the delta is priced and supported by provider-origin charge evidence where required; cache-discount-at-risk from usage plus pricing renders as separate invoice-check exposure. |
| Availability and downtime | Did the provider return a provider-owned error, timeout, overload, or capacity failure? | Time-primary downtime window; provider credits are creditable and capped only where verified cloud-SLA provenance applies. |
| Latency and time loss | Did the measured call exceed the active threshold for its segment and workload? | Time-primary excess wait; no configured provider latency credit basis unless supported service-tier/contract context or a published provider SLA applies. Dollar translation stays editable. |
| Tool-call validity | When a tool schema or tool choice is declared, did the model emit a valid tool name and valid arguments? | Exercised only when tool evidence exists; otherwise the row is labeled not exercised. |
| Retry amplification | Did retries or repeated request identifiers create extra wait or possible extra billed attempts? | Triage until provider billing evidence proves duplicate charge; time remains part of the standard ledger when measured. |
| Drift, factuality, security, and policy overlays | Did an approved baseline, factuality contract, citation evidence, safety evidence, or security evidence produce a measurable flag? | Reported with explicit limits. No broad automatic refund claim without the missing evidence or commitment. |
Evidence grades
Evidence grade controls what the row is allowed to claim.
-
refundable_candidate
Objective evidence exists for a provider-recognized category. It contributes provider-recognized dollars only when status, pricing, and credit-candidate rules all qualify.
-
triage_only
Real evidence exists, but the row is for review, support, future reconciliation, or a missing provider disclosure surface.
-
pricing_unknown
Evidence is preserved, but price lookup or observed charge evidence is missing or partial. The report must show
pricing_unknown — add model priceinstead of silently treating unknown as a zero-loss row. -
provider-recognized $0 or exposure
A bill-bounded money-loss row can keep provider-recognized recoverable dollars at $0. An exposure row is different: it stays visible for invoice verification, is labeled separately, and does not enter money loss or recognition gap. Only pricing_unknown withholds a dollar figure where pricing is required.
SSOT-bound SLA defaults
Defaults are public assumptions, not hidden fallback behavior.
The numeric thresholds and rate on this page are generated from packages/measure/src/sla-defaults.ts, then checked against the public page so the published standard cannot drift from the measurement defaults.
| Segment | First result threshold | Output pace threshold | Total acceptable formula | Why this source basis |
|---|---|---|---|---|
| Interactive streaming, non-reasoning | Good 1 s; acceptable 10 s. | Good 50 output tokens/sec; acceptable 44 output tokens/sec. | acceptableStartMs + outputTokens * 23 ms/output token | Why: Nielsen 1s/10s response-time limits plus OpenAI/Azure/Bedrock published TPS anchors. Sources: UX-NIELSEN, OPENAI-SCALE, AZURE-PRIORITY, BEDROCK-OTPS |
| Interactive streaming, reasoning | Good 10 s; acceptable 500 s. | Good 50 output tokens/sec; acceptable 44 output tokens/sec. | acceptableStartMs + outputTokens * 23 ms/output token | Why: Reasoning allowance from OpenAI 25,000-token experimentation reserve divided by the 50 TPS anchor. Sources: UX-NIELSEN, OPENAI-REASONING, OPENAI-REASONING-BEST, ANTHROPIC-THINKING, ANTHROPIC-CONTEXT, OPENAI-SCALE, AZURE-PRIORITY |
| Batch, non-reasoning | Good 30 s; acceptable 1 h. | Good 50 output tokens/sec; acceptable 44 output tokens/sec. | acceptableStartMs + outputTokens * 23 ms/output token | Why: Current local 30s non-interactive good boundary plus Anthropic batch typical-completion boundary. Sources: LOCAL-LATENCY, ANTHROPIC-BATCH, OPENAI-SCALE, AZURE-PRIORITY |
| Batch, reasoning | Good 500 s; acceptable 1 h. | Good 50 output tokens/sec; acceptable 44 output tokens/sec. | acceptableStartMs + outputTokens * 23 ms/output token | Why: Reasoning reserve good boundary plus Anthropic batch typical-completion acceptable boundary. Sources: OPENAI-REASONING, ANTHROPIC-BATCH, OPENAI-SCALE, AZURE-PRIORITY |
Computation methodology
The math is deliberately simple.
-
Spend and bill-bounded money loss
spent is provider spend observed by the run for priced calls it saw.
money_loss is bill-bounded loss tied to observed spend, expected charge, or provider-origin charge evidence.
provider_recognized_recoverable includes only rows that qualify under current provider-recognized policy or explicit credit basis.
-
Time-loss standard
latency_time_lost_ms equals observed elapsed time minus the active threshold, floored at zero.
downtime_duration_ms comes from clustered organic provider-failure windows, not from one failed call.
dollar_translation is time lost expressed in hours, multiplied by the active time-value rate. It stays secondary.
Recognition gap and exposure
recognition_gap = money_loss - provider_recognized_recoverable within the same bill-bounded unit. Money gap and time gap are separate. exposure is separate invoice-check evidence, such as cache discount at risk, and is never summed into money loss or recognition gap.
Disclosure annex
Providers should expose the fields needed to settle the ledger.
The annex is not a ranking. It is the disclosure checklist required for evidence-grade accountability: identity, billing semantics, billing surfaces, service objectives, retry behavior, safety receipts, retention receipts, and factuality surfaces.
-
Per-call billing joins
Invoices and usage APIs should join back to request, response, key, model, tier, token category, and time.
-
Timing receipts
Providers should expose queue, prefill, decode, first-token, inter-chunk, and service-tier timing where latency claims are sold.
-
Retry and idempotency
Billable generation endpoints should dedupe safe duplicate attempts and disclose provider-directed retries.
-
Safety and factuality receipts
Provider-attributed categories, retention class, region, citation support, and confidence surfaces are required for stronger evidence.
Adopt and cite
Use the standard, cite the standard, challenge the standard.
The specification is licensed for citation and reuse under the CC-BY license. If you publish a receipt, report, benchmark, or provider dispute using these rules, cite this page as the human-readable methodology.
-
Local benchmark
Run normal local traffic through inferock-bench and export a receipt.
-
Hosted Inferock
Use the hosted service when you want managed collection, scorecards, and recovery workflow instead of a local-only receipt.