Engineering Transparency

Benchmarking & Pricing Verification Methodology

How we collect, verify, and simulate LLM API economics without vendor bias or synthetic marketing inflation.

1. Direct Provider Upstream Ingestion

We maintain direct programmatic monitors against public pricing registries and API endpoints from Anthropic, OpenAI, Google Cloud Vertex, DeepSeek, and Together AI. When a provider adjusts per-token rates (such as DeepSeek R1 release or OpenAI tiered volume discounts), our internal data dictionary is verified via automated assertions before publication.

Assertion: Price changes must match official billing release notes or documentation commit logs.

2. Prompt Caching Economics & KV Ingestion

Prompt caching fundamentally altered token finance in 2024–2026. Rather than assuming static linear pricing, our simulator splits inputs into cached reads vs uncached baseline writes:

  • Anthropic: 5-minute TTL minimum threshold; cached writes cost 1.25x baseline, while cached reads drop to 0.10x baseline ($0.30/M on Sonnet).
  • DeepSeek: Context-aware prefix cache with automatic hits for prompts exceeding 1,024 tokens.
  • Google Gemini: Implicit caching for recurring context windows over 32k tokens.

3. Standardized Workload Normalization

To prevent misleading apples-to-oranges comparisons, all cost estimates normalize across 30-day billing cycles with identical context boundaries. Benchmark scores (MMLU, SWE-bench Verified, and HumanEval) are collected strictly from peer-reviewed evaluations or official technical reports.

Zero Sponsored Placement Policy

Model rankings in the calculator are sorted strictly by total simulated monthly bill based on the user's workload parameters. We do not accept paid placement to promote specific model providers.