DaoXEWhat it measures#
Availability
Share of HTTP 200 responses over N repeats — the success rate.
Latency
p50 / p95 / average round-trip for a tiny fixed request.
Price
$/1M input & output tokens, from a pricing file you fill from the official page.
How it stays honest (methodology)#
- Sanitized by construction. A record contains only: model id, ok flag, HTTP status, latency (ms), total tokens, and a coarse error category. Your key, prompt and response text are never printed or stored.
- Reproducible. Anyone can re-run it against their own account and models.
- A snapshot, not a verdict. A tiny request at one moment isn't a claim about general quality or long-term reliability.
- Fair by design. Add any other OpenAI-compatible gateway and compare side by side.
Why there's no leaderboard on this page
Publishing a table of illustrative numbers is how “sample” quietly becomes “measured” three reblogs later. We haven't run a measurement we'd be willing to defend across your models, your region and your account, so this page hands you the method instead. The tool is open — the numbers should be yours.
Design a comparison that means something#
Most gateway comparisons are invalid before the first request goes out, because the two sides were never given the same job. Get these five right and the results become worth arguing about:
- Same prompt, same parameters. Identical text, identical
max_tokens,temperature=0, samestreamsetting. A longer prompt on one side inflates its latency and its cost. - The same model, not the same model name. Aliases point at different builds across providers. Pin exact IDs from each provider's own model list, and note them in your results.
- Interleave, don't batch. Run A, B, A, B… rather than fifty of A then fifty of B. Batched runs measure the time of day and whatever the upstream was doing, then attribute it to the provider.
- Enough repeats for the percentile you quote. Five samples cannot produce a meaningful p95. If you only have a handful of data points, report the median and the range and don't dress it up as a tail metric.
- Report the spread, not just the middle. A provider with a good median and a terrible p99 will feel unreliable in production even though its average looks fine.
Traps that quietly invalidate a run#
| Trap | What it does to your numbers | How to control for it |
|---|---|---|
| Cold start | The first call on an idle route pays DNS, TLS and upstream warm-up — often hundreds of milliseconds that say nothing about the provider. | Send a throwaway warm-up request per endpoint, or discard the first N results, and do it identically on both sides. |
| Prompt caching | A repeated identical prompt can hit an upstream cache and come back implausibly fast and cheap, so your second run “proves” a speed-up no real workload will ever see. | Vary a nonce in the prompt when you want cache-miss numbers; keep it fixed and say so when you want cache-hit numbers. |
| Your own geography | You are measuring your network path as much as the provider's. A result from Frankfurt says little about the same provider from São Paulo. | State where you ran it, and re-run from the region your production traffic actually originates in. |
| Concurrency | Sequential requests measure best-case latency. Rate limits, queueing and throttling only appear under parallel load — which is what production is. | Measure twice: one sequential pass for a clean baseline, one concurrent pass at your real peak. |
| Streaming vs full completion | Time-to-first-token and time-to-last-token are different metrics, and a provider can win one while losing the other badly. | Pick one and label it. If users watch tokens appear, TTFT is the honest number; for batch jobs it's total completion time. |
| Output length drift | Even at temperature=0 two providers emit different lengths. Longer output means more latency and more cost, which reads as a slower, pricier provider. | Record usage token counts next to latency and normalise to milliseconds per output token when lengths differ. |
Run it#
Each data point is one small POST /v1/chat/completions with a fixed prompt, max_tokens=8, temperature=0, stream=false. Clone the llm-gateway-benchmark repo, then:
# refresh the SAMPLE leaderboard (no key, no spend)
npm run sample
# a real run (sends live, potentially billable requests)
export DAOXE_API_KEY="your_api_key"
export DAOXE_MODELS="exact-id-1,exact-id-2"
npm run measure && npm run aggregateCost & budget
A live run sends models × repeats requests (default 5), each capped at 8 output tokens. Check current pricing and balance first. Test budget: to be confirmed.
Add another gateway#
Any OpenAI-compatible endpoint works: set its /v1 base URL and its model IDs, and add a clearly-labeled column. Comparing honestly is the entire point — including comparing DaoXE against the official APIs.
From claim to number
Once you've run it, you no longer have to trust anyone's marketing — including ours. Pair this with how to detect model swapping to also check model integrity, not just speed.
Frequently asked questions#
Does the benchmark see my API key?
No. It runs locally and never prints or stores your key, prompt, or the model's response — only sanitized metrics.
Can I test providers other than DaoXE?
Yes — it's provider-agnostic. Point it at the official API and any OpenAI-compatible gateway and compare.
Why doesn't this page publish a leaderboard?
Because a performance table published by a vendor about itself is marketing, not evidence — and an illustrative one gets re-quoted as measured. The method and the tool are the deliverable; the numbers should come from your models, your region and your account.
Is one run a verdict on quality?
No — it's a snapshot of a tiny request. Re-run on a schedule and combine with capability probes for a fuller picture.
Try DaoXE — and benchmark it yourself
One key for GPT, Claude, Gemini, DeepSeek and more. Point the open benchmark at us and compare — don't take our word for it.