RS.

Case study · SharpBench · 2026

Assessed quality,
not popularity.

Agents increasingly depend on third-party services, but choosing one still means trusting reputation, a stale comparison, or a single blended score. Launched as ProviderGrade and rebranded in September 2026, SharpBench makes those tradeoffs observable and re-checkable.

My role
Built and run end-to-end
Stage
Live public product
Scope
Harness, tasks, scoring, API, site
Context
Incubated at AI Fund, with direction from Andrew Ng
SharpBench overview showing three benchmark categories, 250 nightly web-search runs, four scoring axes, and agent API access
Live-product evidence · data represented in the source artifact · September 2026View full size

3service categories, one benchmark

250runs per nightly web-search batch

100%published scores trace to raw runs

4separate axes: quality, cost, latency, reliability

01 / The problem

A leaderboard answers the wrong question.

The best service for an agent depends on the job. A web-search provider can be excellent at multi-hop quality and still be wrong for a cost-sensitive workflow. A provider can be fast in aggregate and unreliable on the exact task that matters.

Popularity compresses those differences into a vague proxy. Even a well-measured composite score hides the tradeoffs unless its ingredients stay visible. The product thesis was simple: benchmark the services on realistic tasks, preserve each axis, and let the consumer state what it values.

Key product judgment

“One score is useful only when the tradeoffs behind it remain inspectable.”

Quality, cost, latency, and reliability remain separable. Published weights provide a sensible default; API callers can request recommendations shaped to their own constraints.

Two SharpBench rankings of five web-search providers: Brave leads at 0.868 with published weights, while Serper leads at 0.918 and Exa falls to 0.460 with the cost-sensitive preset
02 / The decision surfaceThe same 250-run batch, weighted two ways: Brave leads under published weights; Serper leads when cost matters most. Portfolio reconstruction of the September 3, 2026 live readout.View full size
Portfolio reconstruction of the September 3, 2026 SharpBench batch, comparing quality, cost, reliability, and latency across five web-search providers, including a quality-versus-cost Pareto frontier
03 / Four separable axesQuality ranges from 0.84 to 0.93 while cost spans $1 to $17 per 1,000 calls. Keeping each axis visible exposes the trade the composite hides. Portfolio reconstruction of the September 3, 2026 batch; values change as new batches publish.View full size

04 / Trust by design

A score should open, not end the conversation.

On judged benchmarks such as web search, every evaluation links to the judge’s reasoning and then to the archived request, response, latency, and pricing snapshot. The audit trail is the product, not a compliance appendix.

A fixed judge with versioned prompts makes changes diagnosable in those judged categories; code-execution quality instead uses deterministic rubric checks. Task suites make “quality” concrete, and published history is never rewritten. No provider pays to be listed, ranked, or benchmarked. The result is useful to a human comparing providers and to an agent calling the recommendation endpoint.

Live SharpBench audit trail for a two-hop question, with scores from 0.20 for Exa to 0.85 for Tavily, judge reasoning, and links to archived raw runs
05 / The audit trailA September 3 live capture—not a reconstruction. One hard multi-hop task reveals a 0.20–0.85 provider spread, with every judgment opening into the evidence behind it.View full size

What shipped

From benchmark harness to an interface agents can use.

  • Versioned task suites and repeatable nightly runs
  • Four-axis scoring with an inspectable composite
  • Archived evidence with judge reasoning or deterministic rubric results, as applicable
  • An MCP surface and recommendation API for agents
  • A public site for comparing providers and individual runs
Explore SharpBench