Introducing Folkbench Model Degradation Detection: Auditing API Relays with Current-Events Probes, Model Fingerprints, and Direct Official Baselines

Folkbench has launched model degradation detection inside the Integrity Monitor. Conventional model leaderboards evaluate what a model can achieve under official or single-host conditions, while typical API relay directories stop at listed pricing, HTTP uptime, and response latency. Neither answers the central operational question for relay users: when a model is served through a third-party API relay, do its reasoning depth, constraint adherence, and model-level behavior remain consistent with the direct official API?

To close that evaluation gap, Folkbench embeds an automated probe suite spanning current-events and model-fingerprint dimensions, calibrated continuously against independent direct official API keys within the same measurement windows. Anchoring third-party relay behavior to a live official baseline converts subjective suspicion into observable, reviewable public signals.

Key elements of this capability include:

• Dual-dimension probe design: Pairs time-updated current-events probes with model-fingerprint probes tracking reasoning habits, token allocation, and returned fingerprint telemetry—defending against static caching and superficial answer imitation.
• Independent official calibration: Runs identical probes through isolated direct official API keys in the same window, separating upstream vendor updates or natural variance from relay-specific drift.
• Model and service-group granularity: Tracks probe agreement, input tokens, reasoning tokens, model fingerprints, and available cache checks separately for each model and service group rather than issuing opaque site-wide averages.
• End-to-end selection and switching workflow: Connects Integrity Monitor telemetry with Folkbench's four public rankings, side-by-side relay comparison, and the open-source desktop client Folkbench Switch.

Why conventional leaderboards and uptime checks miss model degradation

In a standard relay architecture, an intermediary gateway sits between the caller and the upstream model lab. Most relays simply forward requests and meter usage, yet cost pressure or peak-hour routing can introduce three forms of degradation that network telemetry alone cannot distinguish:

• Silent model substitution: The caller requests a flagship model, while peak-hour or long-context traffic is redirected to a cheaper lightweight model or an older snapshot.
• Reduced reasoning budgets: The underlying model family remains unchanged, but reasoning effort is lowered or internal thinking steps are truncated so the endpoint replies prematurely.
• Context or output truncation: Earlier turns are dropped or output caps are tightened to conserve tokens and bandwidth, causing missed constraints or fragmented logic in multi-step tasks.

Figure 1: Green surface connectivity (HTTP `200`, low initial latency, fluent short replies) does not guarantee intact capability delivery. Gateway routing swaps, lowered reasoning budgets, and truncated histories occur below standard network telemetry.

All three share an important property: surface connectivity remains intact. The endpoint still returns HTTP 200, casual prompts produce fluent replies, and shorter reasoning traces can even make responses appear faster. Standard availability and latency checks cannot distinguish a genuine flagship run from a trimmed or substituted one. At the same time, sampling randomness means isolated failures happen even on healthy endpoints—as analyzed in [Did the Model Really Get Dumber?](/en/blog/did-the-model-get-dumber/), individual screenshots cannot separate normal variance from systemic degradation.

Why naive user checks fail against modern routing

When users suspect a route has degraded, two common manual tests fail under real-world relay routing:

1. Asking the model to state its identity: A language model has no direct visibility into the proxy layer forwarding its traffic. A system-prompt directive claiming a flagship identity—or standard phrasing memorized by a smaller model—produces confident self-identification regardless of which weights actually answered.
2. Reusing static benchmark puzzles: Widely circulated brainteasers and classic reasoning checks already appear across public training corpora, allowing compact models to recall expected answers. Worse, repeated fixed prompts are trivial for an intermediary to cache statically or whitelist onto an unthrottled upstream route during checks while degrading ordinary production traffic.

Figure 2: Two structural failure modes of manual relay checks. On the left, a gateway system prompt overrides identity statements regardless of backend weights. On the right, gateway filters recognize repeated static prompts and return static caches or route checks to a clean whitelist while keeping daily traffic degraded.

Reliable degradation detection therefore requires prompts that cannot be pre-cached or memorized, paired with signals that go beyond whether a single final answer happens to look right.

Dual-dimension probes: verifying live understanding alongside model fingerprints

To detect hidden drift accurately, Folkbench separates evaluation prompts and protocol telemetry into two complementary dimensions collected together in each test window.

Figure 3: Same-window architecture combining dynamic current-events probes (blocking static caching and stale fallbacks) with model-fingerprint telemetry (tracking probe agreement, input tokens, reasoning tokens, fingerprint fields, and cache checks) across isolated official direct keys and target `model × service group` routes.

1. Current-events probes: blocking static caching and stale fallbacks

Current-events probes incorporate recent real-world timelines, newly surfaced context, and time-sensitive relationships into structured reasoning tasks. Unlike static trivia or unchanging puzzles, these prompts evolve continuously:

• Prevents static response caching and test whitelisting: Because prompt context tracks recent events rather than a frozen question bank, relays cannot satisfy audits with pre-computed static replies.
• Surfaces outdated snapshots and over-compressed substitutes: For example, when a probe requires ordering recent public events and reasoning over newly introduced constraints, an older fallback snapshot trips over knowledge-cutoff boundaries, while heavily compressed models struggle to preserve multi-condition consistency.

2. Model-fingerprint probes: inspecting reasoning habits and token telemetry

Even if a substitute model answers an ordinary objective question correctly—or the true model runs with half its thinking budget removed—final-answer accuracy alone may miss the shift.

Model-fingerprint probes target behavioral patterns that remain distinct across model families under multi-step constraints, strict formatting boundaries, and structured reasoning paths. Alongside output alignment, Folkbench records protocol-level vital signs returned by the endpoint:

• Probe agreement: Consistency of task execution and behavioral alignment across fingerprint checks.
• Input tokens and reasoning tokens: Token allocation consumed during context ingestion and internal pre-answer reasoning.
• Model fingerprints and cache checks: API-returned model fingerprint fields and cache-check observations wherever data is available.

For instance, when a reasoning endpoint silently lowers reasoning effort behind the scenes, surface prose may remain grammatically polished while reasoning token volume and probe agreement drop together. Likewise, if traffic is routed to a different model series, deviations across fine-grained formatting discipline and fingerprint telemetry surface rapidly even when simple replies pass.

Independent direct official API key calibration: turning raw scores into a live baseline

Probes alone still leave an attribution problem. If a relay route's probe agreement declines during a given window, raw numbers by themselves cannot distinguish between two very different causes: relay-side degradation versus an upstream vendor update, transient lab instability, or the baseline pass rate of newly authored current-events tasks on the official endpoint itself.

Without a synchronized reference point, absolute figures invite false positives. Folkbench resolves this through a core architectural rule: every degradation probe suite is continuously calibrated using independent direct official API keys. These isolated keys communicate directly with model providers—bypassing all third-party relays—and run identical current-events and fingerprint probes in the same evaluation cadence to establish a live official baseline.

Calibration mechanism: Without a simultaneous anchor (left), a drop cannot be attributed cleanly to upstream changes versus relay drift. Dual-track calibration (right) runs isolated official direct keys alongside each `model × group` route inside the same `W_t` window to measure relative agreement, reasoning-token, and fingerprint deltas.

Figure 4 (illustrative schematic, not measured telemetry): Side-by-side attribution logic using an independent direct official baseline. Parallel movement in Case A reflects upstream or task variance; isolated divergence plus shrinking reasoning token bars from `T3` in Case B isolates drift on that specific `model × group` route.

Folkbench applies strict interpretive boundaries to these signals. As noted on the [Integrity Monitor](/en/arena) page, a single response or isolated percentage cannot establish model substitution on its own. Telemetry should be read alongside sample size, consecutive time windows, token distributions, and test conditions—providing an objective, reviewable early-warning signal rather than conjecture.

From degradation detection to full-stack evaluation: what differentiates Folkbench

Most AI evaluation sites fall into one of two categories: model leaderboards that benchmark official weights on static datasets, or relay directories that track advertised token prices and basic HTTP reachability. Adding degradation detection enables Folkbench to move beyond static model ceilings and shallow ping checks toward continuous verification of real-world delivery.

Four structural differences set Folkbench apart:

Figure 5: Folkbench's four-stage operational loop connecting group-level discovery (`/rankings), calibrated degradation auditing (/arena), exact-group comparison (/stations/compare), and diff-previewed local switching (Folkbench Switch`). Web telemetry remains separate from local Switch features.*

1. Model- and group-level measurement instead of site-wide generalizations

A single third-party relay frequently operates distinct pools—discount promo tiers, dynamic-multiplier routes, and higher-cost official-key pools—backed by completely different upstream capacity. Assigning one vague score to an entire domain hides what users actually buy. Folkbench benchmarks at the level of each specific model and service group, ensuring premium routes never mask compromised budget pools.

2. Four specialized public leaderboards across operational dimensions

Capability integrity is the prerequisite; production reliability and real cost matter equally. Folkbench complements Integrity Monitor with four dedicated rankings:

• [Availability rankings](/en/rankings): Looks past HTTP 200 status codes to filter out "fake 200" error payloads, mid-stream cutoffs, and malformed SDK responses.
• [Stability rankings](/en/rankings/stability): Evaluates sustained completion under longer workloads and busy windows rather than single-request pings.
• [Latency rankings](/en/rankings/latency): Separates Time to First Token (TTFT) from ongoing generation throughput so averages never hide multi-second initial stalls.
• [Session cost rankings](/en/rankings/cost): Estimates practical expenditure across realistic coding workflows, exposing true cost efficiency behind headline discount multipliers.

3. Free public Integrity Monitor and side-by-side Relay Comparison

Replicating live current-events probes, fingerprint checks, and independent direct official API calibration on one's own requires maintaining multiple direct billing accounts, updating fresh prompt sets, and running scheduled cross-route checks. Folkbench publishes this telemetry openly in the [Integrity Monitor](/en/arena) and pairs it with [Compare relays](/en/stations/compare), allowing side-by-side inspection of competing routes under the same model on one screen.

4. Closing the loop from evaluation to configuration: open-source Folkbench Switch

Most benchmark sites stop at displaying tables, leaving users to edit local JSON, TOML, or YAML files by hand. Folkbench pairs its public metrics with [Folkbench Switch](https://github.com/Folkbench/folkbench/releases), an open-source desktop client built with Tauri and Rust (macOS .dmg and Windows .exe, supporting Claude Code, Codex, Gemini CLI, Aider, OpenCode, Kimi CLI, and more):

• Configuration diff previews: Shows exact changes before writing local tool files while preserving unrelated custom settings.
• Codex credential protection: Safely parks official OAuth credentials when switching to a third-party relay and restores them cleanly when switching back.
• Public route discovery and local usage estimates: Surfaces public route availability, status windows, and pricing inside the client alongside local session token cost estimates. (Integrity monitoring and cache-compensation tracking remain web-platform evaluation features rather than client-integrated checks.)

Putting degradation signals to work

1. Filter by operational priority: Start with [Availability](/en/rankings), [Stability](/en/rankings/stability), [Latency](/en/rankings/latency), and [Session Cost](/en/rankings/cost) to shortlist specific model-and-group routes that match your workload.
2. Verify baseline alignment in Integrity Monitor: Check probe agreement, reasoning tokens, and model fingerprints on the [Integrity Monitor](/en/arena) against the live official baseline—especially over consecutive windows when quality feels inconsistent.
3. Compare the exact tier before funding: Use [Compare relays](/en/stations/compare) on the specific group you plan to purchase rather than extrapolating from another pool under the same provider.
4. Maintain rapid failover via Switch: Store primary and backup routes in Folkbench Switch so you can inspect the configuration diff and switch cleanly within seconds whenever route telemetry drifts.

As production workflows increasingly rely on third-party API routing, evaluation must verify not only which model leads on paper, but whether the route you pay for actually delivers it. By bringing current-events probes, model fingerprints, and independent direct official calibration into continuous public view, Folkbench establishes a clearer, auditable standard for choosing AI API routes.