01 / GATEWAY BENCHMARK SUITES
Test the route, not only the model.
No single benchmark proves gateway safety or model identity. Combine controlled tests, repeated sampling, and provenance.
01 / GATEWAY BENCHMARK SUITES
Test the route, not only the model.
No single benchmark proves gateway safety or model identity. Combine controlled tests, repeated sampling, and provenance.
Credential & Endpoint Boundary
Checks TLS posture, endpoint handling, error redaction, credential exposure, and retention signals.
Target: safety signal
Claimed Model Consistency
Runs repeated capability probes and response-pair checks against declared model routes.
Target: identity signal
Route Availability Window
Measures connections, timeouts, latency, retries, and variance across windows.
Target: stability + reach
Response Integrity Challenge Set
Uses invariant tasks, structured outputs, and tool behavior to detect route degradation.
Target: integrity signal
Effective Cost per Success
Compares displayed price with completed-task cost after failures and retries.
Target: price signal
Abuse & Policy Handling
Measures preservation of declared provider controls under adversarial prompts.
Target: safety signal
“Suspicious” is not the same as “proven.”
Assessments should report uncertainty, repeated samples, false-positive controls, and an appeal path.
Editorial digest of an external benchmark. Folkbench did not run this evaluation.
What changed in the evidence.
Gateway Identity & Model Consistency
Repeated task pairs reveal consistency signals; consistency does not prove identity.
30-Day Route Reliability Window
Connection, timeout, and availability patterns with sample context.
Effective Cost per Successful Task
Displayed price compared with retry-adjusted completion cost.
Every method and report includes its scope, update time, and evidence limitations.