How to evaluate an AI relay for software engineering

Byline: editorial article

Put an AI relay (API gateway / aggregator) on a software-engineering map and it is an application-layer (L7) proxy with a state-machine parser and a context scheduler.

The job changed when IDE single-line completion gave way to terminal agents such as Claude Code CLI and OpenAI Codex CLI, and to specification-driven development. Set aside generic gateway metrics such as network stability and high-concurrency load. For code generation, the question is whether the proxy breaks syntax, interrupts the agent state machine, drops long-context dependencies, or strips the optimization protocol the coding client sent.

Four engineering dimensions cover that question.

1. Syntax and escape integrity

Generated code is sensitive to characters and formatting. It is not ordinary prose. It has indentation, newlines, escapes, and embedded JSON or XML.

1. Zero-loss escaping

• Why it breaks: an SSE code chunk contains \n, \t, \", a regex \\d, or a JSON string. If the relay unescapes or recodes while it reassembles chunks, a multi-byte UTF-8 character can be cut and the client receives code that does not parse.
• How to test: ask for code with extreme escapes: a heavy regular expression, Python that embeds nested JSON strings, and C++ with pointers and macros.
• Pass bar: write the relay’s stream to a file and parse it with a local linter (eslint, flake8) or an AST parser (Babel or Tree-sitter). AST parse success must be 100%.

2. Whitespace and indentation

• Why it breaks: Python and YAML use indentation for scope. \n versus \r\n changes a Git diff.
• How to test: send a refactor of deeply nested logic, such as a Python function inside 8 loop levels.
• Pass bar: spaces and tabs in the relay output must match the upstream output at the byte level, 100%. The relay must not “prettify” or compress whitespace.

2. Tool calls and structured output

A coding agent does not only emit Markdown. It calls tools such as read_file, edit_file, and run_terminal_command.

1. JSON schema fidelity

• Why it breaks: a local edit is structured JSON with fields such as path, old_str, and new_str. If the relay breaks that serialization or drops a nested field, the agent fails to parse and crashes.
• How to test: run an agent task that edits several files together, for example an interface refactor across 5 TypeScript files, and capture the forwarded tool-call body.
• Pass bar: the forwarded tool-call object must match the upstream official API. No missing field, no type change such as string to number, and no JSON cut off early.

2. Multi-turn state

• Why it breaks: a hard bug often takes 10–30 turns: read a file, try an edit, run tests, read the error, edit again. The relay has to forward tool_use on role: "assistant" and tool_result on role: "user" for the whole chain.
• How to test: run a real agent debug of more than 10 turns.
• Pass bar: at turn N, the relay still forwards every tool-call ID and state node from the previous N−1 turns. History and state must not be dropped or scrambled.

3. Long code context and prompt caching

A medium or large codebase context is often 100k–200k tokens.

1. Cache-control pass-through

• Why it breaks: an architecture spec and shared base code stay fixed. The agent marks them with cache_control so the model service can reuse the KV cache. If the relay strips that mark, a one-line edit prefills hundreds of thousands of tokens again.
• How to test: send a 100k-token engineering context with cache_control, repeatedly.
• Pass bar:
• On the request, the upstream body still contains cache_control.
• On the response, usage returns cache_read_input_tokens.

2. Needle in a codebase

• Why it breaks: a relay may cut the prompt with RAG, compress it, or swap in a smaller model to save cost, and the model’s view of the repo shrinks.
• How to test: in a 150k-token repo, hide an interface definition in file A at the start and the call site in file B at the end, fill the middle with unrelated code, and ask how to change the interface.
• Pass bar: compare the relay’s edit with a direct official API call. If the relay’s cut context misses a cross-file variable or type, the context was damaged.

4. Complete output and instruction protection

The failure mode is a 500-line module that stops at line 300.

1. Max output horizon

• Why it breaks: a single-file refactor can need 4k, 8k, or even 16k tokens of code in one response. A small proxy buffer or a short timeout cuts the file in half.
• How to test: ask for one long module with no filler, such as a parser with hundreds of enums and conversions.
• Pass bar: the relay must be able to carry the model’s maximum max_tokens. The final finish_reason must be "stop" or "length". An early EOF is a fail.

2. System prompt integrity

• Why it breaks: a coding agent puts strict rules in the system prompt, such as TypeScript strict mode or JSDoc on every function. Some relays inject their own global prompt or trim the system prompt.
• How to test: send a system prompt with a distinctive code-style constraint and ask for code.
• Pass bar: the code follows that constraint, and a packet capture shows the system prompt was not changed by a single byte.

Checksheet