Agent-boundary testing
Phasing
What landed in v1, v2, and v3 of agent-boundary testing.
- v1: OpenAI-compatible chat-completions proxy (non-streaming),
app: agentprocess driver, cassette in trace v1 (additive header + step artifacts),assert_tool_callgrammar, trajectory diff on re-record. - v2: Landed - the Anthropic Messages API (
/v1/messages) and streaming replay for both dialects. A request withstream: trueis served the recorded turn as a synthetic SSE stream in the client's own dialect (OpenAI chat-completion chunks, or Anthropicmessage_start/content_block_*/message_delta/message_stopevents), so every existing cassette serves a streaming client with no re-record and no schema change. Chunk boundaries are synthesized rather than recorded (they carry no test signal, and recording them would break turn matching); the assembled turn is still what matches, andstreamis transport, never part of the comparison. Both wire protocols normalize into one neutral cassette, tagged per turn (protocol, defaulting toopenaiso v1 traces are byte-unchanged); a turn recorded in one dialect and replayed in another diverges on that first. To keep record and replay symmetric, the record path forwards non-streaming to the upstream and synthesizes the same stream back to the agent. Also landed: http-target agents (drive an already-running service viaagent.urlinstead of spawning a process; see "Driving a running service" above). v2 is complete. - v3: MCP servers as a second mockable boundary, for systems whose
tools are external MCP processes rather than internal functions.
Landed (v3.1): the stdio transport, with per-tool result mocks and
per-server strict-positional lanes in the trace (see "Mocking MCP tool
servers" above). Landed (v3.2): the streamable-HTTP transport
(
url:/port:), an in-process loopback listener that forwards to the real server at record (readingapplication/jsonortext/event-streamanswers) and replays the lane as single JSON bodies with zero network. The trace shape is unchanged, so a lane is transport-blind: one recorded through stdio replays through an HTTP-declared server and vice versa. Landed (v3.3): server-initiated NOTIFICATIONS and the standalone server-push SSE stream. A notification is recorded (inline in a POST's SSE body, or off the bridgedGETstream) into its lane with anafteranchor, and replayed over theGETstream flowproof now serves (a second concurrentGETis a409); anchors are recorded and replayed but never matched, so the verdict is unchanged. The remaining v3.4 slice is server-initiated REQUESTS (sampling, elicitation, roots-list), which need answer correlation: on both transports a request mid-record fails by name rather than corrupt a lane, and a JSON-RPC batch is a400.