A paired benchmark compares a defined treatment against a baseline on the same tasks. Hidden acceptance checks must be independent of the agent's output, and failed runs must remain in the results.
- 1Fixed tasks
- 2Alternating conditions
- 3Independent scorer
- 4Report all outcomes
Work through the example
Compare within a client before comparing providers. Keep missing token counts unknown and report sample size and retries.
Start with a disposable branch and synthetic data. Write the expected outcome before changing the implementation, then keep the first failing result. This prevents a later repair from quietly redefining the task. The procedure below is grounded in the repository reference; its examples must still be checked against your project and installed toolchain.
Implementation reference
The following focused section is adapted from the maintained project guide. It preserves the source’s examples and limitations.
Protocol
- 20 fixed prompts; one trial per client × prompt × condition.
- Clients: Claude Code and Codex, using each account's default model without a model override.
- Baseline gets the task and source-file constraints. Treatment additionally gets the skill entry point plus local guides, templates, examples and retrieval scripts.
- No MCP server or subagent bundle is supplied to the model. This tests the guidance treatment, not the entire product.
- Each task gets an empty
Sources/Solution.swift, Swift 6 and Foundation. No secrets or personal app source is used. - Hidden assertions are introduced after the client exits, compiled separately with the generated file, and executed independently. The client cannot edit them.
- Alternate baseline-first and skill-first by task. Each call has a timeout.
- Claude uses safe-mode, no session persistence and file tools only; Codex uses ephemeral mode, ignores user config and disables automatic project docs. Client tools/default prompts still differ, so compare conditions within a client.
- Report compilation, assertions, static review blockers and client-reported token usage. Cache/input token accounting differs by client; do not compare raw token totals across providers or convert them into savings/cost without pricing data.
- Provider failures/timeouts are recorded separately from acceptance failures. A missing token field or failed scorer is unknown, not zero.
Acceptance and failure review
| Checkpoint | What to inspect | If it does not match |
|---|---|---|
| Fixed tasks | Confirm the input and environment | Preserve the failure and return to this step |
| Alternating conditions | Inspect the intermediate artifact | Preserve the failure and return to this step |
| Independent scorer | Run the focused check | Preserve the failure and return to this step |
| Report all outcomes | Record the observed result | Preserve the failure and return to this step |
Ask the agent to explain the smallest change that resolves the observed mismatch. Keep unrelated refactors out of the repair. A change that makes a warning disappear is not enough if the behavior or ownership contract has changed. Re-run the same acceptance check so the before and after results are comparable.
Evidence and limits
No new full benchmark run is completed here; no quality or token-saving percentage is invented.
This is an educational guide. Its presence in the series does not certify a completed client-specific lab. The series evidence record separates executed checks from exercises and blocked environments.
Inspect the source used in this lesson.
Related reading
- Muse Code: what our integration actually verifies
Discovery and hooks are useful milestones, but they are not a finished app.
- Review AI-generated Swift before you trust it
A focused review, a small patch and a real test beat a confident completion message.
What to do next
Next: Preparing to verify Xcode 27 agents with external skills