A comparison with Codex only means something when the starting project and acceptance criteria remain the same. Changing the brief after one client struggles measures two tasks rather than two implementations.
- 1Same starting commit
- 2Same task brief
- 3Independent test run
- 4Compare differences
Work through the example
Keep the first client's output in a separate branch. Compare data ownership, error handling and failed assertions before counting changed lines.
Start with a disposable branch and synthetic data. Write the expected outcome before changing the implementation, then keep the first failing result. This prevents a later repair from quietly redefining the task. The procedure below is grounded in the repository reference; its examples must still be checked against your project and installed toolchain.
Implementation reference
The following focused section is adapted from the maintained project guide. It preserves the source’s examples and limitations.
Protocol
- 20 fixed prompts; one trial per client × prompt × condition.
- Clients: Claude Code and Codex, using each account's default model without a model override.
- Baseline gets the task and source-file constraints. Treatment additionally gets the skill entry point plus local guides, templates, examples and retrieval scripts.
- No MCP server or subagent bundle is supplied to the model. This tests the guidance treatment, not the entire product.
- Each task gets an empty
Sources/Solution.swift, Swift 6 and Foundation. No secrets or personal app source is used. - Hidden assertions are introduced after the client exits, compiled separately with the generated file, and executed independently. The client cannot edit them.
- Alternate baseline-first and skill-first by task. Each call has a timeout.
- Claude uses safe-mode, no session persistence and file tools only; Codex uses ephemeral mode, ignores user config and disables automatic project docs. Client tools/default prompts still differ, so compare conditions within a client.
- Report compilation, assertions, static review blockers and client-reported token usage. Cache/input token accounting differs by client; do not compare raw token totals across providers or convert them into savings/cost without pricing data.
- Provider failures/timeouts are recorded separately from acceptance failures. A missing token field or failed scorer is unknown, not zero.
Acceptance and failure review
| Checkpoint | What to inspect | If it does not match |
|---|---|---|
| Same starting commit | Confirm the input and environment | Preserve the failure and return to this step |
| Same task brief | Inspect the intermediate artifact | Preserve the failure and return to this step |
| Independent test run | Run the focused check | Preserve the failure and return to this step |
| Compare differences | Record the observed result | Preserve the failure and return to this step |
Ask the agent to explain the smallest change that resolves the observed mismatch. Keep unrelated refactors out of the repair. A change that makes a warning disappear is not enough if the behavior or ownership contract has changed. Re-run the same acceptance check so the before and after results are comparable.
Evidence and limits
No paired to-do app run or cost comparison is available; do not infer a winning client.
This is an educational guide. Its presence in the series does not certify a completed client-specific lab. The series evidence record separates executed checks from exercises and blocked environments.
Inspect the source used in this lesson.
Related reading
- ChatGPT and Codex: plan the app, then verify it locally
Separate a browser planning session from a connected development environment.
- Five checks for an AI-built SwiftUI app
Use persistence, search and accessible states to make “it works” a testable claim.
What to do next
Next: Prepare an Antigravity app-building trial and identify missing setup