How to Evaluate a Coding Agent — What Actually Matters
Benchmarks, demos, and feature lists don't tell you if a coding agent works on your codebase. What matters is a handful of structural questions about how the tool is built — questions you can answer in an afternoon with a real project. Here's the checklist we use internally, and that you can use too.
1. Model access — can you choose?
Ask: can you pick any model, or are you locked into one vendor's lineup? The best tools let you use hosted providers, open-weight models, and fully local inference interchangeably. If a vendor goes down, raises prices, or deprecates a model, can you switch in one click? A tool that locks you into a single provider is a dependency, not a tool.
2. Retrieval — how does it find code?
There are two camps: embedding-based (upload your code to a cloud index) and search-based (run grep/ripgrep locally, no upload). Neither is inherently better — but the tradeoffs matter. Embedding retrieval can be faster on very large codebases; search-based retrieval keeps your code entirely on your machine and has no index to go stale. Ask: does the tool upload my code? Is there an index that can drift? What happens offline?
3. Approval model — who's in control?
The core question: does the agent change files without asking? Look for: inline diff approval (you see exactly what changed before it lands), a write-confining sandbox for commands (so a stray command can't touch files outside your project), and a separate gate for genuinely dangerous operations. Full autonomy is useful for CI pipelines — but for interactive development, approval before every write is a safety baseline.
4. Data path — where does your code go?
Trace the data flow: when you send a prompt, where does your code travel? Does it go to the model provider only, or does it also pass through the tool vendor's servers? Is telemetry opt-in? Are API keys stored locally in your OS keychain, or synced to a cloud service? For regulated work or proprietary codebases, the answer to 'where does my code go' should be short and verifiable.
5. Offline and air-gapped support
Can the tool start without a network? Can it function fully with a local model and no internet? Tools that require a cloud connection to function — even for local work — introduce a dependency that matters in regulated, remote, or high-security environments.
6. Extensibility — does it connect to your stack?
Check: can you connect the agent to your existing tools (Slack, Jira, Linear, GitHub, CI pipelines)? Is there a standard protocol for extensions, or is it proprietary? Can you define project-specific rules and behaviors that ship with your repo? The tool should fit your workflow, not the other way around.
The evaluation test
Pick a real task from your current sprint — not a toy example. Run it through the agent. Watch for: did it find the right files? Did it propose changes that made sense? Did you feel in control? Could you have switched models mid-task? That one session tells you more than any benchmark or feature table.
The right coding agent is the one that fits how your team actually works — your models, your data path, your approval requirements. Use these six criteria to evaluate any tool, including ours. The checklist matters more than the name on the box.