Human in the Loop AI Code Review
Keep a person responsible for merge while AI drafts code and helps find defects.
Introduction
Coding agents write patches quickly. Shipping those patches safely still depends on a review system with a human decision at the end. Human-in-the-loop AI code review uses automated checks and optional AI review to narrow the search for defects, then a person accepts, rejects, or sends the change back.
The common failure mode is skipping the human step because the diff looks plausible and the tests pass.
Understanding the Concept
Human-in-the-loop development keeps a person responsible for decisions that tools cannot own. Applied to agent output, that means code review of agent changes with a required human checkpoint before merge. Tools can advise. They cannot replace ownership of the change.
Human-in-the-loop sets where judgment stays with a person. Code review is how that judgment is applied to a diff. Together they form a gate for agent-generated patches. Prepare the change with checks and optional AI findings, then have a reviewer decide whether it may land.
Treat each change as belonging to a risk class. The class sets how much attention the human pass needs.
| Risk class | Examples | Review expectation |
|---|---|---|
| Low | Docs, comments, narrow test additions | Skim scope, confirm no unexpected files |
| Medium | Feature work behind clear contracts | Full diff, behavior and tests |
| High | Auth, payments, migrations, shared libraries | Specialist review, explicit acceptance criteria |
| Critical | Production data paths, security controls | Slow review, threat model, no rubber-stamping |
AI review and human review solve different problems. AI review is strong at spotting familiar local mistakes: unchecked nulls, missing error paths, obvious injection patterns, and tests that do not cover a changed branch. Human review of agent output is similar to ordinary human code review. The threshold for "good enough" is substantially different, especially for high and critical risk classes. Plausible agent code can still miss intent, architecture fit, and operational risk outside the diff.
Acceptance criteria set the merge bar for each change.
Definition
Acceptance Criteria
The explicit conditions a change must satisfy before merge, written so a reviewer can verify them against the diff and tests.
A useful loop looks like this:
task + acceptance criteria
↓
agent implements in an isolated workspace
↓
deterministic checks (build, lint, tests)
↓
AI review pass on configured concerns (optional)
↓
human review of intent, design, and residual risk
↓
approve, request changes, or reject and restart
Deterministic checks matter more than they look. Build, lint, and tests remove whole classes of defects before a person opens the diff. That lowers cognitive load in human review and shrinks the residual risk the reviewer must catch by hand.
AI review is optional. When you run it, configure a different model from the one that wrote the code. Different models have different blind spots, so a second model is more likely to notice mistakes the authoring model missed.
Passing one stage does not certify the next. Tests prove the cases they cover. AI review proves nothing beyond the findings it can support. Human approval is the merge gate.
Applying It in Practice
Write acceptance criteria before the agent starts. Include the behavior that must change, the files or modules in scope, and the checks that must pass. Vague prompts produce vague diffs, and vague diffs produce exhausted reviewers.
When the agent finishes, compare three records:
- the original task
- the agent summary of what changed
- the actual diff
Disagreement between those records is a scope bug. Stop and correct the task instead of reviewing a solution to the wrong problem.
Run deterministic tools first. Then, if you use AI review, run a focused pass with a narrow job such as authorization mistakes or missing error handling. Triage every finding as confirmed, rejected, or unresolved. Fix confirmed issues before the human pass.
Human reviewers should:
- Confirm the diff matches the acceptance criteria.
- Trace main success and failure paths.
- Check tests for false confidence, especially tests written by the same agent.
- Leave feedback that names a location, a concrete defect, and the required constraint.
In Treq, comments attach to changed lines and become a prompt you can send back to the agent in plan or edit mode. The reviewer stays in the loop. The agent does the mechanical revision.
Engineering Considerations
AI review reduces time spent on routine defects. It also creates noise when instructions are broad. A short list of usually-correct findings beats a long list that authors learn to dismiss.
Do not present AI review as a pass certificate. No findings means the model did not recognise a problem in the context it received. It does not mean the change is correct.
Silent human rewrites hide process failures. If you frequently rewrite agent output by hand, the task, repository guidance, or model choice is wrong. Prefer precise feedback and a controlled revision round. Restart when the approach itself is wrong.
Keep merge and deploy permissions away from the agent that wrote the code. Generation and release are different roles.
Scaling and Operational Considerations
Review capacity caps how many agents you should run. Reviewers skim when diffs get too long, and defects get shipped. Prefer small PRs that are easy to digest. Do not compensate with so many tiny PRs that the queue becomes overwhelming.
Useful metrics:
| Metric | What it tells you |
|---|---|
| Time to first human review | Whether agent output is outrunning reviewers |
| Revision rounds per change | Whether tasks and feedback are clear |
| AI finding confirmation rate | Whether AI review instructions are tuned |
| Post-merge defects | Whether the gate is actually catching risk |
| Restart rate | Whether bad tasks are stopped early enough |
Use metrics to improve prompts, risk classification, and review ownership. Do not use them to reward merge volume.
For AI-generated pull requests on a hosting platform, apply the same loop: checks, optional AI pass, human approval.
Next Steps
- What is Human-in-the-Loop Development?: where human judgment stays in the loop
- What is AI Code Review?: how AI review finds candidate defects
- Human-in-the-Loop Review Workflow: step-by-step review practice
- Review AI-Generated PRs: checklist for agent pull requests
- Parallel Coding Agents Without Branch Chaos: keep concurrent agent work reviewable