GUIDE-VER-001 · UPDATED
Verify the realized AI patch and its evidence—not the generated explanation.
AI-generated patch verification is a sequence of checks that makes a candidate patch inspectable: apply it through the intended path, materialize the realized diff, re-check it strictly, separate runner health from candidate failures, and record what evidence does and does not establish.
§ 01
A six-part verification workflow
Use the same evidence path for each candidate. The goal is not to manufacture a verdict from incomplete signals; it is to preserve enough context for a reviewer to distinguish an applicable patch, a broken environment, a risky repair, and an unproven behavior change.
- 01
Identify the candidate and lineage
Record whether the patch is raw, repaired, or synthesized and retain its relationship to prior candidates.
- 02
Apply through the intended harness
Check whether the candidate can be applied by the project’s intended path. A failed apply is a clear review signal, not an invitation to silently rewrite history.
- 03
Materialize the canonical diff
Inspect the realized git diff. This prevents fuzzy patch instructions or tool output from standing in for the actual repository change.
- 04
Run a strict re-check
Re-apply the canonical change under strict checks so a permissive or hidden mutation path does not become an approval signal.
- 05
Classify evidence and failures
Separate public or reproducible test evidence, provenance signals, runner-health events, and candidate-caused failures.
- 06
Export an evidence record
Keep candidate lineage, gate outcomes, receipts, risk labels, and claim limits together for the PR review conversation.
§ 02
Failure classification changes the review decision
Candidate-caused failure
Syntax, import, collection, or regression failures attributable to the candidate are hard review signals.
Environment-caused failure
Broken installs, unavailable services, missing dependencies, or infrastructure issues belong to runner health unless evidence shows the candidate caused them.
Unresolved causality
When the available evidence cannot attribute a failure, preserve the uncertainty instead of converting it into a false candidate failure.
Repair without repro
A repaired candidate may be more testable, but without reproducible behavior evidence it remains a higher-risk review path.
§ 03
What a useful evidence record contains
A review artifact should be specific enough to replay the discussion without exposing a hidden selector or collapsing uncertainty. The table below names practical record categories; it is not a claim that one field proves correctness.
| Record | Why it matters |
|---|---|
candidate lineage | Shows which raw, repair, or synthesis path produced the reviewed candidate. |
gate outcomes | Separates apply, canonicalization, strict re-check, and provenance observations. |
runner health | Keeps infrastructure problems distinct from candidate-caused failures. |
receipts and repro evidence | Preserves the source, scope, and limits of test or execution evidence. |
risk labels and hard blocks | Makes review concerns explicit without pretending that every risk is a binary result. |
§ 04
Verification has a claim boundary
Under AdaptOrch CEK S3, a shadow report can verify applicability, canonicalization, runner-health classification, and explicit evidence records. It does not prove semantic correctness, mutate the baseline-selected candidate, or use hidden tests, gold patches, official outcomes, or expected answers as selector inputs.
For the reviewer-facing decision framework, read AI PR review. For the full product boundary, read the claim boundary.
NEXT
Make the first verification path small and reproducible.
Begin with one repository and a small set of representative AI-generated patches. The useful outcome is not an inflated approval rate; it is a clear answer about which evidence can be replayed, which failures are environmental, and what a reviewer still needs to decide.