We May Be Measuring AI-Generated Software Completely Wrong

Why this question

Software teams have long built confidence through some form of independence: someone other than the author reviews the change, tests are written against a specification rather than the implementation, and separate tools look for different classes of defects. The point is that a second look catches what the first one missed.

AI changes what “a second look” means. The same kind of system can now write the implementation, write the tests, review the change, and summarize the evidence for a human approver. Every artifact can look thorough. That isn’t the same as each one being an independent check.

If generation and verification share the same model, training data, prompt, or specification, their errors may be correlated. A large volume of passing evidence could then mean less than it appears to.

What this project investigates

• Verification provenance. Who or what produced each piece of evidence, and from what inputs?
• Same-model versus cross-model review. Does a different model add independent evidence, or does it share the same blind spots?
• Specification independence. Is a check independent if it derives from the same possibly flawed specification?
• Heterogeneous verification. Do mixed methods (tests, static analysis, formal properties, human review) fail in less correlated ways than repeated use of one method?
• Correlated AI failure modes. When AI-generated code is wrong, are AI reviewers systematically likely to miss it?
• Human accountability. What should a human approver be accountable for when most of the evidence in front of them was machine-produced?

How I intend to approach it

The method is still being designed. The plan is to:

1. Review the public literature on software assurance, independent verification and validation, and recent empirical work on AI code generation and AI-assisted review.
2. Define a small, reproducible experimental setup comparing verification configurations that differ in one dimension of independence at a time.
3. Publish code, prompts, and data so the results can be checked and replicated.

Limitations

This is a question, not a conclusion. Any eventual experiments will cover a narrow slice of real-world software development, and results from one setting may not carry over to others. All evidence will come from public sources or from experiments designed and published here. None of it comes from any employer’s internal systems or data.