Formal Verification for AI-Generated Code: Prove It Safer

What Formal Verification Proves - and What It Does Not

Formal verification uses mathematical reasoning to show that code satisfies a specification under stated assumptions. A specification might require that a function never accesses an array outside its bounds, that an account balance never becomes negative, or that a result matches a defined rule.

That differs from testing. Tests run selected inputs and check observed outputs. They can catch defects, but they cannot generally cover every possible input or execution path. A proof can establish a property across all inputs covered by the specification and the verifier's model.

The limit is important: a proof shows that code meets the specification, not that the specification is complete, sensible, or aligned with users' needs. If an AI-generated function is proved to follow a flawed requirement, it is still flawed software. Proofs also depend on assumptions about the language, libraries, environment, and parts of the system outside the proof boundary.

A Practical Workflow for Verifying AI-Generated Code

Write the behavior in terms that can be checked. Design by contract is one useful approach:

• Preconditions describe what must be true before a function runs.
• Postconditions describe what must be true when it finishes.
• Invariants describe properties that must remain true, often throughout a loop or across state changes.

For example, a payment function might require a nonnegative amount and a valid account, and promise that the balance changes by exactly that amount without exceeding a defined limit. Ask a domain expert to review these requirements before treating them as authoritative.

A failed proof is a signal to investigate, not automatic evidence that the code is wrong. The implementation may contain a defect, the specification may be too weak or inconsistent, or the verifier may need additional facts such as a loop invariant. Fix the underlying issue and rerun verification. Do not silence a failure by weakening a requirement unless the revised requirement has been reviewed and is genuinely correct.

A useful safeguard is to make verification repeatable. Store contracts with the code and run the verifier in continuous integration, so a change that breaks a proven property is caught before release. Keep tests in that pipeline too: proof and testing answer different questions.

A Small Example: Proving an Input-Bounds Check

Suppose an AI assistant writes a function that reads an array at index i . A contract sketch could say:

requires: 0 <= i and i < array length ensures: returned value equals array[i] ```

The verifier can check that the access is safe whenever the precondition holds, and that the returned value meets the postcondition. But this does not establish that every caller supplies a valid index. Each caller must satisfy the function's precondition, or the program must first check the index and handle an invalid value.

That distinction makes the proof useful: it exposes exactly where the safety guarantee begins and what callers must do. It also shows why a contract needs human scrutiny. If the desired behavior is to reject invalid indexes gracefully, a precondition alone may be inadequate; the function's specification should instead describe both valid and invalid input cases.

How to Add Verification Without Rewriting an Entire System

For legacy-code migration, verification can help establish that a translated component preserves important behavior. But specifying old behavior may require substantial investigation, and proving equivalence can be difficult. Running old and new versions through the same representative workflows, alongside tests and human review, provides useful comparison - it is not a substitute for a proof of equivalence.

When Is Verification Worth the Effort?

Formal verification is most compelling when a defect could cause significant financial, safety, privacy, or security harm; when the component is bounded enough to specify; and when the property can be checked repeatedly as code changes. For low-risk features with frequently changing requirements, focused tests and review may be a better use of time.

Before starting, ask:

• What specific property must always hold?
• Are the preconditions and expected behavior complete and reviewable?
• What code, libraries, or environmental assumptions sit outside the proof?
• Can the verifier run consistently in development and CI?
• What tests and human reviews will cover the remaining risks?

Choose one high-risk function in your AI-assisted codebase, write down its preconditions and postconditions, and evaluate whether a verifier such as Dafny can prove the behavior you need. Treat the proof as one strong, targeted safety net - not a certificate that the whole system is correct.

Step-by-Step Guide

1. 1

   Choose a bounded, high-risk component

   Select a manageable function or algorithm, such as a permission check, payment calculation, parser, or safety-related state transition. Focus on a component where a defect could cause significant harm.

2. 2

   Write and review a precise contract

   Define the component's preconditions, postconditions, and relevant invariants. Ask a domain expert to confirm that the requirements cover expected behavior and important edge cases.

3. 3

   Implement the component and run a verifier

   Use the reviewed requirements to guide implementation, then check the code with a suitable verifier, such as Dafny. Treat failed proof obligations as signals to investigate the code, specification, or missing proof details.

4. 4

   Add tests and human review

   Review both the implementation and its specification. Test realistic inputs, error handling, integration behavior, and dependencies outside the proof boundary.

5. 5

   Run verification and tests in CI

   Store contracts with the code and run the verifier and tests in continuous integration so changes that break checked properties are caught before release.

Key Statistics

• A design-by-contract specification commonly uses three kinds of conditions: preconditions, postconditions, and invariants.The article's practical workflow identifies these three contract elements as ways to state requirements a verifier can check.
• The article recommends combining two complementary safeguards: formal verification and testing.Its workflow explains that proofs check specified properties, while tests help cover realistic examples, integration behavior, and dependencies outside the proof.
• The article names one example verification tool: Dafny.Dafny is presented as a tool that combines a programming language with constructs for stating properties and proving them; this is an example, not a comparative benchmark.

Frequently Asked Questions