Software quality processes in the age of AI for regulated environments

Regulated software has always lived under one simple demand: show your work. A medical device, a flight control component, a payment platform, or an energy management system must be able to answer, for any line of code, why it exists, what requirement it serves, how it was verified, and who was accountable for the decision. Frameworks differ in vocabulary (IEC 62304, DO 178C, ISO 26262, 21 CFR Part 11, SOC 2, PCI DSS) but they converge on the same expectation: traceability, verification, and controlled change.

AI coding tools do not remove that expectation. They make it easier to violate by accident. A model can produce a plausible module in seconds, and the same speed that makes it attractive makes it easy to skip the part where a human understood what was built. This post describes a quality process we use and recommend for teams shipping software under regulatory scrutiny while still getting real leverage from AI.

The problem is provenance, not capability

The models are capable. That is not the issue. The issue is that regulated quality systems are built around provenance: every artifact has an origin, an author, a review, and a link to a requirement. When a developer pastes a prompt, accepts a suggestion, and commits the result, three of those four links are missing. The code exists, and nobody can say precisely why it takes the shape it does.

An assessor will not ask whether you used AI. They will ask who reviewed the change, what tests demonstrate the requirement, and how you know the change did not introduce an unintended behavior. If the honest answer is "the model wrote it and the tests passed," you have a finding.

Principle one: AI output is a draft, never a deliverable

Treat everything a model produces the way you would treat a contractor's first submission. It enters the process as an untrusted input. It becomes a deliverable only after a named engineer has read it, understood it, and taken responsibility for it.

In practice this means:

• A human author of record for every change. The person who commits generated code is its author for audit purposes. They must be able to explain it in a review without consulting the tool.
• Review by someone who did not prompt. The reviewer should not be the person who generated the code. Two sets of eyes remain the cheapest defect filter there is, and generated code needs it more, not less, because it tends to look finished.
• No direct commits from agents. Agentic tools that open pull requests are fine. Agentic tools with write access to protected branches are not.

Principle two: requirements first, generation second

The most common failure we see is teams generating code and then writing the requirement to match. That inverts the whole model. Traceability only means something if the requirement existed before the implementation.

Write the requirement, write the acceptance criteria, and then use the model to propose an implementation against those criteria. The prompt itself becomes a useful artifact here: a prompt that references requirement identifiers and acceptance criteria produces code that is easier to trace, and the prompt can be stored alongside the change as design rationale.

Principle three: verification you did not generate

If the same model writes the code and the tests, you have one opinion checked against itself. The tests will pass, and they will prove very little.

Our rule: generated tests may be used to broaden coverage, but the tests that demonstrate a requirement are written or at least rewritten by an engineer who read the requirement. Independent verification is not a bureaucratic nicety. It is the thing that catches the model confidently implementing the wrong behavior.

Tools that help here:

• Mutation testing to prove the test suite actually detects defects rather than merely executing lines.
• Property based testing for anything with a specification you can state as an invariant.
• Static analysis and type systems as the tireless first reviewer. Generated code should be held to the strictest linter and type configuration the language supports, with no suppression comments allowed without a recorded justification.
• Contract tests at every service boundary the model touched, because generated code is most likely to be subtly wrong exactly where it had to guess an interface.

Principle four: record the tool in the change record

Your configuration management process already records who changed what and why. Add what. Recording that a change was AI assisted, and which tool and model version were involved, costs nothing and answers a question assessors are starting to ask. It also lets you do something useful later: when a model or tool version is found to have a systematic flaw, you can find every change it touched.

This does not need to be heavy. A trailer in the commit message, a field in the pull request template, or a tag in the ticket is enough.

Principle five: the maintenance plan is part of the deliverable

Generated code has a particular failure mode: it gets shipped by someone who never fully understood it, and then that person leaves. Six months later, a defect surfaces and nobody on the team can explain the module.

Every AI assisted component should ship with a short design note written by a human. What the component does, what it assumes, where the edges are, and what would break if the assumption changed. If an engineer cannot write that note, the code is not ready to ship, regardless of what the tests say.

Where AI genuinely earns its place

None of the above is an argument against using these tools. Used with discipline, they are excellent at:

• Drafting boilerplate against an existing pattern in the codebase, where the human review cost is low and the time saved is real.
• Explaining legacy code during impact analysis, which is one of the slowest parts of change control.
• Generating candidate test cases to be triaged by an engineer, which broadens coverage faster than writing each one by hand.
• Reviewing pull requests as an additional reader that never gets tired, as long as its comments are advisory and a human makes the call.
• Producing traceability drafts by proposing links between requirements, code, and tests for a human to confirm.

The pattern is consistent: the model proposes, a person disposes, and the process records both.

A minimal process you can adopt this quarter

If you need a starting point, this is the shortest version that survives an audit:

1. Requirements and acceptance criteria are written by people before any generation begins.
2. Generated code enters through a pull request opened by a named engineer who is its author of record.
3. Review is performed by a different engineer. Review comments must show evidence of reading, not rubber stamping.
4. Requirement tests are written or rewritten by a human. Generated tests are additive only.
5. Static analysis and the type checker run at maximum strictness with zero unexplained suppressions.
6. The change record notes that AI assisted, with the tool and model version.
7. Every AI assisted component ships with a human written design note.
8. Nothing with a model in the loop has write access to a protected branch.

The advantage of being able to build it yourself

Here is the part most guidance leaves out. All of this is much easier if the team can build the system without the model. Engineers who can write the code themselves recognize when generated output is wrong, when it is merely unusual, and when it is right but unmaintainable. They can push back. They know what "done" looks like, so they can tell when the tool has stopped short of it.

That is the position we work from. We are engineers first. AI is a fast collaborator that we supervise, and in a regulated environment, supervision is the whole job.