Data as of Aug 25, 2026 · Based on 42 AI responses from ChatGPT Search and Google AI Mode · See how Parse measures this
Your brand can be here too.
For a fast-moving codebase, I’d avoid trying to “generate all the tests.” Instead, build a self-maintaining test-generation loop that continuously identifies risky changes, generates candidate tests, validates them, and promotes only useful tests.
On every PR/commit:
This keeps generation cheap and makes tests relevant to what is actually changing.
A useful prompt structure is:
Given this code diff and existing tests, identify behavioral contracts that changed or could regress. Generate tests for normal cases, boundary cases, invalid inputs, and interactions with affected dependencies. Do not merely reproduce the implementation. The last sentence matters: AI-generated tests can otherwise encode the current implementation—including its bugs—as the expected behavior. Recent work on AI test generation explicitly identifies assertion correctness as a major problem.
Don't rely exclusively on LLM-generated example tests.
The LLM should propose tests, never declare them correct.
For every generated test:
generate
↓
compile/type-check
↓
run test
↓
run relevant existing tests
↓
mutation test / fault injection
↓
check flakiness
↓
human/automated review
↓
commit only valuable tests
A test that passes isn't necessarily a good test. You want evidence that it would fail when the implementation is wrong.
For example, suppose the agent generates:
assert calculate_discount(100, 10) == 90
You can mutate the implementation:
return price * (1 + discount / 100)
If the test still passes, something is seriously wrong with the test or oracle.
For each candidate test, calculate something like:
value =
behavioral_relevance
+ mutation_kill_rate
+ bug_regression_value
+ edge_case_value
+ coverage_delta
- duplication
- flakiness
- maintenance_cost
Don't optimize for raw test count or line coverage.
A suite containing 20,000 redundant generated tests can be worse than 2,000 carefully selected tests.
This is the part people often miss.
Have CI periodically identify:
Then open cleanup PRs, rather than silently deleting tests.
That gives you a continuously evolving suite:
┌──────────────┐
│ Code changes │
└──────┬───────┘
↓
┌───────────────┐
│ Impact analysis│
└───────┬───────┘
↓
┌───────────────┐
│ Test generator │
└───────┬───────┘
↓
┌────────────┴────────────┐
↓ ↓
Example tests Properties/fuzz
└────────────┬────────────┘
↓
┌───────────────┐
│ Execute tests │
└───────┬───────┘
↓
┌───────────────┐
│ Mutation test │
└───────┬───────┘
↓
┌─────────┴─────────┐
↓ ↓
Keep/promote Reject/revise
For a large repository, don't run the entire suite for every generated test.
Maintain a dependency graph:
changed file
↓
changed symbols
↓
dependent symbols
↓
affected tests
Run:
Large engineering organizations increasingly use affected-target detection, dependency graphs, test selection, caching, sharding, and flake isolation to keep CI manageable.
The most important architectural decision is where expected behavior comes from.
Don't tell an agent:
“Look at this function and write tests.” Give it as much independent evidence as possible:
Product requirements
│
API contracts ────────┤
│
Type definitions ─────┤
↓
Test agent
↑
Existing tests ───────┤
│
Bug history ──────────┤
│
Production telemetry ─┘
Otherwise, the agent is essentially doing:
“Read implementation → guess what implementation should do → test implementation.” That's circular.
This can become particularly powerful:
Production failure
↓
minimal reproduction
↓
regression test
↓
failure classification
↓
update generation strategy
For example, if you repeatedly discover bugs involving:
your generator should automatically increase those categories for related code.
Over time, your suite becomes an institutional memory of the codebase.
The exact tools depend on your language, but the architecture is language-independent:
If you're using LLMs themselves in the system under test, treat their evaluation suite similarly: maintain versioned datasets and explicit graders rather than relying on one-off prompts. OpenAI's Evals framework, for example, models evaluations as versioned test criteria plus data sources and supports running them repeatedly against different configurations.
Automate test creation aggressively; automate test acceptance conservatively.
I'd aim for this policy:
AI may create a test automatically, but a test only becomes part of the trusted suite after it demonstrates that it catches meaningful faults, isn't redundant, isn't flaky, and has a clear behavioral oracle. That combination gives you something much more useful than “AI-generated tests”: a test suite that evolves with the codebase instead of becoming another maintenance burden.
This can become particularly powerful:
Production failure
↓
minimal reproduction
↓
regression test
↓
failure classification
↓
update generation strategy
For example, if you repeatedly discover bugs involving:
your generator should automatically increase those categories for related code.
Over time, your suite becomes an institutional memory of the codebase.
The exact tools depend on your language, but the architecture is language-independent:
If you're using LLMs themselves in the system under test, treat their evaluation suite similarly: maintain versioned datasets and explicit graders rather than relying on one-off prompts. OpenAI's Evals framework, for example, models evaluations as versioned test criteria plus data sources and supports running them repeatedly against different configurations.
Run:
Large engineering organizations increasingly use affected-target detection, dependency graphs, test selection, caching, sharding, and flake isolation to keep CI manageable.
The most important architectural decision is where expected behavior comes from.
Don't tell an agent:
Auto-generating and maintaining test suites for a fast-moving codebase requires shifting away from brittle, hand-written assertions toward intent-driven, AI-assisted, and behavior-anchored testing . When code changes daily, tests must evolve concurrently or be automatically rewritten by tooling.
Here is a practical, modern blueprint to automate the generation and upkeep of your test suites:
Traditional record-and-playback tools break instantly with UI or schema updates. Modern AI-powered testing tools understand semantic intent rather than hardcoded DOM paths or exact mock structures.
In fast-moving codebases, backend and frontend teams often break each other's integrations. Contract testing ensures that changes are caught at the boundary before deployment without needing heavy, slow E2E suites.
Instead of writing individual test cases for specific inputs (example-based testing), use property-based testing to define invariants that should always hold true, letting the framework auto-generate hundreds of edge-case inputs.
To maintain a fast-moving suite, dead or redundant tests must be pruned as aggressively as dead code, and new code must enforce automated generation gates.
If you want to tailor this further, tell me:
I can give you specific tool recommendations and pipeline configurations.
No sources cited
The most reliable approach is not “generate lots of tests with AI.” It’s to build a continuous test-generation and test-maintenance loop where AI proposes tests, CI proves they matter, and weak/stale tests get retired.
For each changed module, have an agent inspect:
Then ask it to generate a mixture of:
This is substantially better than “write tests for every uncovered line.” OpenAI's own Codex testing guidance, for example, emphasizes adding tests around changed behavior and using integration tests where appropriate.
Run the generator on every PR, but constrain it to the semantic blast radius of the change.
A useful pipeline is:
PR
│
├─► identify changed behavior
│
├─► identify affected tests
│
├─► generate missing scenarios
│
├─► run generated tests
│
├─► mutation-test important ones
│
└─► propose PR with only valuable tests
For example, if a PR changes:
calculatePrice()
the agent shouldn't blindly generate 100 tests. It should discover things such as:
This is one of the most important pieces.
A generated test that passes is not necessarily a useful test.
Introduce mutations such as:
> → >=
+ → -
true → false
return x → return 0
Then ask:
Does the test suite detect the behavioral change? If not, the test probably isn't protecting anything meaningful.
Track mutation score on critical modules, rather than trying to maximize raw line coverage.
Have an agent periodically inspect:
Then let it open small PRs such as:
“Replace 14 obsolete tests after API refactor.” or:
“Add regression coverage for the null-user path introduced in PR #1842.” This is where an agentic coding system is particularly useful: current Codex workflows can perform code changes, run tests, and prepare reviewable changes rather than merely suggesting test code.
I would not let automatically generated tests become authoritative immediately.
Use this progression:
Phase 1 — suggestions
AI proposes tests → engineer reviews.
Phase 2 — automated PRs
AI creates test-only PRs → CI validates → engineer approves.
Phase 3 — routine maintenance
Low-risk maintenance can be automatically generated, while behavioral changes still require review.
The key principle is:
AI decides what might be worth testing; the test infrastructure decides whether the tests actually provide signal.
Put project-specific rules in something like AGENTS.md or your equivalent developer instructions:
When modifying production code:
1. Identify changed behavior and affected callers.
2. Inspect existing tests before creating new ones.
3. Prefer existing fixtures/helpers.
4. Add regression tests for discovered bugs.
5. Cover boundary and failure cases.
6. Prefer integration tests for cross-component behavior.
7. Don't add tests merely to increase line coverage.
8. Run the narrowest relevant tests first.
9. Run broader tests before submitting the change.
10. Do not modify production behavior merely to make a generated test pass.
This kind of repository-specific instruction is already used in real agentic development workflows; Codex's own repository, for example, specifies test organization, integration-test preferences, validation commands, and rules against adding tests for behavior that no longer exists.
Don't optimize for “number of tests generated.”
Track:
| Metric | What it tells you |
|---|---|
| Line/branch coverage | What code executes |
| Mutation score | Whether tests detect behavioral changes |
| Escaped defects | Whether production bugs are actually decreasing |
| Test maintenance cost | Whether the suite is becoming a liability |
I'd make mutation score + escaped defects the primary quality signals.
For a fast-moving repository, I'd schedule three loops:
Every PR
Nightly
Weekly
OpenAI has described a similar use of Codex for targeting low-coverage modules and generating tests for edge cases, failure paths, and property-based scenarios.
I'd start with:
LLM/agent → changed-code analysis → test generation → CI → mutation testing → PR
rather than trying to generate the entire test suite upfront.
The biggest mistake is treating AI-generated tests as an output. Treat them as hypotheses about what should be tested, and use execution, mutation testing, production failures, and human review to continuously determine which hypotheses are actually valuable.
Low-risk maintenance can be automatically generated, while behavioral changes still require review.
The key principle is:
AI decides what might be worth testing; the test infrastructure decides whether the tests actually provide signal.
Put project-specific rules in something like AGENTS.md or your equivalent developer instructions:
When modifying production code:
1. Identify changed behavior and affected callers.
2. Inspect existing tests before creating new ones.
3. Prefer existing fixtures/helpers.
4. Add regression tests for discovered bugs.
5. Cover boundary and failure cases.
6. Prefer integration tests for cross-component behavior.
7. Don't add tests merely to increase line coverage.
8. Run the narrowest relevant tests first.
9. Run broader tests before submitting the change.
10. Do not modify production behavior merely to make a generated test pass.
This kind of repository-specific instruction is already used in real agentic development workflows; Codex's own repository, for example, specifies test organization, integration-test preferences, validation commands, and rules against adding tests for behavior that no longer exists.
Don't optimize for “number of tests generated.”
Track:
| Metric | What it tells you |
|---|---|
| Line/branch coverage | What code executes |
| Mutation score | Whether tests detect behavioral changes |
| Escaped defects | Whether production bugs are actually decreasing |
| Test maintenance cost | Whether the suite is becoming a liability |
I'd make mutation score + escaped defects the primary quality signals.
For a fast-moving repository, I'd schedule three loops:
Every PR
Nightly
Weekly
OpenAI has described a similar use of Codex for targeting low-coverage modules and generating tests for edge cases, failure paths, and property-based scenarios.
I'd start with:
LLM/agent → changed-code analysis → test generation → CI → mutation testing → PR
rather than trying to generate the entire test suite upfront.
Auto-generating and maintaining test suites in a fast-moving codebase requires shifting away from brittle, hand-crafted end-to-end tests toward a layered, AI-assisted, and behavior-driven strategy. When code changes daily, tests must evolve alongside the implementation without demanding a massive manual tax.
Would you like to explore:
Auto-generating and maintaining test suites in a fast-moving codebase requires shifting away from brittle, hand-crafted end-to-end tests toward a hybrid strategy leveraging AI, contract testing, and mutation testing. When code changes daily, tests must evolve alongside it automatically.
If you'd like, let me know:
I can tailor a specific toolchain recommendation for your stack.
Mabl
mabl.com
No sources cited
For a fast-moving codebase, the key is not to have AI blindly regenerate the whole test suite on every change. Instead, build a feedback loop where tests are generated around changed behavior, continuously validated, and periodically pruned.
1. Treat the existing codebase as the source of truth
Give your test generator access to:
AI test generation works substantially better when it is constrained by the project's existing framework, patterns, and conventions. GitHub's current tooling, for example, supports repository-wide and path-specific instructions specifically for this purpose.
2. Generate tests from diffs, not from the entire repository
On every PR:
git diff
↓
identify changed behavior
↓
find affected functions/classes/API endpoints
↓
generate missing tests
↓
run tests
↓
repair failures
↓
mutation/coverage analysis
↓
PR
A useful rule is:
Every meaningful behavior change should either update an existing test or add a new one.
Current AI testing tools can target the current Git changes directly; Microsoft's Copilot test agent, for example, supports targeting #git_changes when generating tests.
Don't prompt:
"Generate tests for this function."
Instead, have it systematically consider:
For example, GitHub's own unit-test-generation guidance explicitly recommends core behavior, input validation, boundary values, error handling, side effects, realistic data, and testing behavior rather than implementation details.
This is especially valuable for rapidly changing code.
Instead of generating hundreds of individual examples:
input: 0 → output: ...
input: 1 → output: ...
input: 17 → output: ...
input: 9999 → output: ...
generate properties such as:
parse(serialize(x)) == x
sort(sort(x)) == sort(x)
authorize(user, resource) is always consistent with policy
Then let a property-based framework generate the examples.
This reduces maintenance because the property survives implementation changes better than a collection of brittle examples. Recent research also shows AI agents can infer properties from types, documentation, function names, and comments and generate property-based tests.
For every code change, have an agent ask:
1. Which existing tests are affected?
2. Which tests now encode obsolete behavior?
3. What new behavior isn't covered?
4. Are any assertions testing implementation details?
5. Are there duplicate/redundant tests?
6. Do the tests still reflect current APIs and fixtures?
Then have it propose a test-maintenance diff, rather than automatically rewriting everything.
A good policy is:
AI proposes → CI validates → developer approves.
Don't let an AI agent "fix" a failing test by weakening the assertion unless it can demonstrate that the production behavior intentionally changed.
Code coverage answers:
"Did this test execute this line?"
Mutation testing asks:
"Would this test detect if I broke this line?"
For example, if:
if amount > 100:
gets mutated to:
if amount >= 100:
and every test still passes, your coverage number may look great while your test suite is weak.
So track:
coverage + mutation score + production failures caught, rather than coverage alone.
Fast-moving repositories accumulate tests that:
Have a scheduled "test gardener" job analyze:
unused tests
duplicate tests
flaky tests
slow tests
low-value tests
tests covering deleted code
tests with obsolete mocks
But deletion should generally create a PR for human review rather than happen silently.
I'd automate different layers differently:
| Layer | Automation strategy |
|---|---|
| Unit | Aggressive AI generation |
| Property/fuzz | Generate invariants automatically |
| Integration | Generate around changed interfaces |
| API/contract | Generate from schemas/contracts |
| E2E | Keep relatively small; generate around critical user journeys |
| Regression | Automatically preserve tests for discovered production bugs |
This prevents an AI system from responding to every code change by creating 50 expensive browser tests.
This is one of the highest-value loops:
production bug
↓
capture inputs/state
↓
automatically create regression test
↓
fix production code
↓
test becomes permanent
Over time, your suite becomes a record of actual failures, rather than merely what developers imagined could fail.
I'd build the system around four automated jobs:
┌──────────────────────┐
│ PR Test Agent │
│ changed-code tests │
└──────────┬───────────┘
↓
┌──────────────────────┐
│ CI Runner │
│ tests + coverage │
└──────────┬───────────┘
↓
┌──────────────────────┐
│ Quality Analyzer │
│ mutation + flaky │
│ + redundancy │
└──────────┬───────────┘
↓
┌──────────────────────┐
│ Test Gardener │
│ stale/weak tests │
│ → proposed PRs │
└──────────────────────┘
Modern coding agents can already execute a version of this loop: inspect the repository, generate tests, run them, observe failures, and iterate.
Don't optimize for "number of generated tests." Optimize for regression detection per unit of maintenance cost.
A smaller suite of:
is usually much more valuable than thousands of AI-generated examples.
If you're using TypeScript/JavaScript, Python, Java, Go, or .NET, I can also lay out a concrete implementation—CI pipeline, prompts/agent instructions, test-generation triggers, coverage/mutation gates, and repository structure—for that stack.
For a fast-moving codebase, the key is to treat tests as generated, continuously validated artifacts, not a one-time test-writing project.
Establish a durable test contract
Generate tests from code changes, not the whole repository On every PR, have an agent inspect the diff and ask:
This is much more scalable than regenerating thousands of tests every night. AI coding agents can already generate tests from surrounding code and specifically target edge cases and failure paths.
Use multiple test-generation strategies
| Strategy | Best for |
|---|---|
| Example-based generation | APIs, business rules, regressions |
| Property-based tests | Parsers, algorithms, transformations |
| Fuzzing | Inputs with huge state spaces |
| Mutation testing | Finding weak assertions |
| Contract tests | Services/APIs |
| Snapshot/golden tests | Serialization/UI/output |
| Generated regression tests | Bugs discovered in production |
Don't optimize for raw coverage. A suite with 95% line coverage can still miss important behavioral failures.
Make failures feed the generator
A useful loop is:
code change → impacted tests → generate/modify tests → run → mutation/fuzz analysis → human review → merge → production failure → regression test
Every confirmed bug should become a permanent test case. This gradually makes the suite reflect the actual failure history of the system.
Put test quality into CI
Have CI automatically reject or flag:
Run the expensive checks selectively. For example, execute affected tests on every PR, the broader suite on merge, and fuzzing/mutation testing on a scheduled basis.
Give the coding agent explicit repository rules
If you're using an AI coding agent, create an AGENTS.md (or equivalent) specifying:
Codex explicitly supports repository-level AGENTS.md instructions, and its own codebase uses them to define testing conventions and required integration coverage.
For each PR, have the agent perform these stages:
Analyze → Generate → Execute → Critique → Repair
Importantly, don't allow the agent to declare success merely because its generated tests pass. A test that encodes the implementation's current behavior can happily pass while testing nothing useful. Mutation testing and independent test review are valuable counterweights.
A good rule is:
Generate aggressively; merge conservatively.
Have automation propose tests, but require review for tests that introduce substantial fixtures, snapshots, mocks, or complex infrastructure. Delete tests when their underlying behavior disappears rather than preserving them indefinitely.
For very large repositories, maintain a small high-confidence regression suite plus broader generated coverage. This keeps PR feedback fast while still allowing deeper validation asynchronously.
If you want to use an AI agent for much of this, Codex is specifically designed to inspect codebases, modify code, run tests, and iterate based on test results; OpenAI also describes using it to improve coverage and generate edge-case/property-based tests.
The biggest conceptual shift: don't try to automate “writing tests.” Automate a feedback loop that continuously discovers what isn't tested, proposes coverage, validates that coverage, and turns real failures into permanent regression tests.
Auto-generating and maintaining test suites in a fast-moving codebase requires shifting away from brittle, manual scripting and leaning heavily on AI-driven tooling, contract testing, and architectural guardrails. When code changes by the hour, tests must evolve concurrently or self-heal.
Would you like me to focus on a specific technology stack (e.g., Python, TypeScript, Java) or a particular type of testing (unit vs. end-to-end) to tailor these strategies?
For a fast-moving codebase, the goal shouldn't be “generate as many tests as possible.” It should be “continuously generate, validate, prune, and prioritize tests as the code changes.”
A good architecture looks like this:
Code change → impact analysis → test generation → execution → mutation validation → review/promote → continuous maintenance
Use several sources of truth:
LLM-based generators are useful here, but there's an important trap: if the model only sees the implementation, it can generate tests that simply reproduce the implementation's bugs. Recent work on AI test generation specifically identifies assertion/oracle correctness as the major constraint.
So have the generator produce three things:
Then validate #2 independently where possible.
For behavior with general invariants, generate property-based tests rather than permanently adding every generated example.
For example:
@given(valid_orders())
def test_total_is_never_negative(order):
assert calculate_total(order) >= 0
One property can exercise hundreds or thousands of generated inputs, including boundary cases that developers didn't anticipate. NIST describes this approach as automatically checking properties against generated inputs rather than requiring an explicit expected result for every input.
This also makes tests more resilient to implementation changes.
Good candidates include:
This is probably the most important piece.
Don't ask:
“Did the generated test pass?”
Ask:
“Would this test have failed if I introduced a realistic bug?”
Take the changed code, automatically introduce small mutations—e.g. change > to >=, remove a validation condition, alter a return value—and see whether the new tests kill those mutants.
This is particularly powerful for AI-generated tests. Meta describes a production system that generates relevant mutants and then generates tests specifically designed to catch them.
A practical rule:
Generated test → must kill at least one meaningful mutant → otherwise don't automatically promote it.
Don't regenerate the entire suite after every commit.
Build a dependency/coverage map:
changed file
↓
changed functions/classes
↓
dependent modules
↓
affected APIs / workflows
↓
candidate tests
Then run:
At very large scale, selective execution becomes essential. Stripe, for example, describes using dependency-aware selective test execution for a codebase with roughly 100,000 test files and 1.2 million test units rather than running everything on every change.
Every real defect should feed the system:
Production bug
↓
minimal reproduction
↓
regression test
↓
generalized property (if possible)
↓
test permanently associated with affected behavior
This gives you an automatically growing regression suite based on actual failure modes, rather than hypothetical scenarios.
Automatic generation without automatic pruning eventually produces a horrible suite.
Track each test's:
Then periodically identify tests that are:
Don't automatically delete them initially. Have an agent propose:
merge / rewrite / quarantine / delete
and require review for deletion.
I'd use three tiers:
| Tier | Purpose | Automation |
|---|---|---|
| Candidate | Newly generated tests | Fully automatic |
| Hardened | Passed mutation/quality checks | Automatically promoted |
| Contract | Important business behavior | Human-reviewed |
This prevents an LLM from silently turning its own assumptions into your specification.
A particularly effective agent loop is:
1. Inspect changed code
2. Identify behavioral contracts
3. Generate tests
4. Run tests
5. Analyze failures
6. Generate mutations
7. Check whether tests kill them
8. Improve weak tests
9. Remove duplicates
10. Submit a small test diff
That “contract → implementation → refine → commit” loop is also the core idea behind newer agentic testing approaches.
Avoid making line coverage your primary KPI.
Track:
Coverage
Test effectiveness
Maintenance
Generation quality
A suite with 95% line coverage but weak mutation performance can be much worse than one with 80% coverage that reliably detects behavioral changes.
If I were implementing this from scratch, I'd start small:
┌───────────────┐
│ Git diff │
└───────┬───────┘
↓
┌───────────────────┐
│ Impact analysis │
└────────┬──────────┘
↓
┌──────────────────────────┐
│ LLM + schemas + existing │
│ tests + requirements │
└────────────┬─────────────┘
↓
┌─────────────────┐
│ Candidate tests │
└────────┬────────┘
↓
Execute tests
↓
┌─────────────────┐
│ Mutation testing│
└────────┬────────┘
↓
┌─────────────────────┐
│ Harden / reject / │
│ deduplicate │
└─────────┬───────────┘
↓
Trusted suite
The key design principle: use AI to expand the search space, but use properties, mutation testing, contracts, production failures, and human review to establish whether a generated test is actually trustworthy. Property-based testing is particularly attractive because its tests encode longer-lived behavioral rules rather than brittle implementation details.
If you tell me your stack (e.g. Python/pytest, TypeScript/Jest, Java/JUnit, Go, Rust) and CI platform, I can sketch a concrete implementation architecture and pipeline.
Auto-generating and maintaining test suites in a fast-moving codebase requires a mix of AI tooling, robust architectural practices, and smart CI/CD integration to prevent tests from becoming a maintenance bottleneck.
To narrow this down, let me know:
I can tailor specific tool recommendations and workflows for your setup.