AI QA and testing that keeps pace with every release
An agent that reads each pull request, writes the tests a careful engineer would, runs them in CI and separates real failures from flaky ones, with your QA lead approving what enters the suite.
Most teams ship weekly or daily, but the regression suite was written for a slower rhythm. Coverage is patchy where the code changes most, flaky tests get muted rather than fixed, and the manual checklist grows until nobody runs it fully. Bugs reach customers not because testers are careless, but because the maths does not work.
Hiring more testers scales linearly. An agent that drafts tests for every change, runs them and explains what broke scales with the codebase. The human role shifts from writing to reviewing, which is where judgement matters.
How it works
From pull request to triaged result
The agent proposes; people approve. Generated tests arrive as pull requests, failures arrive as draft tickets with reproduction steps, and the flaky-test record stops the same noise recurring.
1. Understand the change
For each pull request the agent reads the diff, the linked ticket and the tests that already touch those files. It ranks the risk: new branches, changed validation, altered queries.
2. Write and prove the tests
It drafts unit, API or end-to-end tests in your existing framework, then checks each one fails when the feature is deliberately broken. Tests that cannot fail are discarded.
3. Run, triage, hand over
Tests run in CI. Failures are replayed to separate flaky from real, and real ones get a reproduction, a suspected cause and a draft ticket for an engineer to confirm.
What you get
A testing capability wired into your pipeline
Test-generation agent in CI
Triggered by pull requests, writing in the frameworks your repo already uses. Built on the same foundations as our AI agents and automation work, with scoped credentials and no direct merge rights.
Mutation and coverage gates
Every proposed test is proven to catch a deliberate break. Coverage is reported per pull request so the trend is visible, not just the total.
Failure triage with reproduction
Each failure is replayed, classified as flaky or real, and written up with steps, environment and a suspected cause. Flaky tests are quarantined with a ticket, never silently muted.
Web, mobile and API coverage
Playwright or Cypress for web, native and React Native mobile suites, and contract tests for APIs, the stacks we ship in our own custom software development work.
Evals as tests for AI features
If your product has an LLM in it, we add an eval suite to the same pipeline: labelled inputs, expected behaviours and a merge gate when groundedness or safety scores drop, following our guide to evaluating LLM outputs.
Guardrails we build in
Accuracy thresholds: a generated test enters the suite only after it passes the mutation check and a reviewer approves it; a triage verdict of "real bug" needs a successful reproduction. Human-in-the-loop: the agent has no merge rights and cannot close or assign tickets; QA leads and engineers do. Audit trail: every generated test, run, triage decision and reviewer action is linked to the pull request that caused it. PII handling: test data is synthetic or masked, production data never enters prompts, and secrets are scoped to the CI job. The agent's permissions follow our AI security and governance practice, including prompt injection defence for untrusted content in tickets and diffs.
Typical rollout
One service first, then the rest of the codebase
Baseline (2 weeks)
Connect repos and CI, measure coverage and flaky-test rate, agree the frameworks and the review rules with the QA lead.
Supervised generation (3–4 weeks)
The agent proposes tests for one service. Every proposal is reviewed; rejections teach us the conventions it must follow.
Triage and widen (2–3 weeks)
Failure triage switched on, flaky quarantine in place, agent extended to the remaining services and the mobile apps.
Hand over (1–2 weeks)
Runbooks, prompts and review rules documented. Coverage and escaped-defect trends reported against the baseline. Optional ongoing tuning.
Illustrative example
What a first release usually looks like
Illustrative scenario, not a client case
A SaaS team ships a web app and two mobile apps every fortnight with a small QA group. Their automation suite covers the login and checkout flows well and almost nothing else, and a dozen flaky tests are muted. In the supervised phase the agent proposes tests for the billing service with every pull request; the QA lead rejects roughly a third at first for style and redundancy, then far fewer once the conventions are captured. Triage identifies which muted tests were genuinely flaky and which were hiding a real defect, and both get tickets. By handover, new pull requests arrive with tests attached, and the fortnightly manual checklist shrinks to the exploratory sessions the testers wanted to do all along.
Testing an old codebase before changing it? The same agent generates characterisation tests in our legacy modernisation solution.
No. The agent produces candidate tests and triaged failures; your QA lead decides what enters the suite and what becomes a ticket. In practice QA engineers spend less time writing boilerplate and more on exploratory testing, risk analysis and the judgement calls an agent cannot make.
What stops the agent generating tests that pass for the wrong reasons?
Three things. Every generated test must fail when the feature under test is deliberately broken (a mutation check) before it is proposed. A QA lead reviews it as a pull request. And coverage reports are compared before and after, so tests that assert nothing useful are visible. See our guide to evaluating LLM outputs for the same discipline applied to AI features.
Which stacks and frameworks do you support?
Web, mobile and API test stacks we already build for: Playwright and Cypress for web, XCUITest and Espresso for native mobile, Jest, pytest and PHPUnit for unit and API tests, plus React Native and Flutter projects. The agent writes in the framework your repo already uses rather than introducing a new one.
How do you test the AI features in our own product?
With evals treated as tests. We build a labelled set of inputs and expected behaviours, score each release against it in CI and block merges when groundedness, safety or accuracy drops. Our fine-tuning and evals service covers this in depth.
How long before the agent is running in our CI?
Usually six to ten weeks: two to connect repos and baseline coverage, three to four of supervised generation on one service, and two to extend across the codebase and hand over. Teams with an existing automation suite move faster.
Ready to make AI work in production?
Tell us about your systems, your data and your constraints. We reply within one business day with a straight answer and a suggested next step.