1. Introduction
AI is changing how we build software, and testing is changing too. When AI can create tests in a few minutes, the question for QA is no longer “How can I write tests faster?” The new question is “How do I know I can trust these tests?”
In this post, I share my view as a QA engineer testing software built with AWS AI-DLC. I look at how testing changes, what new challenges appear, and how to use AI while still keeping quality high.
2. What is AI-DLC?
AI-DLC (AI-Driven Development Life Cycle) is a software development method created by AWS. It is based on one simple rule: AI proposes, humans decide. AI makes a plan, asks questions to understand the task, and only starts the work after a person approves it. This loop repeats for every activity.
AI-DLC has three phases:
- Inception: AI turns business ideas into requirements, user stories and units of work. The whole team reviews and confirms them together in “Mob Elaboration” sessions.
- Construction: AI proposes the architecture, code and tests. The team explains and agrees on technical decisions in “Mob Construction” sessions.
- Operations: AI helps with deployment and running the system, and the team supervises.
Instead of sprints that last a few weeks, the work is split into bolts. A bolt is a short work cycle that lasts hours or days.
3. What is testing in AI-DLC?
In AI-DLC, testing is no longer the last step after developers finish the code. AI creates tests during the Construction phase, at the same time as the code. The tests are based on the requirements and acceptance criteria agreed in Inception.
This changes the role of QA. We spend less time writing tests from scratch. We spend more time defining what needs to be proven and reviewing the tests that AI creates. QA can join at three review points: reviewing acceptance criteria before AI writes code, reviewing the test plan before AI writes tests, and reviewing the tests before the code is merged.
4. How is it different from traditional testing?
| Traditional testing | Testing in AI-DLC | |
|---|---|---|
| Who writes tests | QA or developers, by hand | AI creates them, people review them |
| When | After the code is finished | At the same time as the code |
| Cycle | Sprints (weeks) | Bolts (hours or days) |
| Main QA task | Design and write tests | Define what to prove, review tests |
| Main risk | Not enough tests, slow testing | Many tests, but not sure they are correct |
The biggest difference is this: before, the problem was often too few tests. Now, the problem is often too many tests, without knowing what they really prove.
5. Challenges for QA
5.1. A passing test does not always prove the right thing
When I review tests created by AI, I often find assertions that are written to make the test pass, not to make sure the test fails when there is a bug.
For example, after creating an order, the test only checks that a “Success” message appears. But the data might not be saved, or it might be saved with the wrong values.
Another example: a test checks that a button is disabled when the user has no permission. But the same button is also disabled while the page is loading. I only noticed this when I read the disable conditions in the UI code: permission was just one of several reasons. The fix was to make the test wait for the permission check response, assert that the result is “no permission”, and only then check the button.
The same problem happens with APIs. The requirement says Name is mandatory, but the test only checks for status 400. The API might return 400 for a different error. The test should also check that the error is really about the Name field.
5.2. Fast code changes make tests, environments and documents go out of sync
When a bolt lasts only hours, the UI or API can change several times in one week. A test may pass on a QA’s local machine because they are on a new branch, but fail on CI because CI runs older code.
Documents can go out of sync in the same way. Documents written by AI usually read well and look complete, so people trust them without checking. I once saw a merge request description that said “the last commit only changes comments”, but the diff showed that the commit also changed the navigation logic.
6. Adapting and using AI while keeping quality
AI is very good at tasks that take a lot of effort but need little judgement:
- Suggesting test cases and edge cases from acceptance criteria
- Creating test data and boundary values
- Writing automation skeletons and repeated helpers
- Summarising error logs and explaining diffs
Some tasks should stay with people:
- Deciding what needs to be proven and at which level (unit, API or E2E)
- Checking whether an assertion will fail when the behaviour is wrong
- Judging quality based on evidence, not just on a green pipeline
In short: let AI do the “writing”, and let QA do the “thinking”.
7. Tips for working with AI-DLC in testing
- Give AI clear acceptance criteria. The less clear the criteria, the more general the tests will be.
- Ask AI to explain what each test proves. If a test cannot answer this question, review it again.
- Try breaking the code (mutation testing). If the test still passes when the behaviour is wrong, the assertion is not strong enough.
- Compare documents with the diff. The merge request description and test report must match the last commit.
8. Conclusion
AI-DLC helps us create tests much faster, but faster does not automatically mean better. The value of QA is not in the number of tests. It is in knowing which tests are worth keeping, what each test proves, and whether there is enough evidence to trust the result.
AI can write tests for us, but it cannot decide for us what needs to be proven.
Reference: AI-Driven Development Life Cycle: Reimagining Software Engineering, AWS DevOps Blog.