Introduction
With the raise of LLM such as Copilot, ChatGPT, the demand of applying AI into testing is more critical than ever. So, understanding the bright side and flip side to effective using is a must. And in this blog post, we will discuss where AI can give a hand and where it stills fall short.
How AI generate test cases
Companies apply different approaches to generate test cases, most common are using LLM such as ChatGPT, Copilot, ….
Typically, some kinds of documents that we can give AI to analyze requirement are:
- Requirement, user stories, use case
- API specifications and documentation
- User interface requirements such as Figma files
- Production logs and historical data
- …
LLM will extract these texture information while platform integrated AI can generate test case using application metadata. AI acts as an assistant, suggesting test based on observed patterns instead of exact one.
Benefits of using AI to create test cases
There are a lot of benefits using AI in creating test cases, such as:
- Fast and Productivity: It is time-consuming to create test cases, however, with AI, this is just minutes rather than hours.
- Requirement based coverage: If requirement is clear, AI can identify all the boundaries values and paths that human might miss if rush.
- Scalable: AI is excellent at generating multiple combinations and positive, negative test data.
- Domain learning: Some commercial AI tools are pre-trained with domain knowledge such as finance, healthcare, making their suggestion are quite valuable.
Limitations and risks
- Quality of output depends on input: Despite of a lot of advantages, test case’s quality generated by AI depends on what we feed for it. AI derives information from inputted data, so if the requirement is not clear, ambiguous, not detail the generated will be in low-quality. AI can also assume the missing logic instead of discovering or raising concern.
- Missing context and business rules: AI might not understand the real usages scenarios that let to creation technically correct test cases and ignore real business rules. For example, a tester asks the AI to generate test cases for a software which is similar to ATM withdrawals. AI’s output for boundary, error message are very good but it missed cases such as user withdraw many times and the total amount of those are exceed limitation, ….
- Inconsistent output: The same prompt may produce different set of test cases each time. This makes comparison and audit difficult.
- Security and privacy concern: If prompt contains sensitive and intellectual property information, this could be stored in third-party AI service log, exposing security risks.
When should we use AI generated test cases
So, AI is most-efficient when:
- Documents are clear
- Team need test quickly
- Simple functional, simple business logic
Good practice when applying AI in generation test case
Therefore, when applying AI in test case generation, we should remember:
- Always review the output result from a senior view. AI can support us to brush up productivity but should not be the final point.
- The prompt we provide must be clear, concise and complete. We should have acceptance criteria and constraint in our prompt.
- We should use AI in combination with tester, where tester do the role to review and enhance the output to ensure accuracy, completeness and align with business objectives.