Currently, with the raise of AI in technology, there is a concern how AI can support manual testing? So in this post, I will use Bing Chat is built on GPT-4, a Large Language Model developed by OpenAI, a type of Generative AI to evaluate how it can support manual tester in creating test case.
1. What is Generative AI?
Generative AI is a type of AI that can generate a lot of types of content, such as image, text, audio… based on the data trained. In a high level explanation, previous method guide the machine step by step how to solve the problem while Generative AI finds their own unique way to solve based on the data we trained them. Generative AI based on Large Language Model to learn patterns and relationships of the training data then using that knowledge to generate new content.
User can utilize Generative AI to:
- Explore vast amounts of unstructured content and summarize
- Interaction through chat
- Improve productivity but the quality still depends on the use case.
Although Generative AI has a lot of practical usage but risks are also evolving. Some of typical risks are accuracy, transparency, intellectual property… Generative AI don’t understand your query, instead of that, it converts into tokens and vector … to process. Besides, its result based on trained data so if pattern matched, AI produce the content. Therefore, the result needs human oversight.
So, we will use Bing Chat to write test case and let’s examine how Generative AI could help in this case.
2. Using Bing Chat to write test case
According to Bing Chat, Bing Chat is powered by a Large Language Model (LLM) developed by Open AI called GPT-4. Can Bing Chat help us to create test case and enhance our productivity? I will go through some examples to evaluate. As we need the most accurate result so I set the conversation style to “more precise”.
Example 1
In this example, I use a simple prompt:
And the result:
This test case does not meet my expectation as it misses test data and expectation, it is also lengthy. Besides, as the requirement didn’t specify required field, … so Bing don’t intelligent enough to challenge back. So I will use a more enhanced prompt and guidance for Bing. I also want to be able to update the test case generated faster, so I will ask Bing to generate in a table view. After a series of prompt, I got the result:
So, the expectation is now better. I continue to enhance the prompt with more guidance. And the final result I got:
It’s better now although there’s still have room to enhance more. The next example will be more challenge in test data.
Example 2
The first prompt is:
The result:
You can see the test case is lengthy, unclear, duplicate but still missing. So I will use a series of prompt and get the result:
It is better now and we can still continue give more guidance to a more concise version.
Conclusion
Through above examples, we can utilize Generative AI in creating test scenarios, however, the quality of the result is better if the requirement is clear, enough information and also depend on the prompt quality. Generative AI can’t challenge back ill-logical requirement, can’t see impacts, and don’t know about general common rule when creating test cases. Currently, it can only do what you ask it to do. Besides, the output test case of the Generative AI must be verified and checked by experience tester to ensure the quality.