Common Mistakes When Using AI in Testing

AI is rapidly becoming part of the testing workflow. In many teams today, tools like Microsoft 365 Copilot and GitHub Copilot are already integrated into daily work—from writing test cases to generating automation scripts.

These tools help testers move faster, but based on real project experience, they are often misunderstood or misused. Instead of improving quality, they sometimes introduce gaps, false confidence, or poorly designed tests.

In this article, we will go through the most common mistakes when using AI in testing—and how to avoid them, with practical examples using Copilot.

Why AI in Testing Is Powerful (and Risky)

With tools like Copilot, we can:

• Generate test cases directly from requirements in Word or Confluence
• Auto-complete automation scripts in IDEs
• Quickly explore edge cases
• Speed up debugging

However, AI has a critical limitation:

It does not understand your system – it only predicts based on patterns and training data.

This is where most problems begin.

1. Blindly Trusting AI-Generated Test Cases

Using Microsoft 365 Copilot, you can ask:

“Generate test cases for a login API”

You will get a clean, structured list:

• Valid login
• Invalid password
• Missing fields

But in real systems, we also need:

• Account lock after multiple failures
• Rate limiting
• Multi-factor authentication
• Token expiration

Copilot generates generic test cases, not system-aware scenarios.

How to avoid:

• Treat AI output as a starting point
• Review based on actual system behavior
• Add missing scenarios manually

2. Ignoring Business Context

Copilot works well with text, but it does not truly understand:

• Business rules
• User behavior
• Production edge cases

Example: from a requirement document, Copilot may generate:

• “Add item to cart”
• “Remove item”

But miss:

• Discount rules
• Inventory limits
• Payment failure flows

This creates a dangerous illusion of coverage.

How to avoid:

• Provide more context in prompts
• Combine AI output with domain knowledge
• Validate with real use cases

3. Overusing GitHub Copilot for Automation

GitHub Copilot is extremely useful when writing:

• API test scripts
• k6 performance tests
• Automation frameworks

But overusing it leads to:

• Copy-paste coding
• Lack of understanding
• Hard-to-maintain scripts

Example: Copilot suggests a k6 script:

• No thresholds
• No realistic load pattern
• Minimal validation

It “runs”, but does not test anything meaningful.

How to avoid:

• Use Copilot for boilerplate code
• Always review:
  • Logic
  • Assertions
  • Test design
• Keep ownership of the script

4. Not Validating AI-Generated Code

Copilot-generated code often looks correct—but small issues matter.

Common problems:

• Missing assertions
• Incorrect conditions
• Incomplete error handling

Example: a Postman test script generated by Copilot may:

• Check status = 200
• But ignore response body validation

This leads to false positives.

How to avoid:

• Never trust generated code blindly
• Add:
  • Functional checks
  • Data validation
  • Negative scenarios

5. Using Weak Prompts with Copilot

AI is only as good as the input.

Bad prompt:

“Write test cases”

Better prompt:

“Generate API test cases for a login endpoint with JWT authentication, including positive, negative, boundary, and security scenarios.”

When using Microsoft 365 Copilot:

• Be explicit about requirements
• Include constraints
• Mention edge cases

Better prompts → significantly better results.

6. Ignoring Edge Cases and Real Traffic Behavior

Copilot tends to generate:

• Happy path
• Basic negative cases

But real issues come from:

• Concurrent requests
• Large payloads
• Invalid tokens
• Expired sessions

These are rarely generated unless explicitly requested.

How to avoid:

• Ask specifically for:
  • Edge cases
  • Stress scenarios
  • Failure conditions
• Combine AI with your own testing experience

7. Treating AI as a Replacement for Testers

A common misconception is:

“AI can generate everything, so we don’t need deep testing anymore.”

This is incorrect. AI cannot:

• Understand business impact
• Identify real risks
• Make testing decisions

The role of testers is evolving—not disappearing.

Better mindset:

• Copilot = Assistant
• Tester = Decision maker

Best Practices for Using Copilot in Testing

• Use Microsoft 365 Copilot for:
  • Requirement analysis
  • Drafting test cases
• Use GitHub Copilot for:
  • Writing automation code
  • Speeding up scripting
• Always:
  • Review outputs
  • Add system context
  • Validate thoroughly

8. Real Prompt Examples: From AI Output to Better Tests

Using AI effectively is not about asking once and copying the result. It is an iterative process: Prompt → Output → Review → Fix.

Below are real examples using Copilot-style prompts and how we improve the results.

Example 1: Generating API Test Cases

Prompt (too generic):

Generate test cases for a login API

AI Output:

• Valid login
• Invalid password
• Missing username
• Missing password

At first glance, this looks fine—but it is very basic.

What’s missing:

• Account lock after multiple failed attempts
• Rate limiting
• Token expiration
• Multi-factor authentication
• Security scenarios (SQL injection, brute force)

Improved Prompt:

Generate API test cases for a login endpoint using JWT authentication. Include positive, negative, boundary, and security scenarios such as rate limiting, account lock, and token expiration.

Improved Output:

• Valid login with correct credentials
• Invalid password attempts → account lock after N tries
• Missing fields validation
• Expired JWT token handling
• Rate limiting after multiple requests
• Injection attempts in input fields

Lesson: Better prompts lead to more realistic and complete test coverage.

Example 2: Generating k6 Performance Script

Prompt (basic):

Write a k6 script to test an API

AI Output:

import http from 'k6/http'; 
export default function () 
{ 
         http.get('https://api.example.com/users');
 }

This script runs—but it is not a real performance test.

What’s missing:

• Load configuration
• Checks / assertions
• Thresholds
• Realistic behavior

Improved Prompt:

Write a k6 performance test script for a GET /users API with 50 virtual users for 2 minutes. Include response time checks (<500ms) and failure rate thresholds.

Improved Output:

import http from 'k6/http'; 
import { check } from 'k6';
 export let options = { 
    vus: 50, 
    duration: '2m', 
    thresholds: { 
         http_req_duration: ['p(95)<500'],
         http_req_failed: ['rate<0.01'], 
         }, 
  }; 

export default function () { 
    const res = http.get('https://api.example.com/users'); 
check(res, { 

    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
 }); 
}

Lesson: AI can generate usable scripts—but only when we define clear expectations.

Example 3: Generating Postman Test Script

Prompt (weak):

Write Postman test script

AI Output:

pm.test("Status code is 200", function () {
 pm.response.to.have.status(200);
 });

This is not enough for real testing.

Improved Prompt:

Write a Postman test script to validate status code, response time (<500ms), and that the response body contains a non-empty “data” array.

Improved Output:

pm.test("Status code is 200", function () {
      pm.response.to.have.status(200);
 }); 
pm.test("Response time is under 500ms", function () {     
     pm.expect(pm.response.responseTime).to.be.below(500); 
}); 
pm.test("Response has data array", function () { 
          const jsonData = pm.response.json();           
         pm.expect(jsonData.data).to.be.an('array').that.is.not.empty; 
});

Lesson: AI gives better results when we specify validation rules.

Key Takeaways

• First AI output is rarely enough
• Prompt quality directly affects test quality
• Always review and refine
• Combine AI speed with tester judgment

The real value is not in the first answer—it’s in how we improve it.

Conclusion

AI tools like Microsoft 365 Copilot and GitHub Copilot are powerful additions to the testing workflow. They can significantly improve speed and productivity.

However, they do not replace critical thinking.

Used incorrectly, they create:

• Shallow test coverage
• False confidence
• Missed defects

Used correctly, they help testers focus on what really matters:

• Understanding systems
• Identifying risks
• Designing meaningful tests

AI does not make testing easier—it makes good testing faster.

References

• Microsoft Copilot Documentation
  https://learn.microsoft.com/copilot/
• GitHub Copilot Docs
  https://docs.github.com/en/copilot