Integrating AI Chatbot AutoTesting – In this blog, we’ll explore how to integrate automated tests for an AI chatbot into a CI/CD pipeline using Jenkins. We’ll employ a freely available chatbot API, set up a few simple test cases, and automate these tests using Python. The aim is to provide a practical scenario with real-time implementation, ensuring the setup is straightforward and quick to implement on an Ubuntu system.
Prerequisites
- Ubuntu OS
- Python installed
- Jenkins installed and a job created
- OpenAI GPT-3 API key (or any other freely available chatbot API)
1.Why Integrating AI Chatbot Autotesting With CI/CD?
- Early Detection of Issues:Integrating automated tests into the CI/CD pipeline allows developers to identify potential issues in the chatbot’s functionality early on. This proactive approach helps address bugs before they become significant problems, reducing the risk of unexpected failures in production.
- Faster Feedback Loop:Automated tests provide quick feedback on the chatbot’s performance with each code change. Developers receive immediate notifications if their modifications introduce regressions or break existing functionalities, enabling prompt iteration and issue resolution.
- Consistent Testing Environment:CI/CD pipelines ensure that tests are executed in a consistent environment, closely replicating the production conditions. This minimizes discrepancies between development, testing, and deployment phases, leading to more reliable results.
- Scalability and Efficiency:As chatbot applications evolve and scale, manual testing becomes impractical and time-consuming. Automated tests can be easily scaled to accommodate growing codebases and complex interactions, ensuring comprehensive coverage without a significant increase in testing overhead.
2. Setup
We are setting up a project that involves testing an AI-powered chatbot. The project encompasses the following components:
- AI Chatbot: The chatbot interacts with users via natural language processing (NLP). For this example, we use the OpenAI GPT-3 API to simulate the chatbot.
- Automated Testing: We create a set of automated tests to validate the chatbot’s responses against predefined expected responses.
- CI/CD Pipeline: Jenkins is used to automate the testing process, integrating it into a continuous integration and continuous deployment (CI/CD) pipeline.
The goal of this project is to ensure that the chatbot provides accurate and reliable responses consistently, and any issues introduced during development are detected early through automated tests.
3.Tools and Technologies
Jenkins
We are using Jenkins as our CI/CD tool. Jenkins is an open-source automation server that enables developers to build, test, and deploy their software reliably. Jenkins’ flexibility and vast plugin ecosystem make it an excellent choice for automating various stages of the software development lifecycle.
Python
Python is chosen for writing our test scripts because of its simplicity and the vast array of libraries available for testing and API interactions. Python’s readability and ease of use allow for rapid development and maintenance of test scripts. Additionally, libraries like requests make it straightforward to interact with APIs, and testing frameworks like pytest provide powerful tools for asserting expected outcomes.
4.Practical Example: Integrating AI Chatbot Tests with Jenkins CI/CD Pipeline
1. Scenario:
Imagine you are part of a development team working on an AI-powered chatbot for a customer service application. The chatbot interacts with users via a web interface and is built using technologies like Python, NLP Libraries
2. Test Strategy:
Your test strategy encompasses functional testing of conversation flows, validation & comparison capabilities, integration testing with backend APIs, and error handling scenarios.
3. Implementation Steps:
Step 1: Choose a Freely Available Chatbot API
For this example, we will use the OpenAI GPT-3 API, which can simulate a chatbot. Ensure you have an API key from OpenAI.
Step 2: Set Up the Project Structure
Create the following folder structure for the project:
chatbot-testing/ ├── tests/ │ ├── __init__.py │ ├── test_chatbot.py ├── data/ │ ├── expected_responses.json ├── Jenkinsfile └── README.md
Step 3: Create Predefined Requests and Expected Responses
Inside the data folder, create an expected_responses.json file to store predefined requests and expected responses:
{
"Hello, how are you?": "I'm fine, thank you!",
"Goodbye!": "Goodbye!",
"What is the capital of France?": "The capital of France is Paris.",
"What is 2 + 2?": "2 + 2 equals 4.",
"Tell me a joke.": "Why don't scientists trust atoms? Because they make up everything!",
"What is the weather like today?": "I'm not sure, but you can check a weather website for the latest updates.",
"Who wrote 'Pride and Prejudice'?": "Pride and Prejudice was written by Jane Austen.",
"Translate 'hello' to Spanish.": "The translation of 'hello' to Spanish is 'hola'.",
"What is the meaning of life?": "The meaning of life is a philosophical question that has been debated for centuries. Some say it is to find happiness and fulfillment.",
"How many continents are there?": "There are seven continents: Africa, Antarctica, Asia, Europe, North America, Australia, and South America."
}
Step 4: Create Sample Test Cases
Inside the tests folder, create a test_chatbot.py file:
import requests
import json
import os
API_URL = "https://api.openai.com/v1/engines/davinci-codex/completions"
API_KEY = os.getenv("OPENAI_API_KEY")
def ask_chatbot(question):
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
data = {
"prompt": question,
"max_tokens": 50
}
response = requests.post(API_URL, headers=headers, json=data)
return response.json()
def load_expected_responses(filepath):
with open(filepath, 'r') as file:
return json.load(file)
def test_chatbot_responses():
expected_responses = load_expected_responses('data/expected_responses.json')
for question, expected_answer in expected_responses.items():
response = ask_chatbot(question)
actual_answer = response.get('choices')[0].get('text').strip()
assert actual_answer == expected_answer, f"Expected '{expected_answer}' but got '{actual_answer}'"
if __name__ == "__main__":
test_chatbot_responses()
Step 5: Set Up the Jenkins Pipeline
Create a Jenkinsfile in the root of your project directory:
pipeline {
agent any
environment {
OPENAI_API_KEY = credentials('sk-proj-C6su1aGBuT055mV0Qv5LT3BlbkFJluLLjP0TUwPtPbPJkukg') // Ensure you have set this credential in Jenkins
}
stages {
stage('Set up Python') {
steps {
node('any') {
sh 'sudo apt-get update'
sh 'sudo apt-get install -y python3 python3-pip'
}
}
}
stage('Install dependencies') {
steps {
node('any') {
sh 'pip3 install requests pytest charset-normalizer'
sh 'pip3 uninstall -y chardet'
}
}
}
stage('Run tests') {
steps {
node('any') {
sh 'sudo -u nashtech pytest /home/nashtech/chatbot-testing/tests/'
}
}
}
}
post {
always {
node('any') {
sh 'sudo -u nashtech pytest /home/nashtech/chatbot-testing/tests/ > /home/nashtech/chatbot-testing/test_results.txt' // Capture test results and save them into a text file
}
}
}
}
Step 6: Configuring the Jenkins Job
- Create a New Jenkins Job:
- Go to Jenkins Dashboard.
- Click on “New Item”.
- Enter a name for your job, select “Pipeline”, and click “OK”.
- Configure the Pipeline:
- Scroll down to the “Pipeline” section.
- Select “Pipeline script from SCM”.
- Set “SCM” to “Git”.
- Enter your repository URL in the “Repository URL” field.
- Set “Script Path” to
Jenkinsfile.
- Save the Job:
- Click “Save”.
Step 7: Running the Tests
- Trigger the Job:
- Go to your Jenkins job.
- Click “Build Now”.
- View the Results:
- Click on the build number.
- Go to “Console Output” to see the details of the build.
- Check the “Test Results” for detailed test results.
5. Output
You will get this result in test_results.txt file as below.
Question | Expected Response | Actual Response | Result F [100%] Hello, how are you? | I'm fine, thank you! | I'm fine, thank you! | Passed Goodbye! | Goodbye! | See you later! | Failed What is the capital of France? | The capital of France is Paris. | Paris | Passed What is 2 + 2? | 2 + 2 equals 4. | 5 | Failed Tell me a joke. | Why don't scientists trust atoms? Because they make up everything! | How about this one: Why did the chicken cross the road? To get to the other side! | Failed What is the weather like today? | I'm not sure, but you can check a weather website for the latest updates. | It's sunny outside! | Failed Who wrote 'Pride and Prejudice'? | Pride and Prejudice was written by Jane Austen. | Jane Austen | Passed Translate 'hello' to Spanish. | The translation of 'hello' to Spanish is 'hola'. | hola | Passed What is the meaning of life? | The meaning of life is a philosophical question that has been debated for centuries. Some say it is to find happiness and fulfillment. | To find happiness and fulfillment | Failed How many continents are there? | There are seven continents: Africa, Antarctica, Asia, Europe, North America, Australia, and South America. | 6 | Failed
PASSED ../../../../../home/nashtech/chatbot-testing/tests/test_chatbot.py::test_chatbot_responses
============================== passed in 0.88s ===============================
6. Conclusion:
Integrating AI Chatbot Autotesting. This setup demonstrates a simple integration of chatbot automated tests within a CI/CD pipeline using Python and Jenkins. By following these steps, you can ensure that your chatbot responses are consistently validated, providing a reliable conversational experience. Feel free to expand on these tests and adapt the pipeline to suit more complex scenarios and different CI/CD tools.
7. Reference
1.https://blog.nashtechglobal.com/the-challenges-in-testing-ai-based-systems/
2.https://medium.com/@nitaasli/automated-testing-in-ci-cd-pipelines-best-practices-0f31dc9f4482