AIO Tests is a test management tool for Jira, it is a Jira-native app with many installations from the Atlassian marketplace and good reviews from users. The tool has a one-click installation, and you are good to go with the app’s default settings. Minimalistic but has an enjoyable interface. Everything looks logical. There are two types of test cases: classic and BDD. The UX for BDD cases is the best of the compared tools. The tests are created with test suffixes, the same as in Zephyr Scale, so they are not shown by default on the board or backlog.
When compared to other tools, the AIO Tests tool has the following advantages:
- A powerful tool with a straightforward user interface.
- The test cycle may be done offline and the results imported back into AIO.
- The documentation is quite detailed.
- If you have a team of 11- 50 employees, it is less expensive than the other rivals (350$/year, or 29.17$/month). Free for groups of up to ten persons.
- Many widgets are supported to create a nice dashboard.
- Make APIs available for integration with the automation frameworks to update test results.
AIO Tests allows users to import their Cucumber based tests/scenarios into AIO Tests, either by mapping to existing cases or by creating new ones from the Cucumber results JSON file. Since AIO Tests rely only on the JSON report generated by Cucumber, the tests can be written in any of the supporting languages like Ruby, Perl, Java, Javascript, etc.
The basic process will be like this:
Install AIO Test and Create Manual Test Cases
To install AIO Test, simply go to Atlassian marketplace, search it and then follow steps to add it to Jira, the application is free for team up to 10 users. For larger team, you need to pay USD 1.25 per user per month. Noted that you need to have administrator Jira permission to install new application.
After installing successfully, you can access the AIO Test by left menu ‘AIO Tests’ or from top menu ‘Apps’.
AIO Tests support writing test cases in classic or BDD format, for example, I have a test scenario to login success to the bookcart website https://bookcart.azurewebsites.net/login as the image in the right side.
We also created an empty test cycle: LP-CY-1
Mark for automation
In the test case details, take note value below:
– Test case key ‘TP-TC-3’: This key will be used to map between the AIO Test and the automated test scenario.
– ‘To Be Automated’ automation status: This field indicates that the case will be automated.
– Jira requirement: The requirement to which the test case refers.
Mapping AIO Test case and automation scenario
Export .feature file
AIO Test supports exporting scenarios to a feature file (and import an existing feature file to test cases as well), select the login test case above and export it from option menu:
After downloaded and extract the zip file, we have a .feature file like this.
- @JREQ-LP-1: This tag indicates that the test case is linked to requirement ‘LP-1’
- Feature name ‘Implement login feature’: The name same as the Jira requirement that the test case is linking.
- @LP-TC-3: This tag is AIO Test case key. It helps to indicate what test case will update the test result after importing to AIO Test.
Implement test script
Feature file: Rename the feature file which exported from AIO Test to login.feature and copy it to automation framework.
Note: The important thing to remember that the scenario uses the tag ‘@LP-TC-3’ to update result to AIO Test so that the tag should be the same AIO Test case key.
@JREQ-LP-1
Feature: Implement login feature
@LP-TC-3
Scenario: Login should be success with valid username and password
Given User navigates to the application
And User click on the login link
When User enter the username as "binhle"
And User enter the password as "Pass@123"
And User click on the login button
Then The Username should be displayed in homepage
Steps Definition: Now it’s time for you to implement the steps definition, it is depends on your framework, you can apply Page Object Pattern and other technical to write these steps.
An example code that you can refer from my repo on GitHub: https://github.com/Binhlex/playwright-cucumber
Configuration to generate JSON report
Config to generate JSON report in cucumber.json (line 11):
{
"default": {
"formatOptions": {
"snippetInterface": "async-await"
},
"paths": ["src/test/features/"],
"publishQuiet": true,
"dryRun": false,
"require": ["src/test/steps/*.ts", "src/hooks/hooks.ts"],
"requireModule": ["ts-node/register"],
"format": [
"progress-bar",
"json:test-result/json/cucumber-report.json",
],
}
}
- progress-bar is optional, it will display a bar in terminal when running test.
- A cucumber-report.json will be generated in folder ‘test-result/json after the test finished.
In package.json, add a test script:
"scripts": {
"test": "cucumber-js test"
}
Execute and view the JSON report
In VS Code, open terminal (Ctrl + J) and run the command:
npm test
After the test finish, a ‘cucumber-report.json’ was generated in test-result/json folder.
Import result to AIO Test
AIO Tests supports importing of automation tests execution report through the Import Results API. After executing the test suite, the json result file is generated. This results file can then be imported by passing the file as part of a multipart/form-data request.
API specification: https://tcms.aioreports.com/aiotcms-static/api-docs/#/Execution%20management/importTestResults
In src/helper folder, create a new file ‘update-AIO-report.ts’
// src/helper/update-AIO-report.ts
import axios from "axios";
import fs from "fs";
import FormData from "form-data";
import path from "path";
require("dotenv").config();
const jiraProjectKey = "LP";
const cyclekey = "LP-CY-1";
function importResultToAIOTest() {
console.log("...Updating test result to AIO Test");
const promises: Promise[] = [];
fs.readdirSync("./test-result/json").forEach((file) => {
const jsonPath = path.join(
__dirname,
"..",
"..",
"test-result",
"json",
file
);
const bodyFormData = new FormData();
bodyFormData.append("file", fs.createReadStream(jsonPath));
bodyFormData.append("addCaseToCycle", "true");
promises.push(
axios
.post(
`https://tcms.aiojiraapps.com/aio-tcms/api/v1/project/${jiraProjectKey}/testcycle/${cyclekey}/import/results?type=Cucumber`,
bodyFormData,
{
headers: {
Authorization: `AioAuth ${process.env.AIO_TOKEN}`,
"Content-Type": "multipart/form-data",
},
}
)
.then((response) => {
console.log("Response data: " + JSON.stringify(response.data));
})
.catch((e) => console.log("error: " + e.message))
);
});
Promise.all(promises);
}
importResultToAIOTest();
- The method used Axios to make an HTTP call, you can use any HTTP client as well.
- It creates an array of promises, each promise is a ‘post’ request to update the test result from JSON file which was read from the folder “./test-result/json”. The last line ‘Promise.all(promises); will ensure all promises are resolved before ending the process.
- The request URL contains ${jiraProjectKey} and ${cyclekey} which we can get from Jira.
- API token in header: You can generate it in AIO Test UI and manage it as an environment variable using dotenv.
- From AIO Test UI, select setting icon in top right corner
- Select My setting menu
- In new window, select API Token
- Select Manage token and generate the new one.
- To send the data as a multipart/formdata you need to pass a formData instance as a payload. Setting the Content-Type header is not required as Axios guesses it based on the payload type. There are some parameters acceptable from the import result API:
- file: Results xml file (testng-results.xml/cucumber.json/junit.xml
- createNewRun: Set to true if a new run should be created for the case and execution status set against the new run.Set to false to update existing run.
- addCaseToCycle: Set to true if Case from the results file should be added to the cycle, if it is not already existing.If not set, such a Case would be reported as an error.
- createCase: Set to true to create a new case, if no mapping is found. Case is created with method name in title. If not set, such a Case would be reported as an error.
- bddForceUpdateCase: Set to true to update case steps with those found in scenario. Use if automated scenarios get updated and changes need to be percolated to AIO Tests cases.
- examplesAsCases: Affects only scenario outline cases. Set to true to create/update each example as a separate cases and not as a single scenario outline case Note that scenario outline cases need to be created/updated from Case Details and cannot be created/updated from results file otherwise.
Create a new script in package.json
"scripts": {
"test": "cucumber-js test || true",
"posttest": "npx ts-node src/helper/update-AIO-report.ts"
}
From terminal, run the test script again and see if the test result updated to AIO Test:
npm test
Go to AIO Test and check the result, you will see the test ‘@LP-TC-3’ was added cycle ‘LP-CY-1: Sprint1 – Login Feature’ and it also updated the test result as passed. The test case exists in AIO Test but it did not add to the cycle before the automation test run, the import result API was added it to cycle thank to the parameter in request: bodyFormData.append(“addCaseToCycle”, “true”);
Import a non-existing test scenario to AIO Test
Assume that you have a new test scenario ‘Login error if user inputs invalid password’ from your automation framework but it does not exist on AIO Test yet, the feature file has two scenarios as below:
@JREQ-LP-1
Feature: Implement login feature
Background: Go to login page
Given User navigates to the application
And User click on the login link
@LP-TC-3
Scenario: Login should be success with valid username and password
When User enter the username as "binhle"
And User enter the password as "Pass@123"
And User click on the login button
Then The Username should be displayed in homepage
Scenario: Login error if user inputs invalid password
When User enter the username as "binhle"
And User enter the password as "binhle"
And User click on the login button
Then Error message should be displayed
After implementation the steps definition for the scenario, go to ‘update-AIO-report.ts’ and add more field to the request body:
bodyFormData.append("createCase", "true");
Run the test that includes new scenario:
npm test
Go to the AIO Test, we see a new test case was created and added to Cycle as well.
In Cycles menu:
In Cases menu:
Notice that the new test case uses the value ‘implement-login-feature;login-error-if-user-inputs-invalid-password’ in ‘Automation Key’ field for mapping to automation test scenario, its value can found in JSON report, field ‘id’.
Conclusion
AIO Test supports the import of test results from several automated testing frameworks like Cucumber, JUnit, TestNG, NUnit, Cypress, and Playwright. As previously mentioned for Cucumber, AIO Test relies on JSON reports supplied by Cucumber to update test results, it is able to integrate with other Cucumber-supported automated test frameworks, and tests may be written in any of the supporting languages. We can create more operations using its offered APIs, such as generating a new test cycle and pasting the key to import the test script instead of creating it manually. Let’s give it a go and see what happens!!!
Reference:
