As AI agents become increasingly responsible for generating code, test cases, documentation, and technical recommendations, organizations face a new challenge: how can we trust AI-generated outputs before using them in production?
Traditional software is tested using structured test cases and measurable quality criteria. AI agents should be evaluated with the same rigor. In this article, I share a practical approach for using one AI agent to evaluate another AI agent, enabling repeatable, evidence-based assessment of quality, consistency, and compliance with engineering standards.
Why AI Agent Evaluation Matters
AI-generated outputs introduce several risks:
- Incorrect implementation
- Incomplete requirements coverage
- Non-compliance with coding standards
- Security vulnerabilities
- Inconsistent output quality across prompts
- Hallucinated information
Without evaluation, a successful demonstration may hide significant quality issues that appear only in real-world usage.
In this article, I’ll share one practical that using an AI agent to test another AI agent. It provides a structured, repeatable way to evaluate the quality and consistency of AI-generated outputs before they are used in real projects.
Example Folder Structure For a Eval Skill
workspace/
├── ai-agent-skill-evaluator/
│ ├── SKILL.md
│ ├── prompt.md
│ ├── evals/
│ │ └── evals-ai-implement-automation.json
│ └── templates/
│ └── benchmark.json
│ └── evaluation-results-template.json
│ └── feedback-template.json
│
└── ai-implement-automation/ (target Skill to be valuated)
│ └── SKILL.md
│
└── skill-evaluation-workspace/
Where:
ai-agent-skill-evaluator= the skill used to test another skillai-implement-automation= the target skill being evaluatedskill-evaluation-workspace= where evaluation results will be stored
SKILL.md for ai-agent-skill-evaluator
# AI Agent Skill Evaluator
## Purpose
This skill evaluates the output quality of another Agent Skill.
It helps determine whether a target skill produces reliable, useful, and reviewable outputs across realistic prompts, edge cases, and baseline comparisons.
Use this skill when you need to:
- Test whether a target skill works reliably
- Compare output with and without the target skill
- Grade outputs against clear assertions
- Identify quality gaps
- Provide improvement recommendations for the target skill
- Support human review before approving changes
---
## Inputs
The evaluator expects the following inputs:
1. Target skill path
The path to the skill being evaluated.
2. Evaluation cases
A JSON file containing test prompts, expected outputs, input files, and assertions.
3. Output folders
A workspace location where results should be saved.
4. Optional baseline
This can be:
- No skill
- Previous version of the skill
- Another competing skill
---
## Evaluation Workflow
For each evaluation case, perform the following steps.
### Step 1: Read the Evaluation Case
Read the prompt, expected output, input files, and assertions.
Confirm that the test case is realistic and understandable.
If the test case is too vague, mark it as needing improvement instead of forcing a pass/fail result.
---
### Step 2: Run Baseline Evaluation
Run the prompt without the target skill, or with the baseline skill if provided.
Save all outputs to:
```text
/outputs/Case Study
I assume we already have an AI skill that generates maintainable Playwright automation scripts from approved test cases, web pages, or screenshots, while reusing existing project fixtures, page objects, components, and test data whenever possible. The skill is called ai-implement-automation skill.
ai-implement-automation skill
The purpose is to implement maintainable Playwright automation scripts from a web page, screenshot or approved test cases while maximizing reuse of existing project fixtures, page objects, components, and test data.
The skill should help testers create:
- Test scenarios
- Playwright test scripts
- Page Object methods
- Meaningful assertions
- Stable locators
- Clear file structure
What We Should Evaluate
For the Playwright automation skill, we should evaluate more than whether the generated script simply exists or runs successfully. A script can compile and execute but still be difficult to maintain, inconsistent with project standards, or unsuitable for long-term use in a real automation framework.
To assess the true value of the skill, we need to define evaluation criteria that reflect what a high-quality automation engineer would expect from a production-ready Playwright solution. The goal is not only to verify that the AI can generate code, but also to determine whether the generated code is correct, maintainable, reusable, reviewable, and aligned with established automation standards.
Key evaluation areas include:
- Functional Correctness
- Standards Compliance
- Maintainability
- Reusability
- Locator and Assertion Quality
- Reviewer Readiness
- Multi-Run Consistency
By evaluating these areas, we can determine whether the skill consistently generates automation assets that are suitable for real-world projects, not just whether it can generate code.
Design Test case
In software testing, we design test cases to verify that an application works correctly across different scenarios. Likewise, an AI agent should also be tested with a set of well-designed test cases.
Instead of testing application functionality, these test cases evaluate the AI agent’s ability to perform its task consistently and correctly. Each test case typically includes:
Prompt: a realistic user request
Expected Output: the desired result
Assertions: criteria to verify the quality of the AI’s response
For example, when evaluating an AI skill that generates Playwright automation tests, we create multiple test cases covering different scenarios and verify that the generated code follows project standards, uses stable locators, applies the Page Object Model correctly, and includes meaningful assertions.
evals-ai-implement-automation.json
{
"skill_name": "ai-implement-automation",
"evals": [
{
"id": "eval-login-valid-invalid",
"prompt": "Create Playwright tests for login with valid credentials and invalid password. Follow Page Object Model structure. Target url: https://demo.nopcommerce.com/login ",
"expected_output": "Playwright test files with POM structure, meaningful assertions, and stable locators.",
"files": [],
"assertions": [
"The output includes at least one Playwright test file.",
"The test covers both valid login and invalid password scenarios.",
"The output separates test logic from page actions using Page Object Model.",
"The test uses stable locators such as getByRole, getByLabel, or data-testid.",
"The test includes meaningful assertions for success and error states."
]
},
{
"id": "eval-add-single-product-to-cart",
"prompt": "Create a Playwright automation test for adding the product 'Build your own computer' to the shopping cart on https://demo.nopcommerce.com/. Follow the Page Object Model architecture and use stable Playwright locators.",
"expected_output": "A Playwright automation test using Page Object Model that successfully adds a single product to the shopping cart and verifies the cart contents.",
"files": [],
"assertions": [
"The output includes at least one Playwright test file.",
"The output follows the Page Object Model structure.",
"The output creates or uses a Product Page object.",
"The output creates or uses a Shopping Cart Page object.",
"The test uses stable locators such as getByRole, getByLabel, or data-testid when available.",
"The test verifies the success notification after adding the product.",
"The test verifies the shopping cart contains the expected product.",
"The test includes meaningful assertions."
]
},
{
"id": "eval-add-product-guest-user",
"prompt": "Generate a Playwright automation test for a guest user adding a product to the shopping cart on https://demo.nopcommerce.com/. Follow Page Object Model architecture and verify guest shopping behavior.",
"expected_output": "A Playwright automation test validating that an anonymous user can successfully add a product to the shopping cart.",
"files": [],
"assertions": [
"The output follows the Page Object Model structure.",
"The test does not require user login.",
"The test successfully adds a product as a guest user.",
"The test verifies the shopping cart contains the selected product.",
"The output uses stable Playwright locators.",
"The test contains meaningful assertions.",
"The output includes human review notes before merge."
]
}
]
}Running the Evaluations
Each evaluation test case is executed against the AI agent to generate an output. For better comparison, you can run the same test case against different versions(for example, with_skill vs without_skill) to measure whether the quality has improved.
You can call the evaluator skill by using a simple like this:
Use the ai-agent-skill-evaluator skill to evaluate the ai-implement-automation skill.
Target skill path:
workspace/ai-implement-automation
Evaluation file:
workspace/ai-agent-skill-evaluator/evals-ai-implement-automation.json
Workspace output path:
workspace/skill-evaluation-workspace/iteration-1
For each eval case:
1. Run the prompt without using the target skill times and save output to without_skill/outputs.
2. Run the same prompt with using the target skill 3 times and save output to with_skill/outputs.
3. Grade each output against the assertions.
4. Create evaluation-results.json, timing.json, feedback.json, and benchmark.json.
5. Compare with_skill vs without_skill.
6. Recommend improvements for the target skill.
Do not approve the target skill automatically. Include human review notes before final acceptance.This follows the recommended pattern of keeping each evaluation iteration separate and storing with_skill and without_skill results independently.
Multi-Run Consistency Testing
To evaluate stability and predictability, each evaluation test case should be executed multiple times under the same conditions. The evaluator can then compare the results across runs and determine whether the skill consistently meets the expected quality criteria.
For each run, the evaluator should:
- Generate the output using the same prompt.
- Validate the output against the defined assertions.
- Record pass/fail results and supporting evidence.
- Measure execution metrics such as duration and token usage.
- Compare results across all runs to identify variations or recurring failures.
Expected Output Structure
After running the evaluator skill, the result should look like this:
skill-evaluation-workspace/
└── iteration-1/
└── eval-login-valid-invalid/
├── without_skill/
│ ├── outputs/
│ ├── timing.json
│ └── evaluation-results.json
│
├── with_skill/
│ ├── outputs/
│ ├── timing.json
│ └── evaluation-results.json
│
├── feedback.json
└── benchmark.jsonExample evaluation-results.json with skill
{
"eval_id": "eval-login-valid-invalid",
"run": "with_skill",
"assertion_results": [
{
"text": "The output includes at least one Playwright test file.",
"status": "PASS",
"evidence": "Generated file exists: generated/login-with-pom.spec.ts with Playwright test suite."
},
{
"text": "The test covers both valid login and invalid password scenarios.",
"status": "PASS",
"evidence": "Contains tests for valid credentials and invalid password flows."
},
{
"text": "The output separates test logic from page actions using Page Object Model.",
"status": "PASS",
"evidence": "Includes LoginPage class encapsulating open/login/successBanner/errorBanner methods."
},
{
"text": "The test uses stable locators such as getByRole, getByLabel, or data-testid.",
"status": "PASS",
"evidence": "Uses getByLabel for fields and getByRole for button/heading/alert locators."
},
{
"text": "The test includes meaningful assertions for success and error states.",
"status": "PASS",
"evidence": "Asserts dashboard navigation and visible success heading, plus alert text and URL retention for failure."
}
],
"summary": {
"passed": 5,
"failed": 0,
"not_applicable": 0,
"total": 5,
"pass_rate": 1.0
}
}
Example evaluation-results.json without skill
{
"eval_id": "eval-login-valid-invalid",
"run": "without_skill",
"assertion_results": [
{
"text": "The output includes at least one Playwright test file.",
"status": "PASS",
"evidence": "Generated file exists: generated/login.spec.ts with Playwright imports and test blocks."
},
{
"text": "The test covers both valid login and invalid password scenarios.",
"status": "PASS",
"evidence": "Contains two tests: 'login with valid credentials' and 'login with invalid password'."
},
{
"text": "The output separates test logic from page actions using Page Object Model.",
"status": "FAIL",
"evidence": "Actions are directly in spec using page.fill/page.click; no page object class or abstraction present."
},
{
"text": "The test uses stable locators such as getByRole, getByLabel, or data-testid.",
"status": "FAIL",
"evidence": "Locators use CSS selectors like '#email', '.error', and 'button[type=\"submit\"]' instead of role/label/testid locators."
},
{
"text": "The test includes meaningful assertions for success and error states.",
"status": "PASS",
"evidence": "Success checks dashboard URL/header; failure checks invalid credentials error."
}
],
"summary": {
"passed": 3,
"failed": 2,
"not_applicable": 0,
"total": 5,
"pass_rate": 0.6
}
}
Example benchmark.json
{
"skill_name": "ai-implement-automation",
"evaluation_date": "2026-07-18",
"run_summary": {
"with_skill": {
"pass_rate": 1.0,
"average_duration_ms": 15333,
"average_tokens": 2933,
"passed_assertions": 20,
"failed_assertions": 0,
"total_assertions": 20
},
"without_skill": {
"pass_rate": 0.55,
"average_duration_ms": 10000,
"average_tokens": 1967,
"passed_assertions": 11,
"failed_assertions": 9,
"total_assertions": 20
},
"delta": {
"pass_rate": 0.45,
"duration_ms": 5333,
"tokens": 966,
"passed_assertions": 9,
"failed_assertions": -9
}
},
"case_comparison": [
{
"eval_id": "eval-login-valid-invalid",
"without_skill_pass_rate": 0.6,
"with_skill_pass_rate": 1.0,
"delta": 0.4
},
{
"eval_id": "eval-add-single-product-to-cart",
"without_skill_pass_rate": 0.5,
"with_skill_pass_rate": 1.0,
"delta": 0.5
},
{
"eval_id": "eval-add-product-guest-user",
"without_skill_pass_rate": 0.5714,
"with_skill_pass_rate": 1.0,
"delta": 0.4286
}
],
"recommendation": "The target skill materially improves quality and standards compliance, especially POM usage, stable locator usage, and reviewer safeguards. Human review is required before adoption.",
"human_review_required": true
}
Example of feedback.json
{
"human_review_required": true,
"overall_feedback": {
"strengths": [
"Consistent gains in Page Object Model usage and separation of concerns in with-skill runs.",
"Stronger locator strategy using role-based selectors and explicit cart/success assertions.",
"Improved reviewer readiness through explicit human-review notes in guest-user output."
],
"gaps": [
"Without-skill baseline still lacks reusable page objects and stable role/label/testid locators.",
"Some with-skill examples inline page classes in spec files, which may not match larger-suite file organization preferences.",
"Timing and token usage are estimate-based and should be replaced with provider telemetry when available."
]
},
"improvement_recommendations": [
"Require explicit mapping from each generated test to approved requirement and test-case IDs.",
"Mandate dedicated page object files for multi-flow scenarios to improve maintainability.",
"Add deterministic output formatting guidance so generated code layout remains stable across runs.",
"Add a required 'Human Review Checklist' block in all generated outputs before merge recommendation."
],
"review_notes": [
"Do not auto-approve this skill based only on benchmark metrics.",
"Human review is required before adoption to validate maintainability, environment fit, and CI stability."
]
}
Conclusion
As AI-generated artifacts become part of daily software engineering activities, evaluating AI agents will become as important as testing software applications themselves.
By treating an AI agent as a system under test and applying structured evaluation techniques, teams can move beyond subjective demonstrations and establish measurable evidence of quality.
The goal is not to replace human judgment. Instead, AI-based evaluation provides scalable quality signals that help reviewers make more informed decisions before AI-generated outputs are accepted into real projects.