How-To: Read Your Test & Issue Reports

The value of testing lies in the clarity of its results. Here’s how to interpret the reports Pie generates.

Reading the Test Case List

After a run, the “Test Cases” tab will show you a list of every test that was executed.

• Passed Test: The AI agent completed every step and verified every assertion successfully.
• Failed Test: The agent encountered a blocker.
  Click on a failed test to see:
  • Which step failed and why
  • Whether an element wasn’t found or an assertion failed
  • A visual replay showing the exact moment of failure

Anatomy of an Issue Report

This is the most critical report for your development team. Every issue is a self-contained, actionable bug report.

Title & Severity

• Descriptive title generated by the AI (e.g., “Application Crashes When Navigating to Settings”)
• Severity (Critical, High, Medium, Low) is auto-assigned based on impact
• You can manually adjust severity as needed

Expected vs. Observed Behavior

• A clear summary of what was expected vs. what actually happened

Visual Replay

• Video or step-by-step screenshots showing the bug in real-time
• Eliminates “could not reproduce” issues

Reproduction Steps

• Precise, numbered instructions to manually reproduce the bug

Manage the Issue

• Use “Dismiss” if it’s not a real bug (this trains the AI)
• Push valid bugs directly into your development backlog using the Jira integration