Regression Suite

Run test suites against any Apify actor and automatically compare results to previous runs. Actor Regression Suite detects regressions (tests that were passing but now fail), resolved issues (tests that were failing but now pass), and new test results. Get a clear view of what changed between actor versions so you can ship with confidence.

Actor Regression Suite extends the test runner pattern with historical comparison. Provide test cases and (optionally) results from a previous run. The suite runs all tests, classifies each as pass, fail, regression, resolved, new_pass, or new_fail, and produces a report showing exactly what changed. Integrate into CI/CD to block deploys when regressions appear.

What data can you extract?

Why use Regression Suite?

A test suite tells you what's broken right now. A regression suite tells you what broke since last time. This distinction matters:

• A test that has always failed is a known issue. A test that was passing yesterday and fails today is an urgent regression.
• A test that was failing last week but passes now is a resolved issue worth celebrating (or at least noting).
• A brand new test with no prior history needs different attention than an established test.

Without historical comparison, you can't tell the difference. Regression Suite classifies every test result so you can focus on what actually changed.

Features

• Six-state classification for every test case: pass, fail, regression (was passing, now fails), resolved (was failing, now passes), new_pass, new_fail
• Automatic previous result injection -- when used through the ApifyForge dashboard, previous results are automatically loaded from your last run. No manual tracking needed.
• Same assertion engine as Test Runner with six assertion types: minResults, maxResults, requiredFields, fieldTypes, maxDuration, noEmptyFields
• Sequential execution to avoid overwhelming target actors
• Per-case error isolation -- one crashed test doesn't block the rest
• Regression-first reporting -- regressions are highlighted at the top for immediate attention
• Suite versioning with automatic datestamp for tracking changes over time
• Single PPE charge ($0.35) per suite, not per test case

Use cases for regression testing

Post-deploy verification

After pushing a new actor version, run the regression suite with the same test cases used before the deploy. Regressions mean your update broke something. Block the release, fix the issue, re-deploy, re-test.

Upstream change detection

Websites change their HTML structure. APIs deprecate endpoints. Government databases update schemas. Schedule the regression suite weekly to catch upstream changes that break your actor's output without any code changes on your end.

Migration validation

Switching from one scraping approach to another (e.g., Cheerio to Playwright)? Run the regression suite before and after. Any regressions indicate the new approach doesn't cover all the same scenarios.

Team coordination

When multiple developers work on the same actor, regressions from one person's changes are immediately visible. The "resolved" status shows when someone fixes a previously broken test.

Release notes automation

The regression report is structured JSON. Parse it in CI/CD to auto-generate release notes: "2 regressions fixed, 1 new test added, 0 regressions introduced."

How to run a regression suite

1. Enter the target actor -- Provide the actor ID or username/actor-name slug.
2. Define test cases -- Same format as Actor Test Runner: name, input, assertions.
3. Provide previous results (optional) -- Pass results from a prior run. If omitted, all results are classified as new_pass or new_fail.
4. Run the suite -- Click "Start" and wait for all test cases to complete.
5. Review regressions -- Check the report for regression and resolved statuses.

When used through the ApifyForge dashboard, previous results are automatically injected from your last cached run -- no manual tracking needed.

Input parameters

Previous results format

[
    { "name": "Basic search", "passed": true },
    { "name": "Empty input", "passed": false },
    { "name": "Special characters", "passed": true }
]

The name field must match the test case name exactly. Unmatched previous results are ignored. Test cases without a matching previous result are classified as new_pass or new_fail.

Input example

{
    "targetActorId": "ryanclinton/website-contact-scraper",
    "testCases": [
        {
            "name": "Basic scan",
            "input": { "urls": ["https://example.com"], "maxPagesPerDomain": 3 },
            "assertions": { "minResults": 1, "requiredFields": ["url", "domain", "emails"] }
        },
        {
            "name": "Multiple domains",
            "input": { "urls": ["https://example.com", "https://httpbin.org"], "maxPagesPerDomain": 2 },
            "assertions": { "minResults": 2, "noEmptyFields": ["url", "domain"] }
        }
    ],
    "previousResults": [
        { "name": "Basic scan", "passed": true },
        { "name": "Multiple domains", "passed": true }
    ]
}

Output example

{
    "actorName": "ryanclinton/website-contact-scraper",
    "actorId": "abc123def456",
    "suiteVersion": "2026-03-18",
    "totalTests": 2,
    "passed": 1,
    "failed": 1,
    "regressions": 1,
    "resolved": 0,
    "newTests": 0,
    "totalDuration": 30.5,
    "details": [
        {
            "name": "Basic scan",
            "status": "pass",
            "previousStatus": "pass",
            "currentStatus": "pass",
            "duration": 12.1,
            "resultCount": 1,
            "assertions": [
                { "assertion": "minResults >= 1", "passed": true, "expected": 1, "actual": 1 },
                { "assertion": "field 'url' exists", "passed": true, "expected": "present", "actual": "present" }
            ]
        },
        {
            "name": "Multiple domains",
            "status": "regression",
            "previousStatus": "pass",
            "currentStatus": "fail",
            "duration": 18.4,
            "resultCount": 1,
            "error": null,
            "assertions": [
                { "assertion": "minResults >= 2", "passed": false, "expected": 2, "actual": 1 },
                { "assertion": "field 'url' not empty", "passed": true, "expected": "non-empty", "actual": "non-empty" }
            ]
        }
    ],
    "testedAt": "2026-03-18T14:30:00.000Z"
}

Output fields

Status classification matrix

How much does it cost?

Regression Suite uses pay-per-event pricing at $0.35 per suite run. Target actor runs are billed separately.

Run regression suites using the API

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("ryanclinton/actor-regression-suite").call(run_input={
    "targetActorId": "ryanclinton/website-contact-scraper",
    "testCases": [
        {
            "name": "Basic scan",
            "input": {"urls": ["https://example.com"], "maxPagesPerDomain": 3},
            "assertions": {"minResults": 1, "requiredFields": ["url", "domain"]},
        },
    ],
    "previousResults": [
        {"name": "Basic scan", "passed": True},
    ],
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Regressions: {item['regressions']} | Resolved: {item['resolved']}")
    for d in item["details"]:
        print(f"  [{d['status'].upper()}] {d['name']} (was: {d['previousStatus']}, now: {d['currentStatus']})")

JavaScript

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "YOUR_API_TOKEN" });

const run = await client.actor("ryanclinton/actor-regression-suite").call({
    targetActorId: "ryanclinton/website-contact-scraper",
    testCases: [{
        name: "Basic scan",
        input: { urls: ["https://example.com"], maxPagesPerDomain: 3 },
        assertions: { minResults: 1, requiredFields: ["url", "domain"] },
    }],
    previousResults: [{ name: "Basic scan", passed: true }],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const report = items[0];
console.log(`Regressions: ${report.regressions} | Resolved: ${report.resolved}`);

FAQ

How do I get previous results for comparison? Three ways: (1) The ApifyForge dashboard auto-injects them from your last run. (2) Save the details array from a prior run and map it to previousResults. (3) On first run, skip previousResults -- all results will be new_pass or new_fail, establishing your baseline.

What if I add a new test case? New test cases without a matching entry in previousResults are classified as new_pass or new_fail. They establish a baseline for future comparisons.

What if I rename a test case? Matching is by exact name. A renamed test case appears as new_pass/new_fail (new name) while the old name is simply absent from the report. Keep names stable for meaningful regression tracking.

What's the difference between Regression Suite and Test Runner? Test Runner gives you pass/fail per test case. Regression Suite adds historical comparison -- it tells you whether failures are new (regressions) or pre-existing. Use Test Runner for one-off testing, Regression Suite for ongoing quality tracking.

Can I track regressions across multiple actors? Run a separate regression suite for each actor. The structured output can be aggregated in a dashboard or spreadsheet.

Related actors

Support

Found a bug or have a feature request? Open an issue in the Issues tab on this actor's page.