Test Runner

Run any Apify actor through a suite of test cases with different inputs and validate output against assertions. Actor Test Runner executes each test case sequentially, checks result counts, required fields, field types, duration limits, and empty values, then produces a structured pass/fail report. Build confidence in your actors before publishing, after code changes, or as part of a CI/CD pipeline.

Actor Test Runner takes a target actor ID and an array of test cases. Each test case has a name, an input object, and an assertion set. The runner executes the actor once per test case, fetches the output dataset, evaluates all assertions, and reports pass/fail with details. One PPE charge ($0.35) covers the entire suite -- not per test case.

What data can you extract?

Why use Actor Test Runner?

Manual testing means running an actor once with one input and eyeballing the output. This misses edge cases: empty queries that should return zero results, special characters that break parsing, large result sets that trigger pagination bugs, inputs with missing optional fields. These edge cases break in production, not in your quick manual check.

Actor Test Runner lets you define multiple test scenarios in a single suite and run them all automatically. Each scenario has its own input and its own assertions. The suite tells you exactly which cases pass and which fail, with specific details about every failed assertion.

Why an actor instead of a local test script?

• Runs in the same cloud environment -- catches Docker, memory, and network issues that don't appear locally
• No local dependencies -- works from the Apify console, API, or CI/CD without any setup
• Scheduling -- run test suites on a schedule to catch upstream changes
• Single charge per suite -- $0.35 regardless of how many test cases (target actor compute billed separately)
• Structured output -- machine-readable JSON report for CI/CD integration

Features

• Multiple test cases per suite with independent inputs and assertions -- test your actor's behavior across different scenarios in a single run
• Six assertion types: minimum results (minResults), maximum results (maxResults), required fields (requiredFields), field type checking (fieldTypes), duration limits (maxDuration), and empty value detection (noEmptyFields)
• Sequential execution to avoid overwhelming the target actor with parallel runs -- each test case completes before the next starts
• Per-case error isolation -- if one test case crashes, the remaining cases still run and produce results
• Structured pass/fail report with assertion-level detail for each test case
• Single PPE charge ($0.35) per suite run -- not per test case, making multi-case suites economical
• Configurable timeout and memory applied consistently to all test case runs

Use cases for actor test suites

Edge case coverage

Define test cases for boundary conditions: empty input, maximum result counts, special characters, very long strings, missing optional fields. Verify your actor handles each gracefully.

Input validation testing

Test that your actor rejects invalid inputs appropriately: wrong types, missing required fields, out-of-range values. Each test case can assert specific error behaviors.

Multi-scenario smoke tests

Before publishing, run a suite covering your actor's main use cases: different search queries, different filter combinations, different output modes. A passing suite means confidence to publish.

CI/CD integration

Trigger a test suite after every apify push. Parse the JSON report in your CI pipeline. If any test case fails, block the deployment and alert the team.

Performance regression detection

Use maxDuration assertions to catch performance regressions. If a test case that normally completes in 10 seconds starts taking 30, the assertion fails and you investigate before users notice.

Data completeness checks

Use noEmptyFields assertions on critical output fields. If your scraper starts returning empty strings for email or empty arrays for reviews, the test suite catches it immediately.

How to run a test suite

1. Enter the target actor -- Provide the actor ID or username/actor-name slug.
2. Define test cases -- Create a JSON array where each object has name, input, and assertions.
3. Configure runtime -- Set timeout and memory for each test case run.
4. Run the suite -- Click "Start" and wait for all test cases to complete.
5. Review the report -- Check per-case results and assertion details in the Dataset tab.

Input parameters

Test case format

Each test case object has three fields:

{
    "name": "Descriptive test name",
    "input": { "query": "test value", "maxResults": 5 },
    "assertions": {
        "minResults": 1,
        "maxResults": 100,
        "requiredFields": ["name", "url"],
        "fieldTypes": { "name": "string", "url": "string", "rating": "number" },
        "maxDuration": 60,
        "noEmptyFields": ["name", "url"]
    }
}

Assertion types

Input example

{
    "targetActorId": "ryanclinton/website-contact-scraper",
    "testCases": [
        {
            "name": "Basic website scan",
            "input": {
                "urls": ["https://example.com"],
                "maxPagesPerDomain": 3
            },
            "assertions": {
                "minResults": 1,
                "requiredFields": ["url", "domain", "emails"],
                "fieldTypes": { "emails": "array", "url": "string" },
                "maxDuration": 60
            }
        },
        {
            "name": "Multiple URLs",
            "input": {
                "urls": ["https://example.com", "https://httpbin.org"],
                "maxPagesPerDomain": 2
            },
            "assertions": {
                "minResults": 2,
                "requiredFields": ["url", "domain"],
                "noEmptyFields": ["url", "domain"]
            }
        },
        {
            "name": "Empty input handling",
            "input": {
                "urls": [],
                "maxPagesPerDomain": 1
            },
            "assertions": {
                "maxResults": 0,
                "maxDuration": 30
            }
        }
    ],
    "timeout": 120,
    "memory": 512
}

Input tips

• Keep test inputs small -- 1-3 URLs, 3-5 max results. Tests should be fast and cheap.
• Name test cases descriptively -- the name appears in the report. "Basic search" is better than "test1".
• Start with minResults and requiredFields -- these catch the most common failures. Add type checking and duration limits as needed.
• Test edge cases -- empty inputs, single results, maximum limits. These are where bugs hide.
• Order cases from simple to complex -- makes the report easier to read and debug.

Output example

Each suite run produces one report in the dataset:

{
    "actorName": "ryanclinton/website-contact-scraper",
    "actorId": "abc123def456",
    "totalTests": 3,
    "passed": 2,
    "failed": 1,
    "totalDuration": 45.2,
    "results": [
        {
            "name": "Basic website scan",
            "passed": true,
            "duration": 12.1,
            "resultCount": 1,
            "assertions": [
                { "assertion": "minResults >= 1", "passed": true, "expected": 1, "actual": 1 },
                { "assertion": "field 'url' exists", "passed": true, "expected": "present", "actual": "present" },
                { "assertion": "field 'emails' is array", "passed": true, "expected": "array", "actual": "array" },
                { "assertion": "duration <= 60s", "passed": true, "expected": "60s", "actual": "12.1s" }
            ]
        },
        {
            "name": "Multiple URLs",
            "passed": true,
            "duration": 25.0,
            "resultCount": 2,
            "assertions": [
                { "assertion": "minResults >= 2", "passed": true, "expected": 2, "actual": 2 }
            ]
        },
        {
            "name": "Empty input handling",
            "passed": false,
            "duration": 8.1,
            "resultCount": 0,
            "assertions": [
                { "assertion": "maxResults <= 0", "passed": true, "expected": 0, "actual": 0 },
                { "assertion": "duration <= 30s", "passed": true, "expected": "30s", "actual": "8.1s" }
            ],
            "error": "Run status: FAILED"
        }
    ],
    "testedAt": "2026-03-18T14:30:00.000Z"
}

Output fields

How much does it cost?

Actor Test Runner uses pay-per-event pricing at $0.35 per suite run -- one flat fee regardless of how many test cases. Target actor runs are billed separately.

The Apify Free plan ($5/mo credits) covers approximately 14 suite runs (orchestration only).

Run test suites using the API

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("ryanclinton/actor-test-runner").call(run_input={
    "targetActorId": "ryanclinton/website-contact-scraper",
    "testCases": [
        {
            "name": "Basic scan",
            "input": {"urls": ["https://example.com"], "maxPagesPerDomain": 3},
            "assertions": {"minResults": 1, "requiredFields": ["url", "domain"]},
        },
    ],
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Suite: {item['passed']}/{item['totalTests']} passed in {item['totalDuration']}s")
    for tc in item["results"]:
        status = "PASS" if tc["passed"] else "FAIL"
        print(f"  [{status}] {tc['name']} ({tc['duration']}s, {tc['resultCount']} results)")
        for a in tc["assertions"]:
            if not a["passed"]:
                print(f"    FAILED: {a['assertion']} (expected {a.get('expected')}, got {a.get('actual')})")

JavaScript

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "YOUR_API_TOKEN" });

const run = await client.actor("ryanclinton/actor-test-runner").call({
    targetActorId: "ryanclinton/website-contact-scraper",
    testCases: [
        {
            name: "Basic scan",
            input: { urls: ["https://example.com"], maxPagesPerDomain: 3 },
            assertions: { minResults: 1, requiredFields: ["url", "domain"] },
        },
    ],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const report = items[0];
console.log(`Suite: ${report.passed}/${report.totalTests} passed`);
report.results.forEach(tc => {
    console.log(`  [${tc.passed ? "PASS" : "FAIL"}] ${tc.name}`);
});

cURL

curl -X POST "https://api.apify.com/v2/acts/ryanclinton~actor-test-runner/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "targetActorId": "ryanclinton/website-contact-scraper",
    "testCases": [{
      "name": "Basic scan",
      "input": {"urls": ["https://example.com"], "maxPagesPerDomain": 3},
      "assertions": {"minResults": 1, "requiredFields": ["url", "domain"]}
    }]
  }'

FAQ

How many test cases can I include in one suite? There is no hard limit, but each test case runs the target actor sequentially. A suite with 10 cases each taking 30 seconds will take about 5 minutes total. Keep suites focused and fast.

Why are test cases run sequentially, not in parallel? To avoid overwhelming the target actor or exhausting your compute budget unpredictably. Sequential execution gives consistent, reproducible results and predictable costs.

What happens if one test case crashes? The crashed case reports FAIL with the error message. Remaining test cases continue running normally. One bad case doesn't block the rest of the suite.

Can I reuse test cases across actors? Yes, if actors have similar input schemas. Copy the test cases array and change the targetActorId. Useful for comparing replacement actors.

What's the difference between Test Runner and Cloud Staging Test? Cloud Staging Test runs a single input with optional assertions. Test Runner runs multiple inputs with independent assertions per case. Use Cloud Staging Test for quick smoke tests, Test Runner for comprehensive multi-scenario suites.

Can I use this for load testing? No. Test Runner runs one case at a time. For load testing, use the Apify API to trigger multiple parallel runs directly.

Related actors

Support

Found a bug or have a feature request? Open an issue in the Issues tab on this actor's page.