Actor Test Runner — Validate Inputs, Outputs & Error Handling

Run any Apify actor through a suite of test cases with different inputs and validate output against assertions. Actor Test Runner executes each test case sequentially, checks result counts, required fields, field types, duration limits, and empty values, then produces a structured pass/fail report. Build confidence in your actors before publishing, after code changes, or as part of a CI/CD pipeline.

Actor Test Runner takes a target actor ID and an array of test cases. Each test case has a name, an input object, and an assertion set. The runner executes the actor once per test case, fetches the output dataset, evaluates all assertions, and reports pass/fail with details. One PPE charge ($0.35) covers the entire suite -- not per test case.

What data can you extract?

Data Point	Source	Example
Suite result	Aggregated test results	3/4 passed
Per-case result	Individual test execution	Basic search: PASS (12.1s, 5 results)
Assertion details	Evaluated checks per case	minResults >= 3: PASS (actual: 5)
Duration per case	Measured start-to-finish	8.3 seconds
Result counts	Dataset item count	5 items
Total suite duration	Sum of all cases	45.2 seconds
Error messages	Failed run details	Run status: TIMED-OUT

Why use Actor Test Runner?

Manual testing means running an actor once with one input and eyeballing the output. This misses edge cases: empty queries that should return zero results, special characters that break parsing, large result sets that trigger pagination bugs, inputs with missing optional fields. These edge cases break in production, not in your quick manual check.

Actor Test Runner lets you define multiple test scenarios in a single suite and run them all automatically. Each scenario has its own input and its own assertions. The suite tells you exactly which cases pass and which fail, with specific details about every failed assertion.

Why an actor instead of a local test script?

• Runs in the same cloud environment -- catches Docker, memory, and network issues that don't appear locally
• No local dependencies -- works from the Apify console, API, or CI/CD without any setup
• Scheduling -- run test suites on a schedule to catch upstream changes
• Single charge per suite -- $0.35 regardless of how many test cases (target actor compute billed separately)
• Structured output -- machine-readable JSON report for CI/CD integration

Features

• Multiple test cases per suite with independent inputs and assertions -- test your actor's behavior across different scenarios in a single run
• Six assertion types: minimum results (minResults), maximum results (maxResults), required fields (requiredFields), field type checking (fieldTypes), duration limits (maxDuration), and empty value detection (noEmptyFields)
• Sequential execution to avoid overwhelming the target actor with parallel runs -- each test case completes before the next starts
• Per-case error isolation -- if one test case crashes, the remaining cases still run and produce results
• Structured pass/fail report with assertion-level detail for each test case
• Single PPE charge ($0.35) per suite run -- not per test case, making multi-case suites economical
• Configurable timeout and memory applied consistently to all test case runs

Use cases for actor test suites

Edge case coverage

Define test cases for boundary conditions: empty input, maximum result counts, special characters, very long strings, missing optional fields. Verify your actor handles each gracefully.

Input validation testing

Test that your actor rejects invalid inputs appropriately: wrong types, missing required fields, out-of-range values. Each test case can assert specific error behaviors.

Multi-scenario smoke tests

Before publishing, run a suite covering your actor's main use cases: different search queries, different filter combinations, different output modes. A passing suite means confidence to publish.

CI/CD integration

Trigger a test suite after every apify push. Parse the JSON report in your CI pipeline. If any test case fails, block the deployment and alert the team.

Performance regression detection

Use maxDuration assertions to catch performance regressions. If a test case that normally completes in 10 seconds starts taking 30, the assertion fails and you investigate before users notice.

Data completeness checks

Use noEmptyFields assertions on critical output fields. If your scraper starts returning empty strings for email or empty arrays for reviews, the test suite catches it immediately.

How to run a test suite

1. Enter the target actor -- Provide the actor ID or username/actor-name slug.
2. Define test cases -- Create a JSON array where each object has name, input, and assertions.
3. Configure runtime -- Set timeout and memory for each test case run.
4. Run the suite -- Click "Start" and wait for all test cases to complete.
5. Review the report -- Check per-case results and assertion details in the Dataset tab.

Input parameters

Parameter	Type	Required	Default	Description
targetActorId	string	Yes	--	Actor ID or username/actor-name slug to test.
testCases	array	Yes	--	Array of test case objects (see below).
timeout	integer	No	120	Maximum seconds per test case run.
memory	integer	No	512	Memory in MB for each test case run.

Test case format

Each test case object has three fields:

{
    "name": "Descriptive test name",
    "input": { "query": "test value", "maxResults": 5 },
    "assertions": {
        "minResults": 1,
        "maxResults": 100,
        "requiredFields": ["name", "url"],
        "fieldTypes": { "name": "string", "url": "string", "rating": "number" },
        "maxDuration": 60,
        "noEmptyFields": ["name", "url"]
    }
}

Assertion types

Assertion	Type	What it checks
minResults	number	Dataset has at least N items
maxResults	number	Dataset has at most N items
requiredFields	string[]	At least one item has a non-null value for each field
fieldTypes	object	All non-null values for each field match the declared type (string, number, boolean, object, array)
maxDuration	number	Test case completes within N seconds
noEmptyFields	string[]	No items have null, undefined, empty string, or empty array for the listed fields

Input example

{
    "targetActorId": "ryanclinton/website-contact-scraper",
    "testCases": [
        {
            "name": "Basic website scan",
            "input": {
                "urls": ["https://example.com"],
                "maxPagesPerDomain": 3
            },
            "assertions": {
                "minResults": 1,
                "requiredFields": ["url", "domain", "emails"],
                "fieldTypes": { "emails": "array", "url": "string" },
                "maxDuration": 60
            }
        },
        {
            "name": "Multiple URLs",
            "input": {
                "urls": ["https://example.com", "https://httpbin.org"],
                "maxPagesPerDomain": 2
            },
            "assertions": {
                "minResults": 2,
                "requiredFields": ["url", "domain"],
                "noEmptyFields": ["url", "domain"]
            }
        },
        {
            "name": "Empty input handling",
            "input": {
                "urls": [],
                "maxPagesPerDomain": 1
            },
            "assertions": {
                "maxResults": 0,
                "maxDuration": 30
            }
        }
    ],
    "timeout": 120,
    "memory": 512
}

Input tips

• Keep test inputs small -- 1-3 URLs, 3-5 max results. Tests should be fast and cheap.
• Name test cases descriptively -- the name appears in the report. "Basic search" is better than "test1".
• Start with minResults and requiredFields -- these catch the most common failures. Add type checking and duration limits as needed.
• Test edge cases -- empty inputs, single results, maximum limits. These are where bugs hide.
• Order cases from simple to complex -- makes the report easier to read and debug.

Output example

Each suite run produces one report in the dataset:

{
    "actorName": "ryanclinton/website-contact-scraper",
    "actorId": "abc123def456",
    "totalTests": 3,
    "passed": 2,
    "failed": 1,
    "totalDuration": 45.2,
    "results": [
        {
            "name": "Basic website scan",
            "passed": true,
            "duration": 12.1,
            "resultCount": 1,
            "assertions": [
                { "assertion": "minResults >= 1", "passed": true, "expected": 1, "actual": 1 },
                { "assertion": "field 'url' exists", "passed": true, "expected": "present", "actual": "present" },
                { "assertion": "field 'emails' is array", "passed": true, "expected": "array", "actual": "array" },
                { "assertion": "duration <= 60s", "passed": true, "expected": "60s", "actual": "12.1s" }
            ]
        },
        {
            "name": "Multiple URLs",
            "passed": true,
            "duration": 25.0,
            "resultCount": 2,
            "assertions": [
                { "assertion": "minResults >= 2", "passed": true, "expected": 2, "actual": 2 }
            ]
        },
        {
            "name": "Empty input handling",
            "passed": false,
            "duration": 8.1,
            "resultCount": 0,
            "assertions": [
                { "assertion": "maxResults <= 0", "passed": true, "expected": 0, "actual": 0 },
                { "assertion": "duration <= 30s", "passed": true, "expected": "30s", "actual": "8.1s" }
            ],
            "error": "Run status: FAILED"
        }
    ],
    "testedAt": "2026-03-18T14:30:00.000Z"
}

Output fields

Field	Type	Description
actorName	string	Display name of the tested actor
actorId	string	Apify actor ID
totalTests	number	Number of test cases in the suite
passed	number	Number of test cases that passed
failed	number	Number of test cases that failed
totalDuration	number	Total execution time in seconds for the entire suite
results	array	Per-case results (see below)
results[].name	string	Test case name
results[].passed	boolean	Whether the test case passed (run succeeded and all assertions passed)
results[].duration	number	Execution time for this test case in seconds
results[].resultCount	number	Number of items in the output dataset
results[].assertions	array	Assertion results: { assertion, passed, expected?, actual? }
results[].error	string	Error message if the run failed (optional)
testedAt	string	ISO 8601 timestamp

How much does it cost?

Actor Test Runner uses pay-per-event pricing at $0.35 per suite run -- one flat fee regardless of how many test cases. Target actor runs are billed separately.

Scenario	Suites	Orchestration Cost	Notes
One-off suite (5 cases)	1	$0.35	Target actor runs 5 times
Daily CI/CD (30/mo)	30	$10.50	Plus target actor costs
Nightly fleet test (10 actors)	10/night	$105.00/mo	300 suites/month

The Apify Free plan ($5/mo credits) covers approximately 14 suite runs (orchestration only).

Run test suites using the API

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("ryanclinton/actor-test-runner").call(run_input={
    "targetActorId": "ryanclinton/website-contact-scraper",
    "testCases": [
        {
            "name": "Basic scan",
            "input": {"urls": ["https://example.com"], "maxPagesPerDomain": 3},
            "assertions": {"minResults": 1, "requiredFields": ["url", "domain"]},
        },
    ],
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Suite: {item['passed']}/{item['totalTests']} passed in {item['totalDuration']}s")
    for tc in item["results"]:
        status = "PASS" if tc["passed"] else "FAIL"
        print(f"  [{status}] {tc['name']} ({tc['duration']}s, {tc['resultCount']} results)")
        for a in tc["assertions"]:
            if not a["passed"]:
                print(f"    FAILED: {a['assertion']} (expected {a.get('expected')}, got {a.get('actual')})")

JavaScript

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "YOUR_API_TOKEN" });

const run = await client.actor("ryanclinton/actor-test-runner").call({
    targetActorId: "ryanclinton/website-contact-scraper",
    testCases: [
        {
            name: "Basic scan",
            input: { urls: ["https://example.com"], maxPagesPerDomain: 3 },
            assertions: { minResults: 1, requiredFields: ["url", "domain"] },
        },
    ],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const report = items[0];
console.log(`Suite: ${report.passed}/${report.totalTests} passed`);
report.results.forEach(tc => {
    console.log(`  [${tc.passed ? "PASS" : "FAIL"}] ${tc.name}`);
});

cURL

curl -X POST "https://api.apify.com/v2/acts/ryanclinton~actor-test-runner/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "targetActorId": "ryanclinton/website-contact-scraper",
    "testCases": [{
      "name": "Basic scan",
      "input": {"urls": ["https://example.com"], "maxPagesPerDomain": 3},
      "assertions": {"minResults": 1, "requiredFields": ["url", "domain"]}
    }]
  }'

FAQ

How many test cases can I include in one suite? There is no hard limit, but each test case runs the target actor sequentially. A suite with 10 cases each taking 30 seconds will take about 5 minutes total. Keep suites focused and fast.

Why are test cases run sequentially, not in parallel? To avoid overwhelming the target actor or exhausting your compute budget unpredictably. Sequential execution gives consistent, reproducible results and predictable costs.

What happens if one test case crashes? The crashed case reports FAIL with the error message. Remaining test cases continue running normally. One bad case doesn't block the rest of the suite.

Can I reuse test cases across actors? Yes, if actors have similar input schemas. Copy the test cases array and change the targetActorId. Useful for comparing replacement actors.

What's the difference between Test Runner and Cloud Staging Test? Cloud Staging Test runs a single input with optional assertions. Test Runner runs multiple inputs with independent assertions per case. Use Cloud Staging Test for quick smoke tests, Test Runner for comprehensive multi-scenario suites.

Can I use this for load testing? No. Test Runner runs one case at a time. For load testing, use the Apify API to trigger multiple parallel runs directly.

Related actors

Actor	How to combine
Cloud Staging Test	Quick single-input validation. Use Test Runner for multi-case suites, Cloud Staging Test for fast one-off checks.
Schema Validator	Schema compliance checking. Run Schema Validator for type/field analysis, Test Runner for functional correctness.
Regression Suite	Historical comparison. Regression Suite extends Test Runner with diff logic to detect tests that were passing but now fail.
Actor Health Monitor	Runtime failure monitoring. Test Runner validates output quality; Health Monitor catches crashes and error rates.

Support

Found a bug or have a feature request? Open an issue in the Issues tab on this actor's page.