A/B Tester

Run two Apify actors with the exact same input in parallel and get a structured comparison report covering result count, field coverage, execution speed, and compute cost. Actor A/B Tester tells you which actor delivers more data, faster, and cheaper -- so you can make informed decisions about which scraper, enrichment tool, or data pipeline to use in production.

Features

• Runs two actors simultaneously with identical input for a fair comparison
• Compares result count, execution duration, compute cost, and output field coverage
• Identifies shared fields and unique fields between both actors' output schemas
• Declares a winner based on a weighted scoring system (results, speed, cost, fields, status)
• Returns a sample record from each actor so you can visually inspect data quality
• Works with any Apify actor -- scrapers, enrichment tools, API wrappers, or custom actors

Use Cases

• Scraper evaluation: Compare two competing scrapers (e.g., Cheerio vs. Playwright-based) on the same target URL to see which extracts more data fields faster.
• Cost optimization: Test whether a cheaper actor produces comparable results to a premium one before committing to a production pipeline.
• Migration validation: When replacing one actor with another, verify the new actor returns equivalent or better results before switching.
• Actor development: Test your custom actor against an existing Store actor to benchmark performance during development.
• Vendor selection: Compare third-party actors from different developers on the Apify Store to pick the best one for your use case.

How to Use

1. Go to Actor A/B Tester on the Apify Store.
2. Click Try for free.
3. Enter your Apify API Token.
4. Enter the two actor IDs or names (e.g., apify/web-scraper and apify/cheerio-scraper).
5. Provide a Test Input JSON that both actors accept.
6. Optionally adjust timeout and memory settings.
7. Click Start and wait for both actors to finish.
8. View the comparison report in the Dataset tab.

Input Parameters

Output Example

{
    "actorA": {
        "name": "apify/web-scraper",
        "status": "SUCCEEDED",
        "results": 15,
        "duration": 8.2,
        "cost": 0.003,
        "fields": ["url", "title", "email"],
        "sampleRecord": {
            "url": "https://example.com/page1",
            "title": "Example Page",
            "email": "[email protected]"
        }
    },
    "actorB": {
        "name": "apify/cheerio-scraper",
        "status": "SUCCEEDED",
        "results": 12,
        "duration": 12.5,
        "cost": 0.005,
        "fields": ["url", "name", "contact_email", "phone"],
        "sampleRecord": {
            "url": "https://example.com/page1",
            "name": "Example Company",
            "contact_email": "[email protected]",
            "phone": "+1-555-0100"
        }
    },
    "comparison": {
        "winner": "actorA",
        "reasons": [
            "25% more results (A: 15, B: 12)",
            "34% faster (A: 8.2s, B: 12.5s)",
            "40% cheaper (A: $0.003, B: $0.005)"
        ],
        "sharedFields": ["url"],
        "uniqueToA": ["title", "email"],
        "uniqueToB": ["name", "contact_email", "phone"],
        "resultCountDiff": "+25%",
        "costDiff": "-40%",
        "speedDiff": "-34%"
    },
    "testedAt": "2026-03-18T10:30:00.000Z"
}

Output Fields

How It Works

1. Parallel execution. Both actors are started simultaneously via the Apify API with identical input, timeout, and memory settings. This ensures a fair comparison under the same conditions.

2. Polling. If either actor is still running after the initial request, the tester polls every 3 seconds until the actor completes or the timeout expires.

3. Data collection. For each actor, the tester fetches up to 1,000 items from the output dataset, extracts all unique field names, and records the first item as a sample record.

4. Cost retrieval. The total USD cost is fetched from the Apify run details API, which includes compute and platform usage.

5. Comparison scoring. The winner is determined by a point system:

   • Results count: 2 points to the actor with more results
   • Speed: 1 point to the faster actor
   • Cost: 1 point to the cheaper actor
   • Field coverage: 1 point to the actor with more output fields
   • Status: 3 points if one actor succeeded and the other did not

6. Report generation. The comparison includes percentage differences, field overlap analysis, and human-readable reasons for the winner.

How Much Does It Cost?

Actor A/B Tester uses Pay-Per-Event pricing at $0.15 per A/B test. This covers the orchestration and comparison logic. The two sub-actor runs are billed separately at their own rates.

The Apify Free plan includes $5 of monthly credits, enough for approximately 30 A/B tests (orchestration only, excluding sub-actor compute costs).

Tips

• Use the same memory and timeout for fair comparisons. Different memory allocations can significantly affect both speed and cost. The A/B tester applies the same settings to both actors by design.
• Start with small inputs. Use a limited test input (e.g., 1-2 URLs) to keep sub-actor costs low during initial evaluation.
• Check the sampleRecord. Raw result counts do not tell the full story. The sample record lets you compare actual data quality -- field richness, data completeness, and value formats.
• Compare field overlap. Two actors might return different field names for the same data (e.g., email vs contact_email). The shared/unique field analysis helps you spot this.
• Run multiple tests. A single run can be noisy due to network conditions or API rate limits. Run 3-5 tests for reliable benchmarks.

Frequently Asked Questions

Q: Can I compare actors with different input schemas? A: The test input must be valid for both actors. If Actor A expects startUrls and Actor B expects urls, you will need to structure the input to satisfy both schemas, or test with an input format both accept.

Q: What happens if one actor fails? A: The tester still produces a full report. The failed actor will show status FAILED (or ERROR/TIMED-OUT), zero results, and the successful actor will receive 3 bonus winner points for completing successfully.

Q: Does the $0.15 fee include the sub-actor run costs? A: No. The $0.15 covers the A/B Tester orchestration. Each sub-actor run is billed separately based on that actor's pricing model (compute time, PPE events, etc.).

Q: Can I compare more than two actors at once? A: Not in a single run. Run multiple A/B tests to create a tournament bracket -- compare pairs and then compare the winners.

Q: Is it legal to compare actors from other developers? A: Yes. You are running actors through the standard Apify API using your own token and credits. This is no different from running any public actor on the Apify Store.

Q: Why is the winner sometimes "tie"? A: A tie occurs when both actors score equally across all comparison dimensions. This can happen when both produce the same number of results with similar speed and cost.

Integrations

Use this actor with:

• Zapier for automated weekly benchmarking
• Make for complex evaluation pipelines
• Google Sheets for tracking comparison results over time
• The Apify API for programmatic access

Programmatic Access

Python

from apify_client import ApifyClient

client = ApifyClient("apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx")

run = client.actor("ryanclinton/actor-ab-tester").call(
    run_input={
        "apiToken": "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "actorA": "apify/web-scraper",
        "actorB": "apify/cheerio-scraper",
        "testInput": {"startUrls": [{"url": "https://example.com"}]},
        "timeout": 120,
        "memory": 512
    }
)

dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items
result = dataset_items[0]

print(f"Winner: {result['comparison']['winner']}")
for reason in result["comparison"]["reasons"]:
    print(f"  {reason}")

JavaScript

import { ApifyClient } from "apify-client";

const client = new ApifyClient({
    token: "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
});

const run = await client.actor("ryanclinton/actor-ab-tester").call({
    apiToken: "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    actorA: "apify/web-scraper",
    actorB: "apify/cheerio-scraper",
    testInput: { startUrls: [{ url: "https://example.com" }] },
    timeout: 120,
    memory: 512,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const result = items[0];

console.log(`Winner: ${result.comparison.winner}`);
result.comparison.reasons.forEach((r) => console.log(`  ${r}`));

cURL

curl -X POST "https://api.apify.com/v2/acts/ryanclinton~actor-ab-tester/runs?token=YOUR_API_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "apiToken": "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "actorA": "apify/web-scraper",
        "actorB": "apify/cheerio-scraper",
        "testInput": {"startUrls": [{"url": "https://example.com"}]},
        "timeout": 120,
        "memory": 512
    }'

Related Actors

• ryanclinton/apifyforge-quality-monitor -- Score actor quality across five dimensions
• ryanclinton/actor-competitor-scanner -- Find competing actors in the Apify Store
• ryanclinton/actor-revenue-analytics -- Track actor revenue and usage metrics
• ryanclinton/cost-watchdog -- Monitor spending and detect cost anomalies