Cut your token costs by 40-70%
ApifyForge LLM Output Optimizer is a data optimization tool that analyzes your Apify actor output and identifies low-value fields that waste LLM tokens: raw HTML, internal IDs, timestamps, and debug data. Scores every field by information density, recommends what to drop, and shows exactly how many tokens you'll save — typically 40-70% reduction at $0.20 per analysis.
Actor output is designed for data storage, not LLM consumption. Fields like rawHtml (2,000+ characters), internalId, and crawledAt consume tokens without adding information value for AI pipeline tasks like summarization, extraction, or classification. ApifyForge LLM Output Optimizer identifies these fields so you can filter them before the LLM API call.
Calculates token cost for every field across your sample output using character-based approximation (~4 chars/token). Shows which fields consume the most tokens.
Scores each field as high-value (name, url, email), medium-value (generic text), or low-value (IDs, timestamps, raw HTML). Based on information density for LLM tasks.
Fields with >80% null values are flagged for removal — they consume tokens on empty data in most items while providing value in a minority of cases.
Fields averaging >500 characters (raw HTML, page content, base64 images) are flagged for truncation or removal. A single rawHtml field can consume 500+ tokens per item.
Generates a recommended field list that keeps high-value data and drops the rest. Use directly in your pipeline to filter output before sending to any LLM API.
Original vs optimized token count with percentage savings. See the exact impact before making changes — typical savings are 40-70% of total token consumption.
There are several ways to reduce token consumption when feeding web scraping data to LLMs. Each trades off effort, precision, and ongoing maintenance.
| Method | Typical savings | Setup time | Cost |
|---|---|---|---|
| ApifyForge LLM Output Optimizer | 40-70% with field-level analysis | Under 30 seconds | $0.20/analysis |
| Manual field review | 20-50% (varies by developer knowledge) | 30-60 minutes per actor | Free (time cost) |
| Custom preprocessing script | Variable — depends on implementation | 1-3 hours per actor | Free (development time) |
| No optimization (raw output to LLM) | 0% — full token waste | Zero | 2-3x higher LLM API costs |
{
"actorName": "ryanclinton/website-contact-scraper",
"originalTokens": 4200,
"optimizedTokens": 1680,
"savingsPercent": 60,
"fieldAnalysis": [
{ "field": "rawHtml", "tokens": 2800, "value": "low", "action": "drop" },
{ "field": "url", "tokens": 45, "value": "high", "action": "keep" },
{ "field": "emails", "tokens": 120, "value": "high", "action": "keep" }
],
"optimizedSchema": ["url", "domain", "emails", "phones"],
"recommendations": ["Drop 3 low-value fields — saves 60%"]
}Connect your Apify token and enter the actor ID to analyze
ApifyForge LLM Output Optimizer reads output from a recent run and scores every field by information density
Get an optimized field list with exact token savings — ready to apply to your LLM pipeline
Several approaches exist for reducing token consumption in LLM pipelines fed by web scraping data. The right choice depends on how many actors you optimize and your team's LLM expertise.
Examine actor output JSON, identify large or irrelevant fields, and manually create a filter list. Requires understanding both the data and LLM tokenization. Takes 30-60 minutes per actor and produces inconsistent results across team members.
Best for: developers who deeply understand their data and LLM requirements.
Write a Node.js or Python script that filters, truncates, or transforms actor output before sending to the LLM API. Fully customizable but requires 1-3 hours per actor and ongoing maintenance as actor schemas change.
Best for: teams with dedicated data engineering resources and stable actor schemas.
Tools like LangChain's text splitters or LlamaIndex's node parsers manage context windows but don't optimize the source data. They chunk and paginate large inputs rather than removing low-value fields.
Best for: handling large text documents, not structured JSON from web scrapers.
Feed the full actor output to the LLM. Simple but wastes 40-70% of tokens on low-value data. At GPT-4 prices ($0.03/1K input tokens), this costs $1.26 extra per 1,000 items with 4,200 tokens each.
Best for: prototyping only — not sustainable for production pipelines.
Automated field-level analysis with value classification, null ratio detection, and exact token savings calculation. Generates an optimized field list in under 30 seconds. $0.20 per analysis, model-agnostic.
Best for: developers building LLM pipelines with Apify data who want fast, data-driven optimization.
Every optimization analysis executes on your own Apify account at the standard pay-per-event rate of $0.20 per analysis. ApifyForge has no platform fee or subscription. The $0.20 analysis pays for itself after a single LLM API call with the optimized output.
ApifyForge LLM Output Optimizer analyzes every field in your Apify actor's output and scores it by information density for LLM consumption. It classifies fields as high-value (names, URLs, emails), medium-value (generic text), or low-value (raw HTML, internal IDs, timestamps, debug data). Then it recommends which fields to keep and which to drop, with exact token savings calculations. Typical savings are 40-70% of token consumption.
Each ApifyForge LLM Output Optimizer run costs $0.20, charged as a pay-per-event (PPE) fee on your own Apify account. The tool reads actor output data from a recent run — it does not trigger new actor runs. The optimization report pays for itself if your LLM API costs exceed $0.50/month, since a 40-70% token reduction compounds across every subsequent LLM call.
ApifyForge LLM Output Optimizer uses character-based approximation at approximately 4 characters per token, which aligns with tokenizer behavior for GPT-3.5/GPT-4 and Claude on English text. While exact token counts vary by model and content, the approximation is accurate within 10-15% for typical web scraping output. The relative savings percentage (original vs optimized) is consistent regardless of the exact tokenizer used.
ApifyForge LLM Output Optimizer flags fields as low-value based on several signals: raw HTML content (>500 characters average), internal system IDs, timestamps that don't add semantic meaning, debug/meta fields, fields with >80% null values, and fields that repeat identical content across items. These fields consume tokens without adding information value for LLM processing tasks like summarization, extraction, or classification.
Yes. ApifyForge LLM Output Optimizer outputs an optimizedSchema array listing only the fields recommended for LLM consumption. You can use this list directly in your pipeline to filter actor output before sending it to the LLM API. The optimized schema is a subset of the original fields — no data transformation required, just field filtering.
Yes. ApifyForge LLM Output Optimizer reduces the data volume before it reaches any LLM API. The token savings apply equally to OpenAI (GPT-4, GPT-3.5), Anthropic (Claude), Google (Gemini), and any other LLM that charges per token. The optimization is model-agnostic — it reduces input data, not model-specific tokens.
ApifyForge LLM Output Optimizer provides recommendations, not requirements. If your use case genuinely needs raw HTML or timestamp fields, keep them. The value classification helps you make informed decisions: you might keep a field classified as low-value if it's critical for your specific LLM task. The savings calculation shows the impact of each field so you can decide individually.
ApifyForge LLM Output Optimizer checks the percentage of items where each field is null, empty string, or undefined. Fields with >80% null values are flagged for removal because they consume tokens on empty data across most items while providing value in only a minority of cases. For example, a 'faxNumber' field that is null in 95% of items wastes tokens on 95 null representations for every 5 actual values.