Concepts
Jobs, batches & webhooks
Every scrape is a job underneath. The synchronous endpoints are a convenience wrapper that waits for a fast one — so nothing here is bolted onto a request/response design that cannot take it.
When to use which
- Synchronous GET — you need one thing and you need it now.
- Batch job — you have a list. One call, up to 500 targets, results as they land.
- Async job — one piece of work that will take longer than a request should. A full-channel pull, a deep comment tree.
- Subscription — you would otherwise run the same call on a cron. Do this instead.
| POST/v1/jobs | One long-running call, queued. Returns 202 with a job id. |
| POST/v1/jobs/batch | Up to 500 targets against one endpoint, in a single request. |
| GET/v1/jobs/{id} | Status, per-target progress, and results as they land. |
| GET/v1/jobs | Recent jobs for your team. |
| POST/v1/subscriptions | Watch a target and take a webhook when it changes. |
| GET/v1/subscriptions | What you are currently watching. |
| DELETE/v1/subscriptions/{id} | Stop watching. |
Batches
A batch is one request that fans out across workers. There is no request timeout above it, so the size of the list is a throughput question rather than a design constraint.
curl -X POST "$API/v1/jobs/batch" \
-H "x-api-key: $KEY" -H "content-type: application/json" \
-d '{
"endpoint": "youtube.transcript",
"targets": ["8XkPqR2nLvE", "dQw4w9WgXcQ", "..."],
"webhook_url": "https://you.example/hooks/batch"
}'{
"success": true,
"data": {
"jobId": "job_01JB8Z3M6QW2K9",
"status": "queued",
"endpoint": "youtube.transcript",
"totalTargets": 412,
"estimatedCredits": 412,
"note": "Failed and empty targets are refunded automatically.",
"pollUrl": "https://api.example/v1/jobs/job_01JB8Z3M6QW2K9"
}
}estimatedCredits is the worst case. The number that gets charged is lower whenever a target fails, returns nothing, or is served from cache — each of those is refunded as the batch runs, not reconciled afterwards.
{
"success": true,
"data": {
"jobId": "job_01JB8Z3M6QW2K9",
"status": "running",
"progress": { "total": 412, "completed": 287, "failed": 9, "percent": 70 },
"creditsCharged": 278
}
}In that example 287 targets have completed and 278 credits have been charged. The gap is nine targets that produced nothing — refunded automatically, visible in the number.
Subscriptions
A subscription is a standing instruction to watch one target. We run the polls; you get called when the data moves. Polls we run on your behalf are free, so watching a thousand creators costs what the changes cost rather than what the checking costs.
curl -X POST "$API/v1/subscriptions" \
-H "x-api-key: $KEY" -H "content-type: application/json" \
-d '{
"endpoint": "youtube.channel",
"params": { "handle": "@mkbhd" },
"webhook_url": "https://you.example/hooks/social"
}'Delivery
- Every delivery is signed, so you can verify it came from us before acting on it.
- Failed deliveries retry with backoff.
- Deliveries that never succeed land in a dead-letter queue rather than disappearing.
- Your endpoint should return quickly and do the work afterwards.
Idempotency
Delivery is at-least-once, which means you will occasionally see the same event twice. Deduplicate on the event id rather than assuming exactly-once — every durable queue makes the same trade.