Jobs, batches & webhooks

When to use which

• Synchronous GET: you need one thing and you need it now.
• Batch job: you have a list. One call, up to 500 targets, results as they land.
• Async job: one piece of work that will take longer than a request should. A full-channel pull, a deep comment tree.
• Subscription: you would otherwise run the same call on a cron. Do this instead.

Batches

A batch is one request that fans out across workers. There is no request timeout above it, so the size of the list is a throughput question rather than a design constraint.

Each entry in targets is an object of parameters, not a bare string. It is merged over the shared params and validated against the endpoint’s own schema at creation, so a batch that is half-invalid is refused now rather than refunding half of itself later.

estimatedCredits is the worst case. The number that gets charged is lower whenever a target fails, returns nothing, or is served from cache. Each of those is refunded as the batch runs, not reconciled afterwards. The same target repeated inside one batch is fetched once and charged once.

In that example 287 targets have completed and 278 credits have been charged: nine of those completions came back empty and were refunded. The nine counted under failed are separate again, and cost nothing either. completed and failed never overlap, so a batch is done when they add up to total.

Subscriptions

A subscription is a standing instruction to watch one target. We run the checks; you get called when the data moves. Each check is a normal request at the endpoint’s price, and a check that fails or comes back empty is refunded like any other call. You pay for the checks, and you hear from us only when something changed. Creating, listing and deleting a watch cost nothing.

interval_seconds runs from the endpoint’s minimum up to 604800, seven days. The minimum is set per endpoint: every operation in openapi.json publishes it as x-min-interval-seconds, and a shorter interval is refused with invalid_request naming that minimum. Endpoints that re-rank the same query on every request (search, trending, popular) are refused here rather than sold: every check would register as a change and call your webhook for nothing having happened.

Leave interval_seconds out and the watch uses the endpoint’s suggested interval, published as x-suggested-interval-seconds. It follows how often that kind of data moves: prices, stock and news hourly; new posts and comments every six hours; profiles and companies daily; music catalogues and transcripts weekly. It is a suggestion, not a limit, and any interval in range is accepted. It matters because every check is billed, so checking data more often than it moves costs money for nothing.

A team has a limit on how often its watches check in total, per hour. It is weighted by interval rather than counting watches, so an hourly watch uses more of it than a daily one. A create past that limit is refused, and the error states the limit. Paused watches count toward the limit, since they resume on their own; watches stopped after repeated failures do not.

Delivery

• Every delivery carries x-webhook-signature and x-webhook-timestamp, plus x-correlation-id, the id of the watch that fired.
• Failed deliveries retry with backoff.
• A delivery that never succeeds is given up on and recorded on our side. There is no queue you can drain later, so treat a webhook as a notification and not as the record; the job or watch can always be read back over the API.
• Your endpoint should return quickly and do the work afterwards.

Idempotency

Delivery is at-least-once, so you will occasionally see the same change twice. There is no per-event id to deduplicate on. x-correlation-id identifies the watch, not the delivery, and is the same on every one it sends. Deduplicate on something from the payload instead: for a subscription.changed, the data it carries is the changed object, so its own id and timestamp are the stable pair to key against.

When your balance cannot cover a check

A check is charged before it fetches. If your balance cannot cover the endpoint’s price, the check is refused: nothing is fetched, nothing is charged, and no webhook is sent for it.

After the second refused check the watch pauses, and your webhook receives subscription.paused with reason: "unpaid". A paused watch keeps its schedule. Top up and it resumes on its own at its next scheduled check once the balance covers the price. That is not the moment the payment lands, and there is nothing to create again.

GET /v1/subscriptions tells the two kinds of stopped watch apart. pausedReason: "unpaid" is paused for credit and resumes after a top-up. active: false with pausedReason: null was stopped after repeated failed checks and does not resume.

A daily credit cap is not a pause. Checks past your team’s cap are skipped until the next day, and the watch carries on from there.

Doing it without writing any code

Everything above has a form behind it. Jobs & watches in the dashboard runs the same handlers as the routes in the table: one code path, so billing cannot differ between what you click and what you curl.

Run a job

Create a watch

A job you started in the dashboard and a job you started with a key are the same job, on the same team, spending the same balance. The dashboard is a second door, not a second system.