Polling Is a Tax: Change Detection for Platform Data

Here is a workload that exists in a few hundred companies right now, most of them in clipping, UGC payouts, or creator analytics.

You are tracking 10,000 pieces of content. View counts decide what someone gets paid, so you check hourly. That is 240,000 requests a day and 7.2 million a month. On any given hour, roughly 97% of those clips have not changed in a way you care about — the older ones are dead, the newer ones update in bursts, and the middle of the distribution is flat.

At a competitive rate of $0.99 per 1,000, that month costs $7,128. About $6,914 of it bought you nothing. You paid for 6,984,000 responses that were byte-for-byte equivalent to the ones you already had.

That is the tax. It is not a bug in anyone's system, it is what happens when the only verb available is fetch and the question you actually have is did this change.

---

The cost is not the requests. It is the requests that told you nothing

Three separate bills arrive, and only the first is obvious.

The money. Above. It scales linearly with your polling frequency and with your target count, and it is almost entirely waste by construction.

The rate-limit budget. 7.2 million requests a month against a handful of platforms is about 2.8 requests per second, sustained, forever. Every one of those is a chance to be blocked, and being blocked has a nasty property: the requests that get you blocked are overwhelmingly the ones that were going to return nothing anyway. You are spending your entire risk budget on non-events. Worse, when the platform does start pushing back, it will often do so by returning 200 and an empty payload rather than an error — so your polling loop keeps running, keeps costing, and quietly stops detecting anything.

The latency floor, which nobody prices at all. Your polling interval is your worst-case staleness. Hourly polling means a creator can post and you find out 59 minutes later. The only lever polling gives you is frequency, and frequency is the thing that costs money. To halve your detection latency you double your bill, and 97% of the new spend is on the same non-events. That is a bad trade offered as if it were a feature.

---

Three ways to detect change, ranked by what they cost you

1. Re-fetch and diff on your side

The default, because it requires nothing from anyone. Pull the payload, compare it to the last one you stored, act if it moved.

Everything above applies: you pay for every fetch, you carry all the rate-limit risk, and you store a full copy of every payload you have ever seen in order to compare against it. The one thing it has going for it is that it works everywhere and you control the comparison logic entirely.

If you are going to do this, the thing to get right is what you compare. Never diff raw response bytes — platform responses contain request IDs, server timestamps, cache-age counters and signed CDN URLs with expiry parameters, every one of which changes on every response and means nothing. Diff a normalised projection of the fields you actually care about, or you will detect a change every single time and conclude that change detection does not work.

2. Conditional requests, where they are supported

HTTP has had the answer to this since 1999. Send If-None-Match with the ETag you got last time, or If-Modified-Since with the timestamp, and a server that supports it replies 304 Not Modified with no body.

This is the right mechanism and you will mostly not get to use it. Static assets and well-behaved REST APIs honour it. The internal JSON endpoints that back social platform pages generally do not — they are not designed for third-party consumption, they are designed for a first-party client that does not conditionally re-request, and many of them regenerate a response per call with fresh volatile fields that would break the ETag anyway.

Test it before you assume either way. One curl -I against your actual target tells you whether an ETag comes back, and a second request with If-None-Match tells you whether it is honoured. When it works, it is the cheapest option on this page by a wide margin — you still pay for a request, but the response is a header and your bandwidth bill collapses. It is worth ten minutes to find out.

3. Server-side fingerprinting, with push delivery

Move the loop to whoever is already paying for the fetch. They poll on your behalf, hash a normalised form of each payload, and contact you only when the hash moves.

The economics change shape entirely: you are billed per change, not per check. Take the workload at the top of this post — 216,000 actual changes out of 7.2 million polls. At the same $0.99/1,000 that is $214 a month instead of $7,128, and your detection latency is now a scheduling parameter rather than a budget line.

That model only works if the fingerprint is trustworthy, which is the next section and the reason this is harder than it looks.

---

Push versus pull: webhooks, WebSockets, and why the answer is about frequency

The moment you stop polling, you have to decide how the news reaches you, and the two options get compared as if it were a matter of taste. It is not. It is a function of how often events happen.

Webhooks — the server POSTs to a URL you own. You need a publicly reachable endpoint, signature verification, and idempotent handling. Each event costs a full HTTP request-response. That overhead is irrelevant when events are rare and discrete, which is exactly the shape of "a creator posted something" or "this ad went live."

WebSockets or SSE — you hold a connection open and events stream down it. Per-event overhead approaches zero and latency approaches the network. That matters when events are frequent, ordered, and time-critical: order books, live chat, collaborative editing.

For platform-data change detection at hourly intervals, webhooks win and it is not close. Holding a socket open for 3,600 seconds to receive nothing is a connection you are maintaining, reconnecting, and monitoring for the privilege of avoiding an HTTP handshake you make once an hour. WebSockets earn their complexity above roughly one event per connection per second. Below that, they are a persistent-connection problem you have chosen to have.

The genuinely useful heuristic: if you would be comfortable with the event arriving as an email, use a webhook.

One increasingly common wrinkle: the thing choosing between these is often not a person. An agent asked to "keep an eye on" a set of creators will build a polling loop by default, because a loop is the obvious composition of the tools in front of it. Telling it otherwise is not a docs problem you solve on a marketing page — it belongs in the machine-readable surfaces the agent actually reads, stated as a negative instruction.

---

What "billed per change" has to guarantee before you trust it

A vendor who charges you when data changes has a correctness obligation that a vendor charging per request does not, and it runs in a direction people find counterintuitive.

The failure everyone designs against is a missed change — the follower count moved and the hash did not. That is bad, and it is the cheap failure. Nobody is billed. The feed under-reports and someone notices.

The expensive failure is a phantom change: the fingerprint moves when nothing meaningful did. Now the webhook fires, the charge lands, and it repeats every polling interval, for every subscription, forever. Every individual event looks exactly like a real one. That is a recurring invoice for a non-event, and it corrupts the customer's data and their bill at the same time.

So the property to ask about is not "does it detect every change." It is "does identical data always produce an identical fingerprint." Three questions get you most of the way:

1. Is key order normalised at every nesting depth? JSON objects are unordered by specification. JSON.stringify() on two semantically identical objects is not guaranteed to produce the same string, so hashing its output is a fingerprint that works most of the time — which is worse than one that fails loudly.
2. Are volatile fields stripped before hashing? This is the one that bites. Our own payloads carry a fetchedAt stamped fresh on every scrape; hashing the payload raw would have made every single poll look like a change and billed on every check. The fix is a separate content fingerprint that strips volatile fields first, and it is the kind of thing you only find because you went looking for it.
3. Is array order deliberately preserved? It should be. Feed order is meaningful — a reordered timeline is a real change. Sorting arrays to make hashing stable would suppress exactly the changes people subscribe for. Objects get normalised; arrays do not. Same data, opposite treatment, and the reasoning is worth reading in full if you are building one of these.

Two more things a change-billing model has to state in writing before it is safe to depend on:

Is the first check billed? It must not be. There is nothing to compare against on the first poll — it establishes the baseline. Any vendor billing you for it is billing you for creating the subscription.

What happens when your balance is empty at the moment a change happens? The wrong answer is to skip recording the new fingerprint, because then the same change is detected again on the next poll and you are billed twice for one event once you top up. The subscription should record the new state regardless.

---

The half nobody plans for: getting the event to you

Detection is the interesting problem. Delivery is the one that generates support tickets.

A push system owes you at least four things, and you should ask about each specifically rather than accepting "we have webhooks."

Signing. An HMAC over the payload plus a timestamp, sent in headers, so you can verify the request came from the vendor and is not a replay. Without it, your webhook endpoint is an unauthenticated write path into your system that anybody who learns the URL can drive.

Retries with backoff. Your endpoint will be down at some point. A single delivery attempt turns your five minutes of downtime into permanently lost events.

Idempotency. Retries mean duplicates. Every event needs a stable identifier so you can safely process it twice.

A dead-letter path. After the retries are exhausted, the event has to go somewhere you can inspect and replay. If it only goes to the vendor's logs, then from your side it did not happen.

Here is where we are on that, honestly. We sign payloads with an HMAC over the timestamp and body, and we retry. Our behavioural spec explicitly puts webhook delivery and retry semantics out of scope — they are not specified, which means they are not contractual and they can change. And there is no dead-letter queue: when delivery ultimately fails, the failure is logged and that is the end of it. Recovering a missed event today means noticing the gap yourself and re-querying.

That is a gap, it is the exact gap this section says to ask vendors about, and we would rather write it down than let you discover it during an incident. If change detection is load-bearing for you, ask every vendor on your list the four questions above and make them answer in writing.

While you are at it, ask what the loop is tested to do. Ours has both halves under test — a scheduled poll charges nothing, and fingerprints are stable across key reordering — but the integration between them, the poll → diff → charge path end to end, is not yet covered. Our own spec names that as a known gap. "Billed only on change" is a claim that should come with a test you can point at, from us or anyone else.

---

One thing to notice about "watching is free"

Somebody still pays for the upstream request.

When a vendor tells you unchanged polls cost nothing, that is a statement about your invoice, not about physics. The fetch happened. Bandwidth was spent, a proxy was used if the platform needed one, and a worker was occupied. The vendor is absorbing that cost because it is far smaller for them than for you — they can batch, cache across tenants, and amortise a single fetch of a popular creator across every customer watching them — but it is not zero, and a model that gives it away has to be underwritten by something.

Which is worth knowing for two reasons. It tells you the interval you are offered is a business decision as much as a technical one, and it tells you that if you push a change-billing vendor to poll thousands of unique, unpopular targets every sixty seconds, you are on the wrong side of the economics that make the model work, and the pricing will eventually reflect that. Which pricing model fits which workload is a longer argument, and change billing is genuinely the wrong shape for several common ones.

---

When polling is genuinely correct

This post is an argument against a default, not against a technique. Poll when:

• The data changes on every check anyway. A view counter on an actively watched video moves every minute. Change detection has nothing to suppress and adds a hash for no benefit.
• You need the data at a fixed time regardless of change. A nightly rollup that runs whether or not the inputs moved does not care about your notification; it needs the current value at 02:00.
• You cannot accept an inbound connection. No public endpoint, no tunnel, hard network policy. Pull is the only option and that is a legitimate constraint, not a failure of imagination.
• The target set is small. Fifty targets checked hourly is 36,000 requests a month. That is a rounding error and the operational complexity of subscriptions, signatures and replay is not worth it. The tax only bites at scale.
• You are doing one-shot or backfill work. There is no "change" in a backfill. Use batch requests — and if your provider makes you loop because their synchronous endpoints have a hard timeout ceiling, that constraint is the actual problem and it is worth understanding before you architect around it.

The rule underneath all of those: poll when you need a value, subscribe when you need an event. Most teams paying this tax are polling for values they already have, in order to discover events, and the two are not the same operation.

---

For completeness, since it is the thing we build: our subscriptions run a poll with charging disabled, fingerprint the result with volatile fields stripped, and charge only when the hash moves. First poll is a free baseline. A run of failures backs off rather than hammering a dead target, and twenty consecutive failures deactivate the subscription instead of polling something that no longer exists forever. The gaps in the delivery half are the ones described above, unedited.

---

Sources. The billing and change-detection behaviour described here is specified in SPEC.md §6, including the known end-to-end test gap named above; the out-of-scope status of webhook delivery and retry semantics is SPEC.md §7. The $0.99/1,000 rate used in the arithmetic is ScrapeCreators' published Business-tier price, recorded in research/03-pricing-and-unit-economics.md, 2026-08-20.