Polling Is a Tax: Change Detection for Platform Data

Here is a workload that exists in a few hundred companies right now, most of them in clipping, UGC payouts, or creator analytics.

You are tracking 10,000 pieces of content. View counts decide what someone gets paid, so you check hourly. That is 240,000 requests a day and 7.2 million a month. On any given hour, roughly 97% of those clips have not changed in a way you care about: the older ones are dead, the newer ones update in bursts, and the middle of the distribution is flat.

At a competitive rate of $0.99 per 1,000, that month costs $7,128. About $6,914 of it bought you nothing. You paid for 6,984,000 responses that were byte-for-byte equivalent to the ones you already had.

That is the tax. It is not a bug in anyone's system, it is what happens when the only verb available is fetch and the question you actually have is did this change.

---

The cost is not the requests. It is the requests that told you nothing

Three separate bills arrive, and only the first is obvious.

The money. Above. It scales linearly with your polling frequency and with your target count, and it is almost entirely waste by construction.

The rate-limit budget. 7.2 million requests a month against a handful of platforms is about 2.8 requests per second, sustained, forever. Every one of those is a chance to be blocked, and being blocked has a nasty property: the requests that get you blocked are overwhelmingly the ones that were going to return nothing anyway. You are spending your entire risk budget on non-events. Worse, when the platform does start pushing back, it will often do so by returning 200 and an empty payload rather than an error, so your polling loop keeps running, keeps costing, and quietly stops detecting anything.

The latency floor, which nobody prices at all. Your polling interval is your worst-case staleness. Hourly polling means a creator can post and you find out 59 minutes later. The only lever polling gives you is frequency, and frequency is the thing that costs money. To halve your detection latency you double your bill, and 97% of the new spend is on the same non-events. That is a bad trade offered as if it were a feature.

---

Three ways to detect change, ranked by what they cost you

1. Re-fetch and diff on your side

The default, because it requires nothing from anyone. Pull the payload, compare it to the last one you stored, act if it moved.

Everything above applies: you pay for every fetch, you carry all the rate-limit risk, and you store a full copy of every payload you have ever seen in order to compare against it. The one thing it has going for it is that it works everywhere and you control the comparison logic entirely.

If you are going to do this, the thing to get right is what you compare. Never diff raw response bytes. Platform responses contain request IDs, server timestamps, cache-age counters and signed CDN URLs with expiry parameters, every one of which changes on every response and means nothing. Diff a normalised projection of the fields you actually care about, or you will detect a change every single time and conclude that change detection does not work.

2. Conditional requests, where they are supported

HTTP has had the answer to this since 1999. Send If-None-Match with the ETag you got last time, or If-Modified-Since with the timestamp, and a server that supports it replies 304 Not Modified with no body.

This is the right mechanism and you will mostly not get to use it. Static assets and well-behaved REST APIs honour it. The internal JSON endpoints that back social platform pages generally do not: they are not designed for third-party consumption, they are designed for a first-party client that does not conditionally re-request, and many of them regenerate a response per call with fresh volatile fields that would break the ETag anyway.

Test it before you assume either way. One curl -I against your actual target tells you whether an ETag comes back, and a second request with If-None-Match tells you whether it is honoured. When it works, it is the cheapest option on this page by a wide margin. You still pay for a request, but the response is a header and your bandwidth bill collapses. It is worth ten minutes to find out.

3. Server-side fingerprinting, with push delivery

Move the loop to whoever is already making the fetch. They check on your schedule, hash a normalised form of each payload, and contact you only when the hash moves.

What this changes is everything around the fetch. You stop storing a copy of every payload you have ever seen, stop carrying the rate-limit risk, and stop writing the comparison logic that option 1 makes so easy to get wrong. Your detection latency becomes a scheduling parameter.

What it does not change on its own is the fetch count, and so it may not change the bill. Some vendors charge only when the hash moves and absorb the unchanged checks; others, us included, charge every check like any other request, for reasons the section on "watching is free" further down sets out. On per-check pricing, the workload at the top of this post still costs what 7.2 million requests cost.

That model only works if the fingerprint is trustworthy, which is the next section and the reason this is harder than it looks.

The lever under all three: check as often as the data moves

Whichever option you pick, the interval does more to your bill than the mechanism does. The 97% at the top of this post is mostly clips being checked far faster than they change.

Say 8,000 of those 10,000 clips are more than a week old and move a few times a day at most. Check those daily and the newest 2,000 hourly: 56,000 checks a day, 1.68 million a month, $1,663 instead of $7,128. Nothing in the arithmetic depends on who runs the loop. It depends on refusing to ask a question faster than the answer can change.

---

Push versus pull: webhooks, WebSockets, and why the answer is about frequency

The moment you stop polling, you have to decide how the news reaches you, and the two options get compared as if it were a matter of taste. It is not. It is a function of how often events happen.

Webhooks. The server POSTs to a URL you own. You need a publicly reachable endpoint, signature verification, and idempotent handling. Each event costs a full HTTP request-response. That overhead is irrelevant when events are rare and discrete, which is exactly the shape of "a creator posted something" or "this ad went live."

WebSockets or SSE. You hold a connection open and events stream down it. Per-event overhead approaches zero and latency approaches the network. That matters when events are frequent, ordered, and time-critical: order books, live chat, collaborative editing.

For platform-data change detection at hourly intervals, webhooks win and it is not close. Holding a socket open for 3,600 seconds to receive nothing is a connection you are maintaining, reconnecting, and monitoring for the privilege of avoiding an HTTP handshake you make once an hour. WebSockets earn their complexity above roughly one event per connection per second. Below that, they are a persistent-connection problem you have chosen to have.

The genuinely useful heuristic: if you would be comfortable with the event arriving as an email, use a webhook.

One increasingly common wrinkle: the thing choosing between these is often not a person. An agent asked to "keep an eye on" a set of creators will build a polling loop by default, because a loop is the obvious composition of the tools in front of it. Telling it otherwise is not a docs problem you solve on a marketing page; it belongs in the machine-readable surfaces the agent actually reads, stated as a negative instruction.

---

What change detection has to guarantee before you trust it

A vendor who tells you when data changes has a correctness obligation that a vendor answering single requests does not, and it runs in a direction people find counterintuitive.

The failure everyone designs against is a missed change: the follower count moved and the hash did not. That is bad, and it is the cheap failure. The feed under-reports and someone notices.

The expensive failure is a phantom change: the fingerprint moves when nothing meaningful did. Now the webhook fires, your pipeline acts on it, and it repeats every interval, for every watch, forever. Every individual event looks exactly like a real one. It corrupts the customer's data, and under per-change billing it bills them for a non-event on every check as well.

So the property to ask about is not "does it detect every change." It is "does identical data always produce an identical fingerprint." Three questions get you most of the way:

1. Is key order normalised at every nesting depth? JSON objects are unordered by specification. JSON.stringify() on two semantically identical objects is not guaranteed to produce the same string, so hashing its output is a fingerprint that works most of the time, which is worse than one that fails loudly.
2. Are volatile fields stripped before hashing? This is the one that bites. A payload that carries a fetchedAt stamped fresh on every scrape will, hashed raw, make every single check look like a change and fire on every one. The fix is a separate content fingerprint that strips volatile fields first, and it is the kind of thing you only find because you went looking for it.
3. Is array order deliberately preserved? It should be. Feed order is meaningful. A reordered timeline is a real change. Sorting arrays to make hashing stable would suppress exactly the changes people subscribe for. Objects get normalised; arrays do not. Same data, opposite treatment, because one reordering is noise and the other is the event.

Two more things any watch has to state in writing before it is safe to depend on:

What exactly is billed? Per change or per check, and what happens to a check that fails or comes back empty. Either model can be fair. A vendor that will not answer plainly is the one to worry about, because the bill is where a vague answer shows up.

What happens when your balance runs out? The wrong answers are a silent stop, which you discover weeks later as a gap in your data, and carrying on for free, which cannot last and will be taken away without warning. The right one is a pause you are told about, followed by a watch that picks back up by itself once you top up.

---

The half nobody plans for: getting the event to you

There is a specific hole in the obvious version of detection: an empty response is a success, so it hashes like content, and reports a change once on the way down and again on recovery.

Detection is the interesting problem. Delivery is the one that generates support tickets.

A push system owes you at least four things, and you should ask about each specifically rather than accepting "we have webhooks."

Signing. An HMAC over the payload plus a timestamp, sent in headers, so you can verify the request came from the vendor and is not a replay. Without it, your webhook endpoint is an unauthenticated write path into your system that anybody who learns the URL can drive.

Retries with backoff. Your endpoint will be down at some point. A single delivery attempt turns your five minutes of downtime into permanently lost events.

Idempotency. Retries mean duplicates. Every event needs a stable identifier so you can safely process it twice.

A dead-letter path. After the retries are exhausted, the event has to go somewhere you can inspect and replay. If it only goes to the vendor's logs, then from your side it did not happen.

If change detection is load-bearing for you, ask every vendor on your list the four questions above and make them answer in writing.

While you are at it, ask what the loop is tested to do. Ours went a long time with both halves under test (the scheduled check on one side, fingerprint stability across key reordering on the other) and nothing covering the integration between them. Closing that gap found two real billing bugs immediately: a transient empty result that billed once on the way down and again on recovery, and two overlapping sweeps each charging for the same work. The loop is now driven end to end by its own tests. Whatever a vendor's billing model is, it should come with a test you can ask about, from us or anyone else.

---

One thing to notice about "watching is free"

Somebody still pays for the upstream request.

When a vendor tells you unchanged checks cost nothing, that is a statement about your invoice, not about physics. The fetch happened. Bandwidth was spent and a worker was occupied. A vendor can absorb that for a while (batching, caching across tenants, amortising one fetch of a popular creator across everyone watching them), but it is not zero, and a model that gives it away has to be underwritten by something. Push it to check thousands of unique, quiet targets and you are on the wrong side of the economics that make it work.

When this post was first written, our watches gave it away. They no longer do. Every check is now billed like any other request, refunded if it fails or comes back empty, and the webhook still fires only when the data actually changed. It is the less flattering price and the more honest one: what a watch costs is now visible to you, and set by the one thing you control, the interval, rather than hidden in a margin that was always going to be repriced. To help with that, every endpoint suggests an interval from how fast its kind of data moves (hourly for prices, daily for profiles, weekly for transcripts), and you can override it.

A watch that runs out of credit pauses after two refused checks, tells you so, and resumes by itself once you top up. Which pricing model fits which workload is a longer argument, and per-change billing is still the right shape for some of them, from a vendor who can sustain it.

---

When polling is genuinely correct

This post is an argument against a default, not against a technique. Poll when:

• The data changes on every check anyway. A view counter on an actively watched video moves every minute. Change detection has nothing to suppress and adds a hash for no benefit.
• You need the data at a fixed time regardless of change. A nightly rollup that runs whether or not the inputs moved does not care about your notification; it needs the current value at 02:00.
• You cannot accept an inbound connection. No public endpoint, no tunnel, hard network policy. Pull is the only option and that is a legitimate constraint, not a failure of imagination.
• The target set is small. Fifty targets checked hourly is 36,000 requests a month. That is a rounding error and the operational complexity of subscriptions, signatures and replay is not worth it. The tax only bites at scale.
• You are doing one-shot or backfill work. There is no "change" in a backfill. Use batch requests. If your provider makes you loop because their synchronous endpoints have a hard timeout ceiling, that constraint is the actual problem and it is worth understanding before you architect around it.

The rule underneath all of those: poll when you need a value, subscribe when you need an event. Most teams paying this tax are polling for values they already have, in order to discover events, and the two are not the same operation.

---

Sources. The $0.99/1,000 rate used in the arithmetic is ScrapeCreators' published Business-tier price, read 2026-08-20.