The Phantom Change Problem: Why JSON Key Ordering Became a Billing Guarantee

There is a class of bug that only exists once you attach money to a diff.

If you write a change-detection system for a dashboard, and it occasionally reports a change that did not happen, someone refreshes the page and shrugs. If you write the same system and bill on every change it reports, the same bug is an invoice for work that produced no information. The code is identical. The blast radius is not.

We build one of these — scheduled polls that cost nothing and charge only when the payload changes — so this is a writeup of the part of it that turned out to matter far more than expected: what a fingerprint has to guarantee, and the two cases where the obvious implementation is wrong in opposite directions.

No product pitch in this one. It is thirty lines of code and one uncomfortable finding about our own.

---

Wrong in the safe direction, wrong in the expensive direction

A change detector has two ways to be wrong.

It can miss a change. The follower count went from 100 to 101 and your fingerprint did not move. The customer's webhook does not fire. They find out later, from somewhere else. This is bad, it is the failure everyone designs against, and — critically — it is the cheap one. Nobody is billed. The system under-reports.

It can invent a change. Nothing about the data is different in any way a human would recognise, and your fingerprint moved anyway. The webhook fires. The charge lands. Repeat every polling interval, for every subscription, forever.

The second one is worse, and not by a little. A missed change is a gap in a feed. A phantom change is a recurring charge for a non-event, delivered with a webhook that tells the customer something happened when nothing did. It corrupts their data and their bill, and it does it silently — every individual event looks exactly like a real one.

So the primary correctness property is not "detects all changes." It is "identical data always produces an identical fingerprint." Under-reporting is a bug. Over-reporting is a refund and an apology.

---

Where the non-determinism comes from

The reason this is not automatic is that platform responses are not stable in the way you would like them to be.

Key order is not guaranteed. JSON objects are unordered by specification, and serialisers are free to emit keys in whatever order their internal map produced. Most of the time you get the same order twice because the same code path ran twice. But responses cross load balancers, get regenerated by different service versions during a rolling deploy, and pass through JSON layers with their own opinions. JSON.stringify() on two semantically identical objects is not guaranteed to produce two identical strings, and hashing the output of JSON.stringify() is therefore not a fingerprint. It is a fingerprint most of the time, which is worse than one that fails loudly.

Absent and null are different, and platforms treat them as interchangeable. A field that is null in one response and simply missing from the next is extremely common — it usually means an upstream service timed out and the serialiser dropped the key. Whether that should count as a change is a judgement call. Whether it should count as a change inconsistently is not.

Some fields change on every single response and mean nothing. Request IDs, server timestamps, cache-age counters, signed CDN URLs with expiry parameters embedded in the query string. Every one of these is a guaranteed fingerprint change on every poll if it reaches your hash function.

That last category is where the real danger lives, and we will come back to it, because it is where we found our own problem.

---

Normalising key order, at every depth

The fix for key order is to serialise canonically instead of trusting the platform's serialiser. Sort keys, recurse, hash the result:

export function fingerprint(value: unknown): string {
  return createHash('sha256').update(canonicalize(value)).digest('hex');
}

export function canonicalize(value: unknown): string {
  if (value === null || value === undefined) return 'null';
  if (typeof value !== 'object') return JSON.stringify(value);
  if (Array.isArray(value)) return `[${value.map(canonicalize).join(',')}]`;

  const entries = Object.entries(value as Record<string, unknown>)
    .filter(([, v]) => v !== undefined)
    .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));

  return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${canonicalize(v)}`).join(',')}}`;
}

Thirty lines including the whitespace. Three decisions in it are worth stating out loud, because each is a judgement call rather than an obvious truth:

undefined values are filtered out, so an undefined key is identical to an absent key. A platform dropping a field it has no value for should not read as a change. But null is not filtered — {a: null} and {} produce different fingerprints. The reasoning is that an explicit null is a statement ("we looked, there is nothing there") and an absent key is often an accident. You may disagree; the important part is that the choice is made once, in one place, and tested.

Numbers and strings are distinguished. JSON.stringify(1) is "1" and JSON.stringify("1") is "\"1\"". A platform that starts returning follower counts as strings has changed something, and you want to know.

Arrays are left alone. Which is the interesting half.

---

Why array order must not be normalised

Having just argued that key order is meaningless noise to be normalised away, the natural next step is to sort arrays too. It is the same shape of problem, and it would eliminate another class of phantom change.

It would also be wrong, for a reason specific to this domain: in social data, array order is the data.

A creator's recent-posts array is a feed, and its sequence is the chronology. A search result array is a ranking, and its sequence is the ranking. If you sort those arrays before hashing, then a video that jumps from position 8 to position 1 produces an identical fingerprint, and you have silently deleted the single most interesting signal in the payload. Whoever built the subscription was almost certainly watching for exactly that.

So: object keys are a serialisation detail and get normalised; array order is semantic content and is preserved exactly.

// Same object, different key order → same fingerprint.
fingerprint({ id: 'abc', followers: 100 })
  === fingerprint({ followers: 100, id: 'abc' });     // true

// Same items, different sequence → different fingerprint.
fingerprint([1, 2, 3]) === fingerprint([3, 2, 1]);    // false

Two rules that look contradictory, applied to the same payload, for a defensible reason. If someone new reads the code and does not immediately see why, the comment is not good enough.

---

Making it testable

A property this load-bearing needs tests that read like the guarantee, not like the implementation. Ours assert the behaviour a customer would be able to describe:

it('IGNORES key order — the phantom-change guard', () => {
  const a = { id: 'abc', followers: 100, name: 'X' };
  const b = { name: 'X', id: 'abc', followers: 100 };
  const c = { followers: 100, name: 'X', id: 'abc' };

  expect(fingerprint(a)).toBe(fingerprint(b));
  expect(fingerprint(b)).toBe(fingerprint(c));
});

it('ignores key order at every nesting depth', () => {
  const a = { outer: { inner: { x: 1, y: 2 }, other: 3 } };
  const b = { outer: { other: 3, inner: { y: 2, x: 1 } } };

  expect(fingerprint(a)).toBe(fingerprint(b));
});

it('respects array order — a reordered feed IS a change', () => {
  expect(fingerprint([1, 2, 3])).not.toBe(fingerprint([3, 2, 1]));
});

it('treats an undefined value the same as an absent key', () => {
  expect(fingerprint({ a: 1, b: undefined })).toBe(fingerprint({ a: 1 }));
});

it('distinguishes null from absent', () => {
  expect(fingerprint({ a: null })).not.toBe(fingerprint({}));
});

Note what the nesting-depth test is really for. A first implementation that sorts only top-level keys passes the flat test and fails in production on the first nested object a platform reorders — which, in social payloads, is roughly everything. The depth test is not thoroughness. It is the actual bug.

The other test worth having is the boring one: throw null, undefined, {}, [], 0, '', and false at the function and assert it returns a 64-character hex string for every one of them without throwing. Fingerprinting sits on the polling hot path, and a fingerprint function that throws on an empty payload turns a quiet day into a wave of subscription failures.

---

We found this in our own code, while writing this post

Here is the uncomfortable part, and it is the reason this post exists rather than being a code comment.

The function above is correct and well tested. The call site was where the problem lived.

Our unified schema puts a fetchedAt timestamp on every object it returns — a plain ISO string, set to the current time by the scraper on every live fetch. It is genuinely useful: it tells a consumer how old the data in front of them is. And the subscription worker fingerprinted the response payload, fetchedAt included.

Follow that through. Every live fetch produces a new fetchedAt. A new fetchedAt produces a different fingerprint. A different fingerprint is, by our own definition, a change. Every poll would have billed, and fired a webhook, for nothing.

We assumed at first that the cache would mask it — a cache hit replays the stored payload with its original timestamp, so within a TTL the fingerprints hold still. It does not help. Our default cache max-age is zero, and a subscription poll inherits it unless the caller explicitly set one, so the cache lookup is skipped and the poll takes the live path every time. The masking we were counting on only applied to traffic that was never at risk.

(A subscription created with an explicit cache_max_age in its params would have been partly shielded, since those params flow straight through to the execution pipeline. That is the uncommon case, and relying on it would have meant a billing guarantee that held only for customers who happened to pass an unrelated tuning parameter.)

The fix is not clever — fingerprint a payload with volatile fields stripped, as an explicit deny-list (fetchedAt, requestId, durationMs, cacheAgeSeconds, timestamp, retrievedAt), applied recursively so a nested post object in a feed is cleaned too. The list has to be explicit rather than heuristic, because "looks like a timestamp" would also match createdAt, and a post's creation date genuinely is data.

The regression test asserts both directions: that the raw fingerprint still moves when only the timestamp changes — so nobody "simplifies" the strip away later — and that the content fingerprint does not. It uses the payload shape our YouTube scraper actually returns, rather than a hand-made object, because that was the whole problem.

Our behavioural spec had already flagged the shape of this: the poll → diff → charge loop was named there as a known end-to-end test gap. Both halves were tested — a free poll charges nothing, and fingerprints are stable across key reordering — and nothing exercised the integration between them. Testing the primitives was not enough, and the spec said so before the bug was found. Two correct components, one incorrect system. The gap is still open; the specific bug is closed.

---

What the deny-list still does not cover

Stripping by field name is the easy half, and on at least one platform it is not sufficient. The harder case is a field whose name is unimpeachable data and whose value is volatile.

Instagram is the example we have. Meta serves profile pictures and post media from signed, expiring CDN URLs — scontent.cdninstagram.com/...?stp=...&_nc_ohc=...&oh=<signature>&oe=<expiry> — and those parameters are re-minted per response. They land in avatarUrl, thumbnailUrl, and mediaUrls: three fields no deny-list should ever strip, because the media URL is one of the main things a subscriber is watching.

So on Instagram, phantom changes plausibly survive the fix, and survive it silently. The fingerprint moves, the webhook fires, the payload looks entirely reasonable, and the only thing wrong is that nothing changed. The principle a deny-list has to grow into is normalising volatile values, not just dropping volatile names — for signed URLs that means fingerprinting the path and discarding the query string, which is fiddly precisely because sometimes the query string is the content.

Being honest about the evidence. That paragraph reasons from the known structure of Meta's CDN URLs, not from a diff of two live payloads. We could not fetch one: Instagram blocks our network end to end, returning a rate limit on the JSON endpoint and a rendered error page on the embed. Someone with residential IPs needs to fetch the same post twice and diff the results before anyone — us included — states it as fact. This is a strongly suspected gap, not a measured one.

Twitter looks clean on this axis: pbs.twimg.com avatar and media URLs are unsigned and stable. The Ad Library has no signed URLs surviving into our mapping, although Meta does not document ads_archive result ordering as stable, so a list-endpoint fingerprint could flip on a reorder with no content change. Also unverified, also worth checking.

---

Which is the moral of the whole post, really. Fingerprint stability is not a property of your hash function. It is a property of everything that reaches your hash function — and the things most likely to break it are the fields nobody classified as data, followed closely by the fields that are unambiguously data and happen to carry a signature.

It is also not the only place where billing for work rather than for requests turns a design question into a correctness question. The async version of the same problem — quoting a price before you know what the work cost — is one of the billing consequences of moving to a job resource, and it has the same tell: the number the customer sees and the number you charge are computed at different times, by different code, from different information.

Three questions worth asking of your own diff, whatever you have built it on:

1. Does anything in the payload change on every response by construction? Timestamps, request IDs, signed URLs, cache-age counters.
2. If a platform reorders keys mid-deploy, does your fingerprint move?
3. If someone adds a field to the response schema next quarter, does your normalisation notice, or does it silently start reporting a change every poll?

If the answer to the third one is "we would find out from the invoices," that was our answer too — right up until we went looking.