Scraping and GDPR: \"Not a ToS Breach\" Is Not a Lawful Basis

A developer wins the argument with their manager by citing hiQ and Meta v. Bright Data, starts collecting public profiles, and three months later a customer's security reviewer sends over a questionnaire asking for the lawful basis, the retention window, the sub-processor list and the data processing agreement (the DPA — the contract Article 28 GDPR requires between a customer and a vendor that processes personal data for them). None of the case law helps, because none of it was about that.

This is the gap the whole category talks around. The US decisions everyone cites resolve two questions — was this unauthorised computer access, and did you break a contract. GDPR asks a third one that those cases never reach: are you allowed to process this data at all? A win on the first two is not an answer to the third, and if any of the people in your dataset are in the EU or UK, the third is the one that carries regulatory fines.

This is not legal advice. It is written by engineers who have had to answer these questions in vendor reviews, not by lawyers. Data-protection obligations depend on your jurisdiction, your role, your purpose and your specific processing. Get advice on your situation before relying on any of it.

Public does not mean not personal

Article 4(1) GDPR defines personal data as any information relating to an identified or identifiable natural person. There is no exemption for information the person published themselves, and no exemption for information anyone can see. A public display name, a profile photo, a bio, a follower count attached to a real person, a comment — all personal data. Scraping it is processing it.

The enforcement record is not theoretical. Clearview AI built its face database entirely from publicly accessible images, and European regulators have fined it repeatedly on that basis: €20m from the Italian Garante and €20m from France's CNIL in 2022, and €30.5m from the Dutch data protection authority on 3 September 2024, along with orders subject to penalty payments requiring it to stop processing EU residents' data, inform those people, answer their access requests, and appoint an EU representative. The Dutch authority held that the biometric codes derived from those public photos were special-category data under Article 9, and that no Article 9 exception applied.

That last point matters beyond faces. Article 9(2)(e) permits processing special-category data that has been "manifestly made public by the data subject," and it is the exception scrapers reach for. Courts and regulators read it narrowly: "manifestly" implies a deliberate act by the person, with awareness of the consequence. Data being visible in a public place is not the same as the person having manifestly made it public.

And you do not need a European office to be in scope. Article 3(2) applies the regulation to controllers outside the EU who offer goods or services to people in the Union, or who monitor their behaviour. Systematically collecting and analysing the public posting activity of people in the EU is a reasonable description of monitoring behaviour.

Legitimate interest is available, and it is a test, not a checkbox

Consent is not realistic for scraped data — you cannot ask several million people first. Practically, the basis is Article 6(1)(f), legitimate interests, and it requires you to actually run and record a three-part assessment:

1. Purpose. Is there a real, specific, lawful interest? "We might need it later" fails. "We provide advertisers with competitive creative analysis" is specific.
2. Necessity. Is the processing necessary for that purpose, and is there no less intrusive way to achieve it? Collecting whole profiles when you need follower counts fails this step.
3. Balancing. Do the interests or fundamental rights of the people in the data override yours? The heaviest factor is their reasonable expectations: someone posting publicly expects to be read, not to be enrolled in a permanent commercial database, cross-referenced and resold.

The EDPB's Guidelines 1/2024 set out that test, and Opinion 28/2024, adopted 17 December 2024, applies it to AI models built on scraped data. The direction is important for anyone in this business: legitimate interest is not ruled out, but it survives only with mitigating measures — restricting what you collect at source, excluding categories of data and sites, filtering special-category data, honouring objections, and being able to show your work. France's CNIL built on that opinion with its own 2025 recommendations along the same lines.

The other obligation people are surprised by is Article 14. When you obtain personal data from somewhere other than the person, you owe them a privacy notice — who you are, what you collect, why, on what basis, how long you keep it, and how to object. Article 14(5)(b) excuses this where it is impossible or requires disproportionate effort, which is the exemption every scraper assumes covers them. It is narrower than that, the effort has to arise from the indirect collection itself, and where you rely on it you are still required to take appropriate measures — including publishing the information. In practice, "we cannot email eight million people" is an argument for a published notice, not for silence.

Data minimisation is an engineering requirement

The compliance-document version of minimisation is a paragraph in a policy. The version that survives a technical review is decisions in your code. Four places it shows up:

Field selection. Ask for what you need. If your product counts views, a response containing bios, follower lists and profile photos is a liability with no upside. Raw upstream payloads are the worst offender, since they carry everything the platform happened to return; treating raw passthrough as opt-in rather than default is a minimisation control, not a formatting preference.

Retention windows, per category. Different data has different useful life. A cached response supporting a dashboard needs hours. A change-detection fingerprint needs to outlive the last poll. An engagement metric you are trending needs months. One global "we delete after 12 months" tells a reviewer you did not think about it.

Zero retention as a mode, not a promise. The only durable version of "we don't keep it" is a code path where the write does not happen. A configuration flag that is checked before every persistence call, with a test that fails if a payload lands in storage, is verifiable. A sentence in a policy is not.

Isolation between customers. If your cache is shared across tenants, one customer's query can be served another customer's stored personal data. That is a design decision with a data-protection consequence, and reviewers increasingly ask about it directly.

Retention has a cost dimension too, which is part of what makes cost-per-successful-request the honest metric: caching is the margin lever in this business, and every retention control you add takes something off that lever. Vendors who keep everything forever are not only exposed, they are cheaper. Worth knowing which trade you are buying.

What a data retention policy has to actually say

Most published retention policies are unusable because they state a single number. A reviewer is looking for a table they can map onto your architecture:

It has to state	Not this	This
Categories of data held	"Customer data"	Response payloads · request metadata · account data · job results
Purpose per category	"Business purposes"	Cache serving · billing and abuse investigation · async result delivery
Retention period per category	"As long as necessary"	A number per category, with the mechanism that enforces it
Deletion mechanism	"We delete data"	Expiry field, scheduled job, or code path — something falsifiable
Sub-processors	Unlisted	Named, with location and what they see
Subject-request path	An email address	A route from a person's identifier to every row about them

The last row is the one that catches scraping businesses. If you cannot go from "this person objected" to "here is every row derived from them, deleted," you cannot satisfy Articles 15, 17 and 21, whatever your policy says.

Questions to ask any social data vendor

Take these into a call. The answers are more informative than the answers to your feature questions.

1. For the scraping itself, are you a controller or a processor? A vendor that decides what to collect and how, and sells the result to many customers, is on the controller side of that line for its own collection — no matter what the order form says. If they answer "processor" without hesitating, they have not thought about it.
2. What is your lawful basis, and can I see the legitimate-interests assessment? A vendor with a real one will have it written down.
3. What is your retention window per data category, and what enforces it? Ask for the mechanism, not the number.
4. Is your cache shared between customers? If yes, is isolation available and does it cost extra?
5. Do you offer a mode where responses are never persisted at all?
6. What personal data is in your logs? Payload bodies in application logs quietly undo everything else.
7. Do you have a DPA under Article 28, and does it name sub-processors and their locations?
8. How do you handle a deletion or objection request that arrives from a person, not a customer?
9. What data do you collect logged in, versus logged out? Logged-in collection weakens the "reasonable expectations" side of the balancing test, on top of the contract exposure it creates.
10. What happens to my data when I stop paying you?

A vendor's changelog announcing that it now stores API response data, with no retention window published anywhere, is an answer to question 3 — just not a good one.

What we do, and what we don't have

Stated plainly, because the questions above cut both ways.

What is implemented. Per-tenant cache isolation is a standard flag rather than an enterprise upsell, so an isolated team can never be served another team's stored payload. A zero-retention flag stops response payloads being written to cache at all, checked in the cache-write path and covered by a test that fails if a payload is ever persisted for such a team. A cache-disabled flag always fetches live. Our request log stores metadata only — endpoint, platform, status code, credits charged, duration, error code — and no payload and no parameters, so no scraped personal data lands there. Cached payloads carry an expiry — the ceiling on any caller-supplied cache age defaults to 30 days — and a worker deletes expired rows on a timer rather than leaving them to accumulate. Async job results carry a seven-day expiry by default. Raw upstream payloads are omitted from responses unless a caller explicitly asks for them.

What is not. We do not have SOC 2, a published DPA, SSO, or a contractual SLA today. Those are real gaps for an enterprise buyer and we are not going to describe them as anything else. Code-level expiry ceilings are also not the same artefact as a published retention policy — they are the mechanism a policy would point at, and a buyer should ask for both.

None of the above is legal cover, ours or anyone's. Controls reduce exposure; they do not create a lawful basis. That part is yours, it depends on what you collect and why, and it is worth an hour of a lawyer's time before it is worth six months of engineering.