'5.94 billion TikTok videos' — Hugging Face hosts 4.5B rows of metadata instead

On September 2, u/DataShack posted to r/MachineLearning what reads like every video model's dream training corpus: "I scraped 5.94 billion TikTok videos and 3.23 billion profiles in 3 weeks. Uploaded full dataset to Hugging Face for free." The thread drew roughly 680 upvotes and more than 160 comments in two days, and the repository is real. But the artifact holds no videos, no profiles and no creator identity: 289 GB of post metadata, harvested via TikTok's private app API in a way the collector concedes breached its terms of service. The distance between the headline and the files is the story.

The Claim and the Artifact

The repository, kuben-developer/tiktok-videos-4b, calls itself "the largest public TikTok dataset I am aware of", released "as-is, for research." In practice:

• Claimed (Reddit title): 5.94 billion videos, 3.23 billion profiles, three weeks.
• Claimed (write-up): also 2.8 billion comments, via 24 reverse-engineered endpoints.
• Published (Hugging Face): 4,501,811,789 rows — about 4.5 billion — in 289 GB across 27 zstd Parquet files, ~167M rows per ~10 GB file.
• Published columns (16): content_id, create_time, caption, mentions, duration, is_video, music_id, music_title, views, likes, comments, shares, saves, country, language, is_ad.
• Not in the published files: video files, media URLs, usernames, profile data, comments.

The numbers partially reconcile: 5.94 billion collected rows, cut to 27 of 32 partitions and stripped of the ~10% duplicates the collector says he removed, lands near 4.5 billion. That makes the gaps sharper — the title's 3.23 billion profiles appear in no public file, and neither do the write-up's 2.8 billion comments. Creator identity is absent on purpose: "There is no author ID, username or profile data. You can group videos by sound, hashtag mention or caption, but not by who posted them. This is deliberate."

What Was Actually Collected

The method — set out in the repository and at length in the guide — reverse-engineers the private HTTP+JSON API that TikTok's Android app uses. "Requests are signed the way the app signs them, from anonymous device registrations. There is no login anywhere in the pipeline," the repo states. The pipeline fabricates coherent device profiles — a phone model matched to a real carrier in a real region — to obtain credentials from TikTok, then reproduces the app's signature stack (X-Argus, X-Gorgon, X-Ladon) and a TLS handshake that looks like a phone. Get one wrong, the guide explains, and TikTok answers with a clean HTTP 200 and an empty body: no error message, no status code, just silence.

The README documents the data's limits with unusual candor. Engagement counts are snapshots from the three-week window, not a time series: day-one videos simply had longer to accumulate views. Rows are grouped by creator rather than shuffled — "If you are training on this, shuffle." Country and language are TikTok's inferred labels, "wrong often enough that you should not treat them as ground truth." Media URLs are omitted because TikTok CDN links carry signed expiry parameters and die within days; shipping them would have meant 539 GB of dead links, the author notes.

A Free Dataset, a Paid Pipeline

The Reddit post's own disclaimer is worth reading in full:

"The TikTok app exposes 24 endpoints that can be accessed without a TikTok account, so the data itself is publicly accessible. But accessing it this way is probably still against TikTok's ToS. Also, the full code is not free, I charge a small fee for access to it."

The "small fee" is itemized on the tutorial the post funnels to: $699 one-time for the private Go repository behind the signing stack and all 24 endpoints, or $1,899 for a "done-for-you" tier built on your server and handed over running. Scale adds rotating proxies: roughly $150 a month for millions of records, about $950 a month for billions.

The poster behind u/DataShack also runs SeekSocial (seeksocial.io), a TikTok influencer-marketing platform selling access to "9.9M+ creator profiles" with emails for $99 a month or per-email from $0.0006, plus one-click exports. His own 2024 Reddit history funnels to the same site — a "Go viral on TikTok in 1 week" post for its viral-video finder. The pattern is plain: the mobile-API method behind the "free" dataset powers a paid marketing-data business; the free artifact excludes creator identity while the paid product sells exactly that contact data; and the page announcing the upload sells the code. Asked in its FAQ whether the system is legal, the page answers: "It is against TikTok's terms of service. It is sold for research and educational use."

What the Collector Concedes

The legal exposure is stated in the repository, not inferred by outsiders. "Captions are written by real people and this dataset is personal data under GDPR, the UK GDPR and CCPA regardless of the fact that it was publicly posted," the README warns; the obligation transfers to anyone who downloads it in a covered jurisdiction. The opt-out is a GitHub discussion — "open a discussion on this repository." Whether that scales to billions of rows, for people who never hear of it, is untested. Collection, the repo concedes in one line, "was contrary to TikTok's terms of service."

None of this is new terrain, only a change of scale: in 2020 a r/datasets post shared metadata for 32.5 million TikTok videos; this upload is roughly 140 times larger. TikTok operates an official Research API for academics, and the contrast with a no-login crawl of its private endpoints is the distinction his disclaimer draws. Neither TikTok nor ByteDance has commented on the upload as of this writing, and Hugging Face hosts it with the author's caveats attached.

What Would Settle This

An independent audit needs the logs, proxy records and partition manifests behind the 5.94-billion claim — none ship in the repository — plus a sampling check of published rows against live TikTok data. A statement or technical action from TikTok would show whether it treats anonymous device-registration traffic as a reportable breach or a tolerated one. Hugging Face could state its policy for datasets whose own documentation concedes they were collected against a platform's terms of service. And the opt-out can be tested: whether a discussion-based removal request actually deletes rows at scale, and how quickly.

What can be said now is narrower than the headline. The dataset is genuinely large and free, useful for studying captions, engagement and sound usage at a scale no public TikTok corpus has reached. It is not 5.94 billion videos, it is not 3.23 billion profiles, and no one should train a video model on it expecting frames. Treat it as the repository does: 4.5 billion metadata rows, collected against the platform's rules, the human cost documented by the collector himself. The next viral "free dataset" post deserves the same five-minute check: open the artifact, count the columns, see what the headline left out.