Keep the signal. Keep its origin.
Provenance graphs connect deduplication, contamination review and corpus release into one traceable engineering chain.
Discuss a similar programmeDuplicate forks, leakage, licensing ambiguity and low-signal text.
The hard problem
Web-scale multilingual corpora mixed duplicate forks, benchmark neighbours, uncertain licensing, translation artefacts and low-signal machine text. Removing duplicates alone could not establish a release boundary.
Our advantage
We linked provenance graphs to exact, MinHash and semantic deduplication, executed contamination probes, applied trained quality classifiers and balanced language curricula. Review kept uncertain benchmark-neighbour material separate from a releasable corpus.
Value delivered
The programme delivered governed corpus releases, tokenizer and sampling policy, shard manifests, contamination reports and loss-slice dashboards.
Platform context.
Bring the next
requirement.