CoinScoopResearch: Data improvements drove the majority of pre-training efficiency gains from 2019 to 2025
A study by Dwarkesh Patel and Jerry Han released on September 8, 2026, indicates that data improvements contributed 12.0x to computational efficiency gains, while model architecture improvements contributed 3.7x between 2019 and 2025. The study suggests these effects are largely independent and additive, explaining 88% of the variance in OLMES scores. Data evolution progressed from the 9 billion token OpenWebText in 2019 to more refined datasets like UltraFineWeb in 2025.
Summaries are written by AI from the original article. Not investment advice.