is this brilliance or accuracy leak?

Abstract: Small-scale models offer various computational advantages, and yet to which extent size is critical for problemsolving abilities remains an open question. Specifically for solving grade school math, the smallest model size so far required to break the 80% barrier on the GSM8K benchmark remains to be 34B. Our work studies how high-quality datasets may be the key for small language models to acquire mathematical reasoning. We introduce TinyGSM, a synthetic dataset of 12.3M grade school math problems paired with Python solutions, generated fully by GPT-3.5. After finetuning on TinyGSM, we find that a duo of a 1.3B generation model and a 1.3B verifier model can achieve 81.5% accuracy, outperforming existing models that are orders of magnitude larger. This also rivals the performance of the GPT-3.5 “teacher” model (77.4%), from which our model’s training data is generated. Our approach is simple and has two key components: 1) the high-quality dataset TinyGSM, 2) the use of a verifier, which selects the final outputs from multiple candidate generations. 🔗 Links 🔗 TinyGSM: achieving 80%+ on GSM8k with small language models https://arxiv.org/pdf/2312.09241.pdf ❤️ If you want to support the channel ❤️ Support here: Patreon - https://www.patreon.com/1littlecoder/ Ko-Fi - https://ko-fi.com/1littlecoder 🧭 Follow me on 🧭 Twitter - https://twitter.com/1littlecoder Linkedin - https://www.linkedin.com/in/amrrs/