QwenMath
AI/ML Engineering
QLoRA fine-tuned Qwen3-4B to improve math accuracy from 50.3% to 80.2% through chain-of-thought distillation, placing 3rd among 300+ students
Honestly not too proud of this win because we basically benchmaxxed. The test dataset was made available to us and we were allowed to use it as training data. So naturally, I proposed we generate answers using GPT 5.5 Pro (the best math model at the time) and then intentionally overfit our smaller Qwen model to copy its answers. Many of the questions were also taken from publicly available HuggingFace datasets, so we crawled the web to get the answer keys as well. The result: an intentionally overfit and benchmaxxed model that performs well on the test data but probably won’t generalize well to actual unseen data.
Funny thing I learned, though, was that frontier models are also somewhat benchmaxxed. We used the FrontierMath benchmark to determine the best model to distill from, and we found that GPT dominated the leaderboard. However, if you dig into the website you’ll find that OpenAI has access to almost all their problem statements and solutions. So the training data might as well be contaminated.
If the goal is real general intelligence, then AI researchers should be using test data they never see. AI labs should not have access to the exact problems and answer keys to any frontier benchmark aiming to measure AGI.