DesignQA: A Multimodal Benchmark for Evaluating Large Language Models’ Understanding of Engineering Documentation
1Massachusetts Institute of Technology, Cambridge, MA
2Autodesk Research, San Francisco, CA
3MIT Motorsports, Cambridge, MA
4Autodesk Research, Toronto, ON, Canada
DesignQA is a novel benchmark aimed at evaluating the proficiency of multimodal large language models in comprehending and applying engineering requirements in technical documentation. Developed with a focus on real-world engineering challenges, DesignQA uniquely combines multimodal data—including textual design requirements, CAD images, and engineering drawings—derived from the Formula SAE student competition.
