ERI: GESTURE2CAD: multimodal AI for gesture-driven parametric CAD modeling

The National Science Foundation Division of Civil, Mechanical, and Manufacturing Innovation awarded Chapman University $199,875 on June 1, 2026, to develop GESTURE2CAD, a gesture-driven multimodal artificial intelligence system for parametric computer-aided design modeling. The award funds research under the Engineering Research Initiation program (CFDA 47.041).

The project develops three core research activities: creating the first multimodal CAD dataset integrating hand gesture and text representations; establishing a gesture-driven generative approach that translates natural hand movements into parametric CAD commands and expert-level text prompts; and enabling immersive and explainable human-AI collaborative CAD modeling using hand gestures in extended reality environments. The research addresses the transition toward human-centric Industry 5.0 manufacturing by enabling workers to specify complex design intent through natural hand gesture input rather than traditional interface methods, with feedback from expert designers and empirical studies guiding system design and usability assessment. The system aims to produce high-quality, editable 3D outputs compatible with standard CAD tools for direct integration into existing design workflows.

Performance occurs in Orange, California, with a period extending from June 1, 2026, through May 31, 2028.

Abstract

This engineering research initiation (ERI) award supports research that focuses on the development of a novel gesture-driven multimodal shape description system, GESTURE2CAD, to enable immersive, explainable, and controllable human-AI collaborative design for generating high-quality parametric CAD models from natural hand gesture input. As manufacturing transitions toward a human-centric industry 5.0, new forms of interaction are required to empower workforce creativity and relieve them from repetitive tasks. While prior work has explored hand gestures for sketching or specifying basic geometric forms, its potential as a continuous, structured input modality for conveying complex design intent remains largely unexplored. This project advances fundamental knowledge across gesture-based design representation, AI-driven CAD modeling, immersive design in extended reality (XR), and explainable models to support human-AI collaborations. This ERI award advances theory and methods for intelligent CAD modeling through three research activities: (1) developing the first multimodal CAD dataset integrating gestures and text representations; (2) establishing a gesture-driven generative approach capable of translating natural hand movements into parametric CAD commands and expert-level text prompts; and (3) enabling immersive and explainable human-AI collaborative CAD modeling using hand gestures in XR. Feedback from expert designers and rigorous empirical studies with participants of varying CAD design experience will guide system design and assess usability. This research includes advancing the design and development of the first AI-aided parametric CAD generation system from gesture input, supporting a new paradigm in intelligent CAD modeling. The project aims to produce high-quality, editable 3D outputs compatible with standard CAD tools, enabling direct integration with existing design workflows and lowering the entry barriers to parametric CAD. The project will also release open-source gesture-CAD data and methods to advance human-centered AI, manufacturing, and healthcare applications. In parallel, the project will enhance stem education and workforce development by integrating GESTURE2CAD into coursework to support hands-on, interactive CAD learning across stem and design disciplines. Outreach activities will include interactive workshops to engage middle to high school students in engineering design, AI, and XR, as well as partnerships with a local art school to give secondary students the opportunity to explore AI-assisted CAD modeling, thereby increasing participation and strengthening pathways into the future stem workforce. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.