Circ: DEV: smile: scan to multi-sensorial interactive learning environment

The National Science Foundation awarded The Research Foundation For The State University Of New York $514,915 on October 1, 2025, under the Computer and Information Science and Engineering program (CFDA 47.070) to design and develop SMILE (Scan to Multi-sensorial Interactive Learning Environment), an infrastructure for rapidly constructing virtual-reality environments from 3D scans of real-world indoor scenes and enabling multi-sensory interactions including touch, thermal, olfactory, visual, and aural feedback within those environments.

The award funds nearly automated VR environment construction, development of multisensorial interaction capabilities applied directly to 3D objects, Internet-scale real-time collaboration among remote participants with synchronization tailored to multi-sensorial temporal requirements, and rigorous software testing including coverage-based fuzz testing and configurable static analysis to ensure safe user interactions before deployment.

Performance runs through October 31, 2027, at Albany, New York. The Research Foundation serves as the administrative entity managing federal research grants for the State University of New York system.

Sub-recipients deliver research components across four partner institutions. The University of Texas at Dallas (subaward dated March 17, 2026) develops core SMILE infrastructure for nearly automated VR construction, multisensorial interaction capabilities, and Internet-based real-time collaboration with software testing and verification. The New Jersey Institute of Technology (April 8, 2026) addresses challenges in dense 3D reconstruction, multi-sensorial display integration in collaborative VR environments, and safety verification of VR software working with thermal and haptic devices. The University of Illinois (April 23, 2026) integrates Neural Radiance Fields and 3D Gaussian Splatting content representations into SMILE to enable new multi-view content formats. Dallas College (July 7, 2026) designs and conducts three major research studies using a virtual chemistry laboratory deployed within SMILE, with Prof. Vinu Johnson directing the chemistry lab research protocol.

Abstract

VR (virtual reality)-based immersive and interactive environments can be a great resource for learning and training, especially for concepts that involve safety aspects by interacting with inflammable or breakable objects. The tedious nature of developing these VR experiences continues to be a limiting factor for VR becoming mainstream. The main goal of this project is the design and development of an innovative infrastructure, smile (scan to multi-sensorial interactive learning environment), focusing on: (1) nearly automated construction of VR environments that mimic real-world indoor scenes; (2) interactions with virtual objects involving multiple senses such as touch, visual, aural, and smell. The smile infrastructure will undergo rigorous software testing to ensure safe interactions before being deployed to support internet-scale collaboration among users. In the exploratory phase of this project, the team will address the following: (1) democratize the creation of VR environments for subject-matter experts that are realistic in reflecting the ambiance of a real-world scene while providing the needed interactions with objects in the scene. (2) apply sensorial information directly to 3D objects, enabling multiple sensations during interactions with the virtual objects. (3) internet-scale collaborations in smile through multi-modal inter-sender and inter-receiver skew synchronization, based on the temporal requirements of multi-sensorial interactions. This will be tested by deploying smile for use in the sites of the project partners: the university of texas at dallas, the university of illinois at urbana-champaign, the new jersey institute of technology, and the dallas college. (4) a coverage-based fuzz testing module where numerous inputs can be generated and executed efficiently to test corner cases in the smile components, and configurable static analysis where algorithms can be flexibly tuned to check the most critical safety properties in smile. Virtual chemistry laboratory experiments for assessing the performance of smile will be designed and developed through research collaboration with the dallas college. Through this collaboration, the smile project will have a significant impact on the broad undergraduate student community. Smile could be used for different learning environments, apart from the chemistry lab case study, promoting education involving k-16. The project will involve doctoral students, undergraduate students, and k-12 students. Open-source resources produced through the smile project include a multi-sensory database of material properties for sensorial displays, unity-based VR authoring tools, and automated software testing tools for VR-based applications. These will be made available for a period of five years through the project website at: https://labs.utdallas.EDU/multimedialab/. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.