The National Science Foundation Division of Electrical, Communications and Cyber Systems awarded The Pennsylvania State University $199,444 on October 1, 2025, under the Engineering program (CFDA 47.041) to develop retrieval-augmented generation (RAG)-empowered extended reality systems that integrate large language models with localized, context-rich knowledge databases to enhance context awareness in virtual reality and augmented reality applications.
The project builds and maintains knowledge databases incorporating diverse contextual information sources—3D object data, egocentric images, text inputs, and user-specific data such as head pose, eye gaze, and user preferences—and optimizes context-aware XR performance by leveraging collaboration between XR devices and nearby edge servers to reduce latency and resource consumption. The work spans applications across commerce, entertainment, manufacturing, and social interaction domains, including smart homes, intelligent manufacturing, and collaborative social XR platforms where systems adapt to individual and group user behaviors and dynamic environmental conditions. The project includes training of multiple cohorts of undergraduate and graduate students.
Performance occurs in State College, Pennsylvania, with a period of performance from October 1, 2025, through September 30, 2027.