Collaborative research: vines: track 1: towards AI-native holographic communication for accessible telepresence

The National Science Foundation Division of Information and Intelligent Systems awarded $400,000 to the University of North Carolina at Charlotte on October 1, 2026, under the Computer and Information Science and Engineering program (CFDA 47.070) to develop a generative holographic codec framework for accessible telepresence communication.

The project advances technology to capture a subject using one or a few standard cameras, transmit compact digital cues, and reconstruct high-fidelity 3D holograms on smartphones, smart glasses, and headsets. The framework integrates 3D Gaussian splatting, vector quantization, masked encoder-decoder learning, and analysis-by-rendering. Work encompasses three primary tasks: designing generalizable codecs for sparse multi-view and single-view inputs with recovery mechanisms for lost vector indices; developing sparse computing methods for resource-constrained edge devices using heterogeneity-aware global structured pruning, mixed-precision quantization, and pose- and point-aware attention mechanisms; and creating adaptive control via layered refinement and reinforcement learning to balance quality, bandwidth, latency, memory, and device hardware constraints. Comprehensive evaluation occurs across wireless testbeds, long-distance network links, and large-scale simulations. By reducing data, hardware, and computing resources needed for high-quality holograms, the technology supports applications in remote healthcare, interactive classrooms, emergency response training, workforce development, collaborative design, entertainment, and digital arts, particularly for communities with limited network capacity or equipment access.

Performance occurs in Charlotte, North Carolina, with a period of performance from October 1, 2026, through September 30, 2029. This is a Project Grant assistance award.

Abstract

Today, sending a moving, life size person across the internet as a hologram requires an impractical number of cameras, excessive bandwidth, and massive computing power. To address these limitations, this project introduces a highly efficient framework to capture a subject using only one or a few standard cameras, transmit a compact set of digital cues, and reconstruct a high-fidelity 3D hologram on smartphones, smart glasses, and headsets. Technically, the project advances a novel generative holographic coder-decoder (CODEC) that integrates 3D gaussian splatting, vector quantization, masked encoder-decoder learning, and analysis-by-rendering. The first task designs generalizable codecs for sparse multi-view and single-view inputs, with built-in recovery mechanisms for lost vector indices. The second task develops sparse computing methods for resource-constrained edge devices using heterogeneity-aware global structured pruning, mixed-precision quantization, pose- and point-aware attention mechanisms, and few-step generative sampling. The third task creates adaptive control via layered refinement and reinforcement learning to dynamically balance quality, bandwidth, latency, memory, and device hardware constraints. The framework will be comprehensively evaluated across wireless testbeds, long-distance network links, and large-scale simulations. By drastically reducing the data, hardware, and computing resources needed for high-quality holograms, this project broadens access to next-generation immersive communication. The resulting technology will support critical applications in remote healthcare, interactive classrooms, emergency response training, workforce development, collaborative design, entertainment, and the digital arts, including communities with limited network capacity or limited access to specialized equipment. Furthermore, the project integrates research with education through student thesis work, capstone projects, internships, course modules in networking and machine learning, and outreach demonstrations for kindergarten through grade twelve learners. These activities help prepare a diverse workforce for next-generation networks, artificial intelligence systems, and immersive media. This collaborative project brings together investigators from the university of north carolina at charlotte and the university of central florida to make live three dimensional (3D) communication significantly more accessible and practical. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the foundation's intellectual merit and broader impacts review criteria.- subawards are not planned for this award.