Project Grant 2245652

CRII: RI: UNDERSTANDING ACTIVITIES OF DAILY LIVING IN INDOOR SCENARIOS -IT IS PROJECTED THAT, BY 2050, ONE-IN-SIX PEOPLE IN THE WORLD WILL BE OVER THE AGE OF 65. THIS IS UP FROM ONE-IN-11 IN 2019. THUS, OLDER ADULTS ARE A GROWING DEMOGRAPHIC GROUP IN SOCIETY. BECAUSE AGING IS RELATED TO INCREASED HEALTHCARE UTILIZATION, AS ARE PANDEMICS, THIS INCREASING DEMOGRAPHIC TRANSLATES INTO THE NEED TO INCREASE THE WORKFORCE IN HEALTHCARE. THE RISING DEMAND FOR HEALTHCARE CAN BE COMBATED BY DEPLOYING ACTIVITY MONITORING SYSTEMS, WHICH COULD HELP MONITOR THE HEALTH STATUS OF OLDER PATIENTS AND SUPPORT THE EARLY DETECTION OF HEALTH ISSUES. THUS, BUILDING SUCH MONITORING SYSTEMS REQUIRES AN AUTOMATED UNDERSTANDING OF ACTIVITIES OF DAILY LIVING (ADL) PERFORMED BY HUMANS. MOST OF THE INVESTIGATIONS TOWARDS MODELING HUMAN ACTIVITIES OWING TO THE ADVANCES IN COMPUTER VISION ARE TARGETED FOR GENERIC INTERNET VIDEOS. EXISTING MODELS ARE FABRICATED FOR RECOGNITION IN WEB VIDEOS WHEREAS VIEWPOINT VARIATION AND SUBTLE MOTION THAT CHARACTERIZE ADLS GENERALLY CANNOT HANDLE UNCERTAINTY AND TEND TO UNDERPERFORM IN REAL-WORLD SCENARIOS. MOREOVER, THEY HAVE DIFFICULTIES DISTINGUISHING SIMILARLY LOOKING ACTIVITIES. THUS, THE KEY OBJECTIVE OF THIS PROJECT IS TO BUILD A FRAMEWORK FOR RECOGNIZING ADLS THAT CAN BE DEPLOYED IN MONITORING SYSTEMS. THE PROJECT WILL ALSO PERFORM COMPLEMENTARY EDUCATIONAL AND OUTREACH ACTIVITIES THAT ENGAGE STEM STUDENTS. THIS PROJECT WILL DEVELOP A MULTI-MODAL FRAMEWORK PREDOMINANTLY BASED ON THE RGB COLOR MODEL AND POSE MODALITIES DUE TO THEIR EASY ACCESSIBILITY IN INDOOR SCENARIOS. THIS FRAMEWORK AIMS AT ADDRESSING TWO IMPORTANT CHALLENGES - THE LIMITED AVAILABILITY OF LABELED ADL VIDEOS AND HOW TO COMBINE DIFFERENT HETEROGENEOUS MODALITIES (RGB AND POSES) FOR CLASSIFYING ACTIVITIES. THUS, THIS PROJECT WILL EXPLORE THE INTEGRATION OF TWO INTERRELATED RESEARCH DIRECTIONS: (1) A STUDY ON LEARNING FROM LIMITED TRAINING DISTRIBUTION; AND, (2) A STUDY ON COMBINING MODALITIES LIKE RGB AND DEPTH. IN THE FIRST STUDY, THE PROJECT WILL EXPLORE THE POSSIBILITIES OF MITIGATING THE LIMITATION OF THE SCALE OF AVAILABLE DATA IN THE ADL DOMAIN FOR EFFECTIVE TRAINING OF NEURAL NETWORKS FOR VIDEO UNDERSTANDING. IN THE SECOND STUDY, THE TEAM WILL AIM AT IMPROVING THE EFFECTIVENESS OF RGB-BASED HUMAN ACTIVITY RECOGNITION BY LEVERAGING THE HUMAN LOCALIZED REGIONS WITHIN A SCENE. FINALLY, WE WILL DEVELOP A MULTI-MODAL NEURAL NETWORK MODEL FOR ADL BY INTEGRATING HUMAN LOCALIZED RGB AND 3D POSES OF A HUMAN ACTOR. THIS RESEARCH STUDY WILL REVEAL SEVERAL NEW DIMENSIONS TOWARDS UNDERSTANDING ADLS THAT WILL ALSO BENEFIT THE COMPUTER VISION COMMUNITY. THIS AWARD REFLECTS NSF'S STATUTORY MISSION AND HAS BEEN DEEMED WORTHY OF SUPPORT THROUGH EVALUATION USING THE FOUNDATION'S INTELLECTUAL MERIT AND BROADER IMPACTS REVIEW CRITERIA.