P00001 THIS PROPOSAL LAYS THE FOUNDATIONS OF MULTI-REWARD LEARNING WHERE THE COMPLEXITY OF A MISSION PRECLUDES COLLAPSING THE LEARNING SIGNAL TO A WELL-FORMULATED, SINGLE REWARD OR OBJECTIVE FUNCTION. INDEED MANY MISSIONS REQUIRE AN AGENT (OR A ROBOT) TO BALANCE PERFORMANCE, LONGEVITY, AND SAFETY, AS WELL AS PROVIDE EXPLANATIONS FOR ITS ACTIONS. IN ADDITION, EACH OF THOSE HIGH LEVEL GOALS ARE THEMSELVES COMPOSED OF MULTIPLE SUBGOALS. FOR EXAMPLE, OPTIMIZING PERFORMANCE FOR A ROBOT MAY REQUIRE MOBILITY, ENERGY MANAGEMENT, COMMUNICATION, AND OBSERVATION TASKS WHILE, AGAIN, PROVIDING EXPLANATIONS AND ASSURANCES. THIS NEED TO MANAGE A VARIETY OF POSSIBLY IMPORTANT GOALS TURNS THE WELL-STUDIED LEARNING PROBLEM OF HOW TO DETERMINE ACTIONS TO OPTIMIZE ONE GOAL INTO THE SIGNIFICANTLY MORE COMPLEX PROBLEM OF HOW TO DETERMINE ACTIONS TO OPTIMIZE THE RIGHT GOAL AT THE RIGHT TIME. BUT HOW DOES AN AGENT KNOW WHICH OF ITS GOALS IS IMPORTANT AT ANY TIME? THIS QUESTION IS BEYOND THE CAPABILITIES OF THE CURRENT STATE OF THE ART IN BOTH MULTI-OBJECTIVE OPTIMIZATION (WHICH GENERALLY RELIES ON LINEAR COMBINATIONS OF REWARDS OR PARETO CONCEPTS [8, 15]) AND MULTI-TASK LEARNING (WHICH GENERALLY RELIES ON A WELL-DEFINED PROGRESSION OF GOALS [4, 3, 12]). THE OBJECTIVE OF THIS PROPOSAL IS ENABLE AN AUTONOMOUS SYSTEM TO NOT ONLY LEARN TO CHOOSE WHICH ACTIONS TO TAKE, BUT TO LEARN TO CHOOSE WHICH REWARDS TO SEEK TO SATISFY HIGH-LEVEL MISSION GOALS. THE SPECIFIC RESEARCH OBJECTIVES OF THIS PROPOSAL ARE TO: 1. GENERATE DOMAIN SPECIFIC REWARDS AND ENABLE AN AUTONOMOUS AGENTS TO SWITCH AMONG THEM TO IMPROVE ITS LEARNING SPEED AND PERFORMANCE. 2. FORMALIZE THE CONCEPT OF ALIGNMENT AMONG REWARDS TO CAPTURE BROADER SET OF DESIRED BEHAVIORS (SAFETY, ASSURANCE, EXPLAINABILITY) AND DEMONSTRATE EFFECTIVENESS OF APPROACH IN DOMAINS WITH QUALITATIVELY DIFFERENT REWARDS. THE KEY CONTRIBUTION OF THIS WORK IS TO QUANTIFY THE ALIGNMENT AMONG MULTIPLE REWARD FUNCTIONS SPANNING MULTIPLE TIMESCALES AND USE THAT ALIGNMENT TO DETERMINE WHAT MATTERS WHEN TO GUIDE THE LEARNING OF AGENTS FACED WITH COMPLEX, TIME-EXTENDED MISSIONS. THE IMPACT OF THE PROPOSED RESEARCH IS TO EXTEND AUTONOMY AND LEARNING TO COMPLEX SCENARIOS WHERE MULTIPLE MISSION DIRECTIVES CANNOT BE COLLAPSED INTO A WELL-FORMULATED SINGLE REWARD SIGNAL. THIS RESEARCH WILL LEAD TO EXPLAINABLE (BY SPECIFYING WHICH LOCAL REWARD DROVE PARTICULAR DECISIONS), FAULT-TOLERANT (BY HAVING MULTIPLE REWARDS TO STEP UP IF BEHAVIOR BECOMES QUESTIONABLE), AND ASSURED (BY SPECIFICALLY ENCODING ASSURANCE AS A HIGH-LEVEL REWARD) AUTONOMY. Other Administrative Action $0 7/3/18 Not listed THIS PROPOSAL LAYS THE FOUNDATIONS OF MULTI-REWARD LEARNING WHERE THE COMPLEXITY OF A MISSION PRECLUDES COLLAPSING THE LEARNING SIGNAL TO A WELL-FORMULATED SINGLE REWARD OR OBJECTIVE FUNCTION. INDEED MANY MISSIONS REQUIRE AN AGENT (OR A ROBOT) TO BALANCE PERFORMANCE LONGEVITY AND SAFETY AS WELL AS PROVIDE EXPLANATIONS FOR ITS ACTIONS. IN ADDITION EACH OF THOSE HIGH LEVEL GOALS ARE THEMSELVES COMPOSED OF MULTIPLE SUBGOALS. FOR EXAMPLE OPTIMIZING PERFORMANCE FOR A ROBOT MAY REQUIRE MOBILITY ENERGY MANAGEMENT COMMUNICATION AND OBSERVATION TASKS WHILE AGAIN PROVIDING EXPLANATIONS AND ASSURANCES. THIS NEED TO MANAGE A VARIETY OF POSSIBLY IMPORTANT GOALS TURNS THE WELL-STUDIED LEARNING PROBLEM OF HOW TO DETERMINE ACTIONS TO OPTIMIZE ONE GOAL INTO THE SIGNIFICANTLY MORE COMPLEX PROBLEM OF HOW TO DETERMINE ACTIONS TO OPTIMIZE THE RIGHT GOAL AT THE RIGHT TIME. BUT HOW DOES AN AGENT KNOW WHICH OF ITS GOALS IS IMPORTANT AT ANY TIME? THIS QUESTION IS BEYOND THE CAPABILITIES OF THE CURRENT STATE OF THE ART IN BOTH MULTI-OBJECTIVE OPTIMIZATION (WHICH GENERALLY RELIES ON LINEAR COMBINATIONS OF REWARDS OR PARETO CONCEPTS [8 15]) AND MULTI-TASK LEARNING (WHICH GENERALLY RELIES ON A WELL-DEFINED PROGRESSION OF GOALS [4 3 12]). THE OBJECTIVE OF THIS PROPOSAL IS ENABLE AN AUTONOMOUS SYSTEM TO NOT ONLY LEARN TO CHOOSE WHICH ACTIONS TO TAKE BUT TO LEARN TO CHOOSE WHICH REWARDS TO SEEK TO SATISFY HIGH-LEVEL MISSION GOALS. THE SPECIFIC RESEARCH OBJECTIVES OF THIS PROPOSAL ARE TO: 1. GENERATE DOMAIN SPECIFIC REWARDS AND ENABLE AN AUTONOMOUS AGENTS TO SWITCH AMONG THEM TO IMPROVE ITS LEARNING SPEED AND PERFORMANCE. 2. FORMALIZE THE CONCEPT OF ALIGNMENT AMONG REWARDS TO CAPTURE BROADER SET OF DESIRED BEHAVIORS (SAFETY ASSURANCE EXPLAINABILITY) AND DEMONSTRATE EFFECTIVENESS OF APPROACH IN DOMAINS WITH QUALITATIVELY DIFFERENT REWARDS. THE KEY CONTRIBUTION OF THIS WORK IS TO QUANTIFY THE ALIGNMENT AMONG MULTIPLE REWARD FUNCTIONS SPANNING MULTIPLE TIMESCALES AND USE THAT ALIGNMENT TO DETERMINE WHAT MATTERS WHEN TO GUIDE THE LEARNING OF AGENTS FACED WITH COMPLEX TIME-EXTENDED MISSIONS. THE IMPACT OF THE PROPOSED RESEARCH IS TO EXTEND AUTONOMY AND LEARNING TO COMPLEX SCENARIOS WHERE MULTIPLE MISSION DIRECTIVES CANNOT BE COLLAPSED INTO A WELL-FORMULATED SINGLE REWARD SIGNAL. THIS RESEARCH WILL LEAD TO EXPLAINABLE (BY SPECIFYING WHICH LOCAL REWARD DROVE PARTICULAR DECISIONS) FAULT-TOLERANT (BY HAVING MULTIPLE REWARDS TO STEP UP IF BEHAVIOR BECOMES QUESTIONABLE) AND ASSURED (BY SPECIFICALLY ENCODING ASSURANCE AS A HIGH-LEVEL REWARD) AUTONOMY. $0 7/3/18 Not listed THIS PROPOSAL LAYS THE FOUNDATIONS OF MULTI-REWARD LEARNING WHERE THE COMPLEXITY OF A MISSION PRECLUDES COLLAPSING THE LEARNING SIGNAL TO A WELL-FORMULATED, SINGLE REWARD OR OBJECTIVE FUNCTION. INDEED MANY MISSIONS REQUIRE AN AGENT (OR A ROBOT) TO BALANCE PERFORMANCE, LONGEVITY, AND SAFETY, AS WELL AS PROVIDE EXPLANATIONS FOR ITS ACTIONS. IN ADDITION, EACH OF THOSE HIGH LEVEL GOALS ARE THEMSELVES COMPOSED OF MULTIPLE SUBGOALS. FOR EXAMPLE, OPTIMIZING PERFORMANCE FOR A ROBOT MAY REQUIRE MOBILITY, ENERGY MANAGEMENT, COMMUNICATION, AND OBSERVATION TASKS WHILE, AGAIN, PROVIDING EXPLANATIONS AND ASSURANCES. THIS NEED TO MANAGE A VARIETY OF POSSIBLY IMPORTANT GOALS TURNS THE WELL-STUDIED LEARNING PROBLEM OF HOW TO DETERMINE ACTIONS TO OPTIMIZE ONE GOAL INTO THE SIGNIFICANTLY MORE COMPLEX PROBLEM OF HOW TO DETERMINE ACTIONS TO OPTIMIZE THE RIGHT GOAL AT THE RIGHT TIME. BUT HOW DOES AN AGENT KNOW WHICH OF ITS GOALS IS IMPORTANT AT ANY TIME? THIS QUESTION IS BEYOND THE CAPABILITIES OF THE CURRENT STATE OF THE ART IN BOTH MULTI-OBJECTIVE OPTIMIZATION (WHICH GENERALLY RELIES ON LINEAR COMBINATIONS OF REWARDS OR PARETO CONCEPTS [8, 15]) AND MULTI-TASK LEARNING (WHICH GENERALLY RELIES ON A WELL-DEFINED PROGRESSION OF GOALS [4, 3, 12]). THE OBJECTIVE OF THIS PROPOSAL IS ENABLE AN AUTONOMOUS SYSTEM TO NOT ONLY LEARN TO CHOOSE WHICH ACTIONS TO TAKE, BUT TO LEARN TO CHOOSE WHICH REWARDS TO SEEK TO SATISFY HIGH-LEVEL MISSION GOALS. THE SPECIFIC RESEARCH OBJECTIVES OF THIS PROPOSAL ARE TO: 1. GENERATE DOMAIN SPECIFIC REWARDS AND ENABLE AN AUTONOMOUS AGENTS TO SWITCH AMONG THEM TO IMPROVE ITS LEARNING SPEED AND PERFORMANCE. 2. FORMALIZE THE CONCEPT OF ALIGNMENT AMONG REWARDS TO CAPTURE BROADER SET OF DESIRED BEHAVIORS (SAFETY, ASSURANCE, EXPLAINABILITY) AND DEMONSTRATE EFFECTIVENESS OF APPROACH IN DOMAINS WITH QUALITATIVELY DIFFERENT REWARDS. THE KEY CONTRIBUTION OF THIS WORK IS TO QUANTIFY THE ALIGNMENT AMONG MULTIPLE REWARD FUNCTIONS SPANNING MULTIPLE TIMESCALES AND USE THAT ALIGNMENT TO DETERMINE WHAT MATTERS WHEN TO GUIDE THE LEARNING OF AGENTS FACED WITH COMPLEX, TIME-EXTENDED MISSIONS. THE IMPACT OF THE PROPOSED RESEARCH IS TO EXTEND AUTONOMY AND LEARNING TO COMPLEX SCENARIOS WHERE MULTIPLE MISSION DIRECTIVES CANNOT BE COLLAPSED INTO A WELL-FORMULATED SINGLE REWARD SIGNAL. THIS RESEARCH WILL LEAD TO EXPLAINABLE (BY SPECIFYING WHICH LOCAL REWARD DROVE PARTICULAR DECISIONS), FAULT-TOLERANT (BY HAVING MULTIPLE REWARDS TO STEP UP IF BEHAVIOR BECOMES QUESTIONABLE), AND ASSURED (BY SPECIFICALLY ENCODING ASSURANCE AS A HIGH-LEVEL REWARD) AUTONOMY. Not listed $148.0k 6/6/18 Not listed THIS PROPOSAL LAYS THE FOUNDATIONS OF MULTI-REWARD LEARNING WHERE THE COMPLEXITY OF A MISSION PRECLUDES COLLAPSING THE LEARNING SIGNAL TO A WELL-FORMULATED SINGLE REWARD OR OBJECTIVE FUNCTION. INDEED MANY MISSIONS REQUIRE AN AGENT (OR A ROBOT) TO BALANCE PERFORMANCE LONGEVITY AND SAFETY AS WELL AS PROVIDE EXPLANATIONS FOR ITS ACTIONS. IN ADDITION EACH OF THOSE HIGH LEVEL GOALS ARE THEMSELVES COMPOSED OF MULTIPLE SUBGOALS. FOR EXAMPLE OPTIMIZING PERFORMANCE FOR A ROBOT MAY REQUIRE MOBILITY ENERGY MANAGEMENT COMMUNICATION AND OBSERVATION TASKS WHILE AGAIN PROVIDING EXPLANATIONS AND ASSURANCES. THIS NEED TO MANAGE A VARIETY OF POSSIBLY IMPORTANT GOALS TURNS THE WELL-STUDIED LEARNING PROBLEM OF HOW TO DETERMINE ACTIONS TO OPTIMIZE ONE GOAL INTO THE SIGNIFICANTLY MORE COMPLEX PROBLEM OF HOW TO DETERMINE ACTIONS TO OPTIMIZE THE RIGHT GOAL AT THE RIGHT TIME. BUT HOW DOES AN AGENT KNOW WHICH OF ITS GOALS IS IMPORTANT AT ANY TIME? THIS QUESTION IS BEYOND THE CAPABILITIES OF THE CURRENT STATE OF THE ART IN BOTH MULTI-OBJECTIVE OPTIMIZATION (WHICH GENERALLY RELIES ON LINEAR COMBINATIONS OF REWARDS OR PARETO CONCEPTS [8 15]) AND MULTI-TASK LEARNING (WHICH GENERALLY RELIES ON A WELL-DEFINED PROGRESSION OF GOALS [4 3 12]). THE OBJECTIVE OF THIS PROPOSAL IS ENABLE AN AUTONOMOUS SYSTEM TO NOT ONLY LEARN TO CHOOSE WHICH ACTIONS TO TAKE BUT TO LEARN TO CHOOSE WHICH REWARDS TO SEEK TO SATISFY HIGH-LEVEL MISSION GOALS. THE SPECIFIC RESEARCH OBJECTIVES OF THIS PROPOSAL ARE TO: 1. GENERATE DOMAIN SPECIFIC REWARDS AND ENABLE AN AUTONOMOUS AGENTS TO SWITCH AMONG THEM TO IMPROVE ITS LEARNING SPEED AND PERFORMANCE. 2. FORMALIZE THE CONCEPT OF ALIGNMENT AMONG REWARDS TO CAPTURE BROADER SET OF DESIRED BEHAVIORS (SAFETY ASSURANCE EXPLAINABILITY) AND DEMONSTRATE EFFECTIVENESS OF APPROACH IN DOMAINS WITH QUALITATIVELY DIFFERENT REWARDS. THE KEY CONTRIBUTION OF THIS WORK IS TO QUANTIFY THE ALIGNMENT AMONG MULTIPLE REWARD FUNCTIONS SPANNING MULTIPLE TIMESCALES AND USE THAT ALIGNMENT TO DETERMINE WHAT MATTERS WHEN TO GUIDE THE LEARNING OF AGENTS FACED WITH COMPLEX TIME-EXTENDED MISSIONS. THE IMPACT OF THE PROPOSED RESEARCH IS TO EXTEND AUTONOMY AND LEARNING TO COMPLEX SCENARIOS WHERE MULTIPLE MISSION DIRECTIVES CANNOT BE COLLAPSED INTO A WELL-FORMULATED SINGLE REWARD SIGNAL. THIS RESEARCH WILL LEAD TO EXPLAINABLE (BY SPECIFYING WHICH LOCAL REWARD DROVE PARTICULAR DECISIONS) FAULT-TOLERANT (BY HAVING MULTIPLE REWARDS TO STEP UP IF BEHAVIOR BECOMES QUESTIONABLE) AND ASSURED (BY SPECIFICALLY ENCODING ASSURANCE AS A HIGH-LEVEL REWARD) AUTONOMY. $148.0k 6/6/18