HR001121S0034.pdf

HR001121S0034 KMASS 1

Broad Agency Announcement Knowledge Management at Scale and Speed (KMASS)

Defense Sciences Office

HR001121S0034

June 24, 2021

HR001121S0034 KMASS 2

Table of Contents I. Funding Opportunity Description

A. Introduction B. Background C. Program Description / Scope D. Program Structure E. Technical Area Descriptions F. Schedule/Milestones G. Deliverables H. Government-furnished Property/Equipment/Information I. Other Program Objectives and Considerations

II. Award Information A. General Award Information B. Fundamental Research

III. Eligibility Information A. Eligible Applicants B. Organizational Conflicts of Interest C. Cost Sharing/Matching D. Ability to Receive Awards in Multiple Technical Areas - Conflicts of Interest

IV. Application and Submission Information A. Address to Request Application Package B. Content and Form of Application Submission C. Submission Dates and Times D. Funding Restrictions E. Other Submission Requirements

V. Application Review Information A. Evaluation Criteria B. Review and Selection Process C. Federal Awardee Performance and Integrity Information (FAPIIS)

VI. Award Administration Information A. Selection Notices B. Administrative and National Policy Requirements C. Reporting

VII. Agency Contacts VIII. Other Information

A. Proposers Day B. Frequently Asked Questions (FAQs) C. Collaborative Efforts/Teaming D. Sample Associate Contractor Agreement (ACA) Text

BAA Attachments:

Attachment A: ABSTRACT SUMMARY SLIDE TEMPLATE Attachment B: ABSTRACT TEMPLATE Attachment C: PROPOSAL SUMMARY SLIDE TEMPLATE Attachment D: PROPOSAL TEMPLATE VOLUME 1: TECHNICAL & MANAGEMENT Attachment E: PROPOSAL TEMPLATE VOLUME 2: COST Attachment F: MS ExcelTM DARPA COST PROPOSAL SPREADSHEET Attachment G: PROPOSAL TEMPLATE VOLUME 3: ADMINISTRATIVE & NATIONAL POLICY REQUIREMENTS

HR001121S0034 KMASS 3

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Defense Sciences Office (DSO)

Funding Opportunity Title: Knowledge Management at Scale & Speed (KMASS)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001121S0034

NAICS Code: 541715 Catalog of Federal Domestic Assistance (CFDA) Number(s): 12.910 Research and

Technology Development

Dates (All times listed herein are Eastern Time.)

o Posting Date: June 24, 2021 o Proposers Day: July 7, 2021 See Section VIII.A.

o Abstract Due Date: July 20, 2021, 4:00 p.m.

o FAQ Submission Deadline: August 19, 2021 4:00 p.m. See Section VIII.B.

o Full Proposal Due Date: September 2, 2021, 4:00 p.m.

Anticipated Individual Awards: DARPA anticipates multiple awards for TA B/C/D and one award for TA E.

Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements, or Other Transactions. Award instruments will be limited to procurement contracts and Other Transactions for Proposers whose proposed solution includes Controlled Unclassified Information (CUI).

Agency contacts o Technical POC: Ted Senator, Program Manager, DARPA/DSO o BAA Email: KMASS@darpa.mil o BAA Mailing Address:

DARPA/DSO

ATTN: HR001121S0034

675 North Randolph Street Arlington, VA 22203-2114 o DARPA/DSO Opportunities Website: http://www.darpa.mil/work-with-us/opportunities

Teaming Information: See Section VIII.C for information on teaming opportunities.

Frequently Asked Questions (FAQ): FAQs for this solicitation may be viewed on the DARPA/DSO Opportunities Website. See Section VIII.B for further information.

Security: KMASS will be unclassified. DARPA anticipates that submissions received under this BAA will be unclassified. See Section IV.B.5 for more details.

mailto:KMASS@darpa.mil https://www.darpa.mil/work-with-us/opportunities?oFilter=DSO https://www.darpa.mil/work-with-us/opportunities?oFilter=DSO

HR001121S0034 KMASS 4

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

This Broad Agency Announcement (BAA) constitutes a public notice of a competitive funding opportunity as described in Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016 as well as 2 C.F.R. § 200.203. Any resultant negotiations and/or awards will follow all laws and regulations applicable to the specific award instrument(s) available under this BAA, e.g., FAR

15.4 for procurement contracts.

A. Introduction

The Defense Sciences Office (DSO) at the Defense Advanced Research Projects Agency (DARPA) is soliciting innovative research proposals in the area of collecting, organizing, sharing, and applying organizational knowledge about key tasks and how to perform them.

Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

Organizations, including the military, store massive amounts of knowledge intended for human consumption, e.g., doctrine, policy, procedures, etc. Creating these documents, videos, and other modes is both expensive and time consuming. They are not structured or indexed to support rapid and appropriate application to particular tasks and may be inconsistent or confusing.

Further, this knowledge must be augmented by local and timely knowledge appropriate to the particulars of a task, accounting for any unique context or situations. Capturing the knowledge along with this context as it is created currently requires a dedicated effort on the part of the person performing the task, which often does not happen. Applying stored knowledge requires a user to know that it exists, where it exists, when it is needed and relevant, how to retrieve it, and how to locate the specifics in what typically is a multi-page document or several minute long video with audio. Applying stored knowledge may not always be possible given time constraints.

The Knowledge Management at Scale and Speed (KMASS) program will research, develop, integrate, evaluate, and demonstrate underlying technology that will enable effective use of documented knowledge, acquisition of new knowledge as part of regular workflows, and application of useful knowledge when and where it is required and with necessary granularity.

KMASS technology will scale to a broad set of tasks and contexts across an organization by collecting and modifying knowledge “in-the-flow” as part of regular task execution and applying the knowledge documented for one purpose to other purposes as appropriate. KMASS will deliver user specific knowledge “nuggets” that are useful for a current task—whether the knowledge is requested or not by the user—exactly when needed, while avoiding irrelevant or already known information. This concept is a core tenet of KMASS and may be referred to as the “JustINs” – i.e., just in time, just enough, and just for me1. KMASS systems will contain a persistent knowledge store comprising source documents in human understandable form in multiple modalities (e.g., text, videos, presentations, etc.), augmented with appropriate tags and

1 Rosenberg, M. J. (2001). E-learning: Strategies for delivering knowledge in the digital age. New York: McGraw- Hill.

HR001121S0034 KMASS 5

indexed for identification, retrieval, linking, and application that will update at the speed of task performance. KMASS requires advances in three key complementary areas: Organizing Background Knowledge, Capturing Local Knowledge, and Disseminating Contextualized Knowledge usefully, appropriately, and on time.

B. Background

State of the Practice Intel co-founder Robert Noyce once said “Knowledge is power. Knowledge shared is power multiplied.” In the modern world, the ability to create new knowledge is far outpacing the ability to capture and share it with others who will benefit from its use. However, knowledge capture is inhibited because processes for creating knowledge are separate from processes used to collect it.

To make insights and discoveries available to others, people must disengage from their knowledge-producing workflows and engage in knowledge-sharing workflows such as writing documents or talking with coworkers. Application of documented knowledge is difficult because locating specific knowledge and determining whether and how it applies to the details of a current task are time consuming and often not feasible given constraints of task performance.

Today’s knowledge management (KM) tools focus on storing documents for access across an organization and/or connecting individuals with others who may possess relevant knowledge.

These tools are largely semantic-free, with no understanding of content and, therefore, require continued and extensive curation and maintenance to provide any utility as well as careful and extensive searching to retrieve exactly what would assist with specific task performance.

There are often vast libraries of data both in an organization’s internal systems and in the cloud.

Although tools do exist to extract such data on demand, e.g., key-word searches or personal assistants (such as Google, Alexa, Siri), there is currently no consistent, automatic way for people to acquire just exactly the knowledge they need and no more. Commercial search engines still require a person to read through several retrieved documents to identify, locate, and understand the specific information that would be useful and applicable to the current task. A personal assistant may provide more targeted information, but generally not to the level of understanding required and only in specific domains. Videos and audio tracks provide their own set of constraints because specialized software is often required to retrieve necessary data embedded within them. Combined with the fact that producing these artifacts is time consuming, tedious, and typically only done for the knowledge that is worth the high cost of producing it, organizations lose much of the potential power to which Noyce alludes.

Today’s KM also supports collaboration and other social practices that are oriented towards person-to-person knowledge exchanges within communities of practice (CoPs). For KMASS, CoPs are groups of individuals who have overlapping interests in activities in which their organization participates. They do similar things, and they benefit from sharing knowledge about how these things are done. Person-to-person exchanges of knowledge, whether they be face-to-face conversations, phone calls, emails, or briefings, have many advantages over documents. People can quickly tailor their answers to the details of a given situation and quickly provide additional information that the knowledge seeker may not yet realize they need. A person can answer multiple questions that may be relevant to a given task, or different related task, in ways that stored artifacts cannot anticipate. Through conversation, one can quickly assess the background of a person and limit the knowledge provided to that which the person truly

HR001121S0034 KMASS 6

needs and is directly applicable to the specific situation. Dialogues allow people to refine queries and deliver knowledge in a way that enhances communication and comprehension.

There are limitations to using human experts as knowledge sources, beginning with knowing whom to ask. Over time, people tend to build up a network of contacts, but for newcomers this network is, in the best case, a mentor who needs to take time from his/her other tasks to either answer questions or assist with networking. Even experienced people find themselves needing to go outside of their network on occasion, which can be a difficult process. Identifying the experts in an organization, together with their areas of expertise, and making this information available to others is a focus of the “connecting people with people” branch of current KM practice.

Another issue is providing incentives for people to share their knowledge. Experts are typically highly valued, much-in-demand employees, and it can be a challenge to balance the time they spend helping others against the time spent directly applying their expertise. Finally, this mode of information exchange is not conducive to proactively sharing newly acquired insights, best practices, etc. as they are acquired and developed.

Motivation A motivation for KMASS can be found in the “New Dimensions in Testimony” (NDT) exhibit created by the University of Southern California’s (USC) Institute for Creative Technologies.

(see https://ict.usc.edu/prototypes/new-dimensions-in-testimony/) in collaboration with the USC Shoah Foundation and Conscience Display. Currently being shown at museums throughout the country, this exhibit is based upon hours of video that were taken of Pinchas Gutter, a Holocaust survivor, as he was interviewed about his experiences as a child in Germany. In the exhibit, visitors are able to query a holographic image of Mr. Gutter and get the best match to their question that can be found in the store of videos. NDT was costly to produce, required a specialized room to create the holographic images, and in its current form is static and, thus, the set of questions that it can answer is limited. Nonetheless, it is a compelling example of the power of narrative and the potential for technology to support effective, memorable communication.

Another more familiar motivation can be found in “how to” videos, which are becoming increasingly common on the Internet. Such videos currently comprise the second most common type of video on the popular YouTube site, following only product reviews. The popularity of these artifacts is clear evidence of the value that video demonstrations bring to knowledge consumers. It also suggests that, when using the appropriate modes of input, “teaching while doing” is an easy and natural way for people to communicate their understanding of step-by-step processes. Videos like these are a step towards the KMASS vision, but enhancements are needed.

Like documents, it can be a challenge to find the best, most appropriate video out of a large set of candidates. In their current form, videos are difficult to search. Unlike NDT, people often need to watch an entire clip to find a specific nugget of information. Like NDT, individual videos are essentially static, and there is no easy way to add to them or alter them.

Storytelling and story understanding provide a third motivation for KMASS. Stories are a natural way for people to convey information. They increase comprehension and retention.2 When

2 Dalhstrom, M. (2014) “Using narratives and storytelling to communicate science with nonexpert audience,” Proceedings of the National Academy of Sciences, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4183170/.

https://ict.usc.edu/prototypes/new-dimensions-in-testimony/

HR001121S0034 KMASS 7

directly compared to expository text, narratives were read twice as quickly and recalled twice as well.3 Such stories would link to specific sources and backup information to clarify contexts in which they apply and would be delivered just when needed to illustrate a point.

Illustrative Example This example illustrates some of the real-world knowledge management issues that KMASS will overcome. This example is NOT intended as a use case for KMASS research, evaluation, or application; it is simply illustrative of some of the issues KMASS research must address.

(KMASS research domains will be proposed by research teams and need not, nor are they expected to, be in a military context; KMASS evaluation and application domains will be determined at a later time by DARPA, as described later in this BAA.)

Captain Jones is an Army captain deployed to an Attack Helicopter Battalion as an individual augmentee. When he is informed of his assignment, he is told that he is lucky: he will have a full two weeks of overlap with his predecessor. Since he knows the basic unit Tactics, Techniques, and Procedures (TTPs)4 for helicopter operations, he plans to spend that time learning the most important facts and details of his predecessors’ position. When he arrives in theatre, the man he is replacing, Captain Smith, greets him and wastes no time, immediately telling stories about the job, including a lot of nuts-and-bolts details and stories about when things went wrong or didn’t go as planned. CPT Jones feels like he is drinking from a firehose, as CPT Smith goes on. He realizes that he can’t fully appreciate all the many details, exceptions, and one-offs, because he really doesn’t fully grasp the simpler, probably more basic information that CPT Smith hasn’t spent much time explaining. He knows that CPT Jones is trying to pass on important lessons learned, so he does his best to write it down and hopes he can make sense of it later.

The following day, he takes his first tour of the region. CPT Smith takes him by the mess hall and introduces him to some of the “who’s who” on base. He meets a handful of civilian contractors and is told what they do and how they can be most helpful. CPT Smith’s flight schedule prevents him from being able to hang around as long as he would like, so CPT Jones does his best to memorize names and makes a mental note to stop by after he is more acclimated to the position.

As he follows CPT Smith to the copter, CPT Jones takes mental notes of his surroundings at the same time he is trying to keep track of the new facts and details that CPT Smith provides as they walk. The two officers spend the rest of the day flying to the Forward Operating Bases (FOBs) of the ground forces that the battalion supports. CPT Smith points out features of the geography:

mountainous areas, passes, settled areas, operating boundaries, etc. At one point, CPT Smith points to a small village in the distance and says that it is best to avoid flying near it, as the locals have a habit of filing noise complaints with the Brigade. Over the course of the day, CPT Smith unloads a wealth of knowledge that gets jotted down in CPT Jones’ notebook.

Two days later, CPT Smith tells CPT Jones that it’s his turn to fly and then quizzes him as they retrace the route. As he pilots the helicopter, CPT Jones appreciates the limitations of his

3 Graesser A., & Ottati V. (1995) “Why Stories? Some evidence, questions, and challenges”. Knowledge and Memory: The Real Story, ed Wyer RS, Lawrence Erlbaum Associates, Hillsdale, NJ.

4 TTP’s incorporate the evolving knowledge and experience required of a soldier by capturing the “who”, “what”, “where”, “when”, and “how” of warfighting.

HR001121S0034 KMASS 8

notebook, which is inaccessible. He is thankful that he remembers to avoid the noise-averse village, but he is dismayed by how much of their previous trip’s content he had not yet absorbed.

CPT Smith assures him that he’ll pick up everything he needs to know quickly enough.

The two weeks of overlap pass quickly. Just as CPT Smith has predicted, CPT Jones did eventually learn what he needed to know, though there were many times during his tour when CPT Jones recognized that the costs of some of the lessons he learned could have been lower. As his year-long tour is coming to an end, he decides that rather than have his successor rely solely upon the sort of quick education that CPT Smith gave him, it would be better to write down the important things learned over the year for his successor’s benefit. Perhaps this could become an artifact that grew over time. For CPT Jones, it turns out to be a very difficult task. He can’t remember all the things that he didn’t know at the time he arrived and needed to know post haste. He no longer has the notes from the few days spent with CPT Smith, and the only specific things that he can remember are some of the flying restrictions that CPT Smith mentioned on that first side-by-side flight, together with the fact he had introduced him to a lot of people, and he had needed to learn a lot of names fast. CPT Jones struggles with how to turn those memories into easily digested, usable written guidance. In the end, he has forty pages of written material, much of which is just contact information. A lot of the advice amounts to exceptions from routine processes that aren’t explained. CPT Jones realizes that his replacement will likely not fully appreciate these routine processes when he arrives, but he doesn’t want to write them down, because he recognizes that he is already planning to hand off quite a bit of reading.

Knowledge Management Issues The example presented above illustrates some of the current limitations with respect to today’s KM technology and suggests potential improvements to be provided by KMASS.

While not explicit in the story above, it is assumed that there was much written documentation available that CPT Jones never had time to look at, let alone understand in detail. This might have included briefings, organization charts, personnel directories and biographies, contracts for support personnel, mission plans and reports, geographic or demographic background, technical documentation, or many others. These documents may have been in paper or electronic form, may have been located in different file cabinets in different buildings, may not have been clearly labeled, may have been in different formats, and may not have been known or accessible to him.

Most important, they were not organized according to a common structure, so what he needed to know was hidden in portions of different diverse documents and required him to locate the relevant portions and mentally link facts across multiple documents. Much of what he tried to learn may have been available in these documents, but even if he had known where to look, he would not have had the time.

Both CPT Smith and CPT Jones learned a lot during their tours; these lessons could have benefited other helicopter pilots in other theaters at later times. After CPT Smith departed, he was extremely busy with his next assignment. Consulting with CPT Smith would have been valuable to CPT Jones to review each mission and as an instructor for Lieutenants who had not yet been assigned to theater. But the demands of CPT Smith’s next assignment meant that these potential benefits were unrealized.

The Army, Navy, Air Force, and Marines all use variants of the Sikorsky UH-60 Black Hawk helicopter with different modifications that provide different capabilities and equipment to

HR001121S0034 KMASS 9

support different mission sets. Sharing knowledge about how to employ these capabilities is a challenge. In general, individuals working together in the same Army Battalion will find it easier to locate each other than individuals assigned to Army units in different geographic locations or individuals in other Services. The Army has roughly 50 helicopter Battalions, the Navy at least 40 Squadrons, and the Air Force over 16 helicopter Groups, spread across the country and throughout the world. KMASS systems would have to scale to organizations of this size, covering the scope and diversity of their background knowledge sources and missions. While their missions and deployments are not identical, there would be useful information that could be shared between individuals with the same role in different Battalions, Squadrons, etc., provided that the contexts are clear to avoid misapplication of lessons learned.

C. Program Description / Scope

KMASS requires advances in three key complementary technical areas (TAs): Organizing Background Knowledge (TA B), Capturing Local Knowledge (TA C), and Disseminating Contextualized Knowledge usefully, appropriately, and in a timely manner (TA D). A fourth Evaluation TA (TA E) will be focused on providing the Government with independent assessments of progress for the TA B/C/D research development efforts. While proposers may choose to submit separate proposals for both TA B/C/D and TA E efforts, a team may be selected as a performer only on either TAs B/C/D or TA E.

KMASS research under TAs B/C/D will be conducted by integrated teams that simultaneously address the objectives of all three TAs. It is essential for the success of the program that performers address them simultaneously to achieve the vision of a complete integrated KMASS system because: (1) advances in each area that do not interoperate will not achieve the KMASS vision, (2) component technologies may span multiple areas (e.g., storytelling and story understanding) and have their own deep representations that enable both collection and dissemination of contextualized knowledge, and (3) lightweight knowledge representation (e.g., tagging and the indexing) enables persistent storage of knowledge in human-consumable form as well as interchange between different deep representations. While the research goals of each TA are distinct, component solutions that do not address all of them together will not achieve the goals of the program. For these reasons, R&D teams will need to assemble end-to-end pipelines, including a persistent knowledge store and any other required infrastructure components, to demonstrate their ability to meet program goals. Proposals must justify their selection of any existing technologies, their research approaches for augmenting existing technologies, and their research approaches for new technologies for all components of an integrated KMASS system.

KMASS TA B/C/D research teams will propose their own research domains. Examples of what might constitute such a domain are appliance repair, IT support, first aid, accounting, scientific research, etc. Evaluations will be performed internally by each team in their research domain and program-wide in domains proposed by the TA E external evaluation team, described later in the BAA, and approved by DARPA. Evaluations will measure progress against the objectives of all three TAs. Proposals must justify their selected domain by explaining how its characteristics map to KMASS objectives qualitatively and quantitatively and must explain how their proposed solution will generalize to other domains.

DARPA has identified a set of program-wide metrics against which progress toward TA B/C/D objectives will be assessed. Relative merits of different technical approaches will be compared

HR001121S0034 KMASS 10

throughout the course of the program. These metrics correspond to the specific objectives for TAs B/C/D and emphasize scale and speed. They are further elaborated upon in Section I.E Technical Area Descriptions and shown in Figure 3 of Section I.F Schedule/Milestones. For TA B, DARPA is interested in solutions that can deliver knowledge in increasingly finer granularity, even as the size of the corpus increases. For TA C, the primary metric is the amount of additional time or effort it takes a knowledge producer to correctly add knowledge to a KMASS system while he/she is engaged in another task. For TA D, there is a set of metrics associated directly with the JustINs and another to measure the time distraction caused by a knowledge consumer’s need to interact with a KMASS system.

To focus the research efforts, KMASS will address the use case of an individual newly assigned to a job. He/she may have general knowledge (e.g., from formal education or training) about the requirements of the position (i.e., is qualified for their position) but may not know the specifics of how the job is performed in a particular unit in a particular location under particular constraints and in particular circumstances. He/she may overlap with the person he/she is replacing for a short period of time, or may not; and the person being replaced may or may not be available for consultation after he/she departs. This replacement must learn how to function effectively as quickly as possible. Rotation and retirement use cases are generalizations of this fundamental use case and may be addressed in later phases of the program, depending on technical progress.

DARPA anticipates that effective KMASS solutions will require advances in and incorporation of multiple technologies into a complete KMASS system. DARPA intends KMASS systems to be integrated sets of component technologies that function cooperatively to achieve its goals.

DARPA has identified the following technologies as potential contributors to KMASS systems.

This list is representative but not prescriptive or exhaustive. Proposers are encouraged to suggest and justify alternatives that they believe will be more effective.

Storytelling and Story Understanding. Narratives are an effective way to communicate, and there is a well-established body of research focused upon understanding and/or generating narratives. Good stories have well-understood structure, which can be used as a basis for storing, composing, and accessing knowledge content.

Semantic processing. Knowledge representation and reasoning techniques are well understood and widely applied. They provide a robust ability to represent axiomatic knowledge in ways that support seamless integration with existing content that can be adapted and applied to the specifics of current circumstances.

Large scale contextual models. Recent advances in machine learning have demonstrated an impressive ability to generate text in response to one-word inputs and questions and in conversations and even to generate computer code in response to a short natural language description. The underlying deep-learning technology, known as transformers, can find important associations in large, unstructured corpora. While thus far focused primarily upon text processing, this ability to find important associations suggests a promising approach to inferring context from multiple data streams.

HR001121S0034 KMASS 11

Natural language processing, particularly speech processing, is mature, proven, and affordable. This supports the goal of seamless, in-the-flow capture and dissemination of knowledge.

Process modeling. There is a long history of modeling processes and making inferences from those models, both with respect to which process steps are active and which steps should be taken next. This offers a foundation for representing “how-to” information in a way that will support reasoning about tasks, subtasks, and goal achievement.

Multimodal communication technologies. The explosion of small, affordable sensor technology has created the ability to acquire information from a variety of data streams.

A KMASS system may leverage such technology to assist with the challenge of capturing knowledge, understanding the context in which it is being produced and/or applied, and communicating seamlessly with consumers of knowledge as they perform their tasks.

Out of Scope Research in the following areas is considered out-of-scope for purposes of KMASS:

• Knowledge representation and reasoning (KR&R) – KR&R seeks to enable machines to reason in ways that people already can and do; KMASS is focused upon helping people share the knowledge that they have with other people

• Social incentives (e.g., techniques to encourage users to provide or maintain organizational knowledge)

• Approaches to creating and managing effective teams – KMASS systems provide the benefit of experience to individuals in the same role in different teams, rather than support for teams comprising multiple individuals in distinct roles

• Manual knowledge acquisition techniques that don't scale

• Improvements to enabling technologies (e.g., natural language processing (NLP)) that aren’t specifically in support of and integrated with a KM system

• Computer/Human teaming

• Personal Assistants

• Tacit knowledge elicitation (i.e., knowledge that experts have difficulty in articulating, such as how to make value judgments)

• Distributed systems for knowledge sharing. While individual users can be expected to have workstations or some other form of personal computing device to support user interactions and data collection, KMASS systems will use server-side approaches to persisting and sharing knowledge.

D. Program Structure

KMASS is organized around four TAs: Organizing Background Knowledge (TA B), Capturing Local Knowledge (TA C), Disseminating Contextualized Knowledge (TA D), and Evaluation (TA E). Research teams will address TAs B, C, and D; and a single evaluation team will address TA E. While proposers may choose to submit separate proposals to TAs B/C/D and to TA E, an individual team may be selected for an award only under either TAs B/C/D or TA E. Figure 1 summarizes TA B/C/D problems, objectives, and key challenges. TA E focuses upon assisting with the evaluation of technical progress and is not depicted in the figure. This figure is not intended as a design framework or specification for a KMASS system.

HR001121S0034 KMASS 12

Figure 1 Technical Areas B, C and D

Program Phases and Schedule: KMASS will be a 36-month program, comprised of three successive phases. Phases 1 will be 14 months, Phase 2 will be 12 months, and Phase 3 will be 10 months. Proposers should propose a base effort for Phase 1 and separate options for both Phases 2 and Phase 3. Participation in Phases 2 and 3 is contingent upon successful performance in prior phases as well as availability of funds and Government priorities.

In order to avoid potential funding gaps between decisions regarding progression from one phase to the next and execution of contract options, a decision on whether to continue individual teams’ efforts into Phase 2 will be made at roughly month 12. Decisions regarding progression into Phase 3 will be made at roughly month 24 of the program, which will be the 10th month of Phase

2. The final two months of both Phase 1 and Phase 2 will be used to prepare for the subsequent phases by refining research plans and improving software baselines. For all performers, a final report will be due 60 days after the last phase in which they participate.

TA B/C/D research performers are expected to conduct formal self-evaluations of their efforts at roughly months 5 and 11 of Phase 1 and at later times as they deem useful and appropriate to measure progress toward their research goals and to support design decisions. In Phases 2 and 3, program-wide evaluations will be conducted by the TA E contractor. These will be staggered during the phase, with the expectation that each TA B/C/D team will be evaluated twice in each phase.

TA B/C/D research teams will conduct self-evaluations in their own research domains in Phase

1. If a TA B/C/D proposal anticipates the need for HSR as part of their internal evaluations, they must comply with the approval procedures detailed in Section VI.B.6, to include providing the information specified therein as required for proposal submission. A draft protocol submitted to the Institutional Review Board (IRB) must be included as an attachment to their proposal. (This attachment will NOT count against the proposal page limit.)

file:///C:/Users/vbrowning/AppData/Local/Microsoft/Windows/INetCache/Content.Outlook/K99LWG5L/in

HR001121S0034 KMASS 13

The TA E performer will not be conducting evaluations in Phase 1. During Phase 1, TA E will be responsible for reviewing the TA B/C/D evaluation plans and structures, observing TA B/C/D evaluations, and independently reviewing the results produced by the TA B/C/D researchers to support DARPA’s assessment of their performance in this phase. More details on this plan are provided below in the descriptions of the TAs.

During Phases 2 and 3, the TA E performer will use its own corpora to independently assess each of the TA B/C/D teams’ systems. DARPA expects that these evaluations will require HSR, and TA E proposers must submit their HSR plans in compliance with the guidance provided below in Section VI.B.6. DARPA anticipates that the evaluation of each TA B/C/D effort will need to occur over several days or weeks and recognizes that it is not reasonable for all evaluations to be conducted in parallel. TA E proposers should provide their plan for maximizing the fairness of evaluations, and TA B/C/D proposers must recognize that they should not plan on being the last performer evaluated during any one cycle.

KMASS will also have a military engagement effort that will be led by a Government laboratory, Federally Funded Research and Development Center, or University Affiliated Research Center.

This BAA is not soliciting proposals for this effort. This effort will work with interested parties in the U.S. Armed Services to develop a suitable demonstration of TA B/C/D technologies in a domain of interest to the DoD. While the bulk of this effort will be conducted by the organization chosen to lead this effort, TA B/C/D and TA E performers should expect to support this effort in Phase 3. TA B/C/D teams will need to make their technology available for demonstration, and the TA E team should assist with the preparations for the effort, based upon their experiences with setting up evaluations in Phases 2 and 3.

E. Technical Area Descriptions

Figure 2, described in the subsections below, shows a notional plan for the evolution of technology through the life of the program. These are examples and not requirements.

Proposers should provide their own detailed, constructive plan for research and development of KMASS systems leading towards program goals and provide their own phased vision for how they expect their technology to evolve over the three years of the program, addressing all the aspects depicted in the figure (i.e., TAs B/C/D or E, knowledge source and delivery formats, knowledge producer/consumer levels). The descriptions of each phase in each TA are qualitative and correspond to the quantitative metrics for the program discussed later in the BAA. From this perspective, only the three-phase structure, the expectation of two evaluations per phase, and the division between the TA B/C/D and TA E teams should be viewed as requirements.

HR001121S0034 KMASS 14

Figure 2 Program Structure and Schedule

The sections below outline program objectives for TAs B, C, D, and E, respectively. For each TA, an example is given for a notional development pathway, corresponding with Figure 2.

Anticipated program-wide measures of technical progress for the TA are identified.

Technical Area B: Organizing Background Knowledge This TA focuses upon processing existing multi-media knowledge sources into lightweight representations that support fine-grained dissemination. For KMASS program purposes, these sources will contain knowledge relevant to executing tasks, such as instructions, manuals, textbooks, recipes, how-to videos, etc. Corpora may also contain domain-relevant content that is not directly relevant to executing tasks but is representative of the diverse sources of information that someone working in a domain might need to be familiar with. The key challenge for this area will be to index and restructure sources to enable preservation and management of content at the “nugget” level.

Envisioned Technology Development Path This TA is about developing technologies for managing the large set of extant artifacts that comprise generally available background knowledge. Managing evolving content is also of interest. For example, when a policy document is updated, the old policy document should not be offered as a source of knowledge. Similarly, if the procedure for requesting meeting space in an organization changes, the new procedure doesn’t simply take precedence over the old one; it replaces it completely.

Document processing and indexing is in general more mature than similar processing of video, with audio being somewhere in between. For documents, the anticipated evolution of granularity will start with a state-of-the-art ability to find relevant documents in response to queries and then

HR001121S0034 KMASS 15

proceed to isolating the most pertinent section of the document, followed by delivering the exact snippet of knowledge that a user requires.

For audio and video, the current state of the art is based upon metadata, such as titles and keywords, that content developers supply along with the artifact. Notionally, as shown in Figure 2, during Phase 1 TA B researchers may demonstrate their ability to suggest videos based upon these metadata descriptions. In Phase 2, they could proceed to finding appropriate artifacts based upon contents not described in the metadata. By Phase 3, TA B might be able to find specific segments of audio and video that answer user queries.

The “sources” line across the top of the TA B/C/D section of Figure 2 captures the expectation that researchers will begin with better understood, more readily available sources of inputs, such as documents and existing videos, and add less-well-understood input modalities, such as cameras and microphones, and then personal communication devices such as cell phones, over the course of the program. While documents and videos are primarily sources of knowledge, the additional modalities may be useful to elicit the contextual information needed to tailor information to consumers’ needs.

Evaluation Organizing and indexing extant content is necessary, but not sufficient, for effective dissemination. TA B proposers must provide plans for demonstrating the efficacy of their technical approach independent of their TA D dissemination technology. Separate TA B assessments will be conducted for each mode of content (text, video, etc.) Engineering interfaces that support on-demand queries will suffice for this purpose.

From the perspective of KMASS evaluations, TA B software runs offline prior to and independent of being made accessible to consumers. If proposers expect to require special-purpose hardware to support this process, they must make this clear in their proposal, and they must explain how they intend to coordinate with TA E to ensure that evaluations involving human subjects can be conducted entirely at the TA E site.

Metrics The ability of TA B software to locate accurately knowledge in extant corpora is the focus of this TA. For this reason, the granularity of knowledge delivered is the primary metric.

Another important dimension for this TA is the size and diversity of the corpus that a team’s TA B technology can manage. DARPA expects to measure the number and size of the input artifacts during each program phase and expects the volume to grow by at least an order of magnitude over the life of the program.

Technical Area C: Capturing Local Knowledge This TA will develop technology to make knowledge collection easy and natural, while minimizing distractions from primary tasks. In addition to computer-based graphical interfaces, this TA will incorporate multi-media sources to collect both knowledge and additional context from knowledge suppliers. The objective of this TA is to enable the capture of new knowledge in real-time with minimal additional effort.

This TA encompasses the set of activities needed to support the addition of local knowledge inside of organizations. The line between background knowledge and local knowledge can be a

HR001121S0034 KMASS 16

fine one, but for KMASS purposes assume background knowledge is supplied as entire artifacts (e.g., documents or videos). This TA seeks to develop technologies that support the incremental addition of knowledge.

TA C technology should capture the detail and context necessary for KMASS systems to subsequently determine if and when specific knowledge should be applied. For example, suppose that a user’s task is editing a periodic report and organizational rules (from background policy documents) state that all acronyms must be spelled out when they are first used. KMASS systems should be able to add a verbal annotation in which a user states “in divisional reports, we are allowed to use the XYZ acronym for our divisional name without spelling it out.”

At a minimum, KMASS systems need to store this annotation and be able to retrieve it if/when someone asks a KMASS system why the XYZ acronym was not spelled out in this document.

Preferably, KMASS systems would have some ability to determine which reports were divisional reports and, thus, also be able to associate this annotation with those reports as well. Exactly how TA C researchers do this will depend upon their proposed technical approaches. Researchers may consider such approaches as natural-language dialogues, visual affordances, or instrumented applications to acquire additional context.

Envisioned Technology Development Path The “Capture” line for TA C in Figure 2 shows a likely progression through program phases along two dimensions. One of these captures a notional evolution of the complexity of the knowledge that a user will be able to provide, from simple statements of facts in Phase 1, to modifying existing workflows in Phase 2, to completely overhauling the description of how something is done in Phase 3.

The second Capture dimension reflects an evolution of the input modalities available to knowledge producers, starting with the ability to annotate artifacts with audio tracks in Phase 1, progressing to stationary cameras for video fragments in Phase 2, and proceeding to mobile devices in Phase 3.

Evaluation For Phase 1, TA C should assume that knowledge producers are at the level of the expert whose knowledge should be trusted, both in terms of the content being provided and the expertise required to use the available tools to put that knowledge in the correct place. In practice, DARPA expects these people to be TA C engineers or other staff working directly with the research team.

The level of expertise will progress in Phases 2 and 3, but for those evaluations, users will be supplied by the TA E contractor.

Metrics Consistent with the objective of making collection easy and natural, DARPA has identified tracking the time that it takes new users to add additional nuggets of context to be an important, program-wide metric.

Technical Area D: Disseminating Contextualized Knowledge This TA focuses upon developing the technology and techniques needed to make knowledge delivery satisfy the JustINs. It seeks to recognize consumers’ specific knowledge requirements and assemble exactly what is needed in a way that can be easily understood.

HR001121S0034 KMASS 17

TA D techniques for effectively disseminating content will depend heavily upon how background knowledge is organized in TA B and how TA C manages the addition of local content and contextual information. For this reason, the proposal for TA D must provide the researchers’ vision for how these components will interact to support the goal of targeted delivery of content. The TA D proposal should clearly explain the plan for managing information about users of the system, including things like their background knowledge, previous tasks, current tasks, usage history, preferences, feedback, etc., so that the proposed TA D solution will be able to estimate the knowledge that an individual user needs and when they need it.

TA D should also address the envisioned user experience for knowledge consumers, including plans for managing the form of delivered content (text, audio, video, imagery) and how the proposed technology will choose between these, techniques for minimizing intrusion, plans for allowing the user to influence the behavior of the technology, and balancing the trade-offs between on-demand (pull) and proactive (push) delivery of content.

Envisioned Technology Development Path The “Disseminate” line of Figure 2 shows a notional progression of the kinds of content TA D will need to deliver, starting with relatively simple facts from the background knowledge and progressing through general knowledge about how things are done locally to very specific, detailed content determined to be the best fit from a set of similar alternatives. The “Delivery” line across the bottom of this section reflects the expectation that over the course of the program, technology will evolve from user-driven questions (pull) to more proactive delivery of content.

Evaluation The Phase 1 evaluation, which will be conducted by TA B/C/D performers as part of their TA D efforts in their chosen domain, should assume that knowledge consumers are novices that know very little about the domain or how the organization conducts its business. In Phase 2, consumers are expected to be at the journeyman level and at the expert level in Phase 3. Note however, that Phase 2/3 evaluations are going to be conducted by TA E.

To support an assessment of the time required to interface with TA D technology in Phase 1, the TA D research team should produce baseline results for how long and how well an individual with no access to technology takes to complete the tasks used for in-the-flow evaluations.

Metrics Program-wide metrics for TA D have been identified for each of the JustINs as well as the speed of delivery:

Just in time: Time required to supply a correct answer on demand Just enough: The number of actions a user needs to take to filter a solution Just for me: The percentage of delivered answers that are relevant Speed: Time to complete task baseline improvement

Technical Area E: Evaluation The role of the TA E performer is to provide the Government with independent assessments of progress of the TA B/C/D technologies throughout the course of the program. The primary challenge with performing this role is going to be developing and conducting assessments of TA B/C/D promise, progress, and performance without relying solely upon a single common set of

HR001121S0034 KMASS 18

problems and measures defined by TA E. DARPA expects each selected TA B/C/D technology approach to have various strengths and weaknesses. The TA E performer will be assisting DARPA with understanding the relative merits and shortcomings of the technologies being developed for the program.

This implies a need for TA E to have some understanding of the TA B/C/D technologies in order to develop appropriate technology assessments. DARPA has structured the program to assist with this. In Phase 1 of the program, TA B/C/D researchers will conduct evaluations of their own design, using corpora and application domains that they have chosen to help them assess their own progress and demonstrate the potential value of their technology.

As an important…