03.Translator User Interface SOW 2021-06-10.pdf

Statement of Work Biomedical Data Translator

User Interface

STATEMENT OF WORK

6/10/2021

GENERAL INFORMATION

Title of Project:

Translator User Interface Development

Statement of Need and Purpose:

As the pace of scientific research continues to advance, the problem researchers face now is not a lack of information, but pairing the right data with the right question. The vision of Translator is to help researchers more easily see connections across those data, accelerating discovery and getting more treatments to more patients more quickly. Ultimately, Translator will accomplish this by serving as a resource for computationally assisted exploration of knowledge and construction of new research hypotheses.

As the Translator program enters the Development phase, the design of the interface is critical/imminent to ensure that the development of the core system supports the needs of the users. The awardees of the Biomedical Data Translator: Development funding opportunity are continuing to develop integrated tools that connect diverse types of data to augment human reasoning and inference for understanding the pathophysiology of human disease. Individual teams have subject-matter experts that utilize components or tools to validate findings, however there is no user interface for Translator. Through the Translator User Interface Contract, the program will develop an interface concurrently with the consortium developing the core system.

Background Information and Objective:

The Translator program completed its three-year feasibility assessment phase in January of 2020. The feasibility assessment phase that generated demonstration projects focused on dismantling barriers to integrating diverse biomedical data. Cross-team working groups formed to integrate many disparate data sources which drove the emerging knowledge graph standards. The teams focused on developing mechanisms to transform data, apply reasoning and enable standard communication messaging, resulting in the successful demonstration that questions could be posed as query graphs across disparate data sources and answers were returned as compliant with the graphs. The feasibility phase focused solely on proof-of-concept, and teams were not assigned to develop a user interface or make considerations for usability.

As these efforts progressed, the program focused on augmenting human reasoning with Translator resources. This effort resulted in several tools which support chaining queries together, as well as accepting various Translator knowledge sources as inputs as defined by the reasoner standard Applications Programming Interface (API), to provide the user with a dossier of the most highly relevant information gathered from multiple sources of data and knowledge.

As tools were developed, they were registered in order to allow Translator components to discover and access each other. The reasoning and architecture teams created a feedback loop which helps to identify gaps in the data as well as integrate disparate knowledge sources to demonstrate that a system can semi-autonomously respond to queries and support exploration of important classes of diverse translational research problems.

http://www.ncats.nih.gov/translator https://ncats.nih.gov/files/NCATS-Translator-FY20-COMBINED-FOA-FINAL.pdf https://github.com/NCATS-Tangerine/kgx https://github.com/NCATS-Tangerine/kgx https://github.com/NCATS-Tangerine/NCATS-ReasonerStdAPI https://smart-api.info/registry?q=translator&tags=translator,reasoner

In order to convey the breadth of the capabilities that were evaluated during the feasibility assessment phase, the cross-team working groups produced case studies on specific examples called Tidbits. Each Tidbit is a vignette that tells a compelling story and demonstrates a kind of insight that could be enabled by Translator.

The feasibility assessment phase demonstrated that Translator can accelerate translational research efforts in a manner that is fully transparent and open source. The awardees of the Biomedical Data Translator: Development phase will continue to fill gaps in data and knowledge used, reduce Translator’s dependence on manual analysis, and increase the breadth of data and speed with which users are able to uncover meaningful results to inform or generate hypotheses. The vendor of the User Experience Lifecycle for the Development of the Translator User Interface contract will deliver an intuitive user interface for biomedical researchers to explore knowledge and construct new research hypotheses.

For the back-end Development phase, the funded teams are responsible for the implementation of three components which are described in the Biomedical Data Translator: Development funding opportunity announcement.

1. Knowledge Providers seek out, integrate, and provide high-value AI-ready data sources within a specific scope of knowledge relevant to Translator.

2. The Autonomous Relay Agents consists of agents that determine which knowledge providers to invoke in response to a query from the Autonomous Relay System. The ARS will be built by NCATS and will provide the primary interaction between the user interface and Autonomous Relay Agents.

3. Translator Standards and Reference Implementation component manages the development of the Translator standards as well as utilities that are integrated into many knowledge providers and autonomous relay agents..

A fourth component is the user interface to be developed under this contract that leverages the input and output of the Autonomous Relay System in accordance with the standards and reference implementation.

The Autonomous Relay System and other components are graph-based and, in response to a query graph, will generate multiple answer graphs depicting a summary of relevant relationships discovered from searching over 100 sources of data and knowledge. The provenance of the asserted relationships will also be associated with the graphs. By drawing from so many different data sources, these computer-generated representations of knowledge go far beyond a search of indexed information. At the same time, just as with searches of indexed information, each query will result in numerous answers graphs. Our objective is to develop a user interface that facilitates biomedical researchers interactively exploring Translator resources and constructing new research hypotheses while ensuring that the emerging standards and development of the Translator component tools support user needs.

User Experience Principles https://ncats.nih.gov/tidbit/tidbit_01.html https://ncats.nih.gov/files/NCATS-Translator-FY20-COMBINED-FOA-FINAL.pdf

The development of a user interface poses several distinct design challenges for the vendor, including but not limited to:

- the presentation of multiple complex knowledge graph “answers” in response to a single query;

- establishing confidence in the ability of Translator to address the user’s question by providing provenance of the knowledge returned to the user;

- capture the collection and history of user decisions used to formulate, iterate, filter and merge results as they develop their query; and

- gather feedback from the user.

Translator requires an intuitive interface that encourages trust while enabling the user to consume complex, diverse results. The intended users of Translator – biomedical researchers

– typically have a set of sites and computational tools they use that allow them to explore information in their area of research. Since these users may be super or power users of their current toolset, the interface for Translator will need to familiarize biomedical researchers with different data types from diverse sources that are the result of connections uncovered by Translator. The interface should engage users without being overwhelming or intimidating.

The answers and supporting evidence that Translator will present may be novel and unfamiliar to biomedical researchers, however Translator must present these results in a way that invokes both trust and curiosity while also enabling and encouraging the user to explore (incrementally and wholly) and iterate. The interface must allow the user to formulate their intent and iteratively increase the relevance of the knowledge returned for their questions. Ultimately, we expect “answers” to be critically evaluated by our users, and want to solicit and capture feedback on both systematic and specific complications that are encountered. These obstacles will need to be communicated back to the development teams to address and iterate on.

Translator knowledge providers contribute deep expertise within the different scientific domains that constitute translational research as well as technical expertise to take complex data and turn it into knowledge. The data represented by the knowledge providers ranges from basic science to clinical data and includes gene products, genomics, model organisms, tissues, chemicals, pathways, conditions and patient groups. The autonomous relay agents employ novel analytic approaches that integrate knowledge across the different knowledge providers.

The vendor for the user experience will complement the existing teams by providing a foundation of deep user research experience, creative design history, dependable testing approaches, and technical development expertise that has been applied to the specific challenges of biomedical data and translational research.

The Translator website will provide biomedical researchers a place to ask questions. The website will utilize the autonomous relay system to communicate with and access multiple graph “answer” sets delivered by the autonomous relay agents and knowledge providers. Result sets are available in the form of knowledge graphs, which provide a variety of answers and supporting evidence that range from known knowns to surprising and radical unknowns, though Translator need not present these answers in classic knowledge graph form. Biomedical researchers will need a place to explore the results and the processes and data sources used to generate the answers. The processes and data are complex analysis implementations and rich biomedical data objects which the user may be unfamiliar with and that will require thoughtful presentation and explanation in order to enable the user to trust the methods and further their research. These rich results will need to be presented in such a way that the user can meaningfully decode this abstract information. The visualizations presented will need to optimize cognitive fit and maximize users’ working memory.

While the project methodology will drive the overall framework, NCATS anticipates that the deliverables will be conducted for each interface (question asking, answering, exploration, iteration, feedback, and collaboration with other users). Translator backend development will iterate in parallel with the interface as needed. The vendor will need to demonstrate flexibility in order to change direction based on the priority and availability of the program.

Challenges that have been identified:

- Ability to explore results (large data sets), currently represented as knowledge graphs

- Transparency of the methods and sources that contribute to Translator answers

- Inherently skeptical users

- During the design iteration process, orienting research participants to a new type of representation of biological data connections and relationships

- Inability of users to precisely express the intent of their question and need

- Evolving Translator standards

- Divided attention of program staff

SCOPE OF WORK

General Requirements:

Independently and not as an agent of the Government, the Contractor shall furnish all the necessary services, qualified personnel, material, equipment, and facilities, not otherwise provided by the Government as needed to perform the Statement of Work below:

Specific Requirements:

The contractor shall work with NCATS and parts of the Translator development consortium (currently fifteen different teams) to develop a user interface for biomedical researchers to interact with Translator.

Base Period:

1. Develop project plan

The Contractor shall meet the team and finalize the draft project plan submitted with its proposal.

2. User Research

The Contractor shall perform user research to support the design of the user interface.

o Proposals shall provide a specific description of research methodology that will be employed o Results of the user research shall include a living, iteratively-updated document of requirements

3. User Interface Design

The Contractor shall design a user interface(s) based on the user research insights and approval by the consortium o Design shall include a notional roadmap to guide development from minimally viable product through beta, v.1, and v.2.

o Design and design roadmap shall address five facets of the interface:

question asking, answering, exploration, feedback, and collaboration with other users o The design process will iterate with NCATS and biomedical researchers through a process which includes but is not limited to:

brainstorming, requirements prioritization, parallel design, wireframes and prototypes o Analysis and interpretation of knowledge graphs, including the fundamental science details that support the knowledge.

o The design will ensure cognitive fit of the visualization of complex answer sets .

o The design will account for all components of the user experience from asking questions to visualizing the results and providing a space for exploration .

o Through the design process, requirements will be documented and communicated with the NCATS team and when needed, included into the standards development of the wider consortium.

o Identification of tools and technologies that are expected to be utilized during the development and testing phase.

4. Development and Testing

The Contractor shall develop and test a minimally viable product o Initially the minimally-viable product will provide a space for a biomedical researcher to submit a “templated” question to the autonomous relay system, display the results to the user, allow the user to explore the results and send feedback to Translator.

o Evaluation of design through iterative user research and usability interviews o The design, development and testing phases will be both iterative and incremental in scope to correspond with the Translator Development progress utilizing continuous integration, automated testing and continuous deployment practices.

o Utilize cognitive science to facilitate meaningful and successful interpretation of abstract knowledge presented within the data taking into account cognitive load, working memory, and optimizing cognitive fit to support the biomedical researchers ability to utilize the results to further their research.

o Identification of tools and technologies proposed to develop the interface, automate testing and support the DevOps lifecycle.

o Development of production quality open-source interface that directly interacts primarily with the Autonomous Relay System and other Translator software components.

Option Period 1:

1. Ongoing interface development to include iterative user research, user design, and interface development.

2. Deliver Translator User Interface v. 1.0 suitable for end-user use.

Option Period 2:

1. Ongoing interface development to include iterative user research, user design, and interface development.

2. Deliver Translator User Interface v. 2.0 suitable for end-user use.

GOVERNMENT RESPONSIBILITIES

The government will allow access to all systems needed to accomplish our goal. The NCATS Translator team will be available to the vendor to provide transition-in support. NCATS will make prior research conducted available to the vendor. NCATS shall furnish a public Github repository.

DELIVERY OR DELIVERABLES

Base period deliverables:

1. Proposed project plan to include o Project Methodology o Resourcing o Research and design process and methods o Prototype tools o Development tools and standards o Communications Plan o Status Reporting o Testing and Continuous Integration framework o Transition/Sustainability plan https://cognitiveresearchjournal.springeropen.com/articles/10.1186/s41235-018-0120-9 https://cognitiveresearchjournal.springeropen.com/articles/10.1186/s41235-018-0120-9

2. User Research results

3. Initial user design o Design o Prototypes, models, simulations o Interface roadmap o Test Cases o Wireframes

4. Complete first step in development of the minimally viable product o Query interface for researcher to submit “templated” questions to the autonomous relay system o Release suitable for expert users and beta testers o All source code included in NCATS-designated repository

Option Period 1 deliverables:

1. Translator User Interface v. 1.0 suitable for end-user use.

2. Continued development is expected to include these features:

• Ongoing interface development to include iterative user research, user design, and interface development.

• Expand the query interface to allow for ad-hoc search queries to be submitted to the autonomous relay system

• Presentations of the results to the user, including multiple answer graphs, as well as the processes and data sources used to generate the results o Representation of multiple graphs, taking into account understanding of cognitive science visualization

• Functionality to allow the user to explore within and across answer graphs including but not limited to filtering, sorting, and merging

• Ability for users to iterate on the results, including refining the query

• Feedback mechanism

3. All source code included in NCATS-designated repository

Option Period 2 deliverables:

1. Translator User Interface v. 2.0

• Ongoing interface development to include iterative user research, user design, and interface development.

• Adjustments, bug fixes, updates, tweaks etc. required after working interface has been in use for a period of time.

2. All source code included in NCATS-designated repository

REPORTING REQUIREMENTS

The project methodology will determine the recurring reporting requirements. The final report will reflect the transition plan.

OTHER CONSIDERATIONS

Travel:

The Translator Development awardees meet and collaborate both with NCATS and other awardees on a periodic basis remotely, individually and in working groups. We also plan to have three, five day, consortium-wide, in-person working meetings per year, to facilitate collaborations between teams as needed. The meetings will generally rotate between east coast, west coast, and central U.S. locations.

The contractor that partners with us for the Development of the Translator User Interface will conduct meetings as needed. The vendor will also have the opportunity to attend the face-to-face meetings, both to work with the Translator Development teams as well as to present and iterate on research and prototypes.

The NCATS team is available virtually and at our offices in Rockville, MD and in-person meetings are available on an ad-hoc basis with prior scheduling.