Translator UI Materials - Portfolio.pdf

For the best experience, open this PDF portfolio in

Acrobat X or Adobe Reader X, or later.

Get Adobe Reader Now!

http://www.adobe.com/go/reader

NCATS Biomedical Data Translator Revealing new connections among existing data

Mark Williams

Discovery Research Opportunity

Intervention Development

Disease Classification

Data

December 2020

• Built with the Blackboard architecture as a reference

• Autonomous Relay System (ARS)

• Will systematize serendipity, generating new research hypotheses

• Designed with the user in mind at all levels

Birds-eye View

Biomedical Data Translator Architecture and Workflow 2

Pain Medications: Query "message": {

"query_graph": { "edges": [

"id": "e00", "source_id": "n00", "target_id": "n01", "type": "treated_by"

"nodes": [

"curie": "MONDO:0021668", "id": "n00", "type": "disease"

"id": "n01", "type": "chemical_substance" n00

MONDO:0021668

(“Disorder involving Pain”) n01 (no identifier)

Chemical Substance e00 treated_by

”Locked” node.

Refers to pain as defined by this specific MONDO id

“Free” node:

Can be any chemical substance connected to

Edge: specific type of relationship between nodes

Biomedical Data Translator Architecture and Workflow 3

Pain Medications: Answers

•{id: "ChEMBL:CHEMBL1261", •name: "Citric acid", •type: "chemical_substance" •{id: "ChEMBL:CHEMBL113", •name: "Caffeine", •type: "chemical_substance"

•{id: "ChEMBL:CHEMBL963", •name: "Oxymorphone", •type: "chemical_substance" •{id: "ChEMBL:CHEMBL485", •name: "Codeine", •type: "chemical_substance" •{id: "ChEMBL:CHEMBL521", •name: "Ibuprofen", •type: "chemical_substance" •{id: "ChEMBL:CHEMBL1457", •name: "Hydrocodone", •{id: "ChEMBL:CHEMBL19019", •name: "Naltrexone", The Bad The Good The Interesting?

Biomedical Data Translator Architecture and Workflow 4

Pain Medications: Answers •type: "treated_by", •relation: "treated_by", •target_id: "CHEBI:7465", •source_id: "MONDO:0005546", •id: "MONDO:0005546--treated_by--CHEBI:7465", •provided_by: "SEMMED", •publications:

•["PMID:19453963",

•"PMID:23359310",

•"PMID:25264467",

•"PMID:25336035",

•"PMID:25506132",

•"PMID:28325149",

•"PMID:28536359",

•"PMID:28766982"

•api: "SEMMED Disease API"

Biomedical Data Translator Architecture and Workflow 5

ARS – Autonomous Relay System

• ”Brain” of Translator

Basic Operation:

1. Receive query from user

2. Break query into sub-queries by subject if necessary

3. Send query to Autonomous Relay Agents (ARAs) with domain expertise in relevant subject areas

4. Receive answers to query

5. Merge, rank, and display answers

6. Repeat as necessary with user iteration

Biomedical Data Translator Architecture and Workflow 6

Architecture and Workflow Diagram

ARS

ARA1

ARA2

ARA3

KP1

KP2

KP4

KP5KP3

Biomedical Data Translator Architecture and Workflow 7

What ARS Currently Does

• Submit endpoint takes a TRAPI query and returns a JSON with information about where to find the results among other things

• Messages endpoint returns the JSONs of results sent to the ARS in response to queries it sent to Translator services

• Nested parent/child relationships of messages allow chaining of multiple messages

• Individual responses or all responses to a specific query can be found at the messages/<uuid:key> endpoint

• A more human-readable form of these messages is available at the work-in-progress

/answers/<uuid:key> endpoint

• Status endpoint which tracks uptime, recent results, and SmartAPI registration information for many Translator tools and services

Biomedical Data Translator Architecture and Workflow 8 https://ars.transltr.io/ars/api/submit https://ars.transltr.io/ars/api/messages http://ars.transltr.io/ars/api/messages/%3Cuuid:key%3E https://ars.transltr.io/ars/answer/%3Cuuid:key%3E https://ars.transltr.io/ars/app/status

• Channels endpoint to show channels available for services to subscribe to

• Currently only one channel “general”

• Will be specialized channels for different topics (e.g. genetic, clinical, chemical) in future releases to allow services to only receive queries on topics their service can handle

• Agents/Actors endpoint returns the registration information for the different actors and agents

• Will typically be one Agent for a given service and potentially multiple Actors for different endpoints or functionalities running on the same service

• Documentation on the actor/agent registration process can be found here

What ARS Currently Does - Continued

Biomedical Data Translator Architecture and Workflow 9 https://ars.transltr.io/ars/api/channels https://ars.transltr.io/ars/api/agents https://ars.transltr.io/ars/api/actors https://docs.google.com/document/d/1w54dlbep2BtmLorGaeJ86l1YpSgjqSt1Ra9Ye15hBzs/edit

Example Use Cases

• Pain Points for Pain Meds

• Explaining Finasteride’s Use to Treat Hair Loss: A Translator Use

Case

• Finding Genes with Any Connection to IL-6

• Genes Related to Chemicals that are Correlated with Uterine

Cancer

Biomedical Data Translator Architecture and Workflow 10

NCATS Biomedical Data Translator

Birds-eye View

Pain Medications: Query

Pain Medications: Answers

Pain Medications: Answers (2)

ARS - Autonomous Relay System

Architecture and Workflow Diagram

What ARS Currently Does

What ARS Currently Does - Continued

Example Use Cases

Translator and UI Background

NCATS Biomedical Data Translator Program

What is Translator?

• NCATS launched the Biomedical Data Translator (Translator) program to accelerate biomedical translation by the research community. Through this program, NCATS will integrate existing biomedical data to help reveal new relationships within those data and also identify novel opportunities for research. [From https://ncats.nih.gov/translator/about/action ]

• The knowledge presented to the user will be a network of empirical biomedical knowledge from a variety of sources with the goal to augment human reasoning and inference for understanding the pathophysiology of human disease. [From https://ncats.nih.gov/files/NCATS- Translator-FY20-COMBINED-FOA-FINAL.pdf ]

• Translator in the News – STAT July 31, 2019

• https://www.statnews.com/2019/07/31/nih-funded-project-aims-to-build-a-google-for-biomedical-data/

Translator and UI Background 2 https://www.google.com/url?q=https%3A%2F%2Fncats.nih.gov%2Ftranslator%2Fabout%2Faction&sa=D&sntz=1&usg=AFQjCNHPZg8SwdLVCbZphbAttudqScra4w https://www.google.com/url?q=https%3A%2F%2Fncats.nih.gov%2Ffiles%2FNCATS-Translator-FY20-COMBINED-FOA-FINAL.pdf&sa=D&sntz=1&usg=AFQjCNFnYCMjNapx0bHsPa_VCndrs7OvxQ https://www.statnews.com/2019/07/31/nih-funded-project-aims-to-build-a-google-for-biomedical-data/

Translator UI Development Contract - Background

• The contract concept presented at this Translator Industry Day is for an entirely new scope of work for the UI that would interact with existing Translator tools. It will start with design, continuing through to development, pilot release and initial public release. No work has started on the user interface.

• The original concept for a UI development contract was presented at the Jan. 16 2020 NCATS CAN Review Board Meeting, and a recording of that presentation can be found on the NCATS Advisory Groups > Cures Acceleration Network (CAN) Review Board site [https://ncats.nih.gov/advisory/canboard/meetings#pastmeetings] starting at approximately 1:26:30.

Translator and UI Background 3 https://ncats.nih.gov/advisory/canboard/meetings#pastmeetings

Background – User Research

• NCATS worked with 18F through a Path Analysis engagement. The goal of the Path Analysis was to clearly identify potential ways the Translator program should use technology to most effectively meet its mission, using data analysis and visualization to help researchers and clinicians make connections amongst a variety of data types ranging from molecular interactions to signs and symptoms of disease.

• Interviews were conducted with translational researchers to model the translational research process and identify user problems that the tools will need to address to be successful. These are summarized in the 18F – NCATS Presentation included.

Translator and UI Background 4 https://18f.gsa.gov/

Architecture Overview

• To align with our development consortium, all software and other tools used for this project must be open source and publicly available at the time of application, and all software created during the project period must be made publicly available through program-directed source code repositories. Data access will be provided through a customized API using common messaging standards - no direct graph database access or data ingest can be used. NCATS will host Translator resources in the cloud. Individual components may have existing interfaces, however these will not be adopted for the Translator interface. The Translator interfaces should adopt and comply with guidelines set by U.S. Web Design System (USWDS). Offerors should plan to propose technologies that they are most familiar with.

• All software created during the project period must be made publicly available through program-directed source code repositories. NCATS supports both AWS and GCP.

Components of Translator continue to evolve and utilize ETL, machine learning, databases, graph databases, however, for the interface, data access will be provided through a customized API using common messaging standards - no direct graph database access or data ingest can be used. Translator will be open source and completely publicly available for any user. Logins will not be required, NIH will provide security through our hosting facilities.

We consider tooltips and training videos useful to facilitate adoption of the system and would like to include them as part of the development

Translator and UI Background 5 https://www.google.com/url?q=https%3A%2F%2Fdesignsystem.digital.gov%2F&sa=D&sntz=1&usg=AFQjCNEqmuLqkiBP9XRvU-mGio3iSicbyA

Autonomous Relay System Links

• Status Page

• Submit endpoint takes a TRAPI query and returns a JSON with information about where to find the results among other things

• Messages endpoint returns the JSONs of results sent to the ARS in response to queries it sent to Translator services

• Nested parent/child relationships of messages allow chaining of multiple messages

• Individual responses or all responses to a specific query can be found at the messages/<uuid:key> endpoint

• A more human-readable form of these messages is available at the work-in-progress /answers/<uuid:key> endpoint

• Status endpoint which tracks uptime, recent results, and SmartAPI registration information for many Translator tools and services

• Channels endpoint to show channels available for services to subscribe

• Agents/Actors endpoint returns the registration information for the different actors and agents

Translator and UI Background 6 mailto:https://ars.transltr.io/ars/app/status https://ars.transltr.io/ars/api/submit https://ars.transltr.io/ars/api/messages http://ars.transltr.io/ars/api/messages/%3Cuuid:key%3E https://ars.transltr.io/ars/answer/%3Cuuid:key%3E https://ars.transltr.io/ars/app/status https://ars.transltr.io/ars/api/channels https://ars.transltr.io/ars/api/agents https://ars.transltr.io/ars/api/actors

Important Links

• NCATS Translator Github

• Relay Repo for the ARS

• ncats-translator Github Repos

• SmartAPI - Translator Resources

• Smart API Portal - Translator Meta-KG

• Biolink Model

Translator and UI Background 7 mailto:https://github.com/NCATSTranslator/ mailto:https://github.com/topics/ncats-translator mailto:https://smart-api.info/registry?tags=translator mailto:https://smart-api.info/portal/translator/metakg mailto:https://github.com/biolink/biolink-model

@ncats_nih_gov @ncats.nih.govncats.nih.gov NIH-NCATS

Translator and UI Background

What is Translator?

Translator UI Development Contract - Background

Background – User Research

Architecture Overview

Autonomous Relay System Links

Important Links

18F Discovery research

Translator, April 2019 in partnership with NCATS

18F NCATS 1

Translational researchers / “Key staff” We are building for:

Earlier on in their career; have less job security (eg. untenured)

Member of a team on research projects (key staff) vs. PI

In the lab on a daily basis

More motivation to strike out on their own, chart new territory

Less set in their ways, more open to new methods and ways of working

They’re doing the experiments, info searches, evolving hypotheses & research questions

Example job titles

Untenured Professors

Non-senior Scientists/ Researchers

Postdoctoral Associates

● Mission-driven to solve disease / get treatments

● Curious / early adopter / Open to change

● Motivated to improve people’s lives

● Need for money, or have money that they have ability to direct

18F NCATS 2

Help explain an observation (for example, how are disease X and treatment Y related?)

The first thing we can build to help translational researchers will

18F NCATS 3

● User story:

○ I have an observation. I need to understand how it works / why it happened. I use Translator to display the connections that may explain my observation. I want Translator to show me paths which are interesting, untested or novel so that I know where to go next in my research, and not redo things that have already been done.

● Jobs that need to be done → Function:

○ Search multiple kinds of data → Show [how much / strength] is being each result

○ Understand how a thing works / Fire random literature walk “I think I read this once” → Enables a researcher to design an experiment

○ Show which paths are interesting [unexpected but promising]

Initial user research

18F NCATS 4

Generative user research: interviews

18F NCATS 5

18F conducted user interviews in order to learn:

● What are translational researchers trying to accomplish in their work - what are their goals?

● In what settings do they work?

● How do they describe their research, and their field?

● How do they go about obtaining information and data, when needed, in the course of their research?

● What kind of information and data do they seek, and how do they describe it?

● What do they do with it once they have it? (“tasks”) How do they analyze it, think about it, reason with it? How does it inform their research?

● What are the challenges they face in their research work, and how do currently they attempt to address them?

User interviews 18F interviewed 8 scientists who conduct research in a variety of settings across academia, private research institutes, and industry.

We conducted 6 interviews remotely via videoconference;

and 2 interviews in person, which gave us a glimpse into the context of their work.

Roles included:

● Postdocs

● Principal Investigators

● Assistant professors

● Pharmaceutical industry research manager

618F NCATS

Emerging themes

18F NCATS 7

I’m busy, and have many demands on my time

● Most scientists were interviewed are switching constantly between meetings, lab time, email, writing time, and other tasks. They don’t have a lot of time to explore and learn how to use a new tool.

● One postdoc interviewed explores new tools and resources she hears about in the 5-10 minutes she has while waiting for an assay to finish up.

Potential design implications: How might Translator draw potential users and quickly provide value? / How might we design Translator so that its value becomes clear to potential users in a couple of minutes?

Working styles and patterns

818F NCATS

I search and explore by “browsing and berrypicking”

● When exploring data to support development or refinement of a research question, scientists follow a meandering information exploration process

● This has been described as a “browsing and berrypicking” process in the information sciences, where as the scientist goes along, they learn more and more that guides and shapes the direction they search in

Potential design implications: How might Translator support a “browsing and berrypicking” process? (For example, saving the path a user has followed and allowing them to easily backtrack and go in a different direction?)

Information exploration

918F NCATS

I need to know where this piece of data came from if I’m going to believe it

● Scientists are at heart, huge skeptics

● They want to know where the information or data they’re looking at comes from

– what lab is it from? Who led the research? Is it published? In which journal?

When? – To determine to what degree they can trust it.

Potential design implication: how might Translator allow related metadata to be easily surfaced and explored? (Provide link, summary, abstract, figures, etc.)

Contextual information for trust

1018F NCATS

I need to know what this piece of data is related to

● Scientists also want to know the context within which a particular data point or piece of information occurs.

● This can be as simple as where in a paper their search terms occurred, because that context can tell them if the related research is relevant to their area of interest.

○ “My two queries... might be in there in a completely different context that is not helpful to me.”

● Several scientists we interviewed – though not all – go straight to the figures to understand the research results of a paper.

Potential design implication: how might Translator allow related metadata to be easily surfaced and explored? (Provide link, summary, abstract, figures, etc.)

Contextual information for relevance

1118F NCATS

I might need more information to understand this data

● When in their explorations, they come across information about a topic outside of their field of expertise, scientists seek out supporting information to help them understand what they’re reading.

● This can take many forms: reading review papers; speaking to colleagues in that field; or reading up on topics in Wikipedia.

Potential design implication: how might Translator provide supporting information to help researchers understand the data linkages being made across multiple disciplines, including ones outside of their specialty?

Contextual information for understanding

1218F NCATS

Don’t give me more results than I can easily process

● One interviewee who had the chance to use one of the Translator prototypes described being overwhelmed by a list of 200-300 results, each one of which they could potentially run an assay on

● “Nobody has the time to go through 200-300 transcription factors to check each and every one of them, so that was challenging”

Potential design implications: How might Translator provide succinct results?

Information overwhelm

1318F NCATS

Research process diagrams

18F NCATS 14

Scientists form and reform research questions and hypotheses throughout the research process.

Each time, they gather and analyze data, whether from experiments, the literature, databases, or colleagues, to help shape their investigation.

Diagramming out their research process allows us to understand where Translator fits in that process, and how it can best support the work scientists are doing.

18F NCATS 15

Initial draft of a research process diagram

Translator would support yellow and lavender activities

18F NCATS 16

What we’ve learned in partnership with NCATS

18F NCATS 17

Interpret Translator Value

Translator’s value is its ability to create relationships between concepts that a scientist understands. So it needs to:

● help scientists select and define the concept they’re interested in,

● interpret those inputs, and

● understand how results correspond to the actual words used by scientists.

By developing user-facing and data-focused parts of Translator simultaneously.

18F NCATS 18

Find Interesting Results in a Large Result Set Translator needs to help scientists identify interesting or important results so that it becomes a usable, efficient tool for them. Identifying a few potentially interesting results can guide scientists through an otherwise too large data set.

Translator will need to help users identify when a more specific term may produce more interesting results.

Scientists may have different motivations for the tool. Some may be looking for more novel, unexplored results, while others may benefit from learning about known pathways outside of their own discipline.

However, scientists signaled that they don’t trust a piece of software that asserts itself as being smarter than them without an explanation of how or why.

18F NCATS 19

Translator is new

The kinds of questions that can be asked of Translator are different from the searches on the other digital resources most commonly used — PubMed and Google

Translator creates connections between data points in a transitive way

The information returned can and will span multiple disciplines

Data sources can include non publication-based sources, and many users need to learn what those are and how much they should trust them

18F NCATS 20

Translator Adoption

Respect and leverage patterns that users are already familiar with

Prioritize accessibility, which requires defining and describing

Continue testing how to present knowledge assertions, and how to explain why they are valuable

Invest significant resources in the first-time user experience

18F NCATS 21

Next Phase of Work

18F NCATS 22

18F NCATS 23

18F Discovery research

Translational researchers / “Key staff”

Help explain an observation (for example, how are disease X and treatment Y related?)

Initial user research

Generative user research: interviews

User interviews

Emerging themes

Working styles and patterns

Information exploration

Contextual information for trust

Contextual information for relevance

Contextual information for understanding

Information overwhelm

Research process diagrams

Scientists form and reform research questions...

Initial draft of a research process diagram

What we’ve learned

Interpret Translator Value

Find Interesting Results in a Large Result Set

Translator is new

Translator Adoption

Next Phase of Work

Cycle

Translator Use Case - Explain Finasteride Repurposed for Hair Loss 1

Explaining Finasteride’s Use to Treat Hair Loss: A Translator Use Case https://ars.transltr.io/ars/api/messages/2280ed43-9e1c-4f02-9986-3e09712a600f?trace=y

The task we had for Translator in this instance was to explain how Finasteride, which was originally developed for benign prostatic hyperplasia, affects hair loss. To do this, we first have to “translate” our question into a form that Translator can understand. We use TRAPI, a JSON-based standard for encoding queries and responses developed for use in Translator. This standard uses nodes, representing biomedical entities ( e.g. diseases, symptoms, genes, chemicals etc), and edges representing the connections between these entities (e.g. treats, contraindicated for, causes, etc). Currently the process for converting a question to TRAPI is a manual one which requires some degree of familiarity with both TRAPI and the underlying data sources used in Translator. In this case, the way that we have chosen to represent the query in TRAPI for is "message": { "query_graph": { "edges": [ "id": "e0", "source_id": "n0", "target_id": "n1" "id": "e1", "source_id": "n1", "target_id": "n2" "nodes": [ "curie": "CHEMBL:5062", "id": "n0" "id": "n1", "type":"gene" "id":"n2", "curie":"MONDO:0005339" https://github.com/NCATSTranslator/ReasonerAPI

Translator Use Case - Explain Finasteride Repurposed for Hair Loss 2

The gist of what we are saying is “Show me genes (indicated by node n1) that are connected to both Finasteride (represented by n0, CHEMBL:5062), and male pattern baldness (represented by n2, MONDO:005339).” Note that this is not a direct “translation” of our original query. There is a human decision made to interpret the original query and use shared genes between the drug and the condition to find possible links, but many other interpretations are also possible.

The types of the edges in this instance are not specified. So,Translator will return any type of edge that connects the specified nodes. If desired, the types of these edges could be specified to further refine the returned results.

Once we have our query in TRAPI format, it is POSTed to https://ars/transltr.io/ars/api/submit In response to the POST, the ARS will return a small message with some information about the query, such as the status, HTTP code returned, timestamp and, perhaps most importantly, the primary key (pk) by which the query responses can be retrieved.

"model": "tr_ars.message", "pk": "2280ed43-9e1c-4f02-9986-3e09712a600f", "fields": { "name": "", "code": 200, "status": "Running", "actor": 12, "timestamp": "2021-01-07T20:33:11.766Z", "data": {

"edges": [ "id": "e0", "source_id": "n0", "target_id": "n1" "id": "e1", "source_id": "n1", "target_id": "n2" "nodes": [ "curie": "CHEMBL:5062", "id": "n0" "id": "n1", "type":"gene"

"id":"n2", https://ars/transltr.io/ars/api/submit

Translator Use Case - Explain Finasteride Repurposed for Hair Loss 3

"curie":"MONDO:0005339"

"url": null, "ref": null

The Autonomous Relay System (ARS) for Translator accepts and runs this query (https://ars.transltr.io/ars/api/messages/2280ed43-9e1c-4f02-9986-3e09712a600f), and then sends it to many different Translator tools which send their results (if any) back to the ARS. The status of this task can be found by requesting:

https://ars.transltr.io/ars/api/messages/2280ed43-9e1c-4f02-9986-3e09712a600f?trace=y Within this response, individual result sets are identified by their own message field. Here, the results for the ARAX tool have the identifier f3630ade-2c56-4184-91cd-a4dd6ff1edc5

So, the results for Expander Agent (ARAX) can be found at https://ars.transltr.io/ars/api/messages/f3630ade-2c56-4184-91cd-a4dd6ff1edc5 These results are also in the TRAPI format, which contains:

● Query Graph

○ This is the same graph that we initially sent to the ARS, or if the tool has chosen to return results for an altered query graph, the query graph actually used.

● Knowledge Graph

○ This contains all of the individual node and edge entities found in any of the results. They are found here in full detail only once each. This is to prevent bloat in the Results field in the (fairly common) case that a node or edge is found in multiple results. For example, in our query, Finasteride would be present in all of our results, but there is no need to give the full details on the node every time.

● Results

○ These are the actual results for our query and will contain node and edge bindings which refer by use of the kg_id field to the relevant entry in the Knowledge Graph for the node and edges used in that result.

https://ars.transltr.io/ars/api/messages/2280ed43-9e1c-4f02-9986-3e09712a600f

Translator Use Case - Explain Finasteride Repurposed for Hair Loss 4

It is typical that for a given query, only a subset of the registered Translator tools will be able to return a response. Which tools return responses varies on the specific content of the query.

The results that are returned from Translator answers are grouped (for now) based on which Translator tool returned them. Expander Agent (ARAX) and Ranking Agent (Aragorn) are the two tools that have provided results for this question, and if we look at the results returned from Ranking Agent (Aragorn) https://ars.transltr.io/ars/api/messages/7da56a33-9e65-4352-ab5d-c176c3f90fe5 we see two results being returned.

"results": [ "node_bindings": [ "qg_id": "n0", "kg_id": "CHEBI:5062" "qg_id": "n1", "kg_id": "NCBIGene:6716" "qg_id": "n2", "kg_id": "MONDO:0005339" "edge_bindings": [ "qg_id": "e0", "kg_id": "41d2e35750c8fc5520b93c2c6734cbce" "qg_id": "e1", "kg_id": "efd52a55344e2d4f61711acce588e02d" "timestamp": 5.205526113510132, "score": 20.806633671805507 "node_bindings": [ "qg_id": "n0", "kg_id": "CHEBI:5062" "qg_id": "n1", "kg_id": "NCBIGene:367" https://ars.transltr.io/ars/api/messages/7da56a33-9e65-4352-ab5d-c176c3f90fe5 https://ars.transltr.io/ars/api/messages/7da56a33-9e65-4352-ab5d-c176c3f90fe5

Translator Use Case - Explain Finasteride Repurposed for Hair Loss 5

"qg_id": "n2", "kg_id": "MONDO:0005339" "edge_bindings": [ "qg_id": "e0", "kg_id": "8da802d11d122592a92357c15726a67b" "qg_id": "e1", "kg_id": "fe42b437debd5625f149f15e427e4a07" "timestamp": 5.2120277881622314, "score": 67.56069811364625 "url": "http://robokop.renci.org:5781/query", "ref": "2280ed43-9e1c-4f02-9986-3e09712a600f"

In the first result, we see that the gene connecting Finasteride and baldness is identified as "NCBIGene:6716" and if we go to the Knowledge Graph, we can see that this gene has the name SRD5A2:

"id": "NCBIGene:6716", "type": [ "molecular_entity", "named_thing", "biological_entity", "biolink:NamedThing", "macromolecular_machine", "gene_or_gene_product", "genomic_entity", "gene" "name": "SRD5A2", "equivalent_identifiers": [ "NCBIGene:6716", "ENSEMBL:ENSG00000277893", "HGNC:11285", "UniProtKB:P31213" "category": [ "gene", Translator Use Case - Explain Finasteride Repurposed for Hair Loss 6

"named_thing", "biological_entity", "molecular_entity", "genomic_entity", "macromolecular_machine", "gene_or_gene_product" "synonyms": []

Moreover, if we look at the result set, we see that the relationship between CHEBI:5062 and NCBIGene:6716 is defined by edge "kg_id": "41d2e35750c8fc5520b93c2c6734cbce" edges": [

"id": "41d2e35750c8fc5520b93c2c6734cbce", "source_id": "CHEBI:5062", "target_id": "NCBIGene:6716", "type": "decreases_activity_of", "object": "NCBIGene:6716", "source_database": "PHAROS 6.7", "affinity": "8.38", "edge_label": "decreases_activity_of", "affinity_parameter": "IC50", "relation": "biolink:decreases_activity_of", "publications": [

"PMID:11063622"

"subject": "CHEBI:5062"

From which we see that in the PHAROS 6.7 database, we know finasteride decreases the activity of SRD5A2 with an IC50 of 8.38, which is quite potent. NOTE: this should be a pIC50 of

8.38. It also provides a publication reference for this assertion PMID:11063622, which is extremely important for end-users to be able to use to assess the quality of this assertion.

By following a similar process with the Expander Agent (ARAX )result set ( https://ars.transltr.io/ars/api/messages/f3630ade-2c56-4184-91cd-a4dd6ff1edc5 ) , we find that the same gene is also indicated as being connected by their tool, though it refers to SRD5A2 with a different identifier "description": "-!- FUNCTION: Converts testosterone (T) into 5-alpha-dihydrotestosterone(DHT) and progesterone or corticosterone into their corresponding 5-alpha-3-oxosteroids. It plays a central role in sexual differentiation and androgen physiology. {ECO:0000269|PubMed:10898110}.-!- CATALYTIC ACTIVITY:Reaction=a 3-oxo-5alpha-steroid + NADP(+) = a 3-oxo- Delta(4)-steroid +H(+) + NADPH; Xref=Rhea:RHEA:54384, https://pubmed.ncbi.nlm.nih.gov/11063622/

Translator Use Case - Explain Finasteride Repurposed for Hair Loss 7

ChEBI:CHEBI:13601,ChEBI:CHEBI:15378, ChEBI:CHEBI:47909, ChEBI:CHEBI:57783,ChEBI:CHEBI:58349;

EC=1.3.1.22;Evidence={ECO:0000269|PubMed:10898110};-!- BIOPHYSICOCHEMICAL PROPERTIES:Kinetic parameters:KM=0.9 uM for testosterone (at pH 6.0) {ECO:0000269|PubMed:10898110};KM=8 uM for NADPH (at pH 6.0) {ECO:0000269|PubMed:10898110};Vmax=1.9 nmol/min/mg enzyme with testosterone as substrate with NADPHas cofactor (at pH 6.0) {ECO:0000269|PubMed:10898110};pH dependence:Optimally active at acidic pHs.;- !- INTERACTION:P31213; A8MQ03: CYSRT1; NbExp=3; IntAct=EBI-13130472, EBI- 3867333;P31213; Q14749: GNMT; NbExp=3; IntAct=EBI-13130472, EBI-744239;-!- SUBCELLULAR LOCATION: Microsome membrane; Multi-pass membrane protein.Endoplasmic reticulum membrane {ECO:0000305}; Multi-pass membrane protein {ECO:0000305}.-!- TISSUE SPECIFICITY: Expressed in high levels in the prostate and many other androgen-sensitive tissues.-!- POLYMORPHISM:

Individuals with Thr-49 have an increased risk of prostate cancer (PubMed:10501358). The enzyme with Thr-49 has a higher in vitro V(max) than the Ala-49 enzyme (PubMed:10501358).{ECO:0000269|PubMed:10501358}.-!- DISEASE: Pseudovaginal perineoscrotal hypospadias (PPSH) [MIM:264600]:A form of male pseudohermaphroditism in which 46,XY males show ambiguous genitalia at birth, including perineal hypospadias and ablind perineal pouch, and develop masculinization at puberty. The name of the disorder stems from the finding of a blind-ending perineal opening resembling a vagina and a severely hypospadiac penis with the urethra opening onto the perineum.

{ECO:0000269|PubMed:10718838,ECO:0000269|PubMed:10898110, ECO:0000269|PubMed:10999800,ECO:0000269|PubMed:12843198, ECO:0000269|PubMed:15064320,ECO:0000269|PubMed:1522235, ECO:0000269|PubMed:15528927,ECO:0000269|PubMed:15770495, ECO:0000269|PubMed:16098368,ECO:0000269|PubMed:16181229, ECO:0000269|PubMed:7554313,ECO:0000269|PubMed:8626825, ECO:0000269|PubMed:8768837,ECO:0000269|PubMed:9208814, ECO:0000269|PubMed:9745434,ECO:0000269|PubMed:9843052}. Note=The disease is caused by mutations affecting the gene represented in this entry.-!- SIMILARITY: Belongs to the steroid 5-alpha reductase family.{ECO:0000305}.-!- WEB RESOURCE: Name=NIEHS- SNPs;URL=\"http://egp.gs.washington.edu/data/srd5a2/\";-!- WEB RESOURCE:

Name=Wikipedia; Note=5-alpha reductase entry;URL=\"https://en.wikipedia.org/wiki/5_alpha_reductase\";; Short=S5AR 2;

This gene encodes a microsomal protein expressed at high levels in androgen-sensitive tissues such as the prostate. The encoded protein is active at acidic pH and is sensitive to the 4-azasteroid inhibitor finasteride.

Deficiencies in this gene can result in male pseudohermaphroditism, specifically pseudovaginal perineoscrotal hypospadias (PPSH). [provided by RefSeq, Jul 2008]. --- 3-oxo-5-alpha-steroid 4-dehydrogenase 2 (254 aa, ~28 kDa) is encoded by the human SRD5A2 gene. This protein plays a role in sexual differentiation and androgen metabolism.; UMLS Semantic Type: UMLS_STY:T126;

UMLS Semantic Type: UMLS_STY:T116 --- A protein that is a translation product of the human SRD5A2 gene or a 1:1 ortholog thereof. // COMMENTS:

Category=gene. --- A 3-oxo-5-alpha-steroid 4-dehydrogenase 2 that is encoded in the genome of human. // COMMENTS: Category=organism-gene. --- A protein

Translator Use Case - Explain Finasteride Repurposed for Hair Loss 8 coding gene SRD5A2 in human. // COMMENTS: Category=external.; UMLS Semantic Type: UMLS_STY:T028", "id": "UniProtKB:P31213", "name": "SRD5A2", "node_attributes": [ "name": "equivalent_curies", "value": [

"HGNC:11285",

"NCIT:C17190",

"ORPHANET:119860",

"PR:000015608",

"PR:P31213",

"UniProtKB:P31213", "ENSEMBL:ENSG00000277893", "NCBIGene:6716" "name": "publications", "value": [ "DOI:10.1016/s0140-6736(98)11282-5", "DOI:10.1038/ng1285", "DOI:10.1210/jcem.85.9.6786", "PMID:10718838", "PMID:15489334", "PMID:10898110", "PMID:16181229", "DOI:10.1101/gr.2596504", "PMID:7554313", "PMID:9745434", "PMID:15064320", "PMID:16098368", "DOI:10.1210/jcem.81.5.8626825", "PMID:15770495", "DOI:10.1002/j.1939-4640.2004.tb02808.x", "PMID:9208814", "PMID:1505484", "PMID:14702039", "DOI:10.1111/j.1365-2265.2005.02348.x", "2-t", "PMID:8626825", "PMID:11927504", "DOI:10.1002/(sici)1096-8628(19981116)80:3<269::aid-ajmg18>3.0.co", "DOI:10.1136/mp.50.1.51", "DOI:10.1210/jcem.83.9.5125", "DOI:10.1210/endo.131.3.1505484", "DOI:10.1159/000081893", Translator Use Case - Explain Finasteride Repurposed for Hair Loss 9

"PMID:1944596",

"DOI:10.1038/354159a0", "DOI:10.1038/sj.ejhg.5201101", "DOI:10.1007/s00109-005-0651-7", "DOI:10.1210/jc.2002-021415", "PMID:8768837", "PMID:10501358", "PMID:10999800", "DOI:10.1046/j.1365-2265.2000.00941.x", "DOI:10.1097/00008571-200007000-00004", "DOI:10.1210/jcem.81.8.8768837", "PMID:9843052", "DOI:10.1172/jci115954", "PMID:1522235", "PMID:15528927", "PMID:14560315", "DOI:10.1016/j.urology.2005.02.021", "DOI:10.1111/j.1365-2265.1995.tb01913.x", "PMID:12843198"

"name": "all_names", "value": [ "3-oxo-5-alpha-steroid 4-dehydrogenase 2", "steroid 5 alpha-reductase 2 [Source:HGNC Symbol;Acc:HGNC:11285]", "SRD5A2", "3-oxo-5-alpha-steroid 4-dehydrogenase 2 (human)", "3-Oxo-5-Alpha-Steroid 4-Dehydrogenase 2", "steroid 5 alpha-reductase 2" "qnode_ids": [ "n1" "type": [ "gene" "uri": "https://identifiers.org/uniprot:P31213"

The other gene found in Ranking Agent’s (Aragorn) results (AR "NCBIGene:367"/"UniProtKB:P10275") also appears in Expander Agent’s (ARAX). This overlap is a good indication that these genes are reasonable answers to our query. Other results found in Expander Agent (ARAX) but not Ranking Agent (Aragorn) are not necessarily poor answers, and while there are confidence scores attached to results by the individual tools, Translator Use Case - Explain Finasteride Repurposed for Hair Loss 10 some degree of human evaluation is still necessary at this stage to determine how reasonable or useful any given result is in the context of our question.

Translator Use Case - Finding Genes with Any Connection to IL-6 1 https://ars.transltr.io/ars/api/messages/9af99dbb-7c99-45a2-ac27-67e7eab2a6cb?trace=y The process for navigating this set of results will be the same as is described in the documentation on the use case for repurposing Finasteride for hair loss.

In some cases, especially with broad or simple queries, the amount of data returned by registered services to Translator may be large. In this example, the query was:

"message": { "query_graph": { "edges": { "e00": { "subject": "n00", "object": "n01" "nodes": { "n00": { "id": "NCBIGene:3569", "category": "biolink:Gene" "n01": { "category": "biolink:Gene" This query is searching for all genes connected to IL-6 (NCBIGene:3569). IL-6 is a highly researched gene with connections to rheumatoid arthritis, enterovirus, schizophrenia, aging, depression, asthma and many other conditions. Because this gene is so heavily studied, it is no surprise to find that it is also heavily connected to other genes. The maximum number of results returned from a single tool is 1,274 genes with a variety of different connections to IL-6. This number of results would be time-consuming to analyze manually demonstrating the need for tools to aid in the parsing and analysis of result sets.

https://ars.transltr.io/ars/api/messages/9af99dbb-7c99-45a2-ac27-67e7eab2a6cb?trace=y https://docs.google.com/document/d/10nJIOaYBB079-1CwwmfLIrRGFCT_a_Oe2u9rXrP6R7U/edit

Pain Points for Pain Meds - Detailed Data Spelunking 1

Data Spelunking: Pain Points for Pain Meds The long version …

Let’s start with a simple one-hop query to the ARS for chemicals that treat pain:

https://ars.transltr.io/ars/api/messages/ad4e5b34-f286-45d7-8e25-f3cba50774a8

"edges": [ "id": "e00", "source_id": "n00", "target_id": "n01", "type": "treated_by" "nodes": [ "curie": "MONDO:0021668", "id": "n00", "type": "disease" "id": "n01", "type": "chemical_substance"

NOTE: This query is constructed using version 0.92 of the TRAPI standard https://ars.transltr.io/ars/api/messages/ad4e5b34-f286-45d7-8e25-f3cba50774a8

Pain Points for Pain Meds - Detailed Data Spelunking 2

Let me just google that disease CURIE to make sure I’m using the right one … and of course, you can’t google a MONDO, because Chrome thinks it’s a URL ...

Eventually, you get to MONDO, whose definition looks appropriate:

https://monarchinitiative.org/disease/MONDO:0021668 disorder involving pain MONDO:0021668 Mappings: SCTID:373673007 DOID:0060164 UMLS:C1300028 Exact Synonyms: pain disorder https://monarchinitiative.org/disease/MONDO:0021668 http://purl.obolibrary.org/obo/DOID_0060164 http://linkedlifedata.com/resource/umls/id/C1300028

Pain Points for Pain Meds - Detailed Data Spelunking 3

But when you look at the mapping into DOID, you aren’t too sure, because you didn’t think pain was a “disease of mental health” ...

http://www.ontobee.org/ontology/DOID?iri=http://purl.obolibrary.org/obo/DOID_0060164

The semantics aren’t perfect here … and you could probably dig into “treated_by” and “chemical_substance” as well. But let’s skip that for the time being.

What do our answers look like?

You can query the ARS to see what agents were queried for an answer https://ars.transltr.io/ars/api/messages/ad4e5b34-f286-45d7-8e25-f3cba50774a8?trace=y

"message": "ad4e5b34-f286-45d7-8e25-f3cba50774a8", "status": "Done", "actor": { "pk": 9, "channel": "general", "agent": "ars-default-agent", "path": "" "children": [ "message": "b0a0aa59-a0fb-4fd4-adc3-0dc61820d38d", "status": "Done", "actor": { "pk": 1, "channel": "general", "agent": "ara-aragorn", "path": "runquery" "children": [ http://www.ontobee.org/ontology/DOID?iri=http://purl.obolibrary.org/obo/DOID_0060164 https://ars.transltr.io/ars/api/messages/ad4e5b34-f286-45d7-8e25-f3cba50774a8?trace=y

Pain Points for Pain Meds - Detailed Data Spelunking 4

"message": "fc7414d4-1f64-48f6-a4be-0775a6486227", "status": "Done", "actor": { "pk": 3, "channel": "general", "agent": "ara-bte", "path": "runquery" "children": [

Things don’t always work as planned … that’s OK https://ars.transltr.io/ars/api/messages/e45b3582-0761-45b6-a6fb-224be94d2a4f

"model": "tr_ars.message", "pk": "e45b3582-0761-45b6-a6fb-224be94d2a4f", "fields": { "name": "", "code": 405, "status": "Error", "actor": 11, "timestamp": "2020-10-20T16:30:41.683Z", "data": { "message": { "query_graph": { "logs": [ "{\n "detail": "The method is not allowed for the requested URL.",\n "status": 405,\n "title": "Method Not Allowed",\n "type":

"about:blank"\n}\n", "Tr_ars.server: nginx/1.16.1", "Tr_ars.date: Tue, 20 Oct 2020 16:30:42 GMT", "Tr_ars.content-Type: application/problem+json", "Tr_ars.content-Length: 142", "Tr_ars.connection: keep-alive", https://ars.transltr.io/ars/api/messages/e45b3582-0761-45b6-a6fb-224be94d2a4f

Pain Points for Pain Meds - Detailed Data Spelunking 5

"Tr_ars.reason: METHOD NOT ALLOWED", "Tr_ars.url: https://evidara.healthdatascience.cloud/api/v1/query" "url": "https://evidara.healthdatascience.cloud/api/v1/query", "ref": "ad4e5b34-f286-45d7-8e25-f3cba50774a8" kp-molecular … hey, that’s cheating!

I say this tongue-in-cheek because the ARS connects directly with a knowledge provider in this case. The query is clearly very simple, and it just happens that this KP can respond to this exact query.

https://ars.transltr.io/ars/api/messages/5a9f8d08-b6af-4bd1-b8a3-fa4182769251

In fact, if I swap the curie and types between nodes "n00" and "n01", I get no results back.

When it works, kp-molecular provides 55 results.

In the resultant "knowledge_graph" I get 50 "edges" and 51 "nodes" … 1 disease node and 50 chemical nodes, 5 of which are bound twice in the "results". For example:

"edge_bindings": [ "kg_id": "e34", "qg_id": "e00" "node_bindings": [ "kg_id": "MONDO:0021668", "qg_id": "n00" "kg_id": "ChEMBL:CHEMBL898", "qg_id": "n01" https://ars.transltr.io/ars/api/messages/5a9f8d08-b6af-4bd1-b8a3-fa4182769251

Pain Points for Pain Meds - Detailed Data Spelunking 6

"edge_bindings": [ "kg_id": "e34", "qg_id": "e00" "node_bindings": [ "kg_id": "MONDO:0021668", "qg_id": "n00" "kg_id": "ChEMBL:CHEMBL898", "qg_id": "n01"

In case you were wondering, “ChEMBL:CHEMBL898" is diflunisal, an NSAID developed in the 1970s and marketed under the brand name Dolobid. There’s no other provenance information here, but presumably there’s a reason in the underlying data that this compound is listed twice. I wouldn’t say it is twice as good as any others on the list in terms of treating pain.

Scanning the results, you get the ones you’d expect … ChEMBL:CHEMBL70 (morphine), ChEMBL:CHEMBL596 (fentanyl), ChEMBL:CHEMBL112 (acetaminophen). You don’t find some of the more exotic ones, like https://en.wikipedia.org/wiki/Resiniferatoxin#Research [ChEMBL:CHEMBL17976] or https://en.wikipedia.org/wiki/Botulinum_toxin#Chronic_pain [ChEMBL:CHEMBL2108570 … this is a poor identifier for this b.t.w.] kp-molecular … Naltrexone as a pain treatment?

One interesting result is ChEMBL:CHEMBL19019 (naltrexone). Naltrexone is structurally very close to the drug naloxone, a more popularly known drug used in the context of suspected overdose of some of the actors above. Naltrexone is used to help treat opioid addiction. Again, provenance would be essential to understanding exactly how it winds up here as a treatment for pain.

https://www.accessdata.fda.gov/drugsatfda_docs/label/2013/018932s017lbl.pdf REVIA has not been shown to provide any therapeutic benefit except as part of an appropriate plan of management for the addictions.

https://en.wikipedia.org/wiki/Diflunisal https://en.wikipedia.org/wiki/Resiniferatoxin#Research https://en.wikipedia.org/wiki/Botulinum_toxin#Chronic_pain https://drugs.ncats.io/drug/5S6W795CQM https://drugs.ncats.io/drug/36B82AMQ7N https://www.accessdata.fda.gov/drugsatfda_docs/label/2013/018932s017lbl.pdf

Pain Points for Pain Meds - Detailed Data Spelunking 7 kp-molecular … Citric Acid as a pain treatment?

ChEMBL:CHEMBL1261 (citric acid aka vitamin C). I’m even more skeptical here … but then, like other certain overrepresented terms in the literature, vitamins tend to be connected with all sorts of stuff. I want to see exactly how this one wound up on the list before I grant any credit (but concede that I might have to as I hear scurvy can be pretty painful).

ara-unsecret … haven’t I seen this somewhere else?

So we know that at least one knowledge provider has a good set of answers to this question, how many of the ARAs can take advantage of this? (Un)Secret agent does exactly this - giving us back Molecular Provider’s knowledge graph.

https://ars.transltr.io/ars/api/messages/3f01d797-7b0c-4f9c-adc7-802627e4c9a8

I wish there was a way to know exactly where this knowledge graph came from in the TRAPI response itself and that the ARAs tagged the results as originally being from this or that KP.

In this case, our forensic study strongly suggests that to be the case: 55 results, “ChEMBL:CHEMBL898" (diflunisal) is mapped twice (exactly as above), and our friends ChEMBL:CHEMBL19019 (naltrexone) and ChEMBL:CHEMBL1261 (citric acid aka vitamin C) are included. This is a very nice result, demonstrating we’ve been able to connect end-to-end from ARS to ARA to KP and back.

ara-aragon … third time is … similar https://ars.transltr.io/ars/api/messages/b0a0aa59-a0fb-4fd4-adc3-0dc61820d38d “You said you'd bind yourself to me, forsaking ...”

Like (Un)Secret, this ARA nicely binds the results from Molecular Provider. It also recognizes the duplicate mapping of several nodes, and provides only a single result for these (a total of 51 results). The results furthermore appear to be mapped to canonical CURIEs, so for example CHEBI:39669 (diflunisal) is given instead of CHEMBL898. Again, this is a very nice result, demonstrating we’ve been able to connect end-to-end from ARS to ARA to KP and back.

Perhaps having a breadcrumb trail or citation of the original data would be nice to add to the response here.

ara-arax … audit trail galore Ostensibly, also 50 results. All of the results are bound through the first node in the knowledge graph, a node that ARAX swaps in place of MONDO:0021668 (disorder involving pain).

"nodes": [

"id": "UniProtKB:P22303", "name": "AChE", https://ars.transltr.io/ars/api/messages/3f01d797-7b0c-4f9c-adc7-802627e4c9a8 https://en.wikipedia.org/wiki/Diflunisal https://ars.transltr.io/ars/api/messages/b0a0aa59-a0fb-4fd4-adc3-0dc61820d38d

Pain Points for Pain Meds - Detailed Data Spelunking 8

"qnode_ids": [ "n00" "type": [ "disease"

UniProtKB:P22303 (AChE; Acetylcholinesterase) is a very interesting choice here. It is the protein target of a class of drugs for Alzheimer’s disease and also the target of chemical warfare nerve agents.

I haven’t figured out exactly how this target was chosen based on the original search term.

Luckily, we can look deeper into the results.

ara-arax … a log of how we got here This response includes a full array of logging statements as part of the "knowlege_graph" response under "log".

"level": 10, "level_str": "DEBUG", "message": "The QueryGraph reference template is:

n00(type=disease)-e00()-n01(type=chemical_substance)", "prefix": "2020-10-20 16:30:41.993108 DEBUG: ", "timestamp": "2020-10-20 16:30:41.993108"

And a synopsis of the entire plan of attack under "query_options".

"query_options": { "processing_actions": [ "expand(edge_id=e00, kp=ARAX/KG2, continue_if_no_results=true)", "expand(edge_id=e00, kp=BTE, continue_if_no_results=true)", "overlay(action=overlay_clinical_info, observed_expected_ratio=true, source_qnode_id=n00, target_qnode_id=n01, virtual_relation_label=CP1)", "overlay(action=predict_drug_treats_disease, source_qnode_id=n01, target_qnode_id=n00, virtual_relation_label=P1)", "filter_kg(action=remove_edges_by_attribute, edge_attribute=probability_treats, direction=below, threshold=0.75, remove_connected_nodes=true, qnode_id=n01)", "overlay(action=compute_ngd, source_qnode_id=n00, Pain Points for Pain Meds - Detailed Data Spelunking 9 target_qnode_id=n01, virtual_relation_label=N1)", "resultify(ignore_edge_direction=true)", "filter_results(action=limit_number_of_results, max_results=50)" ara-arax … Methadone CHEBI:6807 (methadone) was given as a result (this was not included in the Molecular Provider’s results above), presumably based on a result from BTE’s SEMMED Disease API.

"confidence": 1, "id":

"0322a6135a640fb89f350a3a93b98b1a5a5ee4f1b448c367a66ba6d8087d1523", "is_defined_by": "BTE", "provided_by": "SEMMED Disease API", "qedge_ids": [ "e00" "source_id": "UniProtKB:P22303", "target_id": "CHEBI:6807", "type": "treated_by"

Methadone is used as an analgesic in chronic pain, often in rotation with other opioids (https://en.wikipedia.org/wiki/Methadone#Pain).

I was not able to identify exactly what disease identifier was used to query this knowledge provider to produce this result, but am very grateful to the developers for this verbosity while I examine some results in detail.

ara-arax … Acetaminophen CHEMBL.COMPOUND:CHEMBL112 (ACETAMINOPHEN) was not included within the 50 "results", but it clearly appears in the "knowledge_graph", with multiple "treated_by" assertions from MyChem.info API and SEMMED Disease API (via BTE). So, it must have been removed at one of the filtering steps.

There’s clearly a lot going on here, and tantalizing messages like "Querying Columbia Open Health data for info about AChE and ACETAMINOPHEN" that make me want to go back dig deeper later.

https://smart-api.info/ui/ed0ee52921c7cbce24033ffd1369922e https://en.wikipedia.org/wiki/Methadone#Pain

Pain Points for Pain Meds - Detailed Data Spelunking 10 ara-bte … Disease expansion https://ars.transltr.io/ars/api/messages/fc7414d4-1f64-48f6-a4be-0775a6486227 This service produces 839 results for this query.

This ARA clearly takes the input MONDO:0021668 and further expands it somehow,…