Attachment_1_-_Draft_Curation_and_Harmonization_SOW_1.pdf

STATEMENT OF WORK

2.1 PURPOSE

The purpose of this Statement of Work (SOW) is to supply specialized metadata development support through curation, harmonization, model annotation, model loading, and program support on behalf of the National Institutes of Health (NIH) National Cancer Institute (NCI) Center for Biomedical Informatics and Information Technology (CBIIT) Semantics Infrastructure (SI) program.

2.2 BACKGROUND

The NCI CBIIT supports informatics, data, and information technology needs of our NCI programs, collaborators, and the cancer and cancer research community. The NCI CBIIT Metadata Content Development Team has supported the cancer research community in understanding, creating and registering metadata standards, content, forms and models for 24 years. This specialized team of experienced metadata content experts organizes metadata through centralized curation of research question and answer sets, creating Common Data Elements (CDEs) and Case Report Forms (CRFs) to make them available through the cancer Data Standards Registry and Repository (caDSR II) II, for wide distribution and use. The caDSR II is a publicly accessible metadata registry and repository where content can be viewed and downloaded by all. Creation of new content must be centrally curated by specially trained metadata curators who go through rigorous training and who adhere to established Business Rules, Best Practices, SOPs and Governance to manage and maintain content as a high quality, trusted resource for the larger public user community. These documents can be found in the Metadata Content Development Team Knowledge Base at https://service.cancer.gov/cadsr-curation?id=nci_ext_kb_homepage.

The caDSR II content is organized in collections by group, or Context, with Stewardship Organizations (SO) who oversee their content through interaction with our Metadata Content Development team. Each Context has collections of classifications and forms.

Classifications are the sets of CDEs organized into similar collections. Forms are Case Report Forms (CRFs) or Data Collection Templates (DCTs). This team also assists in creating content through model annotation and loading for larger sets of data. For more information see: CBIIT Metadata Services at https://datascience.cancer.gov/resources/metadata.

The caDSR II adheres to ISO/IEC 11179 standard for metadata definition and registration.

The semantic structure of all CDEs include the foundational controlled terminologies and ontologies provided by NCI CBIIT Semantic Team partners, the Enterprise Vocabulary Services (EVS): NCI EVS Wiki. The EVS Terminology Team creates and manages content for the NCI Thesaurus (NCIt), NCI Metathesaurus for cross-terminology mapping, and NCI’s terminology services infrastructure that is the source of the concepts for CDEs. NCI’s Metathesaurus is leveraged for cross-terminology mapping. The Metadata Content Development Team works closely with the EVS Team and together constitutes the Semantic Infrastructure (SI) Team.

The SI Team plays a crucial role in providing specialized semantic support to stakeholder communities by ensuring that data and metadata are semantically structured to enhance Big Data collection and analysis, Data Commons initiative operations, enables precise semantic annotation of large-scale biomedical datasets, supports AI-driven insights and advanced analytics for cancer research. Semantic structuring of clinical and pre-clinical research data facilitates harmonization and interoperability across diverse datasets and enhances data integration and sharing within cancer research and clinical care, ultimately driving better research outcomes, discovery, and innovation in precision medicine.

The NCI CBIIT Metadata Content Development Team serve as the registration authority for all content in the caDSR II. Metadata curation ensures data collected for NCI protocols are common and replicable, preserving data through time in a consistent, expected way that is machine readable and structured for aggregation and advanced analytics. Business Rules, Best Practices and Governance, and approaches to maintenance of content are harmonized at the registry level. The caDSR II SOs meet with the Metadata Content Team on at least a quarterly basis to exchange information and enhance data practices as part of the caDSR II User Group. This approach has advanced and specialized the process of creating a unified, shared set of best practices for maintaining CDEs and metadata development in the caDSR II. It provides a community driven process for eliminating redundancy, provide updates on content and new projects, while eliciting input and giving them the opportunity to ask questions and pose suggestions.

The NCI CBIIT Metadata Content Program Manager leads the contractor Metadata Content Development Team and serves to provide leadership and oversight for the team as they create, manage, and maintain metadata content, CDEs, associated metadata and peripherals supporting materials describing operations. Operational support for metadata services is a key function of this Metadata Content Team who serve as consultants to our community and assist with training and collaborative standards work.

caDSR II Metadata Community Training is a specialized training curriculum supported by the Metadata Content Team to ensure the community has the skills and capability to utilize caDSR II, search and browse content, and build Case Report Forms (CRFs). The Metadata Content Development Team provides a multi-dimensional training program developed to support the di erent training needs of the community. The team provides live, instructor led group training, annotated PowerPoint (PPT) training slides, and training videos to support the varied learning needs of the community. These courses are led by the contractor Metadata Curation Team that shall be responsible for producing additional materials on behalf of the government to support evolving needs of the scientific community including the evolution of the model annotation and model loading requirements. Training is moving entirely to a self-driven learning model that will be available 24/7 in the coming years.

The NCI cancer Data Standards Registry and Repository (caDSR II) consists of a database, APIs, and web-based tools for creating and using data standards for cancer research. The infrastructure supports the o icial registration and management of NCI standards for CDEs, CRFs, and data models used in NCI Clinical Trials and other cancer and healthcare studies.

2.3 OBJECTIVE

The primary objective of this Statement of Work is to ensure that cancer research metadata and semantic standards remain robust, interoperable, and adaptable to emerging scientific and technological advancements. The contractor shall supply specialized metadata development support for curation, harmonization, form/template building, model annotation and loading services, and operational program support on behalf of NCI CBIIT. The contractor shall provide support and sta a Metadata Content Development Team and provide additional cancer pre-clinical, clinical and clinical research expertise as detailed in Table 7.1 in section 7.1.1 to support metadata development work. This team shall serve to support daily operations and stakeholder management and support.

More specifically, the objectives are:

• Metadata Standards Curation

• Develop and curate Common Data Elements (CDEs)

• Develop and maintain internal and external metadata standards

• Conduct mapping, harmonization, and model annotation

• Build forms/templates as directed

• Load structured metadata into caDSR II

• Collaborate with EVS content SMEs to support the e ective data flow for development of structured terminology for creation of Common Data Elements.

I. II. Governance and Data Quality Enhancement Support caDSR II Metadata Governance, Business Rules, and Best Practices to maintain metadata quality

• Develop the Yearly Harmonization Plan and Monthly Harmonization Reports to improve data integrity

• Produce the Annual Harmonization Report and present findings to federal lead and

CBIIT Semantic Infrastructure Director

• Expand and promote repository data standards collections

• Enhance methodologies for metadata standards development

II. III. Training and Community Engagement

• Provide community support services, mentorship, and oversight.

• Develop and promote training programs and curriculum for stakeholders.

• Enhance the expertise of the Metadata Content Development Team.

III. IV. Technical and Operational Support

• Provide feedback on caDSR II software and services usability and functionality.

• Support UAT (User Acceptance Testing) for software updates.

• Manage operational change for upgrades, bug fixes, and enhancements to caDSR II and Centralized Curation.

• Ensure smooth knowledge transfer and operational support for contractor transitions.

2.4 SCOPE

The Contractor shall provide several general categories of consulting in support of:

planning, process improvement, management, information technology operations, subject matter expertise, metadata development, curation, mapping, harmonization, form/template building, model annotation and loading, governance, Business Rules and Best Practices, Standard Operating Procedure (SOP) development and maintenance, harmonization planning and reporting, metadata Quality Control (QC) and validation, maintenance, content versioning and retirement, user training, publishing, and support services to CBIIT.

The primary scope includes providing expert, specialized professional services to conduct Metadata Standards Development, Curation, Form/Template Building, Mapping and Harmonization, data model annotation/loading services. Services primarily support NCI branches, such as the Cancer Therapy Evaluation Program (CTEP), Division of Cancer Prevention (DCP), National Cancer Informatics Program (NCIP), Center for Cancer Research (CCR) and other cancer pre-clinical, clinical, clinical research domains. We also support National Cancer Institute-Molecular Analysis for Therapy Choice (NCI MATCH) protocols and Precision Medicine Initiative (PMI) protocols, and support of Cancer Research Data Commons (CRDC) Data Standards Services (DSS).

Contract scope also includes providing additional support to select projects and groups at the direction of the NCI CBIIT Metadata Content Program Manager to include focused NIH Institutes and their stakeholder communities, select trans-NIH programs such as the National Heart Lung and blood institute (NHLBI), Clinical Proteomic Tumor Analysis Consortium (CPTAC), Molecular Characterization of Screen-Detected Lesions MCL consortium, Human Tumor Atlas Network (HTAN), Rare Tumors Groups, NIH CDE Governance Committee, etc. As well as, when directed, individual projects in support of professional organizations such as the Clinical Data Interchange Standards Consortium (CDISC), American Joint Committee on Cancer (AJCC), North American Association of Central Cancer Registries (NAACCR), Surveillance, Epidemiology, and End Results (SEER), College of American Pathologists (CAP), O ice of the National Coordinator for Health Information Technology (ONC), etc., or other government agencies (ex. Department of Health and Human Services (DHHS), Food and Drug Administration (FDA), and standards organizations such as Health Level Seven (HL7) as time and resource allow. In coming years, we shall assume support for the NCI Cancer Research Data Commons (CRDC) Data Standards Services (DSS) and modeling and mapping services along with the representative stakeholder communities.

These services shall include, but not be limited to:

• Providing specialized metadata standards development, curation, mapping, harmonization, forms/template building, model annotation and model loading services; CDE life cycle management and support; Pre-curation activities including supporting stakeholders in content organization, standards selection, and finalization of forms to ready the content for metadata development and curation services.

• Supporting the Semantic Infrastructure (SI) Team and working in parallel with NCI Enterprise Vocabulary Services (EVS) to identify terminology gaps and collaborate with EVS SMEs to support the development of Terminology and Mappings to be utilized for Common Data Element (CDE) curation and to support stakeholders in preparing metadata for curation services.

• Maintaining and evolving the Metadata Development Governance Plan for metadata curation and harmonization. This plan, updated yearly, is designed to direct the Metadata Development Business Rules and Best Practices, SOPs and drive overall decision making about content and curation services.

• Supporting and advising on the creation of the new NCI Metadata Executive Governance Committee which includes resources from other NCI Division, O ices, and Centers (DOCs) that are familiar with metadata use and who provide expertise and insight from multiple domains to best advice on NCI Standards curation, updates, retirement, and maintenance.

• Maintaining and evolving the Metadata Harmonization Plan as the Quality Control (QC) program for the caDSR II. The plan applies metrics of use to metadata content development within the caDSR II supported by the established Business Rules and Best Practices and driven by the Governance Plan. The Harmonization metrics are captured and recorded quarterly and used to guide Stewardship Organizations in metadata versioning, maintenance, and retirement within their Context and reuse of standards across the stakeholder community to ensure high quality, publicly accessible metadata and standards to support the cancer community.

• Producing a yearly Harmonization Report that serves to provide public information about the quality of the metadata maintained in the repository. It serves to validate the services of the Metadata Content Team in support of the community and validates the metadata and standards maintained in the caDSR II as a high-quality, semantically sound resource available to the community and the public at large.

Regular review and update mechanism yearly at beginning of FY.

• Partnering with the NCI CBIIT Metadata Content Program Manager and stakeholder community to, 1) identify metadata standards that need to be promoted within the NCI community, 2) review existing external standards and use cases for maintenance and retirement, and 3) create documentation with supporting details for standards approval, to include recommended details on:

o Enhancing methodology for metadata development, o CDE curation and harmonization as needed, evaluating metadata content annually; participating in NCI, NIH and trans-NIH taskforce activities to improve and enhance metadata standards development and CDE curation operations and services to improve repository data quality.

• Providing and planning for on-going support to SOs and stakeholder communities through participation in stakeholder meetings, user group meetings and directly through one-on-one assistance with curation and curation training, continuing education, support and guidance.

• Providing and promoting training, training curriculum development, and skill development services for metadata standards development, metadata curation and harmonization services, form and template building, and model annotation and loading using caDSR II and integrated tools. The contractor shall leverage their experience and metadata expertise in curation and harmonization and their extensive knowledge base in existing content and contexts while creating and delivering course materials and continuing education materials, to include distance learning and various skill building support to di erent segments of new and existing caDSR II users.

• Training shall be delivered in three modes to meet the multi-dimensional training needs of the community, through trainer led Webex courses, annotated PowerPoint slide modules, and web-based, user driven training videos. This training focuses on the caDSR II aspects that users need to understand how to use the platform and tools to search and browse the repository content; how to prepare and submit projects for curation; develop forms in the form builder; and how to prepare models for annotation and model loading. Information and training shall be developed to support the communities understanding of what metadata is and how to prepare for centralized curation as well as describing how governance, SOPs, business rules, and best practices are used to develop and maintain quality content available in the public repository.

• Engaging in improving and enhancing Metadata Content Development Team expertise enhancement by attendance and participation in training for new standards, new content development practices, or in support of existing projects, as needed, where training will enhance NCI metadata curation and harmonization and at the request of the NCI CBIIT Federal Lead and approval of the Contracting O ice’s Representative (COR)

• Participating in group face-to-face meetings, standards organization meetings or context support meetings, as needed, and as requested by the NCI CBIIT Metadata Development Content Team Program Manager and with the approval of the COR.

• Providing functionality and usability feedback and enhancement identification of caDSR II metadata software tools and services for the caDSR II Software and Development Team based on Metadata Content Development Team use of the software and tools and in support of community requirements. Metadata Content Development Team engagement shall be required for all Usability and Acceptance Testing (UAT) for all updated or new software and services that involve metadata development and tools associated with the NCI metadata repository.

• Providing operational support for NCI CBIIT in operations of caDSR II and integrated tools, to include operational change management support for centralized curation.

This shall require updating website and wiki sites, redesign and creation of new training and support materials for the community, and any artifacts needed to support change management and the needs of the community. The Contractor shall expect to conduct numerous web-based incremental community support presentations, demonstrations, and trainings regarding the centralized curation model and in the use of t infrastructure by the community and Stewardship Organizations.

• Establishing a transition plan for the objectives, tasks, knowledge, and materials required, obtained, and developed under this SOW and awarded contract in the event that another contractor is awarded a follow-on contract for NCI CBIIT Metadata Development, Curation, Harmonization and Support Services. This Includes planning for operational change, support to Stewardship Organizations and stakeholder communities and knowledge transfer to support ongoing projects.

All materials shall be transferred in the formats specified, COR/Federal Lead delivered to the location directed, and in the COR/Federal Lead timeframe requested. This transition shall include the labor categories and level of e ort currently supported should a transition be required.

2.5 TASK AREAS

Independently and not as an agent of the government, the Contractor shall furnish all the necessary services, qualified personnel, material, equipment, and facilities, not otherwise provided by the government, as needed to perform Task assignments at the direction of the Federal Lead and COR in the Task Areas specified below.

2.5.1 Task Area 1: Contract Management

The Contractor shall provide the technical and functional activities necessary for the management of this SOW and all tasks provided under this SOW. The Contractor shall provide a Contract Management Plan outlining the tasks, sub tasks, and approaches to be used to provide the required services including the specific deliverable documents, analyses, and reports to be developed. When approved by the Federal Lead and COR, this plan shall be used to monitor, control, and report progress, issues, and resource expenditure.

The Contractor shall participate in contract and task management meetings, scheduled status updates, and provide oral and written monthly status reports monitoring the performance and progress toward successful completion of this SOW.

The Contractor shall ensure that all personnel are adequately trained. This includes, but is not limited to, the use of standard Microsoft management and documentation applications as well as all applicable confidentiality, privacy, information security, and conflict of interest regulations and corresponding policies. HHS security and privacy policies are found at https://www.hhs.gov/ocio/securityprivacy/.

Contractor/subcontractor sta shall sign Non-Disclosure Agreements as they have access to data that contains personally identifiable and/or business sensitive information.

Contractor personnel shall be certified caDSR II Curators and represent a Labor Skills Experience Mix as detailed in Section 8.1. in Table 8.1.1. Experience Requirements.

2.5.1.1 Contract Start-Up Meeting

Within ten working days of award, the Contractor shall participate in a Contract Start-Up Meeting and present an orientation briefing for the government. The intent of the briefing is to initiate the communication process between the government and Contractor by introducing key Task participants and explaining their roles, reviewing communication ground rules, and assuring a common understanding of the work to be accomplished under this SOW.

2.5.1.2 Status Meetings and Reports

The Contractor shall participate in recurring weekly management meetings to provide status updates, bi-weekly sta meetings, and present monthly status reports monitoring the performance and progress toward successful completion of this SOW. Known changes to work schedules, risks, mitigation recommendations and costs must be identified and discussed in these meetings. Report templates, distribution and posting location shall be established in kick-o meeting.

2.5.1.3 Contractor Interfaces

The Contractor and/or its subcontractors may be required, as part of the performance of this e ort, to interact with other contractors working for the government. Such other contractors shall not direct this Contractor and/or its subcontractors in any manner. Also, this Contractor and/or its subcontractors shall not direct the work of other contractors in any manner.

When appropriate, the government shall establish an initial introduction and contact between the Contractor and other contractors and participate in an initial meeting at which the conventions for the scheduling and conduct of future meetings/contacts shall be established. Any Federal Leads and CORs of other contracts e orts shall be included in any establishment of conventions.

The Contractor shall work closely with other CBIIT contractors to document and implement the procedures to be used to ensure open, timely, and e ective communication, information exchange, and reporting among the contractors and the government.

2.5.1.4 Contractor/Subcontractor Personnel

The Contractor shall be responsible for managing and overseeing the activities of all Contractor personnel, as well as subcontractor e orts used in performance of this e ort.

The Contractor's management responsibilities shall include all activities necessary to ensure the accomplishment of timely and e ective support, performed in accordance with the requirements contained in the SOW.

Contractor Personnel Resumes submitted for key personnel assigned to perform under this SOW shall document experience directly applicable to the functions to be performed.

Further, these prior work experiences shall be specific and of su icient variety and duration that the employee is able to e ectively and e iciently perform the functions assigned.

All contracting team members shall demonstrate proficiency on all aspects of caDSR II tooling and semantic content development for federal lead approval. Training for achieving this proficiency shall be provided by the government.

Contractor/subcontractor sta shall submit to background checks and fingerprinting to secure Personal Verification Validation (PIV) cards which are required for all NIH/NCI laptops and access to all software and services. This is essential for all personnel, including sta working remotely, and is not optional.

2.5.1.5 Project Manager

The Contractor shall provide a Project Manager to facilitate government-Contractor communications. The Project Manager shall be the primary technical and managerial interface between the Contractor and Contracting O icer (CO) and the COR/Federal Lead.

The name of this person, and an alternate(s), who shall act for the Contractor when the Project Manager is absent, shall be designated in writing to the CO/COR.

The Project Manager or alternate(s) shall have full authority to act for the Contractor on all contract matters relating to daily operations. The Project Manager and/or alternate(s) shall be able to read complex and inter-related documents, write professionally, speak clearly, and understand English. The Project Manager or alternate shall be available during normal duty hours, as specified herein and to meet with government personnel within 24 hours to discuss problems.

The Contractor's Project Manager shall meet with the CO, COR and/or Federal Lead as necessary to maintain satisfactory performance and to resolve other issues pertaining to government/Contractor procedures. At these meetings, a mutual e ort shall be made to resolve any and all problems identified. Action items stemming from these meetings shall be documented by the Contractor, ownership assigned, and timeframe proposed, and furnished to the government within two (2) workdays of the subject meeting.

Project management support shall be provided as required to assist CBIIT business operations in developing management processes and planning, costing, and tracking ongoing projects to ensure e ective and e icient use of resources and adherence to CBIIT approved management methodologies, standards, and practices.

2.5.2 Task Area 2 – Project Management Support

The NCI CBIIT Metadata Content Development Project Manager requires contractor operational support to assist with program oversight and significant program change management activities. CBIIT Metadata Services is experiencing a rapidly expanding portfolio of projects and stakeholders while, in tandem, new infrastructure and tools are in development, such as semi-automation tools and visualization tools, that will significantly change how the Metadata Content Team does their work. At the direction of the COR/Federal Lead, additional support for program planning, process reengineering, and community deployment shall be required. The Contractor shall support careful execution with program performance reviews, stakeholder alignment and buy-in, and extensive communications which require support and oversight.

Project Management assistance shall be provided for the CBIIT Metadata Content Program in support of the COR, Federal Lead, and Federal Program Director. Duties shall include providing support for tasks associated with project support assigned by the COR/Federal Lead to include overseeing and planning for Contractor Metadata Content Team e orts at the direction of the COR/Federal Lead to specify, develop, manage, administer or review discrete program and project management activities within and between administrative, technical and scientific domains.

Activities shall include but are not limited to the following tasks:

• Supporting specific individual program and project management activities including, but not limited to, developing strategies, defining scope, estimating resource requirements, planning, scheduling, tracking, and reporting.

• Supporting specific individual program and project governance activities and requirement.

• Assessing and reporting on program and project progress, risks, and issues and recommending approaches, changes, and mitigation strategies to assist program manager.

• Supporting program and project financial planning, tracking, and reporting.

• Ensuring that applicable policies, regulations, standards and processes are followed and adhered to.

• Assessing projects for assignment, forecasting performance against approved plans, and recommending changes and strategies for success.

• Monitoring execution and reporting on program and project progress against approved plans, schedules, and budgets.

• Assisting federal program managers in identifying and implementing best practices, implementation guidelines, and management approaches.

• Recommending risk mitigation strategies, issue resolution approaches, and steps for project management improvement e orts.

• Supporting quality assurance implementation, management, tracking, and reporting.

• Assisting with management reviews and meeting facilitation.

2.5.3 Task Area 3: Transition Planning

The contractor shall develop a Transition-In Plan for submission and discussion at the Contract kick O meeting. This plan shall include background information, workflow and processes, work plans and procedures, and specific activities the Contractor shall use and conduct to adequately and e ectively transfer roles, responsibilities, procedures, and documentation from the out-going contractor and ensure required knowledge transfer is conducted, and smoothly and e ectively transition work and ongoing activities from an outgoing contractor.

In the final period of the Contract, or in the event the government requirements necessitate a re-competition of the requirements of this SOW, and prior to completion of this Contract’s period of performance, the Contractor shall develop and deliver a Transition-Out Plan. The purpose of the Transition-Out Plan is to ensure an orderly and successful transition of ongoing Tasks and functions should the incumbent Contractor not be selected during the re-competition. This Transition Plan shall include processes, procedures, and activities the Contractor shall use, if not selected for a follow-on contract, to adequately educate and train the successful contractor’s employees and smoothly and e ectively transition work activities to the new contractor.

2.5.4 Task Area 4: Program Support Tasks

The contractor shall provide contractor support and as needed subcontractor specialized support to sta the Contractor Metadata Content Development Team for the purposes of providing operational support for Metadata Development, Curation, Harmonization, and Community Support Services. This team’s services shall be delivered in support of NCI CBIIT in conducting metadata standards development services for the pre-clinical, clinical and research community in parallel with the EVS Team to support semantic operations for the stakeholder community. Stakeholder community primarily consists of the cancer research community and associated precision medicine domains but can also support NCI, NIH a iliated projects and other federal standards e orts.

2.5.4.1. Metadata Content Team Operational Support

The Contractor Metadata Content Development Team shall provide operational and community support collaboratively across NCI, CBIIT, CTEP, DCP, other NCI divisions and specified NIH Institutes and Center (ICs) to develop and obtain consensus on procedures supporting Business Rules (BR), Best Practices (BP) and Governance in developing standards and harmonizing and curating content in the caDSR II. Central to the work of the Metadata Content Development Team shall be providing support to collaborations between NCI, NIH, and other government division and branches and their stakeholder organizations in metadata standards development and harmonization. Core stakeholders include NCI Lead Protocol Organizations (LPOs), sponsors, Cancer Centers and the caDSR II SOs and user communities, NIH IC program leads and their communities and select government agencies. Support shall, almost entirely, require remote-based, web-based tele-conference participation in meetings with government and government stakeholders and community stakeholders to accomplish the requirements of this contract.

Occasionally, and upon COR/Federal Lead and/or CO approval, resources may be required to travel to NCI in Rockville, Maryland or other facilities for security, credentialing, technical support, legal proceedings, or to participate in training or meetings that require in-person attendance.

Daily duties shall center around working with stakeholders and community members in preparing content for centralized curation, supporting groups in structuring and organizing their metadata and selecting standards that best support their use cases. Contractor Metadata Content Development Team shall participate in stakeholder and community training that supports caDSR II search and find, and development of forms and templates in caDSR II for the purpose of data collection. Question and answer sets shall be curated as CDEs, bound together in time, gathered into collections of CDEs and rendered as forms for use in data collection for cancer research, clinical trials, precision medicine, or other purposes.

This specialized team shall provide support for CDE life cycle management, BRs, BPs and Governance of the metadata developed and stored in the caDSR II for the purpose of oversite and Quality Control (QC). The BRs and BPs guide content development to reduce duplicate DEs and ultimately creating high quality, structured metadata in a standard, replicable way. The Contractor Metadata Content Team shall also engage in complex, multi-branch, multi-organization projects working with other integrated government and contractor team members and supporting operations` across the NCI cancer research continuum.

Examples of projects that require metadata support include:

• Working with SO’s to organize data for curation, supporting SO’s in structuring and aligning metadata in Clinical Trial data collection forms and eligibility checklists, applying NCI and other Standards.

• Providing support for PROMIS QoL and other validated measures, NAACCR, SEER, Rare Tumors, PMI, HL7 FHIR, mCode, and other standards organizations to harmonize, develop and curate content in support of NCI, NIH, and caDSR II community to better support data collection e orts.

• Providing support for the Clinical Data Exchange Standards Consortium (CDISC) in harmonization of NCI Standards and forms with Clinical Data Acquisition Standards Harmonization (CDASH) for data collection and the Study Data Tabulation Model (SDTM) for reporting data in accordance with the FDA mandates and for supporting working groups within CDISC and the resultant expansion of terminology updates and data model expansion versions to better support the oncology domain.

• Providing support for HL7 for informing committees and providing written feedback on structuring data, standards and metadata for developing guidance and protocols promoting interoperability.

The NCI CBIIT Semantic Infrastructure (SI) Team is made up of the EVS Team, the Metadata Services Team, and the data architecture team. The highly specialized Contractor Metadata Content Development Team personnel are members of the Metadata Services Team who work closely with the EVS Team to provide support for semantic structuring of data for interoperability and analysis. The Contractor Metadata Content Development Team shall work with EVS to define concepts, have them edited and added to caDSR II which make up the foundational components of all CDE development. Working with new projects and organizations, the EVS and Metadata Content Development Teams shall frequently partner in preparing projects for metadata support. As the EVS and Metadata Content Development Team’s infrastructure expands, additional cooperative training and functionality analysis may be required.

Contractor support shall include but is not limited to the following tasks:

• Managing working groups and subcommittee meetings, user group meetings and stakeholder teleconference meetings, scheduling and calendaring the resources, developing the primary presentation materials and supporting documentation, setting up teleconference equipment, recording meetings, taking notes, and following up on action items.

• Attending regularly scheduled stakeholder and community meetings, participating actively, assisting with notes, accepting action item tasks and completing assigned action items or tasks.

• Working with existing and new stakeholders and Content Owners (COs) as Subject Matter Experts (SMEs) to support pre-curation activities, such as review of disparate data and data sources to organize data collection forms and templates for easier CDE curation; assisting with decisions on which standards support their work best;

helping to understand NCI data collection standards, permissible value lists, and codes required across NCI and NIH; and reusing NCI Standards and CTEP CDISC Standards.

• Working with existing and new stakeholders and COs to support organizations SMEs in aligning content with CDISC standards, recommending variable naming conventions; short name and long name recommendations; curating content and reusing existing CDISC variable CDEs; curating custom variables as CDEs;

understanding CDISC Implementation Guidelines; recommending CDEs for retirement; leveraging standards in a proactive manner; and supporting users in versioning changes; assisting stakeholders and COs in documentation needs.

• Providing support for CDE life cycle management, Business Rules, Best Practices, SOPs, and Governance of the metadata stored in the caDSR II.

• Working to eliminate duplicates; mentor and assist SOs with editing, versioning, and retiring metadata according to metric reports and harmonization plan reports.

• Working with Big Data, Data Commons, large biorepository stakeholders to bulk load metadata; assist in preparing metadata for curation; harmonize data sets;

review and analyze data for alignment with standards; mapping multiple data sets;

curating into collections; registering metadata; preparing them to develop forms and data collection templates.

• Assisting stakeholders and COs in building forms and data collection templates in caDSR II carts for exporting in ODM or XML downloads, or annotating ALS for export.

• Assisting stakeholders and COs with data model annotation.

• Reviewing new data standards for potential support of stakeholder communities;

providing feedback on how these might reinforce data standard collection and identify any issues with the new standards.

• Attending weekly Federal Lead meetings, bi-weekly contractor team meetings, and as required COR, Federal Lead, and Semantics Infrastructure Director meetings.

• Active participation and follow-up and accepting and acting on tasks within individual area of specialization expertise are required.

2.5.4.2 Comprehensive Metadata Content Development and Harmonization in the caDSR Plan

The Contractor shall develop and maintain a Comprehensive Metadata Content Development and Harmonization in the caDSR Plan yearly based on the previous year’s plan furnished by the Government and any contract modifications that impact the year’s plan of work. The plan describes a process for the management and maintenance of NCI’s CDE’s, and associated metadata stored in the caDSR II. The Plan shall set forth how the contractor shall plan to monitor the success of curation and harmonization activities and the implementation of BRs, BPs and Governance and include plans for assessing roles and responsibilities, expedited reviews, community vetting, additions and revisions to current standards, and the role of dynamic extensions. The purpose of this Plan is to define the tasks performed to identify obsolete and inconsistent components in both the CDEs and the caDSR II overall, to harmonize and consolidate the registry metadata to reduce redundancy and to surface preferred metadata for reuse.

The Plan shall set forth goals for content improvement and a vigorous metrics program to monitor and measure progress towards these goals utilizing caDSR II software and database resources and ServiceNow tools to measure and analyze caDSR II content.

Contractor shall in the course of this work provide feedback to caDSR II software developers and ServiceNow software team on functionality and usability for the purpose of creating and maintaining the Plan.

The caDSR Metadata Content Team and other applicable NCI Center for Biomedical Informatics and Information Technology (CBIIT) sta and contractors shall be responsible for implementation of this Plan. The Comprehensive Metadata Content Development and Harmonization in the caDSR Plan addresses the full metadata lifecycle including discovery, development, creation, reuse, curation, harmonization, maintenance, and standardization of well-formed CDEs and associated metadata. The metadata collected from diverse sources such as data collection forms, measures, UML models, data dictionaries and legacy computer applications shall be included in the Plan activities.

Harmonization activities are included to address governance and management of the caDSR II metadata supporting NCI programs, NCI partners, the user community, and commercially available Clinical Data Management Systems (CDMS.)

The Contractor Project Manager (CPM) shall review the draft Comprehensive Metadata Content Development and Harmonization in the caDSR Plan with the Federal Lead and make edits as needed per Federal Lead until a draft version is approved and accepted as final. The Comprehensive Metadata Content Development and Harmonization in the caDSR Plan draft shall be delivered within 30 days of contract year kick-o and shall be finalized and approved within 60 days of award. Once approved, the Contractor shall maintain the plan on an ongoing basis for the period of performance. The contractor Metadata Content Team shall use caDSR II software and database resources and ServiceNow tools to implement the Plan.

2.5.4.3 Quarterly State of Metadata Content Development and Harmonization in the caDSR Report

The Contractor Project Manager (CPM) shall monitor the success of metadata content development and harmonization throughout the quarter and produce a written Quarterly

State of Metadata Content Development and Harmonization in the caDSR Report. The CPM shall review report with the Federal Lead and make edits as needed per Federal Lead until the draft version is approved and accepted as final. The report shall include progress toward the Comprehensive Metadata Content Development and Harmonization in the caDSR Plan, any inconsistencies, issues, mitigation strategy, and highlighting changes made within the quarter.

2.5.4.4 Quarterly State of Metadata Content Development and Harmonization in the caDSR Summary PPT Slides

The Contractor Project Manager (CPM) shall produce the Quarterly State of Metadata Content Development and Harmonization in the caDSR Report Summary PPT Slides highlighting the results and findings from the quarterly report. These slides shall be reviewed with the Federal Lead and edits shall be made as needed per Federal Lead until the draft version is approved and accepted as final. The quarterly report PPT slides shall be presented at the next caDSR II Content Team Meeting which is a quarterly caDSR II user group meeting.

2.5.4.5 Quarterly Individual Steward Organization (SO) State of Metadata Development and Harmonization in the caDSR Reports

The Contractor Project Manager (CPM) shall produce written Quarterly Individual Steward Organization (SO) State of Metadata Development and Harmonization in the caDSR Reports for each SO in coordination with the Quarterly State of Metadata Content Development and Harmonization in the caDSR Report. These SO Reports shall be emailed to each SO and the assigned Metadata Content Team member who supports all mentoring activities for that SO in order that they can work together to improve SO metadata quality.

The assigned mentor shall identify inconsistencies and work with SO to edit, version, eliminate, duplicate and retire no longer needed content, highlighting changes and inconsistencies.

2.5.4.6 Year End Comprehensive Metadata Content Development and Harmonization in the caDSR II Report

At contract year end the Contractor Project Manager (CPM) shall submit a detailed written Year End Comprehensive Metadata Content Development and Harmonization in the caDSR II Report that details progress over the year through a thorough impact assessment that articulates the status of all activities, the state of CDE development and harmonization, identifies barriers and risks encountered and provides recommended mitigations. The Report shall include any recommendations for future work. The report shall include a table of the contractors supporting the work throughout the year, detail the trainings performed and a detailed table of CDE counts for the year.

2.5.4.7 Year End Comprehensive Metadata Content Development and Harmonization in the caDSR Leadership PPT and Oral Report

The Contractor Project Manager (CPM) shall prepare a Year End Comprehensive Metadata Content Development and Harmonization in the caDSR Leadership PPT summarizing and highlighting key details from the yearly written report. The PPT shall be provided to the Federal Lead and/or the COR, CO, Semantics Infrastructure (SI) Director, and the Clinical and Translational Research Informatics Branch (CTRIB) Chief. The PPT and oral report serve to support a detailed discussion of the past year’s work, issues, mitigation strategies, and any recommendations the CPM would like to detail for consideration in the next OY deliverables and tasks.

2.5.4.8 caDSR II and Metadata Software and Tooling User Acceptance Testing (UAT) and Reports

The NCI caDSR II infrastructure was modernized on May 22, 2022, made more flexible, easily configurable, standards-based, with secure, role-based infrastructure and integrated tooling. The new infrastructure provided the Metadata Content Development Team the ability to provide the most requested service by the community, centralized curation.

caDSR II shall continue to provide new software, tooling, services, and processes to keep caDSR, metadata curation, and our community at the forefront of scientific modernization.

NCI has invested in additional software tooling and plans to participate in review of other software solutions meant to ease the burden and improve e iciency of curation to better support metadata content creation, harmonizing, and mapping. The contractor Metadata Content Development Team shall participate in meetings, provide feedback, and document recommendations and outcomes with the caDSR II Software and Development Team and other groups to evaluate and provide feedback on the needs of the curation team and the community.

caDSR II and Metadata Software and Tooling User Acceptance Testing (UAT) and Reports Documentation shall include, but not be limited to, requirements and use case documentation, usability testing, workflow tracing, acceptance documentation, and bug reporting to ensure that the software and tooling will support the needs of the NCI in metadata evaluation, curation, harmonization and development. Additionally new training and skill building shall be required both for the contractor Metadata Content Development

Commented [BD1]: Same story with this section. Two deliverables but muddled together so will need to carve it out.

Team and the community for new software and tooling to support metadata development.

Updated training materials and general informational materials shall need to be produced so that the community’s exposure to the new infrastructure is well informed.

The contractor Metadata Content Development Team shall provide a mechanism to collect community use cases and requirements to inform and enhance functionality. Expected future services include centralized form building and new tooling that supports semi-automation and use of AI for key curation tasks.

Contractor activities shall include, but are not limited to, the following tasks:

• Documenting usability issues encountered during daily curation work and support of projects and tasks.

• Providing workflow-based feedback on proposed enhancements

• Validating that software changes support metadata curation best practices

• Reporting functionality gaps that impact curation e iciency

• Attending and actively participating in caDSR II Software Development Team and other software development teams' presentations and/or demonstrations of caDSR software and tooling development for the purpose of evaluating development plans and software for functionality and usability for Metadata Content Development Team and community needs.

• Conducting thorough User Acceptance Testing (UAT) of any software and tooling utilized in support of Metadata Content Development Team and community. The Metadata Content Development Team shall provide caDSR II and Metadata Software and Tooling User Acceptance Testing (UAT) and Reports which consist of detailed, written feedback highlighting what the team felt enhanced or made more e icient, operations and utilization, what worked and why, what didn’t work and why, and any enhancement the team feels might ease use or limit barriers.

• The Contractor Project Manager (CPM) shall harmonize UAT feedback and provide to the caDSR Federal Lead and caDSR II Software Development Team in the template provided by the government within 1 week of presentation.

2.5.4.9 caDSR II Software Issues and Bug Notification Reports

Occasionally, the caDSR II community users or NCI/NIH team users may experience issues or “bugs” in production ready software and tooling. The Metadata Content Development Team members shall fully document each reported issue or bug and then contact the user to obtain more meet with these users when appropriate to document issues and provide detailed, written caDSR II Software Issues and Bug Notification Reports which…