Statement of Work.pdf

CISNET CONSORTIUM FOR CANCER SURVEILLANCE MODELING SUPPORT

BASE/PARENT IDIQ STATEMENT OF WORK

Independently and not as an agent of the Government, the Contractor shall furnish all necessary services, qualified personnel, material, equipment and facilities, not otherwise provided by the Government as needed to perform this Statement of Work.

Background:

The Cancer Intervention and Surveillance Modeling Network (CISNET;

https://cisnet.cancer.gov/) is a consortium of NCI-funded investigators who use simulation modeling to extend evidence provided by trial, epidemiologic, and surveillance data to guide public health research and priorities. CISNET fills a unique niche in the NCI grant portfolio by connecting innovations in cancer research to strategies that most effectively deploy these interventions to maximize their population impact, while minimizing harms and burden. The CISNET modeling approach advances the practice of simulation modeling to inform clinical and policy decisions, including work in support of US Preventive Services Task Force screening recommendations. CISNET currently consists of six U01 cooperative agreements, each focusing on a single cancer site (breast, prostate, colorectal, lung, esophageal, and cervical). The consortium started in 2000, but some of the cancer sites joined more recently. Each U01 project includes a coordinating center and three to six independent modeling groups. This structure promotes efficient collaboration among the modeling groups for that cancer site, including the gathering and sharing of key data resources and critical evaluation of where model results converge or diverge. Affiliate members (who are not formally funded by CISNET) can join the consortium for specific collaborations. The U01 grant mechanism allows substantial programmatic involvement between NCI and grantees.

Features of CISNET’s capabilities and modeling perspective include the following:

• Flexible broad-based disease models: CISNET models incorporate a central cancer model (usually including natural history prior to clinical appearance), which can be modified by evolving cancer control interventions across the cancer control spectrum — prevention strategies, screening modalities, and treatment and follow-up regimens.

Outputs can include the interventions’ benefits, harms, and costs. The models can be readily adapted to characterize new technologies. For example, CISNET breast models, which were originally developed based on film mammography, have been adapted to incorporate the costs, performance characteristics, and logistics of digital mammography and, more recently, tomosynthesis (3D mammography).

• Multiple birth-cohort modeling: Modeling a hypothetical cohort often does not fully capture the intervention’s impact or cost-effectiveness as implemented in a broad population. CISNET modeling captures a series of birth cohorts and shifting risk factor profiles, screening behaviors, and treatments used as each cohort ages, enabling complete representation of an intervention’s impact in the actual US or other populations.

https://cisnet.cancer.gov/

• Comparative modeling: Rather than relying on single-model estimate for each cancer site, CISNET modeling jointly evaluates results from a spectrum of models. While independent modeling often yields disparate results that are difficult to reconcile, a comparative modeling approach can systematically investigate the differences.

Comparative modeling produces a range of results across models. When consensus is achieved, this reproducibility strengthens the credibility of modeling results. When results are disparate, it can highlight knowledge gaps to be addressed by further research.

• Documentation to promote transparency in modeling and assumptions: CISNET has implemented documentation for their models to accommodate audiences who have different objectives and levels of technical familiarity with modeling. The CISNET Model Profiles (http://cisnet.cancer.gov/profiles/ ) comprise detailed templated technical documentation that allows users to either read about a single model or view side-by-side comparisons of model components across different models. The CISNET Model Registry (https://resources.cisnet.cancer.gov/registry ) is designed for those who want a higher-level understanding of models and their results from a practical, less technical point of view. The CISNET Model Stress Tests (under development, although a tutorial is available - https://resources.cisnet.cancer.gov/projects/#compare/tutorial/tutorial) assists in understanding model results by describing how models react to a standardized “stress” (e.g., a single screen at a specified age) and depicting various outputs after the stress (e.g., time to development of new cancers after the screen). These “stress tests” serve as both a diagnostic tool for CISNET investigators to study model differences (van Ballegooijen et al., 2011; van den Broek et al., 2018) and as documentation for others to understand variations in model results. In addition to these three documentation types, many CISNET papers provide comparative tables and/or short descriptions contrasting salient features of models.

The framework for CISNET population modeling (Figure 1) starts with common inputs to the models including risk factor trends, screening behavior, and the dissemination of new treatments.

These common inputs feed into the individual simulation models (between 3 and 6 per cancer site), which simulate multiple instances of each person’s life, i.e. with and without the interventions of interest to be evaluated. The modeling starts with each individual’s demographic and early life, their preclinical history of cancer, the diagnosis and treatment of cancer, and death from cancer or other causes. The transitions between pre-clinical states (e.g.

adenomatous polyps to early stage colorectal cancer) are not directly observable, and they must be indirectly inferred from screening, autopsy, and other studies which provide a glimpse into pre-clinical cancer. This represents one of the major challenges of CISNET modeling, and the varied approaches to modeling natural history represents a real strength of the comparative approach. Finally, comparing the runs in the absence and presence of interventions, and summing across individuals, common outputs across models can be collected for various metrics, e.g., lives or quality adjusted life years saved, proportion of cases overdiagnosed, direct medical costs, etc. Finally, calendar time can be introduced by modeling multiple birth cohorts in order to represent the actual U.S. or other populations.

http://cisnet.cancer.gov/profiles/ https://resources.cisnet.cancer.gov/registry https://resources.cisnet.cancer.gov/projects/#compare/tutorial/tutorial

The consortium members have published over 500 papers in high profile journals, and they can be found sorted by cancer site (https://cisnet.cancer.gov/publications/cancer-site.html) and 14 topic areas (https://cisnet.cancer.gov/publications/topic.html).

Consortium member must apply for refunding every 5 years in an open competition. In early rounds of funding individual modeling groups applied separately. However, in more recent funding rounds modelers for each cancer site applied jointly utilizing a multiple PI grant mechanism with 3-6 models per cancer site. One of the investigators is designated as the coordinating center PI for that cancer site. The major duties of the coordinating center is to:

formulate, prioritize, and coordinate work; negotiate common requests for outside data sources;

prepare inputs and collect and process common outputs for model comparisons and critical evaluation of disparate results; and lead conference calls and plan agendas for meetings. The current round of funding ends on August 31, 2020, and applications have been received for the next round of funding in response to RFA-CA-19-054 (https://grants.nih.gov/grants/guide/rfa-files/RFA-CA-19-054.html) to start on September 1, 2020. Up to 4 additional cancer sites may join the consortium on approximately September 1, 2021, in response to a new CISNET Incubator program, which will foster development of models and a comparative modeling approach in cancer sites beyond the ones currently in CISNET. The incubator program will have fewer modeling groups per cancer site, i.e. 2-3 groups.

The consortium members meet in person twice a year, with an annual meeting usually in November/December at the NCI, and a mid-year meeting in May/June at one of the consortium members sites. The meetings last an entire week, and include 1½ - 2 day sessions for each cancer site, a half day plenary session at the annual meeting, as well as Junior Investigator and Model Accessibility sessions at every meeting. Meetings are planned jointly by a government contracted meeting organizer organization, the coordinating center administrators for each cancer site, organizers of the Junior Investigators and Model Accessibility groups, and the NCI.

Most cancer site groups have monthly calls between meetings, and smaller groups have project specific calls. Projects using multiple models are called “base cases”, and the groups work to establish common model inputs, and a template for model outputs to facilitate comparisons across models. Not every modeling group works on every base case project, and some projects are done using only a single model. Cross-cancer site meetings and projects are conducted on common themes across cancer sites (e.g. the role of comorbid conditions on the age to stop screening; Lansdorp-Vogelaar et al. 2014).

The consortium has been supported by a technical and logistical support contractor. The role of the contractor has been to support a private password-protected CISNET members only website housing e-mail and contact information for members; listservs; a place to post and track manuscripts and data resources; manage shared files, specifications, and templated output spreadsheets; a repository of past meeting presentations; a repository of consortium communication and publication policies, and affiliate member policies; cancer site home pages;

conference call scheduling, documents and reminders; web-connect conference call numbers and account access; consortium logos and PowerPoint background templates; meeting agendas; and a site for members to develop Model Profiles (which would later be made into PDF’s and posted online). Screenshots of the User Agreement and Home Page are shown in Figures 2 and 3. The https://cisnet.cancer.gov/publications/cancer-site.html https://cisnet.cancer.gov/publications/topic.html https://grants.nih.gov/grants/guide/rfa-files/RFA-CA-19-054.html https://grants.nih.gov/grants/guide/rfa-files/RFA-CA-19-054.html contractor has worked with NCI and consortium members to develop and host the different forms of model documentation including the Model Profiles, Model Registry, and Model Stress Tests. In addition to developing the structure and format of these platforms, they have worked with consortium members to facilitate in developing their content. They have worked with consortium member to produce online interactive graphics to supplement fixed graphics available in member publications and other modeling resources (https://resources.cisnet.cancer.gov/projects/). The major portion of the CISNET public website is currently hosted by Information Management Services (IMS), Inc. (https://www.imsweb.com) under https://cisnet.cancer.gov, while specific resources developed by the incumbent technical support contractor for CISNET are hosted by them under https://resources.cancer.gov/registry and https://resources.cisnet.cancer.gov/projects/. Appropriate links between the two domains and information exchange between the two contractors occur under the guidance of the government. For example, information on the papers are hosted on the private CISNET knowledge management system, but when published information is copied to IMS, Inc. to be added to the public facing page (https://cisnet.cancer.gov/publications/).

The contractor has attended the in-person meetings to assist coordinating center administrators in collecting and up-loading meeting presentations and hosting web-connect sessions (which is routinely done for all meeting sessions to facilitate members and guest speakers who could not attend in person), providing back-up general AV support to local AV staff, and attending and participating in the scientific sessions including giving presentations relevant to their technical and logistical support functions. They have provided development support for limited model interfaces, allowing users to view the results of a limited set of model runs for use as individual or policy decision tools. In some cases, funding for the development of these interfaces has been supported by the grantees themselves, rather than the NCI support contract.

Finally, the technical and logistical support contractor has hosted and/or provided technical support to the CISNET Programmer’s Support/Model Accessibility Group. As stated in the

RFA-CA-19-054:

“While it is recognized that there are no standards for the release and accessibility of complex microsimulation models, continued thinking about barriers to accessibility and how to overcome them, and the evolving development of policies, methods and standards for model accessibility are critically important. CISNET is committed to the transparency of models, and dissemination of modeling results, models runs, and software (whenever possible) to the scientific community.”

In this evolving area, CISNET members are asked to address such aspects as:

“An evolving plan for model accessibility within the specific cancer type of focus that may be applicable also for consortium-wide activities and/or standard approaches, including pilot projects to test specific approaches.

Applicable target audiences for model accessibility, including, e.g.:

https://resources.cisnet.cancer.gov/projects/ https://www.imsweb.com/ https://cisnet.cancer.gov/ https://resources.cancer.gov/registry https://resources.cisnet.cancer.gov/projects/ https://cisnet.cancer.gov/publications/

• Members in an individual modeling group, various members across modeling groups within a cancer type, and/or members across cancer type groups within the consortium;

• Outside collaborators; and

• Outside model developers and model users.

Key technical aspects of model accessibility such as:

• Specific criteria under which the model executable or model source code (or model subcomponents) would be shared;

• Cost-effective methods to better document older models with a potentially large source code base;

• Standards for model versioning, release and documentation of complete sets of model parameters associated with specific publications; and

• Types of licenses for model executable and source code sharing, cross CISNET standards for the use of sharing portals such at GitHub.”

References:

Lansdorp-Vogelaar I, Gulati R, Mariotto AB, Schechter CB, de Carvalho TM, Knudsen AB, van Ravesteyn NT, Heijnsdijk EA, Pabiniak C, van Ballegooijen M, Rutter CM, Kuntz KM, Feuer EJ, Etzioni R, de Koning HJ, Zauber AG, Mandelblatt JS. Personalizing age of cancer screening cessation based on comorbid conditions: model estimates of harms and benefits. Ann Intern Med 2014 Jul 15;161(2):104-12.

van Ballegooijen M, Rutter CM, Knudsen AB, Zauber AG, Savarino JE, Lansdorp-Vogelaar I, Boer R, Feuer EJ, Habbema JD, Kuntz KM. Clarifying differences in natural history between models of screening: the case of colorectal cancer. Med Decis Making 2011 Jul-Aug;31(4):540- 9.

van den Broek JJ, van Ravesteyn NT, Mandelblatt JS, Cevik M, Schechter CB, Lee SJ, Huang H, Li Y, Munoz DF, Plevritis SK, de Koning HJ, Stout NK, van Ballegooijen M. Comparing CISNET Breast Cancer Models Using the Maximum Clinical Incidence Reduction Methodology.

Med Decis Making 2018 Apr;38(1_suppl):112S-125S.

Scope:

The Contractor shall work with the National Cancer Institute (NCI), Division of Cancer Control and Population Sciences (DCCPS) to provide technical and logistical support to facilitate collaborative efforts of the CISNET Consortium in cancer surveillance, including cancer simulation modeling using a multicohort approach, flexible broad-based natural history models of cancer, and comparative modeling managed through a web-based Knowledge Management System with adaptable data structures. If the awardee of this contract is not the incumbent, it is the intention of the government to maintain the incumbent contractor through 2021 to transition the current systems until they are operational by the new contractor and facilitate the orderly transfer and understanding of the current structure, content, and logistics of the current support provided.

The Contractor shall perform work in the following areas as outlined in each Task Order.

1. Knowledge Management System

The Contractor shall provide logistical support to facilitate collaborations for projects in cancer surveillance supported by NCI via a web-based knowledge management system. The current knowledge management system, known as the FlexKB, is a Wiki built in Visual Foxpro and runs on Windows 2019 with West Wind Web Connect app pool. Tables are in dbf format with Custom/early Markup syntax with ability to render Latex. It generates thumbnails and extracts text for uploaded documents (e.g. docx, pdf, ppt, etc.). Although modified over the years, this system is reaching the end of its usable life, and is due to be retired in 2021. In 2021 the contractor shall propose and build a new system (or systems) that supports the functions of the FlexKB, and will accept historical content from the current system. Although it is up to the contractor to propose a technical solution, some potential tools to build this could include (but should not be limited to Github, Slack/MS Teams, Zoom, and/or Github’s own wiki platform (https://help.github.com/en/github/building-a-strong-community/about-wikis and https://github.com/d3/d3/wiki), and then customizing tools such as these to meet the stated functional requirements. In future years (2022 and after) the contractor shall adapt and modify the system(s) to meet the developing needs of the NCI and CISNET members. While the current format of pages will be a starting point for development of the system, the contractor will have the opportunity to discuss modifications with current CISNET members, and make appropriate modifications. Note that although the private sites for developing Model Profiles are currently part of the Knowledge Management System, the Model Profile Documentation System is considered separately under item 2 below.

This system shall include the capability to create and maintain email lists and contact information; post and track manuscripts and post data resources; manage shared files, specifications, and templated output spreadsheets; facilitate the changing affiliations of collaborators; and add new collaborators.

The Contractor shall support a variety of data structures that are easily adapted to the needs of different types of projects in cancer surveillance. These data structures can include both fixed choices (e.g. using drop down menus), to the least-structured end of the spectrum using free-form topics in which notes and various ad-hoc artifacts are stored. As common patterns are identified, the Contractor shall add structure to support specific functions such as manuscript workflow, meeting logistics, presentation storage, citation storage, etc. This system shall also support access controls by groups to allow ad-hoc subgroups to share documents securely in cases where results are confidential. Finally, the Contractor shall support manuscript controls including stamping and locking PDFs to the specific user that downloads them.

2. Model Profile and Documentation System –The Model Profile and Documentation System is the most detailed level of technical documentation. Model profiles are standardized templated descriptions that facilitate the comparison of models and their results. Users can read complete documentation about a single model or read descriptions that contrast how https://help.github.com/en/github/building-a-strong-community/about-wikis https://github.com/d3/d3/wiki models address different components of the process (i.e. (Crosslinking intra-model and intra-model to facilitate model comparisons). Journal articles seldom contain extensive model descriptions; links from publications to model profiles provide a more complete model description.

Historically each CISNET team was given a private model profile Web site on which to maintain their model profile information and control what parts of their profile are shared with the rest of the consortium, and ultimately posted to the public website as PDFs (https://cisnet.cancer.gov/resources/profiles.html). Since the core documentation format is the same for each group, the published profile information is readily comparable among models. Historically, the CISNET support contractor guided users to make sure they have the same level as technical specificity. Profiles are uploaded to a group collaborative site (i.e. a site open to selected NCI staff and all appropriate users) and finally the public website.

The contractor shall develop a system to receive the historical content, and allow users to make edits to update and/or enhance content. The same prior functionality should allow users to work on a private site, to share with other members of CISNET, and finally to a public site. The incumbent contractor will provide a basic static archived version of the website to provide a historical reference of the system before it was retired. The contractor will work jointly with the government and consortium members (especially the Model Accessibility team) to solicit feedback for ideas to substantially modify and enhance the capabilities of the system to meet the future needs of the consortium. The contractor shall maintain the system, train and guide users in understanding how to update and maintain their model profile documentation. The contractor shall also assist in curating content for each modeling group as needed to make sure they are organizing and sharing the best and most relevant online content, and have a technical level of specificity similar to other groups (to assist in viewing cross-model comparisons).

3. CISNET Model Registry System

The Contractor shall work with modeling teams to maintain and enhance the public-facing CISNET Model Registry (https://resources.cisnet.cancer.gov/registry) that is a searchable registry of projects in which users briefly describe their projects using a number of generalized project attributes as well as brief free-form descriptions of the projects’ purpose, relevant domain, and development team specifics. In contrast to the Model Profile and Documentation System, which provides a detailed description of the project, the CISNET Model Registry provides a high-level view of the models and is searchable across models. The technical specifications of the CISNET Model Registry includes: Ubuntu 18.04, a basic docker-compose config: python:stretch (+django==2.2.x), postgres:11.5-alpine, nginx:latest; basic bootstrap layout. The contractor shall accept the Model Registration Website from the incumbent contractor, and make it functional in their environment, learn how to modify content as well as enhance structure and format. The contractor shall maintain, enhance, and provide user support for the CISNET Model Registry, with future enhancements guided by discussions with the government and CISNET members.

https://cisnet.cancer.gov/resources/profiles.html https://resources.cisnet.cancer.gov/registry

4. CISNET Publication Support and Modeling Resources (PSMR) Website

The CISNET Publication Support and Modeling Resources (PSMR) website provides online interactive graphics to supplement fixed graphics available in member publications and other modeling resources (https://resources.cisnet.cancer.gov/projects/).

The incumbent contractor shall accept the transfer of the PSMR content and functionality, including the source code used to build the website, to the successor Contractor. The technical specifications of the PSMR includes: Static website: backbone, react, d3, bootstrap; gulp, babel, build and git-based deployment; supports typescript. The contractor shall accept the CISNET Publication Support and Model Resources Page web (https://resources.cisnet.cancer.gov/projects/ ) from the incumbent contractor, and learn how to modify pages and create new pages (interactive graphics that support publications). The contractor shall maintain, enhance, and provide user support for the PSMR, with future enhancements guided by discussions with the government and CISNET members.

5. Model Stress Tests Documentation

Model stress tests provide a unique type of model documentation in which a well-defined intervention or event (i.e. a stress) is imposed on a model, and the model’s response is captured as output. While other documentation focuses on describing model characteristics, assumptions, and inputs, stress tests focus on what can be revealed about a model’s inner workings based on how it reacts. Because the preclinical natural history of disease (i.e. the rate at which tumors are initiated and grow prior to causing symptoms) is not observed directly, stress tests are especially useful for examining this aspect of models. There is a home page and three distinct portions of the Model Stress test documentation: The Model Stress Test Tutorial; A Guide to Comparing Model Stress Test Across CISNET models, and an Interactive CISNET Model Stress Test Explorer. The home page is on the development site of the incumbent contractor, but a screenshot of that page is available as Figure 10. The Model Stress Test Tutorial is currently part of the PSMR website (https://resources.cisnet.cancer.gov/projects/#compare/tutorial/tutorial).

The final two portions of the model stress test documentation are also currently on development sites, but are expected to be available as live sites before the end of 2020. The contractor shall accept all portions of the model the Model Stress Test Documentation from the incumbent contractor, and learn how to modify and enhance the content and format. The contractor shall maintain, modify, and enhance both the content and structure of the Model Stress Documentation in consultation with the COR and members of CISNET consortium.

6. Support of the CISNET Programmers/Model Accessibility Workgroup

The contractor shall engage CISNET members with the Programmers Group to advance the ongoing discussion on model accessibility, Open Science, model documentation and versioning.

Activities shall include:

a) Regular calls with the Programmers Group co-leads

b) Appropriate technology research and evaluation (e.g. GitHub, publishing services, technology for model documentation and versioning)

c) Working with groups that want to improve their publication process https://resources.cisnet.cancer.gov/projects/ https://resources.cisnet.cancer.gov/projects/ https://resources.cisnet.cancer.gov/projects/#compare/tutorial/tutorial

d) Improving model documentation on the CISNET registry (ex. for groups that have public Github repositories with versions/releases)

e) CISNET Mid-year and Annual-meeting break-out meeting planning

f) Providing logistical support for potential webinars (by members of the group, other

CISNET members, or outsiders)

g) Online chat group for questions

7. Meeting Support, Call Support, Onboarding of New Members and General Consortium

Participation

The contractor shall: (1) provide support for the use of an NCI- supplied web-based unified meeting system to be used across CISNET; (2) provide meeting support for the annual and mid-year CISNET meetings; (3) onboard individual new CISNET regular and affiliate members as the join the consortium; (4) participate in a range of consortium activities as determined in consultation with the COR and consortium members. These activities shall include (but will not be limited to) presentations and discussions relevant to inform CISNET consortium members about contractor projects on conference calls, webinars, etc.

8. Limited Model Interface Systems

The Contractor shall provide support for developing web-based interactive public Project Interface Systems for specific projects. These interfaces shall help the general research community to better understand a project and allow users to explore the responses to inputs. In most cases, these shall be limited scenario-based interfaces and shall include only those parameters most likely to be changed (and understood) by the reader interested in learning more about the project. In consultation with the COR, the Contractor shall enhance existing, and develop new, working interface systems. Note that in some cases funding for the development of these limited model interfaces has been supported by the grantees themselves, rather than by the NCI support contract.

9. Hardware and Software Support

The Contractor shall provide all hosting and technical support required to accomplish the scope of work for this project. The contractor shall maintain the proper certifications for hosting Government web sites. In addition, they shall maintain internet connections, hardware, and software licenses, security certificates, security patches, backups, new user accounts, and monitoring for the collaborative knowledge base. All web-hosting infrastructures shall be implemented using NCI and National Institute of Standards and Technology (NIST) standards included in Section H of Base contract. The web-server environment shall include both an active hardware-based firewall and a warm backup server shall be provided to recover from possible primary server failure and provide continued server availability. All websites shall be developed and updated as needed to support versions of Google Chrome, Microsoft Explorer, Mozilla, and Safari to stay consistent with versions supported by NCI. The contractor shall perform all activities and testing to maintain FISMA compliance.