Attachment 01 - Exhibit 2 - CISNET Needs Assessment.pdf

CISNET Needs Assessment

Submission History:

June 30, 2020 – First submission

July 28, 2020 – Minor corrections

Introduction

The NCI requested that Cornerstone Systems conduct a needs assessment for the Cancer

Intervention and Surveillance Network consortium (CISNET). The main aim of the needs assessment is to provide recommendations for replacement systems for the existing knowledge management and model profiling system that CISNET members currently use. The common names for these services are “FlexKB” and “Model Profiler”. The existing system is scheduled to be retired in 2021. The assessment includes a brief overview of the FlexKB and Model profiler, results from a questionnaire, analysis of each feature set, and general recommendations.

1. Brief overview of FlexKB and Model profiler

History

The FlexKB (Including the Model Profiler) were developed and deployed in 2000 at the same time CISNET was born. It has been in service ever since and remains the central knowledge base of CISNET containing thousands of documents, meeting details, presentations, manuscripts, model profiles, and grantee project reports.

Current features/benefits

At its core, the FlexKB is a knowledge base implemented as an enhanced wiki. It provides a secure online space for CISNET members to discuss, document, plan, and archive their research efforts. It also provides a meeting planning service that sends out notices and reminders for events such as conference calls and meetings. The model profiler sub-system provides a private secure site for each of the 20+ research groups around the world. The profiler subsystem has been used to provide structured content that is formatted and wrapped into individual model profiles and joint model profiles for each of the 6 cancer site groups (Breast, Cervical, Colorectal, Esophageal, Lung, and Prostate). The features of the existing FlexKB system can be summarized into five distinct groups which we call feature sets:

1. User management

2. Event management

3. Knowledge management

4. Communications and community management

5. Model documentation, collaboration, accessibility

We briefly describe each of the feature sets:

https://en.wikipedia.org/wiki/Wiki

User Management

The user management feature set includes the following features/services:

1. Identity / Authentication: provides a single point of access for CISNET users and ensures that only authorized users have access to the specific systems assigned to them.

2. Account setup and Policy management: provides an interface for administrators to manage the user lists and additional policy/agreements for each grant

3. Group membership: provides an interface for administrators to add groups, manage permissions, and potentially manage mailing lists

4. Contact information: provides an interface for CISNET members to provide their contact information and links to their publications and presentations

Event management

This feature set includes features/services related to management of conference, biannual meetings, and webinars :

1. Meeting support logistics: provides an interface to help manage local/onsite and remote/offsite meeting support

2. Meeting services: provides details for conferencing, screen share, VOIP, telephone, etc.)

3. Conference call scheduling: help groups and administrators to find mutually available meeting time slots (with time-zone support)

4. Conference call invitations and reminders: provides an interface for administrators and members to send out scheduled and ad hoc meeting details, automatic reminders, and expected attendance feedback.

5. Conference call planning and archival: provides a long-term, searchable repository of upcoming and post meeting details including agendas, action points, and presentations.

Knowledge Management

These features aim to provide a medium to long-term memory of the consortium’s projects, activities, efforts, decisions, presentations, and publications.

1. Archival of presentations, papers, data: provides a long-term, searchable repository of presentations, papers, and model data including datasets, thumbnails, raw text, related presentations, and other information

2. File management: provides a long-term file repository with granular permissions and optional security features for sensitive material

3. Project, manuscript, and publication management: provides online structured/templated documents to assist with grant projects, manuscripts, and the publication process

Communications and Community

In this section, we enumerate features that aim to provide efficient and frictionless unstructured communication between groups, individual members, and administrators. Communications are often timely (requests for assistance, information), references/links to documents/resources in the knowledge base or other repositories, or informational in nature (announcements, policies, aggregated reports from other sources).

1. CISNET-wide home page function: provides links to commonly requested information, announcements, resources, lists of recently changed documents

2. Home pages for each cancer site: provides links to cancer-site specific teams, projects, reports, publications, announcements and conference calls.

3. Mail lists: provides traditional e-mail lists for cancer site groups, teams, and administrators.

4. Summary and progress reports: provides summary lists and details of active and past projects, publications

5. Event reports: provides basic reports of past, current, and upcoming events (meetings and conference calls)

Model documentation, collaboration, and accessibility

Model documentation, collaboration and accessibility could be classified as a subsection of knowledge management. However, given the expressed needs and goals of the consortium to increase and expand efforts around model documentation, we separate this kind of knowledge into its own distinct group. Later on, we will add new features to this section to reflect these goals but for now we have just a single feature that is supported by the current system.

1. Model profiles: Provides a system by which each modeling group can publish detailed documentation in a consistent and comparable manner.

Summary

The FlexKB and model profiler provide CISNET members with an enhanced knowledge base that stores and disseminates essential information about the consortium’s activities. We classified their features into five main categories which will serve as a convenient framework for our analysis later on in this document.

2. Questionnaire

The software evaluation research and questionnaire design and deployment were performed in parallel so that the questionnaire could include some specific technology candidates (e.g. Slack, MS Teams, Zoom) for members to give feedback on. Moreover, preliminary testing with members on some technologies was used to gather additional data about feasibility and goodness-of-fit in the CISNET context. A questionnaire was designed and made available to each Coordinating Center to help identify which features of FlexKB were most beneficial, which features were not as useful, and what new features might offer additional value. Coordinating centers choosing to make use of the questionnaire then invited members within their center to respond to the online questions.

Justification

CISNET members have been using the FlexKB and model profiler systems for 20 years. In order to identify stakeholder needs, a questionnaire was created with input from CISNET Project staff, designed to assess the importance and utility of current capabilities as well as to understand the familiarity and current use and value of new communication and project management tools. An anonymous questionnaire format was chosen to solicit candid feedback from members. The questionnaire content, format and length was designed with the goal of gaining a better understanding of the value of the features summarized in the above introduction and to obtain initial feedback on potential new tools and technologies.

Methods

We designed the questionnaire in collaboration with NCI Project Scientist for CISNET using standard questionnaire controls and question templates for both quantitative and qualitative responses. The Qualtrics survey engine (XM, Qualtrics LLC) was used to help ensure industry standards for usability, accessibility, and reporting.

The questionnaire included 35 questions, including Likert, forced choice and open-ended questions. We included questions to help us identify responses from key subgroups of the consortium, including cancer site, years of CISNET membership, and CISNET role, including

Coordinating Center Administrators (CC admins), Coordinating Center Principal Investigators

(CCPI), non-CCPI Principal Investigators, Junior Investigators, Modelers, and members of the

Programmers Group. CC admins and CCPIs are currently the primary users of the system as they coordinate efforts for each of the cancer sites in CISNET and in particular conference call and meeting scheduling.

We used an 11-point Likert response scale (zero to 10) assessing “how valuable” a variety of current and proposed features are to respondents. In addition, we assessed current use of

FlexKB and workplace use of communication, project management and programming applications with forced choice responses. For example, to assess frequency of FlexKB use, we asked “On average, how often do you log into the FlexKB” with responses of “Daily”, “Several times a week”, “About once a week”, “About once a month”, “About 2-3 times a year”, “about once a year”, and “Never”. In order to efficiently summarize results by CISNET role, we grouped respondents into three mutually exclusive categories, as follows:

Coordinating Center (CC): CC Administrators and CC Principal Investigators

Investigators (Inv): Principal Investigators and Co-Investigators who are neither CCPIs or

Junior investigators.

Junior/Programmers (JI/PG): Junior Investigators and Programmers who are not in the above two groups (CC or Inv).

We computed summary statistics consisting of scale means for measure or Likert responses and percentage responses combining “Never” and “Once or Twice”, and “Weekly” and “Daily” for usage questions.

Results

Of the six coordinating centers, all six chose to invite their members to respond to the questionnaire. Of the 72 members with current center registration, there were 52 total respondents (72%), two were NCI staff, leaving 50 respondents with an identified non-NCI staff role. Some members affiliated with multiple cancer centers. Respondents’ cancer center affiliation, years in CISNET, FlexKB login and post/edit frequency by role are presented in Table

1. Additional results relevant to each of the Feature Sets outlined in the Brief Overview section are provided in the Analysis section following.

Table 1: Respondent characteristics and FlexKB usage by CISNET role

All

Coordinating

Center Investigators

Junior/

Programmers

50 12 24 14

Site*

Breast 11 -- 7 2

Cervical 13 -- 8 5

Colorectal 8 -- 6 2

Esophageal 6 -- 4 2

Lung 7 -- 5 1

Prostate 4 -- 0 3

*Note: 7 responders serve in multiple sites

Years in CISNET

< 1 Year 1 1 0 0

1-3 Years 10 3 0 7

4-10 Years 21 5 11 5

11-15 Years 9 1 6 2

15+ Years 9 2 7 0

FlexKB Login Frequency

Several days/wk 2 2 0 0

Weekly 5 2 3 0

Monthly 21 7 7 7

2-3 times a year 15 1 8 6

Yearly 5 0 4 1

Never 2 0 2 0

FlexKB Post/Edit Frequency**

Several days/wk 1 1 0 0

Weekly 1 1 0 0

Monthly 5 3 0 2

2-3 times a year 8 5 3 0

Yearly 6 0 4 2

Never 28 2 16 10

**Note: One respondent did not answer this question

3. Analysis

Introduction

For each of the five feature sets currently supported (see Brief overview of FlexKB and Model profiler), we combine results from the survey and expertise from Cornerstone’s experience with the CISNET consortium to provide an analysis that attempts to answer the following questions with respect to the FlexKB and Model profiling systems:

1. What existing features/services are valuable enough to continue and/or improve

2. What existing features/services offer little or no value and could be retired

3. What new features/services could be considered

User Management

User Management involves establishing and authenticating identity, setting up and maintaining user accounts, policies, group membership and contact information. These are described in greater detail in the following sections.

Recall that the current User Management features include:

1. Identity and Authentication

2. Account setup and Policy management

3. Group membership

4. Contact information

Identity and Authentication

“I understand that the approach to security is a Federal requirement. But it's a real barrier… so much so that I don’t use it at all”

User identity and authentication are required by FISMA and must be continued and improved upon in order to keep access secure and streamlined. Having a single source of truth for administrators to maintain and monitor is crucial and relevant now more than ever before with the increase in the number of 3rd party services/applications that use Single Sign-On (SSO) to authenticate user access.

However, the current sign-on process itself appears to increase user burden. For example, in commenting on challenges with the current system, two users mentioned that they don’t use the system because of the requirement for a separate, frequently changing password: “I understand that that approach to security is a Federal requirement. But it's a real barrier… so much so that

I don't use it at all: when I want something, I have asked others to retrieve it for me”, and

“FlexKB became difficult early on because there was a frequent need to change passwords and after a while it became a bit much [so I stopped using it]”. The advances that have been achieved in identity and authentication services provide an opportunity to decrease user burden and increase user engagement. For these reasons, we recommend that an appropriate identity and authentication service be selected and implemented as a replacement for the authentication system currently provided by the FlexKB.

Account setup, Policy management, and Group membership

From a usability perspective, the other aspects of user management considered (account setup, policy management, group membership) are equally important to the CISNET consortium.

Group membership and management provides the necessary control over access to collaborative work, fostering trust among group members, and maintaining and securing group work products and communication. Flexible group management is required in order to serve the numerous layers of groups and teams that the CISNET maintains: cancer site groups, project-oriented groups that span across site groups, and administrative groups at various levels among others. Likewise ensuring NCI policies and best practices are communicated to each of the cancer-site groups is a requirement of NCI’s Project Scientist for CISNET. We recommend continuing and improving upon the existing account setup, policy management and group membership components. Streamlining the process by which users are onboarded and offboarded is recommended and closely related with the identity system and should be considered when choosing an appropriate identity system.

Contact Information

Finally, contact information is crucial to reducing silos between groups and to reducing burden for new members as they explore the resources that CISNET has to offer. Currently, contact information such as e-mails, phone numbers, titles, and affiliations are stored in the FlexKB. We recommend improving this feature by integrating a standard searchable directory service. We would also recommend crosslinking between individuals and the CISNET publications they are authors of.

Event Management

Recall that current event management features cover the following aspects:

1. Meeting support logistics

2. Meeting services

3. Conference call scheduling

4. Conference call invitations and reminders

5. Conference call planning and archival

Meeting support logistics and meeting services

How valuable are (0-10): All

Coordinating

Center Investigators

Junior/

Programmers

Onsite meeting support 8.2 9.6 7.8 5.7

Zoom conference calls 9.3 8.9 9.5 7.8

As validated by the high value rating response to questions 29 regarding onsite meeting support, CISNET biannual meetings are perhaps the two most important events for the consortium because they provide a rare opportunity for researchers from disparate geographical regions to meet face-to-face, to discuss together, and to present to one another. Consequently, logistical support for these meetings is of high value to the consortium. Similarly, respondents rated Zoom video-conferencing highly, noting that it was critical to the success of the most recent remote biannual meeting as well as the numerous and more frequent group meetings.

We recommend maintaining and improving these services. Specifically, we recommend streamlining the presentation upload and onsite management of presentations to minimize disruptions during meetings and to minimize the time required to archive the presentations after the meeting is over. We also recommend integrating meeting invitations and reminders into the larger identity system

In addition to onsite and videoconferencing support, the FlexKB currently offers services such as file uploads, meeting agendas, and presentation archiving.

How often do you login to

FlexKB?

All n=50

Coordinating

Center

(n=12)

Investigators

(n=24)

Junior/

Programmers

(n-14)

At least monthly n (%) 28 (56%) 11 (92%) 10 (42%) 7 (50%)

The results regarding the frequency of logging in to FlexKB indicates that the current features of the FlexKB are used and valued particularly amongst members of the Coordinating Center (CC admins and CC PIs). However, given the importance of event management within CISNET and the lower login frequency by non-CC staff, we feel that the numbers suggest that the technology could be improved.

Conference call scheduling, invitations, reminders, planning and archival

Based on our experience, conference call management (scheduling, invitations, reminders, planning and archival) are key activities that CISNET administrators are responsible for.

Consequently, technology has the potential to greatly assist or hinder the group’s activities.

Currently, the FlexKB system provides support for scheduling, invitations, reminders, planning and archival. However, the technology and integrations with other technologies could be greatly improved, and this is reflected in the following question:

How valuable are (0-10): All

Coordinating

Center Investigators

Junior/

Programmers

Conference call reminders 6.2 7.5 5.9 5.1

As an example of the limitations of the current system, many of the CC admins (who are usually responsible for setting up calls and archiving presentations) already use a calendar service to send out invitations. This is easier to do and keep track of than creating invitations in the FlexKB but suffers from not being part of a central repository for others to access and update.

Additionally, decisions about when to have ad hoc calls are often made using a 3rd party service like Doodle which is helpful, but not integrated with the larger system.

We recommend improving and extending the existing event management services. In particular, the following features would be of value:

● Integrate invitation/reminders with a timezone-capable calendar service (either a calendar service as part of the system, or an integration with individual members’ calendar applications

● Integrate scheduling assistant to help groups find optimal meeting times across time zones and able to handle large groups.

● Improve support and archival for call agendas, action points, follow-up, and presentations. Specifically, event content should be searchable, allow previews, allow tagging/@mentions and searching by keywords/tags.

● Render dates and times based on the user’s timezone preferences (reports, reminders, etc.)

Webinar management (New)

The Programmers Group and Junior Investigators Group regularly host webinars about a variety of topics related to modeling and model accessibility for the CISNET consortium and other modelers in order to share knowledge and challenges within and outside CISNET. The current

FlexKB system does not currently support webinar management. Specifically, there is no standard service supporting webinar invitations, publicity, reminders, discussion, or dissemination of webinar recordings and transcripts. We recommend adding these features to the new system.

Knowledge Management

Recall that current knowledge management system includes the following features:

5. Archival of presentations, papers, data

6. File management

7. Project, manuscript, and publication management

Archival of presentations, papers, data

From our experience, having information about presentations from meetings available and accessible after biannual meetings have shown to be valuable both as a historical record of meetings and research discussions, but also as a way to review presentations for reference in the months that follow a presentation. This is born out in the response to the following question:

How valuable are (0-10): All

Coordinating

Center Investigators

Junior/

CISNET Presentation archives 7.2 7.6 6.8 6.5

Technology that enables users to record meetings and obtain searchable text transcripts now exists and is becoming more accessible and affordable for small teams. From this, we conclude that this is an important feature for CC administrators and investigators. Consequently, archiving presentations, papers, and datasets over time will continue to allow CISNET members to get more value from the efforts they make producing these assets. We recommend integrating an archival application for presentations, papers, and data into the new system.

File Management

File management within the FlexKB has offered groups a way to share common files and offers unique security features such as encryption, security stamping, and custom access permissions based on groups or individuals. However, several secure cloud-based services such as

GoogleDrive and DropBox are now available and affordable. They offer more security options for small groups of 3-5 people and are easy to configure and maintain. Consequently, we recommend migrating users to a well-known cloud-based sharing application that could integrate with the identity/authentication system outlined earlier.

Project, manuscript and publication management

Project, manuscript, and publication management, has been a core part of the existing FlexKB and helps inform research teams and administrators about the status of existing tasks and action points as well as offering a structured framework to help researchers document their efforts over time. Questionnaire results indicate that CC staff find project management tools more valuable than other investigators:

How valuable are (0-10): All

Coordinating

Center Investigators

Junior/

CISNET project mgmt tools 5.8 7.8 4.4 7.0

There are a multitude of cloud-based applications that attempt to solve the common challenges of project management, but most are oriented towards software development processes and software development or marketing teams. Further research is required to find a project management system that meets the needs of CISNET members. It is possible that a custom application will need to be developed to meet the specific needs of the consortium. We recommend that project management and reporting features be continued and integrated into the larger information system.

Community and Communications

Recall that the current community and communications features include the following aspects:

1. CISNET-wide home page function

2. Home pages for each cancer site

3. Mail lists

4. Summary and progress reports

5. Upcoming event reports

Home pages

The home pages on the FlexKB have provided an important central and welcoming overview of

CISNET’s activities in general and the activities of each cancer site group within CISNET.

Moreover, the home pages have acted as a portal to other resources and information about upcoming and past meetings. The branded, private, and friendly nature of the home pages provide a sense of community to CISNET members. We recommend improving the existing home pages and integrating them into the identity system’s login process. We’d recommend including a user-defined control panel overview of mini-reports of interest such as upcoming/past meetings, calls, and recently uploaded presentations to help improve user-engagement and awareness of consortium’s activities amongst CISNET members.

Mail lists

E-mail lists included in the FlexKB have allowed administrators and members to quickly explore and validate the recipients of each list to ensure compliance with CISNET policies and increase transparency and awareness. While the e-mail lists might not necessarily reside in the same class of platform (wiki) the ability for users to know and for administrators to manage the lists remains a critical component of the overall system.

Questionnaire results show that group-specific listserv/email lists are valuable to the

Coordinating Center PIs and Administrators for conducting group email communications, with mean score of 9.2, while less important to non-CC Investigators and Junior/Programmers with mean of 6.2 and 7.3 respectively:

How valuable are (0-10): All

Coordinating

Center Investigators

Junior/

Group email lists/listservs 7.3 9.2 6.2 7.3

An example of an existing mailing list is the Junior Investigators group which includes CISNET members who are junior investigators or members who are interested in connecting with junior investigators. We recommend maintaining the e-mail list service and exploring newer technologies that target the team/group communications space that move beyond traditional

SMTP-based e-mail protocols.

Reports

Summary, progress, and event reports provide up-to-date information on the status of the projects and events that CISNET members are involved in. They help tie together summary information from multiple sources to inform planning and for general awareness of the consortium’s activities and progress. We recommend improving the readability of the reports and finding ways to reduce the user-burden required to keep the reports up to date.

Channel-based chat (new)

While the existing communications features are valued by CISNET members, the current implementation does not take full advantage of technological advances and best-practices for remote team communications. In particular, the industry standard for communications in teams of all sizes is now a channel-based chat system such as Slack or MS Teams. We recommend supporting a channel-based chat service to CISNET members as an optional alternative and enhancement to traditional e-mail. We think it would be particularly useful for modeling groups, the Programmers Group, the Junior Investigators Group, and the Coordinating Center

Administrators. It is important to note that there is a significant subset of CISNET members who are not familiar with this kind of technology and others who are strongly opposed to adopting what they perceive to be a “gadget”. As with any technological change in large groups, this is expected and should be respected. We recommend focusing efforts on supporting members who are ready and eager to use this newer technology and reducing the burden of onboarding for others who are interested but need assistance.

How valuable are (0-10): All

Coordinating

Center Investigators

Junior/

Online communication tools 7.3 7.8 7.8 6.7

Model documentation, collaboration, and accessibility

Recall that the current model documentation, collaboration and accessibility features includes just one aspect currently -- model profiles.

Model profiles

CISNET Model profiles offer detailed technical documentation for each model and are publicly available. Each cancer site group has a joint profile (with the exception of the Cervical Cancer

Group). The bookmarking/navigation in joint profiles allow readers to easily compare corresponding sections across models (See Figure 1 below). While these documents are technically versioned, the versioning system is not tied to a particular publication or document release. Also, due in part to the cost of maintaining the profiles and the limited direct benefit to the modelers, keeping the profiles up to date and versioned in a meaningful way is a challenge.

https://cisnet.cancer.gov/resources/profiles.html

Figure 1: Screenshot of the CISNET Breast Cancer’s group joint profile with cross-linking using PDF bookmarks

Continuous integration best practices that combine both source code with documentation and model inputs such as parameter sets are becoming the industry standard. This helps increase reproducibility and transparency in scientific research. Consequently, we recommend that the model profile documentation be integrated with other model documentation and its source code into a source repository with versioning and release features. Readers and modelers will then be able to quickly identify and understand changes in a given model over time and be able to quickly reproduce the outputs obtained for specific publications with far less effort. The source repository should have the capacity to allow groups to manage multiple levels of access control to the source and compiled materials and a well defined and moderated release system for public consumption.

Section B.4 for RFA-CA-19-054, states that “methods and standards for model accessibility are critically important” and that “CISNET is committed to the transparency of models, and dissemination of modeling results, models runs, and software (whenever possible) to the scientific community.” Moreover, CISNET members are asked to address:

“An evolving plan for model accessibility within the specific cancer type of focus that may be applicable also for consortium-wide activities and/or standard approaches, including pilot projects to test specific approaches. Applicable target audiences for model accessibility, including, e.g.:

https://grants.nih.gov/grants/guide/rfa-files/RFA-CA-19-054.html

● Members in an individual modeling group, various members across modeling groups within a cancer type, and/or members across cancer type groups within the consortium;

● Outside collaborators; and

● Outside model developers and model users.

Key technical aspects of model accessibility such as:

● Specific criteria under which the model executable or model source code (or model subcomponents) would be shared;

● Cost-effective methods to better document older models with a potentially large source code base;

● Standards for model versioning, release and documentation of complete sets of model parameters associated with specific publications; and

● Types of licenses for model executable and source code sharing, cross CISNET standards for the use of sharing portals such at GitHub.”

Given these critical aims for future CISNET grantees, feedback from members, and our experience, we identified four new components that could be considered in a new system:

1. Data provenance

2. Automated build, tests, runs, publishing

3. Publication referencing to datasets and versioned models

4. Repository management

Data provenance (new)

Data provenance provides a way for users and readers of model documentation to identify the provenance of source data/code and access over time. Further research is required to determine best practices for data provenance in the context of open science and given the specific needs of CISNET. Once the best practices have been identified, we recommend including a data provenance component that helps streamline and encourage these practices within CISNET.

Publication referencing to datasets and versioned models (new)

CISNET groups often produce publications that are linked to a particular model and input dataset combination. Best practices in the industry suggest that the input datasets, code, and output datasets be stored and formally referenced so they can be cited in subsequent research.

We recommend researching online services such as zenodo.org and identifying the ones that could best meet CISNET’s needs and integrate well with the repository management system.

http://zenodo.org/

Repository management (new)

The core component of the system that supports model accessibility is the repository management service. This system must be secure and provide granular permissions for each cancer site group. Given the popularity and maturity of git-based source control, and the granular and robust access control that Github offers, we recommend that each cancer site group create and manage its own Github organization account. We recommend that the contractor provide support to each group to assist them in getting user accounts and groups configured and then to provide ongoing support to each modeling group on an as-needed basis.

Automated builds, tests, runs, publishing (new)

Automated processes provide a standard and mature mechanism for building, testing, running and publishing source code and datasets. New and recent research projects would benefit most from automated processes. Older codebases that are in the process of being migrated to new languages would also benefit from automated testing to ensure that the new code can reproduce the same or equivalent outputs as with the older codebase. We recommend identifying and implementing suitable and industry standard automated processes into or alongside the repository management system.

Summary

In Section 3 we have provided an analysis that attempted to answer the following questions with respect to the FlexKB and Model profiling systems for each of the feature sets:

1. What existing features/services are valuable enough to continue and/or improve

2. What existing features/services offer little or no value and could be retired

3. What new features/services could be considered

We concluded each assessment component above with an action recommendation. In the next section, we summarize the analysis and provide general recommendations for a replacement system to the FlexKB and Model Profiler.

4. General recommendations

In this section we provide a convenient summary of the more detailed recommendations in the previous section. First, we provide a map of each existing component to a candidate technology and indicate where new components are recommended. Second, we follow with general recommendations for each of the five component groups. Finally, we provide suggested next steps.

Component map: Current technology to candidate technology

Legend for the feature map below

IMPR: Improve existing component

NEW: New component

1. User Management

Current Current Candidate

Service/Feature Technology Provider Technology

1 Identity / Authentication Wiki CSNW IMPR Identity

2 Account setup and policy management Wiki CSNW IMPR Identity integration

3 Group membership Wiki CSNW IMPR Identity

4 Contact information/User profiles Wiki CSNW IMPR Identity integration

2. Event management

Service/Feature Technology Provider Technology

1 Meeting support logistics (on and off site) Wiki CSNW IMPR Knowledge Base

2 Meeting services (screen share, voip, etc.)

Conferencing service Zoom/NCI IMPR Conferencing service

3 Conference call scheduling Scheduling Doodle IMPR Scheduling

4 Conference call invitations and reminders Wiki CSNW IMPR Calendar

5 Conference call planning and archival Wiki CSNW IMPR Knowledge Base

6 Webinar management

Conferencing service Zoom/NCI NEW Conferencing service

3. Knowledge Management

Service/Feature Technology Provider Technology

1 Memory/archive of presentations, papers, data Wiki CSNW IMPR Knowledge Base

2 File management Wiki CSNW IMPR Knowledge Base

3 Project, manuscript, & publication management Wiki CSNW IMPR Knowledge Base

4. Communications and Community

Service/Feature Technology Provider Technology

1 CISNET-wide home page function Wiki CSNW IMPR Knowledge Base

2 Home pages for each cancer site Wiki CSNW IMPR Knowledge Base

3 Mail lists Mail server CSNW IMPR Distribution list

4 Summary and progress reports Wiki CSNW IMPR

Knowledge Base

Report/Integration

5 Upcoming event reports Wiki CSNW IMPR Knowledge Base

6 Channel-based chat N/A N/A NEW Communications hub

5. Model documentation, collaboration, and accessibility

Service/Feature Technology Provider Technology

1 Model profiles Wiki CSNW IMPR

Version control or

CMS

2 Data provenance N/A N/A NEW Version control

3 Automated builds, tests, runs, publishing N/A N/A NEW

Version control integration

4 Publication referencing to datasets, models N/A N/A NEW Publication service

5 Repository management w/ access/permissions N/A N/A NEW Version control

General recommendations

User Management

All user existing management features are either required by FISMA or have high value to the consortium. No new features in this group were identified but a considerable technology gap exists between the FlexKB’s authentication system that was developed 20 years ago and the array of mature identity/authentication services now available. We recommend researching and testing more appropriate technology to help improve the basic staples of user management that are currently supported.

Event Management

Supporting meeting and conference call management remains critical for the CISNET consortium especially given the rise in demand for remote meetings. The existing components have been useful over the years, but newer technologies have emerged and would likely reduce the overall burden on administrators as well as improving the quality of interactions during events. Also, webinar management is a potential new component that could be considered.

Communications and community

The existing community and communications features have helped create group cohesion and personal connections between members. Technological advances over the past 10 years offer even better opportunities for small teams to connect with each other and improve awareness and transparency. The existing features should be adapted and extended to take advantage of these new technologies while maintaining more traditional technologies such as e-mail.

Knowledge Management

Most existing knowledge management features should be continued and improved upon in the new system but not necessarily within the context of a single application. More research is needed in order to determine the optimal combination of applications and how best to integrate them into the larger system of services such as event management and communications. Also, it is possible that the project management aspects might require custom application development.

Model Documentation, collaboration, and accessibility

Model documentation, collaboration and accessibility are clear priorities for the consortium and the current system must be extended significantly in order to better support these initiatives.

More research and a period of testing might be needed to determine what technologies can best help the consortium. In particular, more information is required from the groups regarding their needs and capacity with respect to permissions and the review process and technologies such as git/Github. In addition, clear guidelines about how groups should release model documentation and the minimal requirements need to be developed in collaboration with the

Coordinating Center and the contractor to ensure that a given implementation will meet the consortium's needs.

Suggested next steps

The needs assessment process helped identify a framework for evaluating the value of each existing component in the FlexKB and Model Profiler system and to identify any new components that might improve the system. Further research is required to fine tune the combination of components into a coherent system and to provide the best value to the CISNET consortium. Once the next contractor for NCI has been chosen, we recommend that the contractor use this framework as a starting point from which to continue the research for technology to replace the FlexKB and Model Profiling system. Here are some suggested next steps to consider:

1) Review recommendations and framework

2) Research specific technologies/vendors and identify vendor candidates

3) Determine an optimal mix of technologies to meet the needs of the consortium

4) Develop a clear concept for a new system and request input from the new Coordinating

Center and NCI

5) Revise the concept to include feedback from stakeholders, make final choices about vendors and design the new system

6) Gradually implement services and migrate users over to the new system

7) Retire FlexKB by Sept 2021 at the latest

Conclusion

This needs assessment offers a set of recommendations to be used as a starting point of discussion between the contractor and NCI as they work together to determine how best to migrate users from the FlexKB and model profiler system into a new system. Apart from retiring the FlexKB for life-cycle reasons, the objective of the migration will be to improve usability and user engagement. These recommendations should be considered in the larger context of evaluating and supporting the consortium’s other needs and core aims and in close collaboration with NCI and the Coordinating Center.