SOW-NeurologicalExaminationViaSmartphone6-2020v2.docx

Statement of Work (Neurological Functions Test Suite)

NIH/OD /OA

Statement of Work (SOW) Office of Acquisitions

1.0 INTRODUCTION/OVERVIEW

The goal of this project is to further develop and optimize suite of simple smartphone tests called Neurological Functions Test Suite (NeuFun), originally developed at the Neuroimmunological Diseases Section (NDS) of the Laboratory of Clinical Immunology and Microbiology (LCIM) of the National Institute of Allergy and Infectious Diseases (NIAID), into fully integrated, highly functional tool that reproduces most aspects of neurological examination for telemedicine assessment of subjects with neurological disorders or neurological symptoms.

1.1 BACKGROUND

There is a world-wide shortage of neurologists, which is predicted to only increase as the world population ages. Neurological examination is a powerful tool for diagnosing and monitoring neurological diseases, but it takes 30-60 minutes to perform and years of training to master. Consequently, fewer and fewer neurological patients are being examined. This delays diagnosis of neurological diseases and causes their suboptimal management.

Additionally, emergency situations like COVID19 pandemic requires shift of medical services to telemedicine. Thus far, there are no remote tools that allow diagnosing neurological disorders and monitoring neurological functions.

Therefore, NDS initiated 3 years ago development and testing of the project to recreate most aspects of the neurological examination via simple smartphone tests that can be administered by patients. Our experience thus far demonstrated that the achievement of the stated goal to “Recreate neurological examination via smartphone” is feasible. We demonstrated statistically significant correlations between selected neurological functions measured by trained clinician and machine-learning-optimized outcomes from several of these simple, smartphone-administered functional test:

https://www.frontiersin.org/articles/10.3389/fneur.2018.00740/full https://www.frontiersin.org/articles/10.3389/fneur.2019.00358/full https://www.medrxiv.org/content/10.1101/2020.03.09.20033316v4

There are currently 12 simple tests employed in the current version of the Neurological Functions Tests Suite (will be referred to as “NeuFun”)

1. Tapping test (fingers, both hands)

2. Tapping test (both feet)

3. Balloon popping test (fingers, both hands)

4. Balloon popping test with distraction (fingers, both hands)

5. Level Task (both hands)

6. Spiral Tracing Task (both hands)

7. Vibration test (both hands)

8. Vibration test (both feet)

9. Randomized Symbol Digit Modalities Test (rSDMT)

10. Verbal memory test (Immediate and Delayed)

11. Short walk

12. Long walk

The NeuFun Apps are written in Kotlin, using the latest Android Studio integrated development environment. The code for current Apps is deposited to GitHub. The tests are delivered as a part of an Android package (APK), can be configured remotely by “prescriptions” defined and prescribed to individual subjects by the test administrator (i.e., clinician or his/her designee), and the results are stored in an online database (Google Firebase) under alphanumeric code known only to the investigators and testers. The use of “prescriptions” allows tailoring of the NeuFun to the disability level of the test participant. The NeuFun is run on the latest Android operating system (Android 9 and up) and the graphics are optimized for Google Pixel XL and Google Pixel 2 XL, in colorblind-friendly visualizations.

The envisioned uses of the end-product include a fully optimized and validated, FDA-approved MS test suite for:

1. Primary care physicians: to identify and localize neurological deficit to prioritize referrals to neurologists

2. Neurologists:

a. Focus their neurological examination on most affected neurological functions during initial/diagnostic work-up of neurological disorders

b. Measure different neurological functions in the patient longitudinally, to assess disease course and efficacy of applied therapies

3. Other clinicians (i.e., non-neurology specialties): to identify/measure neurological deficit as a complication of other disease states, or as complication of applied treatments. Identification of neurological deficit would lead to better utilization of neurological referrals/consults.

4. Patients: monitor their neurological functions longitudinally to make more informed decisions about diet, lifestyle, rehabilitation and treatments

5. Researchers, including pharmaceutical companies:

a. To reliably identify, localize and quantify neurological functions for research applications

b. To reliably measure neurological functions longitudinally, e.g., during clinical trials We want to add following new tests to NeuFun:

1. Visual acuity (both eyes)

2. Motor sequencing test (both hands)

3. Go-No go test (both hands)

4. Postural sway test (eyes open, eyes closed)

5. Spatial memory test

1.2 SCOPE

1. Develop/optimize 5 new tests under NDS guidance

2. Further improve usability of already-developed smartphone tests

3. Integrate results from all tests to enhance functionality of subjects’ dashboard

2.0 REQUIREMENTS

2.1 ENHANCE CURRENT APPS

Currently, 6 out of 12 tests provide feedback with subject-specific tracing of longitudinal performance. We want to enhance test-specific feedback (for all tests) for following functions:

2.1(a). Display test results at the end of each testing The final score for each test (optimized based on NDS testing) should be displayed after subjects successfully completes the test.

2.1(b). Motivate ongoing test performance by displaying ongoing test performance against the best previous achievement of the subject, as well as the normative cohort data.

To motivate subjects’ performance during the test taking, the ongoing score should be displayed in non-intrusive graphical format (e.g. as heatmap) with two targets outlined on the heatmap: one target being the personal best score and the second target being age-related normative data displayed as interval. The goal would be to motivate the subject to their best performance, without being distracting. The display of the previous personal record, as well as the range of normative data would then provide immediate feedback to the testing subject. The subject should have an option to turn off this feedback (e.g., in settings) if deemed bothersome.

This motivational scale should be dynamic whenever possible: e.g. as level/heatmap with lowest performance in blue colors, highest performance in green or red colors and the actual performance data as a dynamic arrow between the maximal/minimal points that reflects ongoing performance and motivates the person to achieve best possible score in each sitting.

2.1 (c). Track subject-specific longitudinal data for each test and provide the subject an option to display them on the device in graphical format at the end of the test The option to see test-specific longitudinal data should be displayed only AFTER the subject submits the test results to the remote database. The display of subject specific longitudinal data should include normative data (provided by NDS) so that subject can easily see whether their current and longitudinal performance falls within normal or abnormal range. The subject could select x-axis as actual dates of test performance or as ordinal scale with each subsequent number of test performance, irrespective of elapsed time between tests.

The subject will have an option to display identical test-specific longitudinal data at the end of whole NeuFun testing session from the dashboard.

2.1 (d). Use non-linear regression to analyze subject-specific longitudinal data within each App to identify “learning effect” in subject-specific test performance.

Our analyses of test performance show clear “learning effect” for majority of tests/tasks, that usually peaks after 8 repetitions of the test. This task would identify peak of such learning effect, e.g. by using non-linear regression as described: https://www.medrxiv.org/content/10.1101/2020.03.09.20033316v4 Identification of learning effect is necessary to define new “post-learning baseline” that can be used for measuring true progression of neurological functions. Quantification of learning effect in terms of improvement of the test score and the number of test sittings that led to peak of the learning effect are outcomes that may have clinical value.

Practically, learning effect analysis will be performed only in longitudinal data, only with the most important result per each test (defined by NDS) and can be performed only after 20 test sittings per subject.

2.1 (e). Average post-learning subject specific data to provide more reliable longitudinal test results.

In our previous work, we measured substantial day-to-day fluctuation in test performance. These test-re-test fluctuations make identification of true declines in test performance insensitive. We have shown that sensitivity of the test can be substantially increased if we average several (e.g. 4) different, but adjacent tests: https://www.medrxiv.org/content/10.1101/2020.03.09.20033316v4 It is desirable to perform such test averages locally at the phone and display these averages as more reliable measure of the test performance trajectory. The user should have an option to select the “bin number” in NeuFun settings. This will determine how many adjacent tests will be averaged (ranging from 2 to 6).

2.1 (f). Provide tailored motivational feedback to enhance subjects’ effort, compliance with multiple/repeated testing This task seeks to implement effective motivational feedback to enhance subject’s effort, as well as subject’s willingness to perform multiple/repeated testing. E.g., during the initial learning stage, the motivational feedback should highlight any improvements in test performance in comparison to previous testing. The stability of the test result should also be celebrated, as it means that no functional decline was detected. When the subject’s results fall within age-appropriate normative data, this should be highlighted and celebrated.

Within each cycle of 4 tests, the subject should be motivated to finish the remaining tests of each cycle, so that the average performance of each cycle can be plotted and compared to previous cycle.

2.1 (g). Compare subject-specific performance between tests, to identify unreliable test results and ask the subject to repeat the test We have deliberately designed the tests to measure at least partially overlapping neurological functions. Consequently, outcomes from different tests are usually correlated. The same way the advanced neurological practitioner can measure same neurological functions using different test to identify discrepancies, the expected correlations between different outcomes of different Apps should either assure that the test was performed correctly, and the results are reliable, or, alert to the incorrect test performance, which should prompt retaking of the test(s).

The NDS will provide confidence intervals, expected relationships between different results of different tests and thresholds for alerts. The contractor will codify these relationships and generation of alerts resulting from incongruent relationships between results obtained from different tests.

2.1 (h). Compare subject-specific longitudinal performance to identify deteriorations that exceed test-retest variability and therefore may indicate true worsening of functions. When between-test result comparisons assure test reliability and deteriorations are seen across multiple tests, the App should alert the subject to possibility of real deterioration and instruct the subject to call their clinician and/or to repeat the testing in few hours Compare current test performance to the aggregated test performance (within subject) to identify worsening that falls beyond test-to-test variability and therefore may signify true worsening of neurological functions. If such test result is identified, the subject should be motivated to retake the test (same day or next day) to validate true deterioration. Validated test deterioration that surpasses test-to-test variability should alert the subject to seek medical attention. The thresholds and alert rules will be provided by NDS. The contractor will only codify the comparisons and resulting alerts.

2.2 – ENHANCE THE DASHBOARD CAPABLITIES

The current version of NeuFun has following Dashboard capabilities:

A. Assignment of subject-specific alpha-numeric code under which the testing proceeds. This assures protection of Personal Identifiable Information (PII) B. Informed consent: Brief description of the Purpose of the NeuFun, followed by multiple-choice questions that must be correctly answered to prove subjects’ understanding of the informed consent. Only after subject signs informed consent can s/he proceed to testing. The copy of the signed consent is streamed to cloud database under afore-assigned subject-specific alpha-numeric code C. Collection of crude demographic data: Gender and age (in years) D. NeuFun education suite: videos that explain functionality of each test and the correct way to perform the test E. Practice mode: Ability to perform any tests (with the exception of the delayed version of the verbal memory test), at any time, without collection and streaming of test results F. Testing mode: The test performance is evaluated locally, providing cross-sectional and longitudinal feedback to the testing subject, and raw data are streamed to cloud database under subject-specific alpha-numeric code G. Test Prescriptions: Ability of the test administrator to select only parts of the Test suite to tailor the tests to the disability status of the testing subject H. Updates: Ability to automatically install software updates from cloud database I. Automatic streaming of raw data from each test to cloud database

The Contractor shall enhance the Dashboard capabilities with following functions:

2.2 (a). Test reminders Automatic reminders to the subject to perform the NeuFun if the smartphone detected that the tests were not performed in accordance to prescription.

2.2 (b). Personal NeuFun Achievement Status:

The dashboard should be interactive so that the subject can select to see longitudinal data from each test, bin them to show averages of 2, 3 or up to 6 adjacent tests, change x-axis to date versus sequential numbers of test performance irrespective of date etc.

Also, NDS is using machine learning to aggregate results from multiple tests into a new, highly accurate clinical scale. This optimized and simplified data aggregation from different tests (according to the algorithm derived/optimized and validated by NDS) should occur locally, enabling display of this aggregated clinical scale(s) of the subject in relationship to time and normative data on the dashboard.

Additionally, the App results should display a homunculus, where different neurological functions are captured in space relationship: e.g., a subject may score within 95% of normative data for every function measured by every limb except in the left leg, where the motoric function might be in 5%-tile. Graphical representation of the results from different Apps of NeuFun (e.g. as heatmaps) would show the subject extent and localization of his/her neurological deficit in highly intuitive way.

2.2 (c). Ability to export results of as pdf or Excel/Google Sheets worksheet via email After each completion of the entire NeuFun suite, the subject should be asked if they want to export their test results and offer such export in pdf and Excel/Google docs format via email. The preference of the export format should be available in NeurFun Settings, where the subject may also provide default email address (or several email addresses) for data export and chose option to always export data upon completion of NeuFun without need for additional user intervention.

From the dashboard, the subject should also be able to select export of all results (including longitudinal data and homunculus graphics in pdf format) that can be displayed on the dashboard.

2.3 DEVELOP 5 NEW APPS

2.3(a). Visual acuity App (both eyes) The Visual acuity contrast threshold determination is a smartphone application intended to measure visual acuity of each eye at different contrast levels (default 100% contrast, 5% contrast, 2.5% contrast and 1.25% contrast; see Figure 1). Additionally, the App should treat these contrast levels as a continuum of contrast on non-linear x scale (spacing provided by NDS) and calculate R2, slope and intercept from subject-specific regression model of contrast level versus visual acuity. Figure 1.

Because neurological dysfunction often affects vision, severely disabled patients may not see any letters with 1.25%, 2.5% or even 5% contrasts. This precludes determination of slope and intercept from patient-specific linear regression models analogous to Figure 1. Therefore, the App should dynamically “dial in” increasing contrasts (i.e., 10%, 20%, 30% etc) of the letters, if the tested subject cannot see any letters under 1.25%, 2.5% or 5% contrast. The goal of the App is to identify smallest contrast level at which the subject can see at least 1 letter and then select minimum of 3 additional higher contrast levels, where the 100% contrast is always tested) to derive reliable linear regression model (i.e., with R2 ≥0.7 derived from 4 points; if R2 is <0.6, the test should be repeated with higher contrasts). The slope and intercept from such patient-specific model derived from each eye, as well as maximum numbers seen at 100% contrast would be the outcome of the visual acuity App. All computed results plus all available raw data (including each contrast level tested and number of correctly identified numbers per each contrast) will be automatically streamed to Firebase.

2.3(b) – Motor sequencing test (both hands) The purpose of this App is to recreate “Motor sequencing test” of the neurological examination. The subject will be shown one of 64 (43) different possibilities of the motor sequence. Each motor sequence will consist of 3 defined finger movements (selected from total of 4 possible: e.g., touching smartphone screen with one finger, followed by making a circle with that finger, followed by touching screen with 3 fingers simultanously; the 4th movement: making a vertical up-down line with one finger, would be omitted from this particular sequence). The app will randomly select one of possible 64 sequences for each testing session and display that sequence on the screen so that subject can memorize it. After demonstrating the sequence, the subject would be asked to first perform the sequence simultanously with the App demonstration under slow speed. When subject can perform sequence correctly under slow speed, the demonstration would cease, and the App would instruct the subject to perform the learned sequence on its own as fast as possible in as many repetitions as possible within limited time (30 sec). The outcome of the app is number of mistakes performed, precision of following demonstration at the initial stage of learning and the number of correct motor sequences performed within 30 sec. Simplified results plus raw data will be streamed to Firebase.

2.3(c) – Go-No Go test (both hands) The purpose of this App is to recreate “Go, no-go” test frequently performed as part of neurological examination of frontal lobe functions. The subject is instructed to touch the screen with two fingers simultanously when s/he sees one circle on the screen, while not touching the screen at all when two circles are displayed on the screen. The subject is asked to perform the test as fast as s/he can for 30 sec. The outcome of the test is the number of correct answers per 30 sec. Another outcome is the number of incorrect answers. Test is performed with each hand separately. Simplified results plus raw data will be streamed to Firebase.

2.3(d) – Postural sway and pronator drift test (eyes open, eyes closed) This test requires supplemental hardware (cap/bandana) that attaches the smartphone to the back of subject’s head for postural sway testing and to each hand for pronator drift testing. The subject will be standing, or if severely disabled, sitting, but not touching anything in the environment. The amount of postural sway (i.e., movement of the head per specified time-period) will be measured by smartphone sensors. Subject will be instructed to stay still and perform the measurement first with eyes open, then with eyes closed. Pronator drift test is based on the same principle: measuring sway or downward movement of the phone held in the palm of one, then the other hand (bandana assures that phone does not fall if the grip is weak) when subject starts with outstretched arm, hand facing up. The outcome of the test is movement of the smartphone from each starting position within subsequent 15 second (with laud counting by the App). The movement is registered in all directions, with eyes open and then with eyes closed. The main result is cumulative distance of the phone from initial position within 15 sec period. The test is performed 6 times: 3 (head, and two hands) x for eyes open and 3 times for eyes closed. Simplified results plus raw data will be streamed to Firebase.

2.3(e) – Spatial memory test The purpose of this test is to measure spatial (as opposed to verbal) memory. This test has 2 versions/levels. In the first level, the subject is provided with a randomly generated checker-board pattern, where, on 4x4 grid of while squares, pattern is created by lighting 4, then 5, then 6, then 7, then 8 squares black (Figure 2 shows examples).

The selection of black squares, while random, does need to fulfill certain criteria: each black square must make at least one contact with another black square, where the contact is considered either by sharing side or a corner. The subject is presented with each static pattern for 5 seconds, then the pattern disappears, and the subject is asked to recreate the pattern from memory by touching white squares, which turn into black after touching them. The test starts with 4-square pattern and if correctly recalled, continues to 5-square pattern etc. When the subject makes a mistake, the new pattern is generated of the same length for total of 5 patterns being tested in total. The outcome of this test is the sum of the square dots on patterns correctly recreated, for the maximum score of 30 (4+5+6+7+8). The subject has 10 seconds for recreating each pattern and the time of the response is also recorded. The App should also record the location of black squares that are missed and analyzed it in terms of quadrants, as this information may capture information about visual fields.

The next stage of the App is that instead of subject being presented with a static image, the pattern is generated in random sequence and the subject is supposed to remember both final pattern and the sequence of its generation. Both sequence generation and the final pattern must still fulfill the stated criterion that each black square must make at least one contact with another black square, where the contact is considered either by sharing side or a corner. The scoring is doubled if the sequence is correctly recreated (e.g., 4 square sequence will give score of 8, for maximum achievable score of 60 for this part of the test). Again, time and quadrant mistakes are also collected. Simplified results plus raw data will be streamed to Firebase.

2.2 DELIVERABLES & 2.3 SCHEDULE

Final test w NDS input 3 months 6 months

Final test w NDS input 3 months 6 months

Final test w NDS input 3 months 6 months

Final test w NDS input 3 months 6 months

Final test w NDS input 3 months 6 months

Final function w NDS input 4 months 8 months

Final function w NDS input 4 months 8 months

Final function w NDS input 4 months 8 months

Final function w NDS input 6 months 12 months

Final function w NDS input 6 months 12 months

Final function w NDS input 4 months 8 months

Final function w NDS input 12 months 12-24 months

Final function w NDS input 12 months 12-24 months

Final function w NDS input 4 months 8 months

Final function w NDS input 12 months 12-24 months

Final function w NDS input 6 months 12 months

2.4 ASSUMPTIONS

SOW is based on following assumptions:

1. The contractor will work remotely and will not need access to NIH resources

3.0 GOVERNMENT FURNISHED PROPERTY

No government-furnished property will be provided to the contractor.

4.0 SECURITY REQUIREMENTS

No unique security requirements associated with contract performance were identified.

Refer to - http://www.hhs.gov/asfr/ogapa/acquisition/policies/hspd12-contract-guide-102707.html

Fig 1: Arbitrary spacing of Visual Acuity contrasts based on healthy donor data

100% contrast 2.5% contrast 1.25% contrast

4.9179206566347471 2 1 58.866666666666667 30.426666666666666 20.68 Spacing (units)

Average letters correct image1.png