Attachment J.2, Procedures Manual for State Traffic Records Assessments, October 2018.pdf

Procedures Manual for State Traffic

Records Assessments October 2018 i

Table of Contents

PART 1: INTRODUCTION

PART 2: ASSESSMENT PROCESS

2.1 Requesting an Assessment

2.2 Pre-Assessment Planning Calls

2.2.1 Initial Orientation Call

2.2.2 One-Month Planning Call

2.3 Pre-Kickoff Preparations

2.3.1 Review and Assign Questions

2.3.2 State Document Library

2.3.3 Coordinator & Module Manager STRAP Training

2.4 Kickoff Meeting

2.4.1 Suggested Kickoff Outline

2.5 Conduct the Assessment

2.5.1 Round One Data Collection

2.5.2 Round One Analysis

2.5.3 On-Site Preparation: Week of Review

2.5.4 On-Site meeting

2.5.5 Round Two Data Collection

2.5.6 Round Two Analysis

2.5.7 Drafting the Final Report

2.6 Delivering the Final Report

2.7 Report-Out Webinar

2.8 Requesting Optional Technical Assistance

PART 3: ROLES AND RESPONSIBILITIES

3.1 NHTSA Traffic Records Team

3.2 NHTSA Regional Program Managers

3.3 State Personnel

3.3.1 State Assessment Coordinator

3.3.2 State Module Manager (Optional)

3.3.3 State Respondent

3.4 Assessment Team

3.4.1 Program Manager

ii

3.4.2 Assessment Facilitator

3.4.3 Module Leader

3.4.4 Assessor

PART 4: APPENDICES

4.1 Pre-Assessment Checklist

4.2 Suggested Document Library

4.3 Draft Agenda for Assessment Kickoff Meeting

4.4 Master List of Assessment Questions and Respondents

4.5 GO Team Application

4.6 Glossary of Terms

Table of Figures Table 1: Sample Traffic Records Assessment Timetable Table 2: Breakdown of Assessment Questions Table 3: Guidelines for Assessor Ratings Table 4: Federal Roles and Responsibilities by Assessment Task Table 5: State Roles and Responsibilities by Assessment Task Table 6: Assessment Team Roles and Responsibilities by Assessment Task Table 7: General Documents and Supporting Question(s) Table 8: Crash Documents and Supporting Question(s) Table 9: Driver Documents and Supporting Question(s) Table 10: Vehicle Documents and Supporting Question(s) Table 11: Roadway Documents and Supporting Question(s) Table 12: Citation/Adjudication Documents and Supporting Question(s) Table 13: Injury Surveillance Documents and Supporting Question(s) Table 14: Data Use and Integration Documents and Supporting Question(s)

PART 1: INTRODUCTION

A traffic records system includes the collection, management, and analysis of data within six core State data systems—crash, driver, vehicle, roadway, citation and adjudication, and injury surveillance—as well as data integration, strategic planning, and State traffic records coordinating committee management. The National Highway Traffic Safety Administration (NHTSA) offers a comprehensive approach for assessing these systems based upon the Traffic Records Program Assessment Advisory (DOT HS 811 601). Developed by a diverse group of subject matter experts from across the country, the Advisory outlines the ‘ideal’ as a uniform benchmark for measurement—not necessarily a goal to be met—and provides a set of questions that are used to assess State capabilities in comparison to that ideal.

The assessment is a peer review of a State’s traffic records system following the methodology laid out in the Advisory. It is an opportunity for open dialogue that is constructive and non-punitive. Neither NHTSA nor the State is involved in developing or approving assessors’ ratings or recommendations. Ratings are the assessors’ judgment as to whether or not the State meets, partially meets, or does not meet the ideal set forth in the Advisory. While States must undertake an assessment at least once every five years to remain eligible for §405(c) traffic safety data improvement grant funds, the assessment’s recommendations themselves have no bearing on funding award. It behooves States, however, to be forthcoming during the assessment as the final report’s utility depends on the quality of information the State provides.

Using the State Traffic Records Assessment Program (STRAP), an online data management system, State respondents enter their answers and supporting documentation to questions they have been assigned. During each assessment, State respondents have two opportunities to document their systems’ capabilities. The assessors then review this information and make their initial determinations. At this time, an optional on-site meeting between the Facilitator and respondents is available to clarify questions and ensure the information is accurate and understood by the assessors. The meeting is followed by the second and final round, during which the assessors make their final determinations and considerations before they are all packaged into the final assessment report.

The resulting assessment report is a consensus-based document providing an overview of the State’s traffic records programs in comparison to the ideal set out in the Advisory that identifies the program’s strengths as well as opportunities for improvement. States can use the assessment report to improve long range planning, focus resource allocation efforts, and generate administrative and political support for program improvement. The assessments also enable NHTSA to aggregate regional and national data on State traffic records system performance that is used to better assist States with programmatic improvements.

A successful traffic records assessment requires coordination among the State’s traffic records system stakeholders, NHTSA staff, and the assessors. This document, which is a companion to the Advisory, describes the assessment processes as well as the roles and responsibilities of all participants.

PART 2: ASSESSMENT PROCESS

2.1 Requesting an Assessment

An authorized State official—generally the Governor’s Representative for Highway Safety— should request a STRAP assessment from their NHTSA Regional Administrator. This request should include the State’s top three most desired assessment slots and identify the State assessment coordinator—the individual responsible for overseeing State participation in the assessment.

A State’s request should be made as far in advance of the expiration of the State’s current §405(c) coverage as possible as the assessment process takes at least three months to complete and requires at least one month of lead time. NHTSA’s TR Team works closely with the regional offices to actively solicit requests in a timely fashion.

NHTSA schedules assessments on a first-come, first-serve basis. In their request letters, States should identify their top two or three choices from the calendar of assessment slots provided by their NHTSA regional office. The slots are staggered to enable NHTSA to field as many concurrent assessments as possible. Once all available slots have been reserved, however, NHTSA will not be able to accommodate additional assessment requests.

Once the State’s assessment request has been received by their regional NHTSA office it will be forwarded to the National Driver Register and Traffic Records Division at NHTSA headquarters.

The NHTSA TR Team will review the assessment schedule and provide the State formal, written confirmation of their selected assessment slot. A copy of the Advisory and this manual will be provided at that time.

2.2 Pre-Assessment Planning Calls

Once a State request has been received by the NHTSA TR Team and the assessment slot confirmed, the appointed Traffic Records Team representative for that State will schedule the first of several pre-assessment conference calls. At minimum, there will be two calls: an initial orientation call immediately following confirmation of the State’s assessment slot and a second call one month prior to the kickoff meeting. There may be additional calls should either the State or NHTSA deem them necessary.

2.2.1 Initial Orientation Call

The initial assessment orientation call will include the State Coordinator, the NHTSA TR Team representative, the NHTSA regional office, and other interested parties from the State as determined by the State Coordinator. The primary goal of this call is to familiarize the State Coordinator with the assessment process and finalize the schedule. Specifically, the initial call is used to establish the assessment’s internal timelines, review the pre-assessment checklist items (Appendix 4.1 Pre-Assessment Checklist), and answer any initial questions the State may have.

While the State will already know when its assessment is scheduled to take place in general, having received confirmation of their slot, States must confirm their assessment’s internal timelines during the initial conference call. The active period of an assessment is a 13-week period that runs from the formal kickoff meeting to delivery of the final report. A basic sample schedule can be seen in Table 1. This sample has been optimized to provide State respondents with time during the workweek, to provide assessors with time during weekends, and ensure that transitions between rounds do not occur on weekends to avoid scheduled server maintenance. States are encouraged to follow this model as closely as possible.

Recognizing that many assessment slots will include holidays or other State commitments, a degree of flexibility has been built into the process. During the initial planning call, States may elect to move certain dates within the assessment and expand or contract the State respondent periods. States may not shorten the periods in which the assessors are at work.

Upon NHTSA TR Team receipt of request Initial pre-assessment conference call 1 month prior to kickoff meeting Facilitator introduction pre-assessment conference call

Between Facilitator conference call and kickoff State Coordinator assigns questions, enters contact information into STRAP, and builds initial document library

As se ss m en t

Monday, Week 1 Kickoff meeting Tuesday, Week 1 – 7pm EST, Friday, Week 3

Round One Data Collection: State answers standardized assessment questions

Friday, Week 3 – Wednesday, Week 5

Round One Analysis: Assessors review State answers and rate the responses and, if needed, request necessary clarifications

Thursday, Week 5 – Tuesday, Week 7

On-Site Preparation: State Coordinator and Facilitator gather questions, issues, clarification to prepare for on-site meeting.

Wednesday, Week 7 On-Site Meeting: Facilitator, State Coordinator and selected respondents meet on-site to discuss assessment results.

Thursday, Week 7 – 7pm EST, Friday, Week 9

Round Two Data Collection: State provides final response to the assessors’ ratings.

Friday, Week 9 – Sunday, Week 12

Round Two Analysis: Assessors review additional information from the State and, if needed, adjust initial ratings.

Monday, Week 12 – Sunday, Week 13 Facilitator prepares final report

Week 13 NHTSA delivers final report to State and Region (After completion of assessment, date set by State) NHTSA hosts webinar to debrief State participants

Table 1: Sample Traffic Records Assessment Timetable

Any alterations to the basic schedule must be made during the initial planning call. While STRAP can accommodate schedule changes under certain emergency circumstances, it is generally discouraged. Furthermore, once the assessment has advanced from one phase to the next (from Round One Data Collection to Round One Analysis, for example), the process cannot be reversed.

During the initial orientation call, the State Coordinator will also decide whether or not to designate any module managers. Module managers play a role similar to that of the module leaders on the assessment teams, assisting the State Coordinator with the management of a specific assessment module and ensuring that the questions in their modules are answered adequately. Module managers have the same view and edit rights that the State Coordinator possesses. Module managers are optional, and States may elect to have managers for all, some, or none of the modules in their assessments. Many States choose to have at least one or two module managers in areas that have many questions—injury surveillance, for example—or that the State Coordinator is not directly involved with. This helps to reduce the State Coordinator’s workload, ensures that all questions are answered in each module in a timely manner, and provides a familiar point of contact for questions or concerns.

Participants in the initial planning call will also review the following critical topics:

• State Traffic Records Assessment Procedures and Policy Manual

• Basic STRAP operating procedures

• Identification of a venue for the kickoff meeting that will accommodate the appropriate number of attendees and provide requisite connectivity (Cat5 Internet in addition to a landline)

• Identification of State attendees for the kickoff meeting

• Pre-Kickoff Meeting Checklist (see Appendix 4.1 Pre-Assessment Checklist)

• Scheduling details for the one-month call

In the time between the initial planning call and the one-month planning call, the NHTSA TR Team will identify the assessment Facilitator and assessors from a pool of qualified subject matter experts.

2.2.2 One-Month Planning Call

The one-month planning call will generally occur one month prior to the assessment kickoff meeting and will include the assessment Facilitator, the State Coordinator, the NHTSA TR Team representative, the NHTSA regional office, any module managers designated by the State, and other interested parties as determined by the State Coordinator. Participants in the one-month planning call will be introduced to the Facilitator, confirm the identity and participation of any module managers, review preparations to date, and address any outstanding logistical issues regarding the kickoff meeting.

The one-month planning call marks the beginning of the assessment Facilitator’s active involvement in an assessment. These individuals possess broad expertise in traffic records and exceptional management skills. Facilitators coordinate assessor activities and provide the State Coordinator with hands-on support from the one-month planning call through the conclusion of the assessment. This support includes assistance with identifying State respondents, entering their information into STRAP, leading the kickoff meeting, clarifying any confusion respondents may have about their assigned questions, and offering advice on how to elicit appropriate responses from State respondents in a timely fashion. At the end of the assessment, the Facilitator will also package and review the assessment report. The Facilitators do not, as a matter of course, directly influence the assessors’ ratings of State responses.

If the State has elected to use module managers, their identities and participation should be confirmed at the one-month planning call. A special STRAP training webinar for these individuals and the State Coordinator will also be scheduled if requested. While there will be a STRAP demonstration at the kickoff meeting, the State Coordinator and module managers have expanded rights and responsibilities so this targeted training is highly recommended. The module manager training webinar is generally scheduled for a week prior to the kickoff meeting at the convenience of the State.

The logistical arrangements for the kickoff meeting itself—to include date, time, agenda, and any audio-visual needs—should be finalized and confirmed at this time. The one-month call also marks the beginning of a more active phase of preparations—particularly for the State Coordinator and Facilitator.

2.3 Pre-Kickoff Preparations

State Coordinators are encouraged to coordinate closely with the Facilitator during this critical period as adequate preparation will improve the assessment process immeasurably for the State participants, the assessors, and those assigned to manage the process. State Coordinators are expected to, with the Facilitator’s assistance, undertake the following items between the one-month call and the kickoff meeting:

• Review the list of questions (Appendix 4.4 Master List of Assessment Questions and Respondents), identify the State respondents, enter their information into STRAP, and assign the questions to them.

• Review the list of suggested resource documents (Appendix 4.2 Suggested Document Library) and upload applicable documents to the STRAP Document Library. STRAP may be accessed for this purpose by the State Coordinator using the token sent via email.

2.3.1 Review and Assign Questions

As soon as the assessment has been scheduled, the State Coordinator should review the questions in the Advisory and begin to identify appropriate respondents for each. Shortly after the one-month call, and after consulting with the Facilitator, the State Coordinator will be granted access to STRAP and should begin entering the respondent’s information into the system and assigning them specific questions. It’s a good idea to have at least one person assigned to each question by the kickoff meeting.

The responses provided by the selected State personnel determine whether or not the State’s data and data systems meet the standard described in the Advisory. Thus, it is imperative that these State respondents are knowledgeable about the data system they are being asked to describe, that they enter their answers into the STRAP software within the scheduled timeframes, and that they include the appropriate standards of evidence documents. Appendix

4.4 Master List of Assessment Questions and Respondents lists the assessment questions and suggests roles or job titles of individuals that should be able to answer each question.

In selecting respondents and assigning questions, NHTSA encourages States to consider the widest possible distribution of questions as it helps ensure that the question is not only answered, but answered by the most appropriate, knowledgeable State personnel. In many cases, this will mean that single questions are assigned to multiple respondents. Identifying multiple respondents is particularly important for questions involving data use/analysis, data collection, and data maintenance and management. The knowledge and experience of all three groups—users, managers, and collectors—help the assessors understand the data, its availability, and how it is used within the State. This then enables them to determine data quality and the potential for data improvement in each component of the system.

The State Coordinator should contact each respondent prior to the kickoff meeting so they are aware of their role in the assessment and which questions will be asked of them. This is vital to ensure that the chosen respondent is capable of answering the assigned questions.

Additionally, the State Coordinator should make him or herself available to review the pertinent questions and suggested evidence requirements with each respondent. This review provides respondents with a more complete understanding of the amount of time and effort required to complete their assigned questions and gather the necessary evidence documentation.

Awareness of the effort required will help to prevent respondent’s waiting until the last day the STRAP system is available, then finding that not enough time has been set aside to complete comprehensive responses and upload the required documentation.

Respondents are expected to provide appropriate evidence and documentation as specified in the Advisory for each question they answer. The time commitment necessary to complete these tasks must be taken into account by the State Coordinator when determining the number of questions assigned to each respondent.

A State’s assessment will generally be set up in the STRAP system two weeks prior to the kickoff meeting. Providing the State Coordinator has already identified the respondents and collected their contact information, it is then quick work to enter the respondents and assign questions in-system. While the STRAP system can add additional respondents mid-assessment, it is very much preferred that as many of the respondents as possible be entered into STRAP with contact information confirmed and questions assigned prior to the kickoff meeting and the initiation of Round One Data Collection.

2.3.2 State Document Library

State Coordinators can also prepare for their traffic records assessment prior to the kickoff meeting by beginning to assemble the document library—the documents that the respondents and assessors will need to refer to during the assessment. These can then be uploaded to Document Library once STRAP is made available to the State Coordinator. This will make it easier for respondents to find and cite critical evidence from basic documents like crash forms and data dictionaries. If the State has completed a previous assessment in STRAP, all documents uploaded during the prior assessment will be available within the Document Library.

Responses that do not include the appropriate evidence documents will be rated negatively, whether or not the State’s answer meets the Advisory standards.

2.3.3 Coordinator & Module Manager STRAP Training

Particularly in States that have will use module managers to assist in the State Coordinator’s oversight and management of the assessment, additional training on STRAP’s management tools can be very useful. This one-hour webinar is generally scheduled one to two weeks prior to the kickoff meeting.

2.4 Kickoff Meeting

The on-site kickoff meeting is hosted by the State Coordinator, led by the Facilitator, and—to the extent possible—attended by the State’s assessment respondents, TRCC members (both executive and working level), and any other key State personnel. The State Coordinator should also invite the State’s FHWA and FMCSA representatives. The kickoff meeting explains why and how the assessment is being undertaken, demonstrates the STRAP system for the respondents in particular, and provides an opportunity for face-to-face interaction with the assessment management team. Generally, the meeting itself takes between 90 minutes and two hours, though the assessment team is happy to spend more time with the State Coordinator or other State staff should they need further assistance.

The kickoff meeting is tailored to the State’s needs and the type and order of events may be altered based on State preferences so long as these core objectives are achieved. While the Facilitator leads the kickoff meeting, NHTSA will be represented by either the appropriate TR Team member or a regional program manager, if not both, as funding permits.

It is imperative that the State Coordinator secures a room suitable for the kickoff meeting. At minimum, it must be large enough to accommodate all participants; have full teleconferencing capabilities; high-speed, hardwired internet access; and associated AV equipment to accommodate the STRAP demonstration (laptop, television, projector and screen, etc.).

2.4.1 Suggested Kickoff Outline

To date, NHTSA has found the following to be the most efficient and effective way to conduct the assessment kickoff, but per the above, the State Coordinator and Facilitator may make alterations to suit State-specific circumstances. The meeting begins with a general presentation and discussion session appropriate to both management and staff-level participants, followed by a more in-depth STRAP demonstration for State respondents. This arrangement encourages greater attendance by allowing higher-level staff to attend the first portion of the meeting, but not all of it. A meeting agenda corresponding to this outline can be found in Appendix 4.3 Draft Agenda for Assessment Kickoff Meeting.

Set-Up & Pre-brief The first session is a pre-brief at the meeting’s primary location to review the day’s agenda, troubleshoot any issues, and finish set-up for the kickoff. This informal session generally will include the NHTSA TR Team member, the NHTSA regional program manager, the Facilitator (via webinar), the State Coordinator, and the State traffic records coordinator (if not the same person as the Assessment Coordinator), along with any other State representatives the Coordinator deems necessary.

At minimum, the Facilitator and State Coordinator or designee should arrive on-site at least half an hour before the start of the meeting to ensure that the room is ready for the meeting. Open square or other non-classroom style seating arrangements are preferred. Internet connectivity and AV display abilities should also be tested at this time.

Presentation & Discussion Following the set-up and pre-brief session, the formal portion of the kickoff will begin with the Facilitator delivering a high-level presentation that covers the administrative aspects of the system as well as the content of the Advisory and assessment. The session will include a presentation that provides an overview of Traffic Records Assessments, explains the purpose of the assessment, the procedures and schedule, important deadlines, and describes the contents of the final report. This session is geared towards all TRCC members, data system managers, users, and respondents.

The discussion period that follows is used to address any of the State’s questions or concerns about the assessment process and any current issues that would help give the assessors a clear picture of the State’s situation at the time of the assessment. If executive-level committee members and other managers are not going to serve as respondents, they may wish to leave following the conclusion of this session. If they will be answering questions themselves, they should participate in the next session as well.

STRAP Demonstration

Following the presentation and discussion session, a demonstration of the State Traffic Records Assessment Program (STRAP) will be delivered. When possible, this will be a live demonstration delivered by the STRAP user support specialist via webinar. The webinar will present in detail the functions of the STRAP interface, how to use STRAP to respond to assessment questions, and information flows during the assessment.

Wrap-Up & Question Assignment Review Following the STRAP demo, the formal portion of the kickoff will come to an end. The Facilitator, State Coordinator, and NHTSA TR Team rep will remain available to address any further questions from the respondents and engage in a more detailed review of the assessment questions for each section of the Advisory. Most question assignments should have been completed prior to the kickoff meeting, but any questions remaining unassigned must be assigned by the end of the kickoff meeting.

2.5 Conduct the Assessment

Following the conclusion of the kickoff meeting, the active phase of the assessment begins. The assessment consists of 328 standardized questions that the State will answer, providing appropriately cited evidence to support their responses. These questions rooted explicitly in the Advisory text and are the basis of the exchange between the State respondents and assessors that is spread over two iterative response cycles.

State respondents are reminded that the assessment process is not meant to be an audit or judgment of the State’s data, but a means by which to determine where the State excels and where progress or upgrades are needed and feasible. Such ratings are meant to assist the state in prioritizing system upgrades, based on need and criticality of the data. As a result, it is imperative that responses paint an accurate picture of the state of traffic records. Anything less will hamper efforts to improve traffic safety within the State.

If a State respondent requires more information on specific question, he or she should first examine the suggested “standard of evidence” that accompanies the question and refer back to the germane portion of the descriptive Advisory text. If additional support is needed, the respondent should contact the State Coordinator, who will then contact the Facilitator and the NHTSA TR Team as required.

Questions TRCC Management & Strategic Planning 27 Crash 48 Roadway 34 Driver 41 Vehicle 36 Citation and Adjudication 50 Injury Surveillance 80 Data Use and Data Integration 12 Total 328

Table 2: Breakdown of Assessment Questions

If any State response is unclear to the assessor, a request for clarification will be included in the module leader’s finding. The State Coordinator will be able to view the module leader’s finding and specific request for clarification. After two exchanges and an on-site meeting, the assessors make their final rating for each question. Specifically, the assessors will examine how States address each question to determine how closely a State’s capabilities match those described in the ideal. For each question, the assessors will determine if a State (a) meets the description of the ideal traffic records system, (b) partially meets the ideal description, or (c) does not meet the ideal description.

If no response to the question is received, STRAP will automatically generate a negative finding.

However, it must be noted that States who do not provide an answer to EVERY question by the end of the second-round response cycle of the assessment will not be eligible for Section 405(c) grant funding. Any appropriate answer is acceptable, but every question must be answered.

The State Coordinator will have access to a number of management reports that will be organized by module. The reports will show the number of questions assigned out of the total for that module, whether or not all have been assigned, the number of respondents assigned to questions within the module, the number of responses submitted out of the number expected and whether or not all responses have been submitted. These reports will assist the State Coordinator in monitoring the progress of the assessment.

2.5.1 Round One Data Collection

Each respondent entered into STRAP and assigned a question will be emailed a link (or token) which will authenticate their identity and allow them to log into STRAP and answer their assigned questions. Each person’s token is unique (delineates identity and access) and may not be shared with others. Questions may be assigned to more than one respondent; nevertheless, respondents should submit their responses to each question individually. All assigned questions can be found on the respondent’s tab. Within this tab, questions can be narrowed down by module and sub-module.

Respondents may partially enter an answer and save their work prior to submission. This is particularly useful when respondents discover that they need to attach additional information prior to submitting their answers. Once the respondents begin to submit their responses, their list of questions can be further narrowed down to show only those that have or have not been submitted.

Past Responses If the State has completed a previous assessment in STRAP, Respondents will have access to view the past responses for their assigned questions. The only exception being questions new to the 2018 Advisory.

The past response(s) will be displayed on the Respondent Answer/Update screen along with any attached documents.

Share, Forward, Decline Respondents may feel that they are not the appropriate person to answer an assigned question, or that there is a person more qualified to answer the question. In these cases, respondents have several options. They may answer the question and then share it with another respondent, by selecting the “share” button and choosing a respondent already in the system or by entering a new respondent’s name and email. If the respondent has nothing to contribute to a question, he or she may forward the question to a better qualified respondent by selecting the “forward” button and choosing a respondent already in the system or by entering a new respondent’s name and email. As a last resort, when the respondent cannot answer the question or think of anyone to defer it to, he or she may decline the question. This will remove the question from their queue completely. Should respondents need to decline a question, they should notify the State Coordinator so a replacement respondent can be found.

As all questions must be answered in order for the assessment to be validated for grant purposes, declining questions is strongly discouraged.

The Traffic Records Program Assessment Advisory should be reviewed by respondents in order to assist with the context of questions that they are assigned. If more assistance is required, the State Coordinator and the Assessment Facilitator may be contacted for clarification.

Providing Evidence To ensure that assessments are uniform and reliable, STRAP provides respondents and assessors with suggested standards of evidence to support the responses for each question. In most cases, State respondents are asked to document their answers to the assessment questions.

Evidence should be uploaded to the document library and linked to the question response in STRAP. If the evidence required is a multi-page document, the response must include the page number and location of the relevant part of the document (or the respondent may cut-and-paste the relevant part of the document into a separate document, listing its source, then upload and link it to the question). States are encouraged to provide screen shots and sample data runs instead of web links. States are free to provide alternative evidence, but in order to garner a “meets” rating it must support the State’s assertions with enough clarity that the assessor can make an accurate judgement and, further, that it would pass muster with any program auditors.

Providing High Quality Answers The assessment questions are designed to elicit information that will allow the assessors to make a comparison of the State’s systems with the ideal system described in the Advisory. As an example, the Advisory describes an ideal crash system in part:

The State maintains accurate and up-to-date documentation—including process flow diagrams—that details the policies and procedures for key processes governing the collection, submission, processing (e.g., location coding), posting, and maintenance of crash data.

Question 47 relates directly to that ideal description:

Does the State maintain accurate and up-to-date documentation detailing the policies and procedures for key processes governing the collection, reporting, and posting of crash data—including the submission of fatal crash data to the State FARS unit and commercial vehicle crash data to SafetyNet?

The suggested standard of evidence related to this question:

Please document key processes governing the collection, reporting, and posting of crash data—including the submission of fatal crashes to the State FARS unit and commercial vehicle crashes to SafetyNet. If the State has existing documentation (process flow diagrams, reports, etc.), please submit the relevant document or an excerpt thereof. If the State does not have existing documentation, a brief narrative may be submitted instead.

When answering, respondents should keep in mind that the assessors are unfamiliar with the State’s processes and they should strive to provide complete information, rather than short or one-word answers.

For example, an inadequate response to Question 47 would be:

Yes. Policy attached.

While this is responsive to the question, it does not provide all the details that the assessor needs to make a determination. For example, does the policy provided apply to all law enforcement officers within the State or is it a State Police/Patrol policy only? A better response would be:

Yes, the State uses a single Uniform Crash Report and Officer’s Manual for Crash Reporting. The Manual is updated whenever the report is revised. FARS reporting is included in the Manual as is commercial motor vehicle crash reporting.

SafetyNet reporting is done centrally at the crash data repository and information about that process is included in the crash data entry policy and procedure manual which is updated semi-annually, with interim changes being added to the Manual in the form of memoranda to the staff until the change is formally incorporated by the semi-annual update.

2.5.2 Round One Analysis

The assessment team evaluates the State’s performance compared to the ideal specified in the Advisory. At the end of the first State response period (Round One Data Collection), access to the STRAP State respondents tab is disabled and the assessor tab is activated. These qualified SMEs review the State’s response to each question and rate each as (a) meets the description of the ideal traffic records system, (b) partially meets the ideal description, or (c) does not meet the ideal description. At least two assessors examine each question.

Upon accessing the assessor tab in STRAP, the assessor should see instructions, reports and a listing of the questions and answers they will need to evaluate. Assessors and module leaders will review the State’s answers and supporting documentation before making their rating selections from a drop-down menu: "meets", "partially meets", or "does not meet". In addition to the rating, assessors also provide a ballot for each question. This brief narrative accompanies and justifies the ballot selection for each question. Assessors may also include clarification requests as part of the ballot. This should be used if the assessor needs more information in order to verify a State’s answer that is not supported by the supplied documentation.

Using the Standard of Evidence & Assessor Guidance The assessors for each module will independently complete ballots to rate whether each State-supplied response indicates that the State meets the standard outlined in the Advisory and findings, which delineate the reasoning of the assessor in making the rating, and note the evidence provided. The assessors use the standards of evidence guidance to help them determine whether there is sufficient evidence to determine whether a capability or critical element exists, is being implemented, or does not exist. It is, however, entirely possible that a State may have provided evidence sufficient for the assessor to make an accurate evaluation about a question without providing the evidence specified in the standard. It is up to the assessor and module leader to make such determinations. In the interests of transparency and efficiency, these standards of evidence and assessor guidelines are also made available to State respondents. See Table 3 for guidelines used by assessors to develop ratings.

Using the guidelines found in Table 3, assessors review the State’s answer and supporting evidence provided for the question. It is imperative that any documentation be linked to the appropriate answer. A rating of “meets the standard” requires that the State indicate it is in compliance with the Advisory ideal and provide sufficient documentation for the assessor to determine the validity of the State’s claim.

In some circumstances, assessors may possess outside knowledge of the State undergoing an assessment. Ratings must be made based solely on the information the State provides via STRAP for the current assessment. Assessors are encouraged to use their external knowledge in requesting additional information of the State, but unless the State actively confirms the assessors’ assumptions and provides sufficient evidence in STRAP, the initial rating must stand.

Such incidents should be highlighted in the narrative module summaries, e.g. “The assessment team is fairly confident that the State meets this requirement of the ideal system, but as no evidence was provided, the team was unable to grant a ‘meets the standard’ rating.”

Assessors will also have access to view the previous assessment’s responses for each question.

These will be displayed for both Assessors and Module Leaders on the Edit/Update Ballot/Finding screens. As with external knowledge, the past response can be used to request additional information of the State, but the rating must be based on the response(s) entered in STRAP for the current assessment.

Assessors are also permitted to include a “clarification request” as part of the ballot if the State’s answer or supporting documentation is unclear. When including a request for clarification, the assessor should provide a narrative of the additional information sought in the space available. Assessors should note that the selected rating was awarded based upon the information supplied and specify what missing information is needed to award a rating in keeping with the State claim.

Prior to the conclusion of Round One Analysis, the Facilitator may host a conference call for all assessors to discuss any major issues and compare notes. While the assessors for each module will already be in regular contact, this is an important opportunity to share insights across modules.

Response Rating Ballot Comment

DNM PM M

No answer provided X System generates response.

Positive answer, no evidence X Request evidence.

Positive answer, substituted evidence other than that required by the Advisory

X X X

Rating will depend on the quality of the substitute evidence, and the information provided by the substitute evidence. If the substitution is verifiable, and serves to prove the response, the rating should be the same as if the required evidence were submitted. If the evidence is lacking, clarification should be requested.

If a system is under development, but has not been implemented

X X

Note with the rating that the State is in process of development in order to both give the State credit and to provide information to future assessments. Many development projects are abandoned or fail. Ratings should not reflect "what might be."

Positive answer, “cannot obtain” evidence

X Seek clarification for the lack of evidence, request alternate evidence. Rate on evidence provided.

Positive answer without adequate information

X Seek clarification. Rate on evidence provided.

For answers to system-wide questions that States claim “meets” except for “one or a few small agencies”, etc.

X

To ensure consistency and equity of ratings, ratings for electronic capture and/or submission are: Meets: States with a pop. of over 6M require 99% electronic capture and submission;

States whose pop. is 2M to 6M require 98% electronic capture and submission; States with less than 2 million population will require 95% electronic capture and submission.

If the question is partially answered. X Request additional information.

Table 3: Guidelines for Assessor Ratings

Providing High-Quality Analysis Assessor ratings and ballots must stand alone grammatically and in terms of content. They should be written in full sentences that explain the evidence provided, and how it was ranked.

An inadequate rating/finding:

The State reported ‘no’.

Or

The State meets the standard. Evidence provided.

An adequate rating/finding might be:

The State does not meet the standard of evidence. No examples of performance measures were provided.

However, it is better that findings are specific to the question/response:

The State indicated that no timeliness performance measures are in place in the Department of Public Safety.

Other examples of appropriate findings are:

The data dictionary does not address the edit checks. Based upon the schema extract provided, it appears that the edit checks and data collection guidelines exist, but the narrative did not include a detailed description to establish that it meets the standard.

Documented procedures are in place for returning data to the individual agency for correction/clarification. The State provided a description of the process used to request data corrections from the collecting agency.

Documents were provided and the responses show that the technical level TRCC's recommendations are referred up to the executive level for approval. The TRCC charter loosely describes this relationship but the State might benefit from tighter coupling in this area.

Combining Ratings and Ballots into Ratings and Findings After the assessors have completed their ratings and ballots, the module leader will review both and incorporate them into a single rating and narrative finding for each question. If the assessors and module leader are unable to reach consensus, the Facilitator will help make the determination.

The module leader will be the assessors’ point of contact for the assessment process. Module leaders will work with the assessment Facilitator to communicate with the State. While the respondents’ names will not be available to the assessors, their job functions and the department in which they are employed will be accessible when available, in order to assist the assessor in determining the respondent’s point of view, either as a data user, collector, or manager; this background information will also help assessors who need clarification to formulate their queries based on the expertise of the respondent.

The combined rating and finding for each “partially meets” and “does not meet” answer will be returned to the State at the beginning of Round Two Data Collection for additional information.

Questions whose Round One answers met the standard are removed from the respondents’ queues. If the Round One answer or evidence was deemed inadequate, the module leader may include a specific clarification request for additional information or clarification. This request will be passed on to the State by the module leader.

The ratings and findings sent to the State are prepared as a synthesis of those made by the assessors and module leader individually. Once consensus has been reached, the module leader will enter in the finalized rating for Round One. At the close of the round, all questions must have rating and narrative finding.

The assessors, module leaders, and Facilitator should agree on timeframes for completion of the work assigned to each. Assessors should be aware that their ratings must be complete prior to the end-date of each round in order to provide adequate time for review by module leaders.

2.5.3 On-Site Preparation: Week of Review

Immediately following the close of Round 1: Analysis, the State Coordinator and Respondents will receive view-only access to the Round One module leader rating and narrative finding for each question. The State Coordinator and Respondents should review the initial findings and determine the questions and areas needing further discussion.

During this time, the Facilitator should also consult with the module leaders to confirm specific questions or topics needing further clarification. The module leaders should also bring attention to any potential issues so they can be addressed and possibly resolved prior to the second and final round.

The Facilitator and State Coordinator should then compare the items relayed by both the respondents and module leaders and create an on-site meeting agenda. This should include time for state participants to provide general feedback on the process as well as an organized schedule for addressing each module. The State Coordinator should invite respondents who wish to discuss their concerns and/or can provide additional information.

2.5.4 On-Site meeting

The On-Site meeting should be held the week following on-site preparation. The length of the meeting will vary, depending on the number of questions to cover, but should last no longer than one business day. The meeting date should be scheduled far enough in advance to ensure respondent participation, ideally prior to the assessment kickoff.

Similar to the Kickoff, the On-Site meeting is hosted by the State Coordinator. The Coordinator should arrange for a meeting room large enough to accommodate all participants; have full teleconferencing capabilities; high-speed, hardwired internet access; and associated AV equipment to accommodate the presentation of notes, responses and other materials.

The State Coordinator and Respondents should come prepared to discuss specific questions, by module, as determined during the on-site preparation. The Facilitator will attend the meeting in person along with a rapporteur who will document the agenda and related discussions.

Following the meeting, the State Coordinator will be provided with the meeting notes recorded by the rapporteur. The meeting notes will also be provided to the assessment team and uploaded to the Document Library.

2.5.5 Round Two Data Collection

Following the close of the on-site meeting, the respondents will have their second and final opportunity to respond and/or provide additional information or evidence to support their responses to the questions that the assessors rated as partially meets, or do not meet the standard described in the Advisory.

Respondents should provide any additional information or clarification that was agreed upon during the on-site meeting or is specifically requested by assessors in the finding narrative. The respondent may contact the State Coordinator to assist with any inquiries about the clarification requested, prior to submitting a response. The State respondent will also have the opportunity to provide additional information in cases where the respondent feels it might impact the rating. If a respondent finds that the answer provided in the previous round was incorrect or incomplete, the initial response should not be deleted, but a notation should be made in the subsequent round response that it is a “correction or clarification” of the previous response. When responses change from one round to the next and the previous response is gone, assessors have no means to account for an upgraded rating.

If the rating is low but correct and respondents have no further information, it would be helpful for the State to indicate that the rating is accurate and no further data will be forthcoming. To do so, respondents may add a note to their answers: “Round Two, no further information” or “Round Two, agree with rating”, etc. This will speed the process for assessors and clarify no further information is available.

This is the State respondents’ last chance to provide information to the assessors and there is no further opportunity for the State to review the assessment report prior to its final release.

2.5.6 Round Two Analysis

At the end of the given timeframe for responses, the respondent tab will again be disabled and the assessors will be able to review…