IAD Technology Evaluations Support-Draft.pdf

STATEMENT OF WORK

Title: Information Access Division (IAD) Technology Evaluations Support Services

I. BACKGROUND INFORMATION

Founded in 1901, National Institute of Standards and Technology (NIST) is a non-regulatory federal agency within the U.S. Department of Commerce. NIST's mission is to promote U.S.

innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life. NIST requires contractor support to fulfill this mission in the Information Technology field.

The Information Technology Laboratory (ITL) is one of NIST’s six research laboratories. ITL focuses on IT measurements, testing, and standards, and is a globally recognized and trusted source of high-quality, independent, and unbiased research and data. ITL’s mission, to cultivate trust in information technology (IT) and metrology, is accomplished using its world-class measurement and testing facilities and encompassing a wide range of areas of computer science, mathematics, statistics, and systems engineering. This non-regulatory role, along with ITL’s deep technical expertise in all fields of information technology, helps increase trust in IT worldwide.

The Information Access Division (IAD) is a division within NIST’s ITL. IAD is known for world-class evaluation-driven research programs that accelerate the readiness of emerging technologies for widespread deployment. For decades, IAD has fostered community evaluations for foundational Artificial Intelligence in research areas such as human language technologies, biometrics, search, information retrieval and natural language processing. IAD advances the measurement science, develops novel evaluation paradigms, and performs research to ensure technology is developed and used correctly and efficiently. Through collaborations with industry, academia, and the federal government, IAD enables the advancement of these technologies and provides the technical underpinnings to support the development of standards.

IAD has four groups that work collaboratively on technology evaluations. IAD technology evaluations can vary in size, purpose, and sector served; in addition to working with various academic and industry communities, IAD’s frequent evaluation partners come from the law enforcement and intelligence communities. The Multimodal Information Group focuses on measurement and evaluation methods to facilitate the use of technologies that provide easier access to multimedia and multilingual information and improve human-technology interface modalities. The Retrieval Group facilitates retrieval research involving large, unstructured text files by providing test collections and organizing the Text REtrieval Conference (TREC) and its proceedings. The group focuses mainly on collections to support specific information retrieval sub-tasks, such as cross-language retrieval and multimedia retrieval. The Image Group focuses on developing standards, measurements, evaluations, and best practice guidelines for biometric technologies. The Image Group promotes accuracy and interoperability by working closely with other government agencies, law enforcement, industry, and academia to support needs in the biometrics community. The Visualization and Usability Group performs research to develop user-centered measurement and evaluation methods, guidelines, and standards by applying human factors, cognitive science, user-centered design, and usability principles to improve human system interaction.

II. PURPOSE

IAD conducts technology measurements and evaluations across a variety of existing and emerging technologies, including technology evaluations and research with a usability testing component. IAD’s technology evaluations can include technologies or algorithms, testing scenarios, multi-modal data (e.g., text/code, image, video, audio, multi-media data), metrics, and can also include humans with various roles and tasks, ranging from general public participants interacting with technology to expert annotators annotating data or providing judgements.

Human judgements can be multi-faceted, including comparison to state-of-the-art technologies.

Evaluation results are communicated back to the larger evaluation community via reports, presentations, and workshops. The purpose of this contract is for the contractor to complete tasks such as creating and refining topic statements, queries, and descriptions, as well as making judgments and annotations on various content types (text, video, audio, images, etc.). The tasks also include assessing the quality and accuracy of content, determining whether content is from the same source or has been tampered with, evaluating the quality of human-produced and technology-generated content, and recruiting humans with specific expertise to complete human-technology interaction tasks. Various combinations of the tasks described in the subsequent

Tasks section are envisioned to support cross-division technology evaluations, measurements, and research for IAD, ultimately serving to improve technology measurement science.

PLACE OF PERFORMANCE

The vast majority of work performed under the resultant IDIQ contract will be performed at the

Contractor’s site/off-site. However, it is possible that circumstances may require the Contractor personnel to work on-site at a federal facility, e.g., NIST campus (for example, if there is certain technology that would be too difficult to transport or ship to the Contractor). The place of performance will be specified in each individual task order.

Regular Business Hours

The following shall apply to on-site work at the NIST Gaithersburg, MD campus:

• Regular business hours are Monday through Friday 8:30 a.m. to 5:00 p.m. Eastern Time with the exception of Federal holidays or other official closures.

• The Contractor shall not be allowed to work on-site at NIST outside of the aforementioned core hours unless specifically stated in the task order.

• In the event of facility closures, Contractor personnel shall not perform on-site services unless otherwise approved by the Contracting Officer.

• All Contractor personnel must comply with the NIST safety prevention regulations. Such regulations include, but are not limited to, general safety and fire prevention.

• Contractor personnel shall wear a visible visitor’s or security badge ID when working on-site at NIST. The badge will be provided by NIST. The Contractor shall immediately report any lost or stolen badges to the Contracting Officer’s Representative (COR). Upon termination/resignation of the Contractor’s services under this contract, the badges shall be relinquished to the COR.

• Contractor personnel shall identify themselves as a “Contractor” in all e-mails, telephone communication, meetings, and conferences.

• All Contractor personnel must register their vehicles in accordance with NIST Security requirements to gain access onto the NIST campus.

• Contractor personnel shall not use Government facilities or Government property in connection with conducting activities not associated with requirements under a given task order.

• At any time during the performance of this contract, any Contractor personnel whose actions (e.g. intoxication, debilitation resulting from drug use, insubordination, theft) result in clear disruption to the work force will be immediate removed by the Contracting

Officer.

PERIOD OF PERFORMANCE

The Period of Performance, or ordering period, of the Indefinite Delivery/Indefinite Quantity

(IDIQ) contract is 5 years from date of award.

TYPE OF CONTRACT

The base contract will be an IDIQ type Contract. All task orders issued against the base IDIQ contract shall be firm-fixed price.

III. SPECIFIC REQUIREMENTS

TASKS

The specific tasks required under this IDIQ contract will be detailed in individual task orders.

However, the following section provides information on the types of tasks that the contractor will be required to perform.

Various combinations of the following tasks are envisioned to support technology evaluations and research for IAD.

1. Technology evaluations are composed of the following tasks:

• Creating initial topic statements;

• Creating refined topic statements and relevance assessments;

• Creating queries (a short sentence that identifies a video uniquely);

• Creating descriptions of videos;

• Making judgments of whether a given segment of video contains a given feature

(e.g.

• person; place, thing or event) or meets the information need specified in a topic

(information request);

• Making annotations of the presence of entities or other linguistic structures in text;

• Creating usable topics with topic statements;

• Selecting newswire articles for each usable topic;

• Writing summaries for each usable topic and making assessments on the quality of the automatic and manual summaries;

• Reviewing topics, queries, descriptions, assessments, or annotations created within the task, for purposes of quality control

• Creating initial test sets across multiple data modalities, developed by both humans and technologies for specific tasks.

• Writing descriptions and annotations for a variety of content types, including text, code, audio, images, and videos.

• Summarizing articles, producing both accurate and inaccurate versions.

• Assessing various characteristics of the data, for example, accuracy and data source.

• Assessing whether two content (e.g., face or speech) are from the same source.

• Assessing whether content has been tampered with.

• Evaluating the quality of both human-produced and technology-generated content.

2. Usability testing is defined as: evaluation that involves representative users performing specific tasks with the system to enable the measurement of efficiency, effectiveness, and/or user satisfaction. Usability testing is composed of the following tasks:

• Research on user/participant demographics

• Recruiting and screening of test participants;

• Providing usability testing facilities, including remote usability testing capabilities;

• Developing test materials and test instrumentation;

• Facilitation of test protocols;

• Proctoring of surveys;

• Facilitating or conducting interviews and contextual analysis;

• Evaluation of user engagement

• Annotating data;

• Analyzing data;

• Compensation of test participants;

• Reporting/Documenting the results of the tests;

• Project management

3. Biometric data collections to support research and technology evaluations are composed of the following tasks:

• Research on user/participant demographics;

• Obtain required data collection approvals;

• Evaluation of user engagement;

• Recruiting and screening of test participants;

• Providing biometric data collection facilities;

• Collecting biometric data;

• Protect biometric data;

• Compensation of test participants;

• Analyzing/error checking data;

• Annotating data;

• Project management

Homeland Security Presidential Directive (HSPD)-12

All work that shall be performed under this contract is designated as HSPD-12 Security Risk Level:

Low Risk.

The Contractor (and/or any subcontractor) and its employees who require physical access to

NIST facilities or logical access to NIST IT networks or systems must comply with Homeland

Security Presidential Directive (HSPD)-12, Policy for a Common Identification Standard for

Federal Employees and Contractors; OMB M-05-24; OMB M-19-17; FIPS 201, Personal

Identity Verification (PIV) of Federal Employees and Contractors; NIST HSPD-12 policy; and

Executive Order 13467, Part 1 §1.2.

The Contractor (and/or any subcontractor) must submit a roster by name, position, e-mail address, phone number and responsibility, of all staff working under this acquisition where the

Contractor will develop, have the ability to access, or host and/or maintain a government information system(s).

The roster must be submitted to the COR and/or CO within 14 days of the effective date of this contract. Any revisions to the roster as a result of staffing changes must be submitted within 14 days of the change. The COR will notify the Contractor of the appropriate level of investigation required for each staff member. If the employee is filling a new position, the Contractor must provide a position description and the Government will determine the appropriate suitability level.

Position Sensitivity Designations Requirement

All Contractor (and/or any subcontractor) employees must obtain a background investigation commensurate with their position sensitivity designation that complies with Parts 1400 and 731 of Title 5, Code of Federal Regulations (CFR).

The following position sensitivity designation levels apply to this solicitation/contact: LOW.

IV. GENERAL REQUIREMENTS

Government Furnished Property

All property, data and information provided by the Government in the performance of this task remains the property of the Government and shall be surrendered to the government upon completion or termination of this requirement. Likewise, all data and deliverables generated under this requirement remain the property of the Government.

Inspection & Acceptance

In addition to the inspection and acceptance terms articulated in the specific FAR clause, the

Government reserves the right to perform such performance tests and evaluations as defined below to verify specified system performance. Such tests and evaluations, if performed, shall be conducted within the environment that the system is to be operated. The Contractor has the right to be present during the tests and evaluations, if performed, at the Contractor’s expense.

The Contractor deliverables within each task order will be inspected by the Technical Point of

Contact (TPOC) or COR to determine their acceptability. Deliverables will be accepted if they meet the requirements set forth within each task order.

Specific performance standards and acceptance criteria for inspection and acceptance will be defined and included within each task order.

Safety

The Contractor employee must be responsible for knowing and complying with all installation safety prevention regulations. Such regulations include, but are not limited to, general safety, fire prevention, and waste disposal.

Security

NIST is a restricted campus. An identification badge is required for access for entry into buildings and is shown to the armed Security Police when entering the campus.

Identification Badges

Contractor employees must comply with NIST identification and access requirements. Each

Contractor employee must wear a visible identification badge provided by the NIST Security

Office.

V. REQUIREMENTS FOR CONTRACTOR PERSONNEL

Base IDIQ Contract Key Personnel:

As identified below, the Contractor shall assign a Supervisor/Program Manager as key personnel at the base IDIQ contract level for managing all tasks issued under the IDIQ contract. The Contractor’s

Supervisor/Program Manager may or may not be one of the key personnel performing the tasks identified in task orders issued under the IDIQ contract. The Supervisor/Program Manager shall be responsible for managing the work across all task orders. The Supervisor/Program Manager will be designated as key personnel at the base IDIQ contract level.

At the IDIQ contract level, the contractor shall identify and provide a key personnel candidate for the following position. The key personnel will be identified in the resultant IDIQ contract and shall be subject to the terms and conditions of clause 1352.237-75 Key Personnel. The following position is the ONLY contractor key personnel position required at the base IDIQ contract level:

1. Supervisor / Program Manager – [Key Personnel on IDIQ base contract]

Description of Duties and Responsibilities: Responsible for managing the work across all task orders issued under the IDIQ contract.

Minimum Qualifications: Contractor personnel responsible for supervision and program management for task orders issued under this IDIQ contract shall have a formal program management certification, or demonstrable equivalent training and experience, that demonstrates a professional's knowledge, skills, and abilities in managing programs, which involves coordinating multiple projects and activities to achieve strategic objectives, and directing and managing program/project schedules, resources, and outcomes.

The Contractor shall be responsible for employing technically qualified personnel to perform the work specified in this SOW. The Contractor shall maintain the personnel, organization, and administrative control necessary to ensure that the work delivered meets task order requirements.

Personnel assigned to, or utilized by, the Contractor in performance of work shall be fully capable of performing the contemplated functions of the respective labor categories in an efficient, reliable, and professional manner.

Individual task orders issued under the IDIQ contract may contain contractor key personnel requirements. The contractor’s personnel under a given task order shall fall within the labor categories proposed and agreed upon by the contractor and Government, incorporated into the

IDIQ contract.

The following labor categories are categories that must be made available under the base IDIQ contract, at a minimum.

Allowable Education Equivalents:

For the following labor categories, five (5) years of relevant work experience is an allowable equivalent to a Bachelor’s degree. Two (2) years of relevant work experience plus an Associate’s degree is also an allowable equivalent to a Bachelor’s degree.

Required IDIQ Labor Categories:

Assessor

Level I:

Minimum Qualifications: All Contractor personnel working under task orders issued under this

IDIQ contract for technology evaluations involving a text retrieval component shall have experience in intelligence analysis. This experience shall include work for the intelligence community such as the Department of Defense, National Security Administration, Central

Intelligence Agency, Defense Intelligence Agency, and others gathering intelligence by examining large amounts of data from various sources.

Responsibilities: Responsible for performing IR research support as described in individual task orders.

Level II:

Minimum Qualifications: All Contractor personnel working under task orders issued under this

IDIQ contract shall have experience in intelligence analysis. This experience shall include work for the intelligence community such as the Department of Defense, National Security

Administration, Central Intelligence Agency, Defense Intelligence Agency, and others gathering intelligence by examining large amounts of data from various sources.

Responsibilities: Responsible for performing IR research support as described in individual task orders, and assistance to new Assessors.

Project Manager

Minimum Qualifications: Bachelor’s degree or equivalent, 3 years minimum experience as project manager.

Responsibilities: Specific duties include defining project scope and requirements, managing project budget, acquiring creative and technical resources from internal and external resource pools, organizing team meetings, facilitating client meetings, product reviews and approvals, creating and maintaining production schedules, troubleshooting problems and communicating project status to internal clients, team members and Senior staff as required. Project Manager will also work with other departments within client company environment as project liaison to ensure successful implementation of product and interdepartmental support as needed. May participate in planning marketing strategies for product and provides input into the development of marketing materials. Works with all departments involved to implement the product on time and within budget.

Research Manager

Minimum Qualifications: Bachelor’s degree or equivalent, 2 years minimum experience as research manager.

Responsibilities: Manages all aspect of the research facility, including scheduling the various stages of studies, setup and maintenance of the facilities and testing rooms, and running test-day operations. Will set up all cameras, on-screen recorders, audio sources, etc. for each study.

User Experience (UX) Researcher

Minimum Qualifications: Bachelor’s degree or equivalent, 3 years minimum experience as UX researcher.

Responsibilities: Assesses the current user experience and seeks to understand future user experience needs. Plans, moderates, and summarizes reporting of a wide range of research activities. Coordinates with the design team to ensure design assets are in alignment with the goals of the studies being conducted. Conducts and evaluates qualitative and quantitative user research to understand needs, goals, and challenges to make recommendations for maximum project benefit. Conducts data collection and analysis through ethnographic, in-depth interviews

(IDIs), focus groups, usability testing, surveys, persona creation, and experience mapping research techniques. Develops personas for each audience segment to enable a common understanding and adoption of each audience segment. Identifies, portrays, and evaluates the existing as well as to-be-created journey for a particular persona to help understand users and their challenges. Oversees UX lab operations including participant recruitment and scheduling, incentive disbursement, facility and equipment set-up, recording of research sessions.

Usability Test Moderator

Minimum Qualifications: Bachelor’s degree or equivalent, 3 years minimum experience as usability test moderator.

Responsibilities: Oversees recruiting, can write the screener, and help identify the audience.

Recommends the methodology and writes the moderator’s guide. Moderates the study. Analyzes the data and makes recommendations. Writes a report based on the findings and can present the findings and recommendation to stakeholders.

Focus Group Moderator

Minimum Qualifications: Bachelor’s degree or equivalent, 3 years minimum experience as focus group moderator.

Responsibilities: Oversees recruiting, can write the screener, and help identify the audience.

Recommends the methodology and writes the moderator’s guide. Moderates the study. Analyzes the data and makes recommendations. Writes a report based on the findings and can present the findings and recommendations.

Senior Developer

Minimum Qualifications: Bachelor’s degree or equivalent, 6 years minimum experience as senior developer.

Responsibilities: The Senior Developer creates, codes, and maintains large-scale web sites and applications. Senior Developers will troubleshoot bugs and deal with cross-browser and interface testing as well as discuss requests with users for clarification. Senior Developers will recommend modifications to lower costs, enhance usability or expand functionality. Senior

Developers create time and expense estimates to fulfill user requests. The Senior Developer uses in-depth knowledge of multiple technologies, user-centered analysis and evaluation techniques, and new programming standards to develop world-class products, sites and applications.

Developer

Minimum Qualifications: Bachelor’s degree or equivalent, 4 years minimum experience as developer.

Responsibilities: The Developer creates, codes and maintains user experiences for high profile web sites. Developers keep abreast of developing web technologies and issues and adhere to and develop new programming standards. They understand browser specific compatibility issues and ensure that they are delivering software which is fully compatible & fully exploits available capabilities.

QA Engineer

Minimum Qualifications: Bachelor’s degree or equivalent, 2 years minimum experience as QA

Engineer.

Responsibilities: Test web sites and applications for technical and functional accuracy. Identify and evaluate tools that will streamline testing. Create test plans and cases by analyzing product requirements and system architecture; execute test plans accordingly. Document test results.

Report and track bugs and discrepancies between requirements and the product, facilitate remedies to any issues found.

Software Engineer

Minimum Qualifications: Bachelor’s degree or equivalent, 4 years minimum

Responsibilities: The Software Engineer is responsible for developing, implementing, and deploying a framework that facilitates the installation and execution of systems with large datasets on a GPU-accelerated cluster. The engineer will create a baseline algorithm to support various evaluation series. Additionally, the Software Engineer will conduct and implement experimental designs, curate datasets, and manage data by querying and translating data formats and specifications.

Computer Science/Electrical Engineer

Minimum Qualifications: Bachelor’s degree or equivalent, 4 years minimum

Responsibilities: Responsible for configuring, installing, and deploying the systems/hardware that performs the data collection. Assures that the data is properly collected, formatted, ground truthed, and stored (including backups) before it is made available to researchers.

Senior Writer

Minimum Qualifications: Bachelor’s degree or equivalent, 6 years minimum experience as senior writer.

Responsibilities: To write, proofread and rewrite technical manuals, training materials, user documentation and presentations. May also write marketing materials, subject matter content, product narratives, descriptive text, or fulfill other writing needs. Maintain a library of technical publications and journals. Recommend changes to document formats to make them easier to develop and use. Coordinate the production and presentation of documents, text, artwork, photographs and other media requirements. Maintain and modify any documentation as needed because of changes to products. The Senior Writer may also be called upon to coordinate and/or manage teams of writers, as well as assist in setting marketing strategy.

V. SCHEDULE OF DELIVERABLES

Each task order placed under this contract will include deliverables and deliverable due dates to the maximum extent practicable.

Standards of Acceptance: The NIST COR shall review all the above deliverables and respond with an acceptance or request for revision email to the Contractor Point of Contact (POC) within the period specified in the individual task order.

Performance Requirements Summary

Individual task orders under this contract will include a Performance Requirements Summary as applicable. The following is a sample Performance Requirements Summary for a task order involving usability testing components.

Task Desired Output Required Service Performance

Standard

Monitoring

Method

Research on user/participant demographics

A report on user/participant demographics

Research and report user/participant demographic

Contractor personnel shall document any data collected in accordance with the task protocol

100% inspection by

COR.

Recruiting of

Task Participants

A schedule of task participants

Recruit and screen task participants and manage scheduling of the HF and usability tasks.

Contractor personnel shall recruit and screen the required number of task participants according to the task protocol provided.

100% inspection by

COR.

Usability Testing

Facilities

On site configured usability lab and audio/video recordings of test sessions

Contractor personnel shall configure the onsite usability lab according to the specifications of the test protocol to include screen capture and audio

Contractor personnel shall record

(audio/video and screen capture) of each usability test session—where a session corresponds to a participant

100% inspection by

COR.

and video capture of each usability test session.

according to the test protocol provided.

Developing task materials and task instrumentation

Prepared task materials and instrumentation in place for the usability sessions

Contractor personnel shall develop task materials and any instrumentation in accordance with the specifications of the task protocol for use in the usability sessions.

Contractor personnel shall provide the task materials and demonstrate that the instrumentation meets the specifications in the task protocol.

100% inspection by

COR.

Developing software to support HF and usability tasks

Developed software code in support of HF and usability tasks

Contractor shall develop software code in accordance with the requirements of task protocol

Contractor personnel shall develop the software code following well-defined and documented SDLC process.

100% bug-free code inspected by

COR.

Conducting or assisting in conducting task protocols

Facilitated usability task protocols

Contractor personnel shall configure any data collection mechanisms or task materials as specified in the task protocol.

Contractor personnel shall document any data collected by the task materials as specified by the task protocol

100% inspection by

COR.

Implementation and proctoring of surveys

A fully-developed survey that is disseminated

Contractor personnel shall develop and disseminate surveys in accordance with the task protocol for data collection.

Contractor personnel shall document how the surveys were designed, implemented, distributed and provide the data collected by the survey instrument.

100% inspection by

COR.

Facilitating interviews

Facilitated and completed interviews

Contractor personnel shall perform interviews in accordance with the task protocol – including taking notes and transcribing audio recordings.

Contractor personnel shall provide the interview approach and data.

100%

Analyzing data Completed data analysis

Contractor personnel shall perform data analysis in accordance with the task protocol

Contractor personnel shall provide the results of the data analysis

100% inspection by

COR.

Compensating task participants

Compensated task participants who have completed

HF and usability tasks

Contractor personnel shall compensate task participants in accordance with the task protocol.

Contractor personnel shall provide a documentation of the participant’s task schedule and their compensation.

100% inspection by

COR.

Project

Management

Successful management of all specific tasks and assurance of timely provision of all acceptable deliverables.

Manage the quality and timeliness of all tasks/deliverables to ensure compliance with the Performance

Work Statement and

Task

Protocol

Contractor personnel shall provide an activity report detailing the progress of work being performed and any pertinent issues.

100%

Individual task orders under this contract will include a Performance Requirements Summary as applicable. The following is a sample Performance Requirements Summary for a task order involving technology evaluation components.

Task Desired Output Required Service Performance Standard Monitoring

Method

Creating topic statements

Developed and refined topics

Create initial topic statements, explore the collection, select the final set of topics to be used in the experiments, and refine the topic statements.

Contractor personnel shall create XX-XX initial topic statements at a rate of X per hour.

From the original set, they will select the final set of topics for experiments and refine them into XX refined topic statements at a rate of X topic(s) per hour.

100% inspection by COR.

Topic relevance assessments

Completed relevance assessments

Create initial topic statements, explore the collection, select the final set of topics to be used in the experiments, and refine the topic statements.

Contractor personnel shall judge documents for X-XX topics at a rate of X topic(s) per hour.

100% inspection by COR.

Homepage relevance assessments

Judged homepages

For each topic created that was selected to be part of the final collection, the

Contractor personnel will be provided a pool of pages which they shall assess as being relevant (or not) homepages for the topic.

Contractor personnel shall judge pools of homepages for each topic. Contractor personnel shall judge pools of homepages at a rate of X topic(s) per hour.

100% inspection by COR.

Creating known- item queries for video search systems

Created known-item queries.

Contractor personnel shall create video queries.

Contractor personnel shall create video queries at a rate of X queries per hour.

100% inspection by COR.

Assessment of videos Judged video clips.

Contractor personnel shall view and assess video clips.

Contractor personnel shall view and assess

XX short video clips per hour on average.

100% inspection by COR.

Descriptions of videos

Short text descriptions.

Contractor personnel shall view video clips and compose a short description of each.

Contractor personnel shall view and describe

XX short video clips per hour on average.

100% inspection by COR.

Entity annotations

Annotations of entities or other linguistic expressions.

Contractor personnel shall read the document and annotate each mention of an entity.

Contractor personnel shall view and annotate

XX documents per hour on average.

100% inspection by COR.

Developing topics for summarization

Develop usable topics and selected chronologically ordered sets of newswire articles for each topic.

Contractor personnel shall develop usable topics and select chronologically ordered sets of newswire articles for each topic.

Shall develop X usable topics at an average rate of X topic(s) per hour.

Contractor personnel shall select chronologically ordered sets of newswire articles for each topic, where each set contains

XX relevant articles.

100% inspection by COR.

Handwritten

Summaries

Hand-written 100-word summaries for each topic that satisfies the information need expressed in the topic statement.

The Contractor personnel shall write

XXX-word summaries for each topic that satisfies the information need expressed in the corresponding topic statement or list of required aspects.

Shall write X XX-word summaries for each topic that satisfies the information need expressed in the topic statement or list of required aspects.

Contractor personnel shall write summaries at an average rate of X topic(s) per hour.

100% inspection by COR.

Assessment of the quality of automatic and manual summaries

Assessment of the quality of automatic and manual summaries.

The Contractor personnel shall assess the quality of the automatic and manual summaries.

Shall assess the quality of the automatic and manual summaries at a rate of XX topic(s) per hour.

100% inspection by COR.

Review of

Annotations

Review of the quality of annotations of entities or other linguistic items.

The Contractor personnel shall review the annotations and correct any errors found.

Contractor personnel shall review annotations in documents at the rate of

XX documents per hour.

100% inspection by COR.

VI. TRAVEL

Travel will be reimbursed under each individual call order in accordance with Federal Travel

Regulation. The Contractor shall not exceed funding for travel as specified in each individual call order.

VII. CONTRACTOR-GIVERNMENT INTERACTION

All correspondence and oversight shall be provided by the Government through the Contractor’s designated individual. The Government will not provide direct or indirect supervision to any

Contractor personnel.

VIII. PAYMENT SCHEDULE

The Contractor will be paid, in accordance with the payments clause in the contract and as otherwise noted in the task order, upon receipt of a proper invoice.

IX. RISK ASSESSMENT

The risk assessment for individual task orders under this contract ranges from None to IT-

Moderate.