Request_for_Information_-_Sources_Sought.pdf

Request for Information/Sources Sought USDA/National Agricultural Statistics Service

User Interface, Data Model and Metadata Standardization Improved NASS Data Repository and User Interface Design (DRUID)

1.0 Description

This Request for Information (RFI) seeks qualified contractors that can provide development and implementation support for a modern adaptable enterprise-wide data repository, data user access system/tools and data release system for the US Department of Agriculture, National Agricultural Statistics Service (NASS). Sources Sought Notice and Request for Information (RFI) No 1232SA19R0011. The proposed acquisition classified under the North American Industry Classification System (NAICS) Code is 541511 Custom Computer Programming Services (Size Standard of $27.5 Million). This requirement could also fall under NAICS 51820

– Data Processing, Hosting, and Related Services (Size Standard of $32.5 Million).

NASS is seeking information on how vendors will implement improved access to NASS’s published data, serving its diverse set of data users. NASS needs improved methods to release and provide access to the data currently published in its reports and in its publicly available online database. This includes a diverse set of data users who will have diverse requirements in how they obtain NASS data, what data is important to them and the format in which they require data. Many NASS reports are designated Principal Federal Economic Indicators of the United States, so pre-release security combined with almost-instant release timing are primary considerations. Solutions may include commercial off the shelf (COTS) solutions applied to new or existing NASS databases, or custom development if COTS solutions do not fulfill all requirements.

DISCLAIMER:

This Request for Information/Sources Sought Notice issued solely for information and planning purposes and does not constitute a solicitation. All information received in response to this Request for Information/Sources Sought that is marked as proprietary will be handled accordingly. Responses to this notice are not offers and cannot be accepted by the Government to form a binding contract. Responders are solely responsible for all expenses associated with responding to this RFI/Sources Sought Notice.

2.0 Background Information:

The National Agricultural Statistics Service (NASS) of the U.S. Department of Agriculture conducts hundreds of survey data collections every year which are used to produce official statistics on U.S. agriculture. These statistics are published in over 400 reports annually covering virtually every aspect of U.S. agriculture, according to the published Agricultural Statistics Board calendar. NASS’s mission is to provide timely, accurate, and useful statistics in service to U.S. agriculture. As an independent statistical agency, NASS serves its diverse user communities by providing its official statistics and services that are accurate, objective, relevant, timely, and accessible. To uphold this mission, NASS:

Reports the facts on American agriculture, facts needed by people working in and depending upon U.S. agriculture.

Provides objective and unbiased agricultural statistics on a preannounced schedule that are fair and impartial to all market participants.

Conducts the Census of Agriculture every five years, providing the only source of consistent, comparable, and detailed agricultural data for every county in America.

Serves the needs of data users and customers at a local level through a network of Regional Field Offices and cooperative relationships with Universities and State Departments of Agriculture.

Safeguards the privacy of farmers, ranchers, and other data providers, with a guarantee that confidentiality of individual reports and data security are top priorities at NASS.

All reports and data are prepared under secure conditions and then released publically on the NASS website (https://www.nass.usda.gov/) according to the date and time previously published in the Agricultural Statistics Board (ASB) calendar (https://www.nass.usda.gov/Publications/Calendar/reports_by_date.php). Data for release in the reports is prepared in typically short time frames (often less than 24 hours). Releases typically consist of a report in PDF, text (TXT), and comma separated value (CSV) formats, including graphic, narrative and (primarily) tables of statistics.

FIGURE 1 EXAMPLE OF A TABLE FROM THE PDF DAIRY PRODUCTS REPORT

The statistics (but not the narrative or any explanatory footnotes) in these reports are also added to the online Quick Stats database (https://quickstats.nass.usda.gov/). NASS’s products are described in the Guide to Products and Services (https://www.nass.usda.gov/Publications/catalog.pdf). Charts and maps published in some reports are also available separately on the NASS website in formats such as PNG, GIF and PDF.

(See for example, https://www.nass.usda.gov/Charts_and_Maps/index.php)

NASS operates under two major program areas: the annual Agricultural Estimates Program and the quinquennial Census of Agriculture Program. Our Agricultural Estimates Program primarily includes estimates of acreage, inventory, production, price, and stocks based on data collected in sample surveys with weekly, monthly, quarterly or annual reference periods, primarily with estimates typically published at both the state (differing by program) and national level. It also includes some county-level estimates as well as forward-looking commodity forecasts. Data for these programs is typically gathered through surveys, then compiled and released in short time frames, typically days or hours. Census data, released every five years, represent a complete enumeration of every farm and ranch, or agricultural producer, in the U.S. and is our most complete source of information on minor commodities, farm demographic and economic characteristics, as well as many other facets of agriculture. Statistics for the census reflect reference years ending in 2 and 7. Census information is published at the county, state, and U.S.

level as well as by watershed, zip code and congressional district. See https://www.nass.usda.gov/Publications/AgCensus/2012/index.php. Data preparation for the Census releases is typically somewhat longer than typical agricultural estimate program releases.

Reports from the two program areas are mostly prepared and released independently of each other and contain mostly different types of data. While these 2 program areas operate separately internally at NASS, data users should not be required to understand this to access the data they need.

The Quick Stats online database contains data from nearly all our current reports as well as some data series beginning as early as 1850. It will soon contain over 42 million records. The current Quick Stats database provides no easy way to understand what data it includes or easy access to the individual statistics desired. NASS is seeking an easier way to provide data to users in a simplified user interface.

Some advanced data users currently download Quick Stats datasets to access detailed NASS data available through an API. However, despite holding nearly all of NASS’s published statistics, the Quick Stats database is difficult to use and unintuitive to many other data users.

Current data users can query the NASS Quick Stats database to access the published data from both the census and survey based programs (https://Quick Stats.nass.usda.gov/). Data from Quick Stats is also used to generate a number of “canned queries” (https://www.nass.usda.gov/Data_and_Statistics/Pre-Defined_Queries/index.php) of commonly requested data. In addition, there are a number of charts, maps and tables available on numerous other pages within the NASS website. On the “Data and Statistics” page of the NASS website (https://www.nass.usda.gov/Data_and_Statistics/index.php), users can choose to go to Quick Stats to retrieve custom datasets, view (unspecified) county maps, explore statistics by subject, explore the COA, access pre-defined queries, and more. Many of these links can be used to access specific data in different formats and may meet the needs of some data users (although they are difficult to follow and do not provide immediate access to relevant data). However, data users visiting the NASS website are not presented with relevant data easily or quickly. On the “Data and Statistics” opening page of the NASS website, NO data is presented. At a minimum, a data user must make 5 clicks before any data is presented. Items are also not sorted or filtered to present more relevant or commonly requested items for the data user. It is also unclear to the typical data user what data are available, where within the NASS website the data can be found, how to find the data they desire, what data will be returned from a query selection, and the best way to access data.

The structure of the information on the NASS website (overall and within Quick Stats) is not intuitive to most data users and the existing taxonomy does not follow a standard organizational structure or most data users’ conceptual understanding of NASS data. Category labels often appear inconsistent, and some items included within broader categories seem counterintuitive or illogical. For example, Quick Stats users must know whether the source of the data they are retrieving is the census or a survey based program. Further, they must negotiate a hierarchy of Sector/Group/Commodity/Location and Time to query the data. In addition, the categories users select from within this hierarchy are often confusing and unintuitive. For example, a data user wishing to find statistics on dairy products is confronted with selecting the Sector “animals and products” followed by a choice to next select from a group including animal totals, dairy, livestock and specialty. Following this selection, the user will still be required to make several more selections before retrieving data. See example below.

FIGURE 2. EXAMPLE OF QUERYING QUICK STATS FOR DAIRY STATISTICS

Alternatively, users can use a keyword search, but the results of most searches return an extremely broad set of records, often in the thousands (see figure 2 for an example). While a wide variety of data are available, it is unclear to data users which of these results is most relevant and filtering these results is difficult. Further, once data are accessed, particularly in Quick Stats, it is often in a format that is difficult to interpret (i.e. cryptic labels), difficult to use (i.e. many items returned in a single table, columns of data unnecessary or empty), and requires expertise to manipulate (i.e. data requiring sub-setting or pivoting). It is also difficult to know if data returned in queries is the same as data included in specific reports. In addition, Quick Stats is limited in ways to graphically display or interactively explore the data.

FIGURE 3 EXAMPLE OF USING THE KEYWORD SEARCH FOR "DAIRY"

FIGURE 4 EXAMPLE OF OUTPUT FROM PREVIOUS KEYWORD (DAIRY) QUERY

In addition, the Quick Stats database does not include supplementary information such as narrative, footnotes or explanatory material included in PDF reports. NASS desires to improve the data user experience, making access to NASS’s published data more intuitive, easier to access and understand, and available in formats suitable to each data users’ needs. The current Quick Stats data model and metadata are provided in the appendix.

While Quick Stats presents a very rich trove of data, it also increases the probability that finding a particular data point can be cumbersome. In fact, in order to accurately interpret this information it can be necessary for a data user to understand NASS’ organizational structure and/or individual survey programs in order to find or accurately distinguish between multiple data points. The potential exists to misinterpret data, so many data users find they must reference pdf version of reports to verify data returned from a Quick Stats query.

NASS has a number of different types of data users who should be considered in redesign of data access tools. These include: agricultural policy makers (e.g. USDA policy staff, industry associations, State Departments of Agriculture, Federal, state and local governments); power users/academia; media; agricultural producers (including data providers); and the general public.

Data should be available and presented in a way that is easy for all data user groups to understand and digest. Users should be able to find the data they require, discover what data is available in an intuitive way and be confident they have obtained the information they have sought. They should also be able to understand what data is available and retrieve it in the format that is most useful to them. Data access should also include engaging, interactive data visualizations in addition to data tables or static documents. Data users should also be able to download data easily from the NASS website in their desired format or through an API. Further, the proposed system should include a standard method to generate and release a “report” including the data tables and information in current NASS PDF reports for release according to the ASB calendar.

This RFI is seeking information on how vendors would evaluate and understand data user needs and provide recommendations for how to improve data user access to the NASS published statistics. Information is sought on how vendors would determine if replacement or modifications to the existing database should be made as well as determining the features that should be included in a user interface and how it would be designed. In addition, information is sought on how best to implement proposed solutions.

The development and implementation of tasks outlined in Section 4.2 below will be performed with Government owned, leased or licensed resources.

3.0 Instructions for Contractors

Please review the requirements and relevant attachments and respond if you are an interested party.

Include both a capabilities statement and complete all sections of the Market Research Questionnaire in Attachment 1. Your response should not exceed 30 pages. All contractors responding to this announcement must demonstrate their experience in the tasks outlined in Section

4.2 below. Responses must be specific to each of the listed tasks to demonstrate that the respondent has the requisite skills and experience. Where possible, respondents are asked to include proposed technology they would leverage to complete these tasks and the timeframe necessary for completion. Respondents should include a high-level work plan outlining a proposed phased approach. Pertinent factors for consideration of qualifications are the primary contractor’s demonstrated experience with the following:

Data model and metadata standardization Building large scale data repositories Developing systems for querying large data repositories (developing user interface) Data migration activities

Both large and small businesses are encouraged to participate in this market research. Joint ventures or teaming arrangements are encouraged and are up to each industry partner. Multiple teaming arrangements are acceptable. Please express your intentions in the questionnaire and capabilities package. If applicable, please specify your small business classification in any response.

If, you desire to participate in this market research request for information, you should reply via email with the corresponding questionnaire responses and your capabilities statement to Jaki.McCarthy@nass.usda.gov no later than 2:00 pm May 16, 2019. Proprietary information should be clearly marked. All submissions shall be submitted via e-mail. The naming convention for submission should include the company name and NASS DRUID RFI, For example –

ABCCompany – NASS DRUID RFI.docx. The email subject line shall be “Company Name DRUID RFI” The electronic format should be in PDF or other common document format.

Questions about this requirement should be submitted no later than close of business on the tenth day after this RFI posting to the email listed above. The email subject line shall be “Question:

Company Name DRUID RFI.”

4.0 General Requirements and Task Summary of Requirement (Subject to Change):

4.1 General Requirements:

An enterprise-wide data repository and access system that is used to release data to the public according to the ASB calendar and houses all NASS public data allowing NASS staff and the public ease of access.

Statement of Objective (SOO):

All NASS published data are available in one data repository that supports both internal NASS program processes as well as outside data users’ needs. The system must be flexible, user friendly, store data and manage data in compliance with Confidential Information Protection and Statistical Efficiency Act (CIPSEA) and National Institute of Standards and Technology (NIST) Special Publication (SP) 800-171 or NIST SP 800-53.

4.2 Tasks:

Detailed business requirements for each of the following tasks are included in Attachment 2.

4.2.1 Data Model and Metadata Standardization

The current data model for the Quick Stats database is included in Attachment 3. This RFI includes a request for a description of an evaluation of the suitability of this model and metadata needed. In addition, recommendations are requested for methods to determine if replacement or any needed modifications to this model or metadata are necessary to improve the data user interface and data access. The new user interface to NASS data should be intuitive and serve all NASS users (i.e. advanced data users as well as casual or first time NASS data users).

Your response to this RFI should provide information on your experience and proposed approach to developing a data model and metadata standards to improve the access to NASS data for all types of data users. This model should support new methods of displaying, searching, compiling, and disseminating NASS published data.

4.2.2 Data Repository

Your response to this RFI should also describe your development approach for a data repository to house all NASS public data and relevant metadata. The repository will serve as the official source of NASS data. Vendors may propose modifications or alternatives to the current NASS Quick Stats database.

The repository needs to be flexible to accommodate new data, and corresponding metadata.

Further, the repository needs to accommodate potential revisions to existing data. Specifically, the repository must be able to hold, and identify, original versions of data as well as the updated versions. Users should also be able to access supplementary information currently found in published reports along with statistics. A Federal Risk and Authorization Management Program (FEDRamp) compliant cloud solution is acceptable.

Your response to this RFI should provide information on your qualifications and experience developing similar data repositories.

4.2.3 User Interface

All of NASS’s major types of data users should be accommodated by the user interface. This should include advanced users who wish to download data for further analysis, users who wish to see current data when released, members of the media or commodity associations representing specific agricultural sectors, and casual users who may want information specific to a set of commodities, their local area or other specific criteria. Ideally, NASS data providers should also be able to retrieve data relevant to the estimates to which they have contributed and other NASS data relevant to their operations. The improved user interface should display relevant data to users, match their conceptual understanding of the data, and reduce the number of clicks or manual intervention needed to obtain the data they need.

Your response to this RFI should describe your experience with user centered design and your approach to developing a user interface that includes user-friendly query tools, data display and visualizations, and innovative methods to access NASS data. The user interface will help data users understand what data are available in the database, zero in on and select the data of most interest, and retrieve selected data and accompanying metadata and statistical measures. Your response should also describe how the information currently contained in scheduled reports would be released in the redesigned system.

Discuss how you would accommodate different user groups in the interface and how user input will be obtained or usability testing could be conducted for these user groups. Please describe how ASB scheduled releases would be incorporated into the user interface. Please also discuss the suitability of COTS products to provide the user interface.

4.2.4 Data Migration

Contractor support would include database redesign or proposal of necessary modifications and implementation of any redesign.

Your response to this RFI should also describe your experience and approach to migrating from existing databases to new or redesigned databases (and their associated user interface).