Attachment_2_Business_Requirements.pdf

Attachment 2: Business Requirements

Executive Summary

Included in this document are the business requirements for a data model and relevant metadata for an adaptable National Agricultural Statistical Service, NASS-wide data repository and access system.

Overarching requirements regarding the data repository:

1. The data repository will have both public-facing and internal-facing components. The public-facing component will contain:

• All publicly available observations

• Statistical measures or supplementary information (currently included in NASS reports)

• Revised data

• Metadata (further explained in Appendix section) The internal-facing component will allow secure loading of release data prior to its scheduled release. At the release date and time, as published in the ASB calendar, data will be made accessible to the public.

2. The data repository must be able to:

• Control access to pre-release information

• Load large data sets quickly

• Make simple calculations

• Return query results quickly

• Allow users to replicate the tables currently appearing in NASS published reports

• Handle new data sets with minimal effort

3. The data model and the metadata used to describe each observation are central to this repository. Implementing a proposed model and metadata schema may require significant resources and a different way of thinking about and accessing NASS data. Despite the high investment, the proposed model opens up many more possibilities in terms of data retrieval and product development. The desired data model should allow access to all data as currently published in NASS scheduled releases in addition to new capabilities such as custom queries and tables, the ability to combine data from separate releases, data visualizations and other data products useful to NASS data users. Users should be able to discover data that share commonalities, such as discovering different measures that relate to a specific commodity or geographic area.

I. Technical Requirements

The technical requirements that must be met to ensure a user-friendly and efficient customer experience while delivering system and data security are identified below. The business requirements that follow must satisfy the overarching technical requirements found in this section of the report.

Security Requirements

Security of the data prior to official release is of paramount importance. The new system will be housed in accordance with all NASS security mandates and external requirements such as Federal Information Security Management Act (FISMA), Confidential Information Protection and Statistical Efficiency Act (CIPSEA), NASS Agriculture Statistics Board procedures, etc.

The proposed system will have both an external-facing system to meet the needs of public data users and an internal-facing system for use by authorized NASS staff only in order to allow NASS to load data prior to scheduled releases. This section of technical requirements applies to the internal facing system.

Data will be loaded internally to the system prior to the public ASB scheduled release of the report containing the data. At the scheduled ASB release date and time (but not before), the data will be made available to the public.

The system will only allow access to the internal repository to authorized NASS users.

Access to data will be defined by business rules.

The system will be easy to access for authorized NASS staff: signing onto the NASS network will provide access to authorized data.

The system must accommodate and store an appropriate number of backups, logs, and audit trails. The implementation team will determine the appropriate number and mix.

General operating requirements

The following requirements apply to both internal and external systems.

The system will be well documented and easy to maintain.

The system will meet Section 508 compliance and accessibility requirements.

The system will operate with minimal planned downtime with targets of 99.5 percent uptime 24 hours per day and 6 days per week for internal users and 99.5 percent uptime 24 hours per day and 7 days per week for external users (consistent with current downtime standards for NASS applications).

The system will have response times equal to or faster than the systems it is replacing.

For searches of the data repository, the response time should be 5 seconds or less. For the data extract, the response time will vary based on the quantity of data selected.

The system will collect and store metrics to assess user demand to allow for identification of most frequently accessed data or web pages.

II. Business Requirements

Data Model and Metadata

Data Model and Metadata

The system must include an appropriate set of metadata to inform customers of important characteristics of the data they are selecting. Data should be organized in the interface for users in a way that makes sense to them, which may differ for different types of users, and may or may not reflect the current organizational structure of Quick Stats.

The system must implement a logical data model or structure, including a consistent coding structure for key identifiers that connect data across reports, efficiently accommodates all reports’ published data, metadata, and includes any agreed upon set of supplemental information.

Metadata and the data model should be sufficient to allow unique identification of each NASS published statistic.

Data model and metadata requirements

REQUIREMENT 1. DATA MODEL

The logical data model structure for storing and organizing data within the NASS repository will be based on common definitional elements found in all NASS data.

An evaluation of the suitability of the current NASS Quick Stats model for an improved data user interface will be required. Changes or additions to this model can be proposed if appropriate or an alternative model required for an improved interface may be proposed.

FIGURE 5 EXAMPLE OF THE EXISTING DATA MODEL

REQUIREMENT 2. MINIMUM SET OF HIGH-LEVEL METADATA CATEGORIES

A list of the required high level metadata needed should be provided, along with justification for the proposed categories. An evaluation of whether the existing metadata in Quick Stats is sufficient or what modifications would be needed is required.

REQUIREMENT 3. MINIMUM METADATA

Requirement 2 must be implemented in such a way that each data value is uniquely identified.

Examples of how these would be used to uniquely identify related data items should be provided.

The repository must accommodate additional metadata as appropriate for any given dataset. If additional metadata not currently existing in Quick Stats is required, this should be described.

REQUIREMENT 4. METADATA CLASSIFICATION

Requirements for a distinct metadata classification systems for the repository for NASS data should be proposed.

Query tool

Query deliverable

Develop and implement one or more high-level approaches that will enable NASS staff and the public to:

• Understand what data are available;

• Zero in on and select the data of most interest to them; and

• Retrieve these data with accompanying metadata and statistical measures.

Both NASS staff and the public should have the ability to query multiple dimensions of data (e.g., across programs, time series) and save their queries for use at a later time.

The query tool user should be able to understand the differences when multiple results appear to be similar or identical. Recommendations for COTS solutions to provide a user interface should be provided if suitable.

Query tools should operate rapidly enough to provide a reasonable user experience.

Query introduction

This requirement outlines the basic functionality of a modern NASS data query tool.

At a high level, the query functionality must be designed to:

• Allow users to easily find data relevant to their interests

• Present data most likely to be relevant most prominently (for example, most recent, popular searches, sorted by size, highest level of aggregation if not specified, etc)

• Allow users to filter results based on the information being queried (e.g., commodity, State and/or characteristics of the information such as geography, reference period, date of release, data type) to obtain a manageable number of results

• Allow for the addition or modification of any data model elements. For example, if a new geographic area or way of classifying information is added, the query must reflect those

• Meet Section 508 standards

• Allow users to query data in the repository by:

1. Filtering

2. Free-form text searches

3. Entering publication/release title (unique identifier)

4. Repeating previous queries

Query results requirements

REQUIREMENT 5. QUERY RESULTS

Once a user has queried the repository, the results will:

• Display meaningful labels and descriptions of the data with enough information to differentiate between similar items.

• Display in default sort order based on machine learning algorithms (derived from previous searches and associated selections, popularity, etc.) and specified rules (such as most recent, in rank order sorts, state alphabetic sorts, etc.) to optimize users finding the appropriate data in the top most results.

• Allow the user to view information about the data, including a plain language description of the measure and a description of the group of people, places, and/or things the measure applies to. This information could be viewed via hover over, expanding features, or similar Section 508-compliant functionality.

• Allow users to sort results. The query results will use a default sort as described above, but give users the option to sort and filter by other features such as data values, date of last update, series/program, state, etc.

• Allow users to query the initial results using the query functionality described above.

Users will be able to continue to refine their query results in an iterative manner until they have found the data of interest.

o For example, a user could search for “goats” and get thousands of results. They then could use the filters to refine the results to only those related to geography.

They then could search those results for those that show only inventory or sales to get the most relevant results.

Assembly and Delivery

Assembly and delivery deliverables

Develop and implement standard layouts NASS publishes (e.g., Crop report, Hogs report, and data visualizations.) These should include developing approaches by which customers can themselves retrieve the data that reports currently create.

Implement the initial formats of products (e.g., Excel, CSV). Note these formats are likely to change over time so that any solution must be flexible enough to incorporate new product formats as the IT environment and public demand changes.

Assembly and delivery introduction

The assembly engine takes the results of queries (discussed in the previous section) and gives them shapes, such as tables and visualizations. It also allows the users multiple options for presenting the data. Ideally, we want customers to explore the data and find the type of outputs that satisfy them. The ability of the assembly engine to offer a broad variety of useful options is crucial if we are to meet the sponsors’ goal of reducing the number of NASS-produced tables and visualizations.

The delivery engine takes the results of the assembly engine. It allows the users to download the visualization or table and to share on social media. Visualizations and tables will have different delivery options. There will be a 508-compliant output available from the system. Standards evolve over time, and the system needs flexibility to accommodate these changes.

Default(s) for Assembly

Options for the type of data outputs should be provided, but should include tables, maps, and a variety of charts (for example, time series line charts, bar charts, etc.) Outputs should be interactive allowing users to change data displayed in a dynamic manner.

For example, a default line chart should be interactive. Specifically, it should be presented on a screen that offers the user a widget or scroll bars for changing the beginning and ending periods.

This will make it easier for users to find and focus on the periods that interest them. For example, Federal Reserve Economic Data (FRED) offers three ways of changing the time period illustrated, as exhibit 2 below illustrates.

EXHIBIT 1. FRED EXAMPLE

Users can use a widget to select the last 1, 5, or 10 years of the series (blue oval on the left). Or they can enter specific dates (blue oval on the right). Or they can use sliders (indicated by the blue arrows). On zoom, the data values displayed should be official published statistics.

The requirement is not to copy the FRED interface. It is instead that the new NASS user interface offers tools or scroll bars to adjust features such as time periods. It might also be desirable to offer at least one other direct way to specify the start and end periods. If the time series have different start and end dates, the chart should show the earliest start date and the latest end date.

Other features could include allowing multiple series on one chart; by scrolling over the chart, users could be able to see data values; and by clicking on items in the legend users could change the data that are displayed. See exhibit 3.

EXHIBIT 2. INTERACTIVE CHART MAKER

REQUIREMENT 6. FEATURES OF THE DEFAULT MAP

The default map assembly can be for states, counties, regions, or the US, depending on the data.

Default maps should have the interactive features such as by hovering over an area the user should be able to see the data values. See Exhibit 4.

EXHIBIT 3 MAP EXAMPLE

REQUIREMENT 7: FEATURES OF THE DEFAULT TABLE

The default table for a non-time series will have a title, row labels, and one or more columns of numeric data. This would look like the tables in a typical NASS release. Row labels will be descriptive, written in plain language. Users would be able to sort the data in a variety of ways (e.g., A-Z, aggregation level, highest to lowest).

REQUIREMENT 8. CUSTOM TABLE GENERATOR

A tool will exist for internal and external users to generate tables by querying metadata. The tool should be simple enough for expert and novice users to navigate.

REQUIREMENT 9. TABLE DEFINITION STORAGE

Table definitions must be able to be stored for reuse.

Options for Assembly and Delivery

The default assemblies should each have one link that takes users to a page where they can edit the visualization or table.

REQUIREMENT 10. ADDITIONAL FEATURES FOR CHARTS

For charts, the edit options will allow the ability to select from several alternative types of charts.

Below is a minimal list of alternatives:

• Area

• Columns/bars (including stacked versions)

• Scatter

• Bubble

• Columns with ranges

• Box plots

• Maps (for data available by state, county, or metropolitan areas)

Users should also be able to chart derived series—that is, series of data derived from the query results.

Users should be able to create dynamic charts, with a sliding widget that allows the user to see how a visualization changes over time. Google’s public data explorer, for example, creates dynamic bubble charts, and the user can use a widget to show how the bubbles move over time.

(See exhibit 5.)

EXHIBIT 4 GOOGLE DATA EXPLORER

REQUIREMENT 11. DELIVERY OPTIONS FOR VISUALIZATIONS

For visualizations, users will be have three delivery options:

• Download the image itself in one of several formats (jpg, png, ppt, or pdf and possibly others).

• Download the data and shapefiles for the chart (along with selected metadata) in one of several formats (xlsx, HTML, XML, and possibly others for the data).

• Share image (or a link to an image on www.NASS.gov) on social media (LinkedIn, Facebook, Twitter).

If the user selects a data download, the download should include at a minimum: series titles;

time periods; some characteristics of the measure (such as units, base periods for indexes, etc.);

and a web address for the data. Users will be able to save the formatting choices as part of a saved query (discussed in the query requirements). More specifically, if a user wants a line chart with particular series for a particular period and wants to download that image in a particular format, the user would be able to save those choices with the query.

REQUIREMENT 12. EDIT OPTIONS FOR TABLES

The edit options will allow users to switch rows and columns (in other words, to transpose the table). Users will also be able to add and remove columns and rows. Users will be able to add columns of derived data (such as a confidence interval). In addition, users will be able to add more columns in several ways: such as selecting an additional time period, a different geography, or a different population group. (This step may require querying the database again or re-fetching the data.)

The options for tables will allow users to create visualizations, other than line charts, for cross-sectional data. In particular, for data with geographic cross sections, maps will be an option.

Columns and bar charts should also be available.

REQUIREMENT 13. DELIVERY OPTIONS FOR TABLES

Users will have two delivery options for tables:

• Download the data for the table along with selected metadata in one combined file. This would be in xlsx, HTML, XML, at least one print-friendly format, and other formats. In this case, a limited amount of metadata would be incorporated in titles, subtitles, or footnotes of the table.

• Download the data and metadata in separate files, possibly with different formats (csv should be required, json, rdf and others should be considered and will be determined in the development phase, as well as the formats listed above). This would allow users to access all the metadata associated with series, which will facilitate the importing of NASS data into more sophisticated packages for data analysis and visualization.

However, the formats for exporting data should be such that they can be read and used by a variety of users.

As is the case with visualizations, users will be able to save formatting options for delivery.

Any additional features beyond those already described should be described.