Copy of Final Data Catalog Requirements.pdf

# Functional Area Requirement

1 Access Management Establishes and assigns roles, organization relationships and user access privileges for functions related to data and analytics governance.

2 Access Management Supports capabilities such as user authentication, password policy management, two-factor authorization, single sign on and role-based access.

3 Business Glossary Supports development and usage of a glossary to enable policy analysis and development. Supports taxonomies and ontologies to address semantic variations.

4 Business Glossary

Ability to document standardized business rules for a business system (e.g. REALD - SOGI has identified

6,500 living lanugages which must always be presented as an option). The business glossary should track the standardized list and also identify any systems or data sources which are non-compliant.

5 Business Glossary Ability to reference statutes, rules, and implications.

6 Classification Organizes data by relevant categories so that it may be used and protected more efficiently. Enables data to be more easily located and retrieved. Enhances risk management, compliance and data security.

7 Classification

Supports user tagging of content or automatic detection. Provides context (such as tagging and rating) to enable data analysts, data scientists, data stewards and other data consumers to identify and integrate access to additional relevant datasets.

8 Data Catalog Need to identify who the registration authority is who can has the final authority and decision-making responsibility for the data asset.

9 Data Catalog Ability to review, manage, and maintain the completeness of the data.

10 Data Catalog Ability to review and monitor the accuracy, consistency and relevancy of data.

11 Data Catalog Ability to create, edit, and manage entries in a centralized data catalog.

12 Data Catalog Ability to "easily" update information for each data asset.

13 Data Catalog Ability to track versions between updates, including who updated what and when.

14 Data Catalog Ability to track the entire worfklow status of any change to any record in the data catalog.

15 Data Catalog Ability to log connections between the catalog and why the business is using the data, and what the business is using the data for.

16 Data Catalog Ability to define custom roles (e.g. Data Custodian), the responsibilities for said role, and link these roles to each data asset.

17 Data Catalog Ability for data catalog to be accessed by non-technical users to review and update information.

18 Data Catalog Ability to define and document different types/levels of data assets in a hierarchy (e.g. database, data table, fields, etc.) and require or limit the level of definition by unique departmental needs.

19 Data Catalog Ability to perform a keyword search across all fields in the data catalog, including technical and business labels.

20 Data Catalog Ability for end-users to add comments, reviews, and other information regarding the data asset.

21 Data Catalog

Inventories and curates data assets. Leverages machine learning and automatic detection of relationships with other data assets. Requires a user-augmented process for validating and resolving any ambiguity in the automated inventory process.

22 Data Dictionary Ability to reconcile data definitions.

23 Data Dictionary Ability to track changes to data dictionary entries and metadata with audit trails and approvals.

24 Data Dictionary Ability to export data dictionary entries in various formats for documentation and sharing.

25 Data Dictionary Ability to easily import data dictionary entries in various formats for documentation and sharing.

26 Data Dictionary Ability to attach reference manuals to the data dictionary which provide additional organizational background.

27 Data Dictionary Ability to use equations within the dictionary.

28 Data Dictionary Ability to distill to plain language for non-technical users.

29 Data Dictionary Ability to capture and understand business justification for data for data fields, tables, schemas, etc.

30 Data Dictionary Ability to tag data elements within a data asset as being considered PHI and PII or other protected data asset classes.

OREGON HEALTH AUTHORITY

Data Cataloging Business Requirements These requirements were gathered as part of an information gathreing intiative to both better understand programmatic needs in the area of data cataloging and to support the development of data catalog persona. These are in no way intended to be comprehensive or final. Their inclusion with this RFI is for informational context purpose only.

OREGON HEALTH AUTHORITY

Data Cataloging Business Requirements These requirements were gathered as part of an information gathreing intiative to both better understand programmatic needs in the area of data cataloging and to support the development of data catalog persona. These are in no way intended to be comprehensive or final. Their inclusion with this RFI is for informational context purpose only.

31 Data Dictionary

Capturing the context of the data, when it was collected (if available based on the data source and transaction volumes), what were the conditions under which it was collected. i.e. what options were available to select from for gender, when data was collected.

32 Data Dictionary Ability to update the documentation automatically by reading the data assets directly from source systems

(and potentially track the edits).

33 Data Dictionary Ability to define and see related data assets to each data asset.

34 Data Dictionary

Ability to segregate PHI & PII data through multiple lenses for regulations and account for the regulation that the existance of a protected element in a data asset may elevate the entire data asset to a greater security classification.

35 Data Dictionary Ability to connect through APIs (open & secure).

36 Data Dictionary Ability to tie to application inventory for all data assets and data lineage to identify that any data asset was sourced from a specific source system in the application inventory.

37 Data Dictionary

Enables collecting names, definitions and attributes about data elements being used or captured in a database, information system or part of a research project. Describes the meanings and purposes of data elements within the context of a project.

38 Data Governance Integration of DQ into other modules (DI, Catalog, etc.).

39 Data Governance Ability to sync with data tables (e.g., SQL).

40 Data Governance Ability to easily share data/models across the enterprise.

41 Data Governance Collaboration features allowing data stewards to contribute, validate, and maintain business terms.

42 Data Governance Protecting sensitive and personal data (PHI/PII) through privacy policies, encryption, access controls, and compliance.

43 Data Governance Defining who can access what data and under what circumstances including user roles, permissions, and regular auditing of data access.

44 Data Governance Ensuring compliance with industry-specific regulations and standards and generating reports as required to demonstrate compliance.

45 Data Governance The ability to provide an audit trail of all interactions with the data, including who accessed it, when, and how they interacted with it ‘data transparency’.

46 Data Governance Data Stewards and responsibility matrix around data 'ownership'.

47 Data Governance Automated system to examine table/column changes and update Data Dictionary accordingly

(automatically or via prompt).

48 Data Governance Early indicator for changes to data pipelines in order to to mitigate downstream impacts.

49 Data Governance Dashboard or other ability to monitor connections.

50 Data Governance Change management - changes to data systems through subscriptions with notifications (potentially ahead of time) to downstream applications and systems.

51 Data Governance Managing data ownership/stewardship from external third party (i.e., tribal, etc.).

52 Data Governance Ability to apply input from / supply access to to community organizations.

53 Data Governance A dashboard for third parties to see the data catalog information.

54 Data Governance Ability to use open data standards (non-proprietary).

55 Data Governance Ability to track user agreements for data shared with external parties.

56 Data Governance Ability to track the responsible party for administering the data sharing.

57 Data Lineage Ability to link data dictionary entries to data lineage to provide context for attributes' origins.

58 Data Lineage Cyclical exports to third party tools, and open data science languages (i.e., Data tool -> Excel -> Data tool).

59 Data Lineage

Supports the depth and breadth of data lineage for identifying data provenance. Audits all the steps, applications and transformations that any data element has gone through from its original source to all the possible endpoints.

60 Data Lineage Ability to trace data lineage inside and outside of the audit system

61 Data Lineage

For data assets where the data lineage is unknown (e.g. receive external data source that may have been previously transformed), show a "black box" to indicate the lineage is only known beginning upon a specific point.

62 Data Lineage Ability to trace data lineage across modules.

63 Data Lineage Visual representation of data lineage.

64 Data Lineage Ability to see downstream impacts of data changes.

Data Cataloging Business Requirements These requirements were gathered as part of an information gathreing intiative to both better understand programmatic needs in the area of data cataloging and to support the development of data catalog persona. These are in no way intended to be comprehensive or final. Their inclusion with this RFI is for informational context purpose only.

65 Data Lineage Notification system (Early indicator) (in plain language, agreed upon data language).

66 Data Lineage Audit log of notifications to track changes.

67 Data Lineage Linking data quality metrics and alerts to the data lineage for better understanding of quality issues.

68 Data Lineage Documenting data transformations applied at each stage of the data lineage.

69 Data Lineage Clear picture of source systems and related data.

70 Data Lineage Ability to triangulate between data, warehouses, catalogues, and reports / dashboards.

71 Data Lineage Ability to trace lineage to downstream reporting.

72 Data Lineage Ability to understand schematic layout and load data in a programmatic fashion.

73 Data Lineage Ability to see value changes at an individual level.

74 Data Lineage Ability to see who made the change to the information.

75 Data Lineage Can be easily visualized to show how data can be used.

76 Data Lineage Keyword search for data types, with lineage.

77 Data Lineage Ability to automatically scan and identify in what reports a given data asset is used from multiple reporting tools (SSRS, Power BI, Tableau, etc.).

78 Data Lineage Ability to quickly and easily review the timliness and refresh frequency of each step of the data pipeline.

79 Data Lineage Identifies the impact of a change on any metadata element. Allows for drilling down or analyzing to the finest level of detail, such as column-level or transformation logic.

80 Data Management Ability to maintain and audit master data and reference data assets for authorized users.

81 Data Management Ability to define an ETL fault tolerance.

82 Data Management List of data owners with a subsequent list of fields, tables, schemas per owner.

83 Data Management Automated updates when non-owners request a change to data.

84 Data Management Ability to capture customer feedback on data dictionary elements.

85 Data Management Data Equity and Data Justice - Data ownership/Data Stewardship.

86 Data Management Ability to determine authorized users for data.

87 Data Management Ability to determine agreements (storage, purging, auditing).

88 Data Management Succession planning for departed data owners/stewards (user audits for data owners), easy to accomplish.

89 Data Management Automation and discovery, scanning of source systems and application source code to build catalog.

90 Data Management Scheduling for periodic scans - looking for changes/deltas.

91 Data Management Supports matching, linking and merging of related data entries within or across diverse datasets using a variety of traditional and new approaches such as rules, algorithms, metadata and machine learning.

92 Data Management Enables ability to review, edit, explore and otherwise interrogate various models (e.g., data, policy, rule, organizational) and various states and conditions over time.

93 Data Management

Supports augmented data management, which uses machine learning to automate data quality, data integration, workload management, data catalog, data preparation, insight discovery, model development and insight sharing.

94 Data Management Enables business process modeling, data flow modeling and documentation. Supports analytics key performance indicators and other benchmarking efforts.

95 Data Management Provides required data storage capacity, file types and locations. Supports processes such as disaster recovery, rollbacks, extraction or eradication.

96 Data Management Enables monitoring, reporting and management of data sharing. Supports encryption and security for data at rest and in motion.

97 Integration

Loads and exports metadata, including roles, in a fast, efficient and accurate manner with other third-party tools. Provides a communication backbone for the bidirectional flow of metadata between the central repository and the data sources.

98 Integration Integrates with all relevant applications, data sources and technologies.

99 Integration

Sends data to external systems such as large language models and uses insights from connected engines such as generative AI to deliver product functionality. Enables users to manage all aspects of data sharing, including full disablement.

Data Cataloging Business Requirements These requirements were gathered as part of an information gathreing intiative to both better understand programmatic needs in the area of data cataloging and to support the development of data catalog persona. These are in no way intended to be comprehensive or final. Their inclusion with this RFI is for informational context purpose only.

Metadata

Management Ability to scrape metadata.

Metadata

Management

Ability to document, maintain and share metadata and data elements definitions with team members and business users.

Metadata

Management The ability to capture detailed metadata such as description, data source, and usage.

Metadata

Management The ability to maintain version history for metadata definitions and governance policies.

Metadata

Management Establishing workflows for reviewing, approving, and managing changes to metadata.

Metadata

Management

Open metadata repository, with the ability to share metadata bidirectionally with other tools or other instances of tools.

Metadata

Management Ability to see data type availability.

Metadata

Management Ability for data stewards to update the metadata and connections.

Metadata

Management Ability for a 'smart' search.

Metadata

Management Requirements for metadata information, ability to determine which fields need this requirement.

Metadata

Management Metadata management dashboard with scoring.

Metadata

Management Ability to identify and/or recognize Open data standards.

Metadata

Management

Supports continuous analysis of all available user, data management, systems and infrastructure, and data governance experience reports to determine the alignment and exception cases between data as designed versus actual experience.

Metadata

Management

Automates the enforcement of business rules tied to data elements and associated metadata. Supports dedicated interfaces for creating, providing order of execution and offering links with information stewardship for effective governance.

114 Roles Supports Role Based Access Control.

115 Roles

Assembles organizational models and associated user IDs with key roles across workflows and intersections of work related to policy setting and policy enforcement. Sets up models that tag real people to data elements, tasks, workflows and rules.

116 Roles

Establishes, assigns and reassigns tasks across the organizational roles involved in policy setting, enforcement, and external roles and users. Provides management tools such as dashboards and work-to lists to monitor status of tasks.

Security &

Compliance

Provides certain data security policies through data risk assessment and orchestration of data security controls available on the governance platform. Enacts certain data access privileges, audit or monitoring to enable certain levels of security.

Security &

Compliance

Enables configurable controls that extend data and transaction security and compliance to third-party platforms or the solution's hosting providers. Documents security policies, audits, attestations or evaluations for compliance needs.

Security &

Compliance

Complies with relevant standards like CCPA, GDPR and third-party or government certifications such as

SOC 2, ISO 27001 and FedRAMP.

System

Administration Provides proactive alerts on system events. Enables logging and resolution reporting on all issues.

System

Administration Enables processes such as disaster recovery, rollbacks and version control.

System

Administration

Leverages network technologies like software-defined wide area networks and over-the-top monitoring to ensure the optimal performance of the solution.

System

Administration

Allows customization of the standard deployed solution with custom user interfaces, data tables, process components and business logic.

Training and

Enablement Robust Online help and reference available.

Data Cataloging Business Requirements These requirements were gathered as part of an information gathreing intiative to both better understand programmatic needs in the area of data cataloging and to support the development of data catalog persona. These are in no way intended to be comprehensive or final. Their inclusion with this RFI is for informational context purpose only.

Training and

Enablement Automation and templates to reduce manual input.

Training and

Enablement

Strong documentation and training aids (fit for purpose. Technical vs. Business) (Plain language) (Easily findable).

Training and

Enablement Team specific, tailored to each team and how that team interacts with data.

Training and

Enablement Ability to connect to wide selection of data source types and platforms, including on-premises or cloud.

Training and

Enablement

The ability to embed robust knowledge management (KM) practices within the governance layer to foster availability and clarity of data definitions, processes, and guidelines to enable enhanced self-service.

Training and

Enablement Ability to partition out roles - report writer vs. data curious, etc.

Training and

Enablement Ability to output data in JSON format, or other various formats outside .csv.

Training and

Enablement Ability to support multiple cloud hyper scalers.

Training and

Enablement Ticketing system and support system.

Training and

Enablement

Ability to integrate requests and incidents with a third-party ticketing and service desk management system.

Training and

Enablement Data owners to see the data in the context of their business context.

Training and

Enablement Templatized Access Requests.

Training and

Enablement

Data Sharing Access templates and track data sharing agreements, including all stakeholders and terms of the agreement.

138 User Interface Ability to provide a user interface which is "easy" for non-technical stakeholders.

139 User Interface User friendly interface for searching data within the system, also including outside assets.

140 User Interface User friendly and intuitive interface.

141 User Interface Provides easily understood, friendly interfaces with intuitive designs to facilitate user engagement.

142 User Interface

Supports the skills and needs of various roles and provides them with collaborative workflows. Addresses various users with an easily navigable, engaging interface. Enhances users' experience while interacting with the solution or product.