Appendix 11 - Data Engineering Foundations (Functional).pdf

Data Engineers Foundations Course Description: This course is designed to provide participants with a comprehensive overview of the fundamental concepts, principles, and technologies that define and guide Army data operations.

Attendees will leave the course with a starting foundation of skills, lexicon, concepts, and experiences that they can build upon to increase their capabilities through further self-study and application at future assignments.

The three general goals of the Data Engineers Foundations course are as follows:

1. Establish a foundational knowledge and skill set for data engineering

2. Improve FA26B and 255A skillsets for ongoing Army-wide incorporation of more data-focused tasks and missions

3. Provide learning in topics relevant to an organization’s data operations allowing talent to deliver the most robust and relevant products to their organization in the future.

Lesson 1: Data Engineering Roles and Responsibilities (4 Hours) TLO: Identify the Roles and Responsibilities of the Data Engineer

In this lesson, students will explore their core roles and responsibilities as Data Engineers functioning within the Army’s data environment. There will be a review of data literacy fundamentals for those who are completely unfamiliar with the concepts, as well as a discussion of various aspects of a Data Engineer’s duties and challenges.

• Identify data literacy fundamentals.

• Identify the duties and functions of a Data Engineer.

• Identify the challenges of performing duties as a Data Engineer.

Lesson 2: Data Governance (4 Hours) TLO: Define Data Governance Within the Army Enterprise

This lesson will focus of data governance within the DoD and the Army, and how the Data Engineer fits into various roles.

• Identify how data governance functions within the Army enterprise.

• Identify how the Army benefits from good data governance.

• Define the fundamental concepts, principles, and components of data governance.

• Identify the processes, policies, roles, metrics, and standards established by existing data governance bodies.

• Define the principles, standards and practices that render data consistent and reliable.

Lesson 3: Data Modeling (8 Hours) TLO: Develop a Data Model

This lesson introduces the basic concepts of data modeling. Students will learn about various data types and sources, the reason why data is visualized, and how models are used for predictions.

• Identify various sources of data, data types, and formats that data can take.

• Define the components of a data model as compared to machine learning or AI.

• Define data classification and the conceptual overview of model fitting.

• Develop a data model for a scenario. (Practical Exercise)

• Develop a data model within Power BI. (Practical Exercise)

Lesson 4: Zero Trust Fundamentals (4 Hours) TLO: Identify Zero Trust Concepts

This lesson helps students understand the ideas, principles, and components of the Zero Trust concept.

The discussion includes implementation of Zero Trust Architecture (ZTA), use cases, and deployment scenarios.

• Identify the concepts, principles, and components of zero trust.

• Identify the importance of zero trust concept to the Army enterprise.

• Identify procedures and techniques for implementing a ZTA.

• Review case studies and deployment scenarios for ZTA from a military lens.

Lesson 5: Data Tagging (4 Hours) TLO: Apply Data Tagging to Existing Data

Within this module the importance of data tagging is presented to the students. Just as knowing the parts of a rifle is important in understand how it operates, data must be tagged correctly if it is to be worked with easily and efficiently. Students will get practice in tagging data, identifying errors, and correcting them.

• Define the fundamental concepts and principles of data tagging.

• Identify various examples of data tagging.

• Identify data tagging best practices.

• Define the mechanisms of scalability and their implications in a cloud network.

• Apply data tagging and correction in a practice scenario.

Lesson 6: Intro to Programming (4 Hours) TLO: Apply Programming Techniques to Create a Software Application

This module introduces the student to basic programming concepts. The intent is to lay a foundation to build upon later in the course. It is ideal if the student has prior programming experience, preferably in Python, but it is not necessary.

• Define basic computing programming concepts.

• Identify development tools to edit, compile, run, and test software.

• Identify basic language constructs, such as data types, output, input, constants, variables, assignment statements, and Boolean expressions.

• Implement software solutions with control and data structures.

• Implement functional, procedural, object-oriented, concurrent, and logical programming.

Apply programming techniques to a given scenario. (Practical Exercise)

Lesson 7: Engineering in the Cloud (4 Hours) TLO: Identify Data Engineering Responsibilities Within a Cloud Platform

This module is where students begin seeing how the pieces are put together. By using prior course knowledge and skills, they will begin conceptually connecting the various aspects of working with data: organization, cleaning, access, curation, and data tools together in ways that can be useful.

• Identify the impacts of conducting data management via cloud platform vs local infrastructure.

• Define the four main cloud vendors on the market: AWS, Azure, Oracle, and GCP.

• Define cloud computing’s key principles, models, strengths, and weaknesses.

• Define the basic technological underpinnings and infrastructure components of cloud services.

• Identify the practical applications of cloud technology in data management.

• Identify requirements for effective design and implementation of a data model within a cloud infrastructure as part of a complete data management plan.

• Identify strategies to address challenges encountered in data operations within a cloud-based infrastructure.

Lesson 8: Data in Practice (8 Hours) TLO: Apply Data Analytics Concepts Within a Scenario

In this section, students will use Power BI, a visualization tool that can tell a story with raw data. This is important because the insights gained from data don’t matter unless they can be communicated effectively. Students will be given a chance to practice their skills in a provided scenario, with ample time to work through problems and ask questions of the instructor.

• Define the attributes of an effective data story.

• Identify the use of data visualizations for effective decision making.

• Identify visualization tools to communicate data.

• Identify the benefits and limitations of Power BI to communication of data insights.

• Apply various data engineering concepts to a provided sample dataset (inspecting, cleaning, managing, curating, etc.).

• Implement principles of effective data visualization design to transform raw data into a visual representation.

• Apply data management functions and the Power BI tool to produce appropriate visualizations for a given scenario. (Practical Exercise)

Lesson 9: Understanding Data Pipelines (8 hours) TLO: Establish a Data Pipeline

Now that the student understands what data is and how it can be visualized, the student must learn about how it is stored in bulk and how it is accessed. As Data Engineers, managing data stores and establishing secure and reliable pipelines to the data they oversee is a critical aspect of their duties.

• Define data storage and data movement.

• Implement the serving layer.

• Monitor data storage and processing.

• Implement a data pipeline for a given scenario. (Practical Exercise) A practical exercise that involves curation of a dataset and establishment of a secure data pipeline to a target entity.

Lesson 10: API Creation (8 Hours) TLO: Establish an API for an Existing Data Set

This module begins providing the student with a discussion to ensure an understanding of API’s and ASP.NET. Students will also configure a newly built API. Students define API interfaces according to the OpenAPI specification and build SOAP and REST based APIs along with a GraphQL API. After building and testing APIs students learn to secure APIs.

• Identify Web APIs

• Define the ASP.NET core.

• Identify API and Returning Resources.

• Measure Resources and Validating Input

• Define the Entity Framework Core

• Manage API Resources

• Implement an API for a given scenario to include the creation, building, and testing of an API.

(Practical exercise)

Lesson 11: Building ETLs (8 Hours) TLO: Implement the Extract, Transform, and Load (ETL) Process

In this module the student learns how to manage the entire Extract, Transform, and Load (ETL) process from start to finish with a given dataset.

• Employ Microsoft SQL Server Platform for ETL Pipelines.

• Record data into dimension tables.

• Record data into fact tables.

• Deploy an ETL pipeline project to the SSIS catalog.

• Apply the ETL process to an Army-related scenario. (Practical Exercise)

Lesson 12: Big Data Processing (8 Hours) TLO: Implement Big Data Processing

This module discusses big data processing, engines, and their characteristics. Big data processing has become a trending technology, and big data tools play a huge role in the organizational data analysis process. The usage of big data tools to store, process, and analyze help to develop techniques to process data and learn techniques for masking and validating data.

• Define big data processes and the data processing engine.

• Identify techniques for processing different types of complex data resources including relational data and unstructured data.

• Identify how to handle complex data challenges and designate data-driven decisions.

• Identify techniques for processing data, including techniques for masking data and techniques for validating data using data rules.

• Manage large datasets in a given scenario. (Practical Exercise)

Lesson 13: Capstone (4 Hours)

• Complete a major project designed to reinforce of all learning outcomes covered in the course.

• Consists of a Go/No-Go format

Administration Time (4 hours)

• Time set aside for schoolhouse / instructor tasks such as working out student login issues, in-processing, issuance of graduation certificates, etc.