Description
Who should attend
- Data Engineer
- Data Analysts
Prerequisites
Complete “Fundamentals of Big Data and Machine Learning.”
Course Objectives
- Identify the need for data integration,
- Understand the capabilities of Cloud Data Fusion as a data integration platform,
- Identify use cases for possible implementation with Cloud Data Fusion,
- List the major components of Cloud Data Fusion,
- [Design and execute batch and real-time data processing pipelines,
- Work with Wrangler to build data transformations.
- Use connectors to integrate data from different sources and formats,
- Configure the runtime environment; monitor and troubleshoot pipeline execution,
- Understand the relationship between metadata and data lineage
.
Outline: Data Integration with Cloud Data Fusion (DICDF)
Module 00 – Introduction
(in English)
Module 01 – Introduction to Data Integration and Cloud Data Fusion
- Data integration: what, why, challenges
- Data integration tools used in the industry
- User personas
- Introduction to cloud-based data fusion
- Critical Data Integration Capabilities
- Cloud Data Fusion user interface components
Module 02 – Building Pipelines
- Cloud Data Fusion architecture
- Basic concepts
- Data pipelines and directed acyclic graphs (DAG)
- Pipeline Life Cycle
- Designing pipelines in Pipeline Studio
Module 03 – Designing Complex Pipelines
- Branches, merges and joins
- Actions and Notifications
- Error handling and macros Pipeline configurations, scheduling, import and export
Module 04 – Pipeline Execution Environment
- Scheduling and triggers
- Runtime environment: Compute profile and provisioners
- Pipeline Monitoring
Module 05 – Building transformations and preparing data with Wrangler
- Wrangler
- Guidelines
- User-defined directives
Module 06 – Stream Connectors and Pipelines
- Understand the data integration architecture.
- List the different connectors.
- Use the Cloud Data Loss Prevention (DLP) API.
- Understand the streaming pipeline reference architecture.
- Build and run a streaming pipeline
.
Module 07 – Metadata and Data Lineage
- Metadata
- Data lineage
Module 08 – Summary
- Course summary
