Get in Touch

Course Outline

Databricks Platform and Lakehouse Fundamentals

  • Lakehouse architecture and core components of Databricks
  • Organizing workspaces and catalogs

Databricks Workspace and Notebooks

  • Navigating the workspace and developing via notebooks
  • Structuring code into reusable notebook modules

Apache Spark Architecture and Execution

  • Sparse runtime architecture and execution model
  • Lazy evaluation and the job DAG

PySpark DataFrames and the DataFrame API

  • DataFrame abstractions and schemas
  • Core DataFrame operations and column expressions

Translating SQL to PySpark DataFrames

  • Converting core SQL clauses into DataFrame operations
  • Implementing window functions and aggregations in PySpark

Reading and Writing Data in Databricks

  • Reading from various file and database sources
  • Writing and partitioning data within the Lakehouse

Delta Lake and Table Management

  • Understanding Delta tables and ACID transactions
  • Utilizing time travel and schema evolution features

Data Cleaning and Transformation Patterns

  • Data cleaning techniques and type conversion
  • Developing reusable transformation logic

User-Defined Functions and Modular Code

  • Python UDFs and pandas UDFs
  • Modularizing procedural logic into functions

Performance Tuning and Optimization

  • Partitioning and caching strategies
  • Identifying bottlenecks using the Spark UI

Structured Streaming Fundamentals

  • Differences between batch and streaming processing models
  • Streaming DataFrames and basic aggregations

Databricks Jobs and Workflow Orchestration

  • Scheduling notebooks as jobs and tasks
  • Constructing multi-step workflows with dependencies

Unity Catalog and Data Governance

  • Unity Catalog architecture and namespaces
  • Managing access control and data lineage

Testing, Debugging, and Production Practices

  • Unit testing PySpark logic
  • Debugging techniques and code quality standards

End-to-End Financial Services Use Cases

  • Developing an end-to-end banking ETL pipeline
  • Converting legacy SQL processes to PySpark

Migrating SQL Workloads to PySpark

  • Migration strategies and planning patterns
  • Incremental conversion of SQL workflows to PySpark

Requirements

  • Experience with Python programming, particularly regarding functions and data types
  • Understanding of SQL concepts, including joins, aggregations, and subqueries
  • No prior experience with Databricks or PySpark is necessary

Audience

  • Data engineers, data analysts, and related data professionals
  • Teams seeking to migrate existing SQL-based workflows to Databricks and PySpark
 35 Hours

Number of participants


Price per participant

Testimonials (1)

Upcoming Courses

Related Categories