Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Fundamentals

• Overview of the Big Data ecosystem and Spark’s role in modern data platforms
• Exploring Spark architecture: driver, executors, cluster manager, lazy evaluation, DAG, and execution planning
• Distinguishing between RDD and DataFrame APIs and determining when to apply each approach
• Creating and configuring SparkSession to understand the basics of application configuration

Module 2: PySpark DataFrames

• Reading and writing data from enterprise sources and formats: CSV, JSON, Parquet, and Delta
• Manipulating PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations such as window functions, timestamp handling, and nested data processing
• Applying data quality checks and writing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Basics for Spark Users

• Understanding the relationship between Docker images and Kubernetes pods in relation to Spark driver and executor processes — a conceptual view for cluster consumers rather than administrators
• What happens when a Spark job runs on Kubernetes: pod scheduling, resource requests and limits, and how session settings map to them
• Identifying what you control through Spark session parameters versus what the platform team manages via cluster provisioning — and why this boundary is crucial for tuning
• Hands-on exercise: Examine a running Spark-on-Kubernetes job in the sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — In-Depth

• Understanding the trade-off between executor count and executor size: balancing memory and cores and how to reason about these configurations
• Allocating memory for drivers and executors, managing memory overhead, and translating these into pod resources
• Dynamic allocation: its behavior on Kubernetes, when it conserves resources, and when it does not
• Hands-on exercise: Execute the same job with different executor and core configurations to compare runtime and resource usage

Module 5: Scaling Behavior & Cost Optimization

• Understanding how adding or removing nodes affects job behavior, completion time, and resource consumption
• Comparing many small executors against fewer large ones: analyzing performance and cost trade-offs
• Shuffle behavior and tuning shuffle partitions, including estimating the cost impact of configuration choices
• Hands-on exercise: Scale the sandbox cluster from five to ten nodes and observe the impact on job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• Understanding the small-files problem: why sources split across many 1–5 MB Parquet files degrade performance and distort partitioning
• Strategies for repartitioning and coalescing
• Controlling partition sizes during read and write operations
• Writing Parquet efficiently to prevent the recurrence of the small-files problem downstream
• Hands-on exercise: Load a dataset composed of many small Parquet partitions, apply various repartitioning strategies, and compare performance before and after optimization

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying the root causes of out-of-memory errors in Pandas and recognizing their symptoms
• Applying memory-efficient conversion patterns between Spark and Pandas
• Preventing memory spikes when writing large datasets to CSV
• Utilizing chunked processing and dtype optimization for constrained environments
• Hands-on exercise: Reproduce a common Pandas out-of-memory scenario and resolve it using chunking and dtype optimization

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: performance characteristics, lazy evaluation, and memory behavior
• Understanding where Polars fits alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Hands-on exercise: Rewrite a Pandas-heavy transformation in Polars to compare memory usage and processing speed

Module 9: Applying Optimization to ETL & ML Workloads

• Applying configuration, partitioning, and memory-management principles across a realistic ETL pipeline
• Understanding optimization considerations specific to machine-learning workloads running on the same cluster
• Applying a practical tuning workflow to systematically diagnose cost and performance issues
• Hands-on exercise: Complete an end-to-end mini-project combining data loading, transformation, and a simple model-training step, with participants tuning the Spark configuration themselves