Spark Core Concepts Interview Questions

Apache Spark is a distributed data processing engine designed for large-scale batch processing, streaming, machine learning, and big data analytics.

Spark processes data in parallel across multiple machines using an MPP-style distributed architecture.

Spark Architecture consists of:

• Driver Program
• Cluster Manager
• Executors
• DAG Scheduler
• Stages
• Tasks

Goal:

• Process massive datasets in parallel
• Distribute computation across clusters
• Optimize execution using DAG and lazy evaluation

Spark Architecture Overview

Apache Spark is a distributed data processing engine designed for large-scale batch processing, streaming, machine learning, and big data analytics.

Spark processes data in parallel across multiple machines using an MPP-style distributed architecture.

Spark Architecture consists of:

• Driver Program
• Cluster Manager
• Executors
• DAG Scheduler
• Stages
• Tasks

Goal:

• Process massive datasets in parallel
• Distribute computation across clusters
• Optimize execution using DAG and lazy evaluation

Spark Architecture Overview

1. Driver Program

The Driver Program is the brain of Spark.

Responsibilities:

• Runs the user code
• Creates SparkSession
• Builds execution plan
• Creates DAG (Directed Acyclic Graph)
• Schedules stages and tasks
• Coordinates executors
• Collects final results

Example:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("SalesApp").getOrCreate()

The Driver runs this code.

2. Cluster Manager

Cluster Manager allocates resources for Spark applications.

Common cluster managers:

• YARN
• Kubernetes
• Spark Standalone
• Databricks Managed Cluster

Responsibilities:

• Launch executors
• Allocate CPU and memory
• Manage cluster resources

Example:

Driver requests:

8 Executors
8 Cores each
32 GB memory each

Cluster manager allocates containers/VM resources.

3. Executors

Executors are JVM processes running on worker nodes.

Responsibilities:

• Execute tasks
• Read data partitions
• Perform transformations
• Store cached data
• Handle shuffle operations

Each executor contains:

• multiple CPU cores
• task execution threads
• executor memory

Example:

Executor-1
8 cores
32 GB RAM

Executor can run:

• 8 tasks in parallel
• one task per core

4. DAG (Directed Acyclic Graph)

Spark converts transformations into a DAG execution plan.

Example:

df.filter(col("amount") > 1000).groupBy("region").sum("amount")

Spark does NOT execute immediately.

Instead:

• Spark builds a DAG of transformations
• Optimizer analyzes execution strategy
• Execution starts only when an Action is called

Example Actions:

.show()
.count()
.write()
.collect()

This concept is called Lazy Evaluation.

5. Catalyst Optimizer

For DataFrames and Spark SQL, Spark uses Catalyst Optimizer.

Catalyst performs:

• Predicate Pushdown
• Column Pruning
• Join Reordering
• Constant Folding
• Filter Optimization

Example:

Instead of reading all columns:

df.select("customer_id", "amount")

Spark reads only required columns.

This reduces IO dramatically.

6. Stages

Spark divides DAG into Stages.

A new stage is created at shuffle boundaries.

Narrow transformations:

• do NOT create shuffle
• stay in same stage

Examples:

• map()
• filter()
• select()

Wide transformations:

• create shuffle
• create new stage

Examples:

• groupBy()
• join()
• distinct()
• orderBy()

Example:

df.groupBy("region").sum("amount")

Execution:

Stage 1:
Read partitions + local aggregation

Shuffle Boundary

Stage 2:
Aggregate same regions together

7. Tasks

Each stage is divided into Tasks.

One task processes:

• one partition

Example:

100 partitions
→ 100 tasks

Tasks run in parallel across executors.

If cluster has:

8 Executors
8 Cores each

Then:

64 tasks can run simultaneously

End-to-End Spark Execution Flow

Example PySpark Code:

df = spark.read.parquet("/sales")

result = (
    df.filter(col("amount") > 1000)
      .groupBy("region")
      .sum("amount")
)

result.show()

Execution Flow:

1. Driver receives PySpark code
2. Driver builds Logical Plan
3. Catalyst Optimizer optimizes query
4. Spark creates DAG
5. DAG split into stages
6. Each stage split into tasks
7. Cluster manager allocates executors
8. Tasks sent to executors
9. Executors process partitions in parallel
10. Shuffle happens for aggregation
11. Final result returned to Driver

Real Production Example

Suppose:

Daily Processing:

• 500 GB sales data
• Read from ADLS/Delta Lake
• Filter + Join + Aggregation pipeline

Execution:

df.groupBy("region").sum("amount")

Spark Flow:

• Driver creates DAG
• Executors read partitions in parallel
• Shuffle groups same regions together
• Aggregation performed on executors
• Results returned to Driver