Spark & Big Data quiz

Question 1

What is the difference between RDDs, DataFrames, and Datasets in Apache Spark?

Accepted Answer

RDDs are unoptimized; DataFrames use Catalyst optimizer; Datasets add compile-time type safety

Answer

RDDs are faster; DataFrames are for streaming; Datasets are for ML

Answer

DataFrames replace RDDs entirely in Spark 3.x

Answer

All three have identical performance characteristics

Question 2

What is the difference between narrow and wide transformations in Spark?

Accepted Answer

Narrow use one input partition per output; wide require shuffle across multiple partitions

Answer

Narrow are faster always; wide are for joins only

Answer

Narrow transformations run on the driver; wide run on executors

Answer

Both require a shuffle; the difference is only in serialization format

Question 3

How does Adaptive Query Execution (AQE) improve Spark job performance?

Accepted Answer

Reoptimizes plans at runtime using actual execution statistics for partition coalescing, join switching, and skew handling

Answer

A static pre-execution optimizer added in Spark 3.0

Answer

A replacement for the Catalyst optimizer

Answer

A GPU acceleration layer for Spark SQL

Question 4

How do repartition and coalesce differ and when should you use each?

Accepted Answer

repartition does a full shuffle for balanced partitions; coalesce merges without shuffle but only reduces count

Answer

coalesce is always faster and preferred over repartition

Answer

repartition and coalesce are identical except for API signature

Answer

repartition is for DataFrames; coalesce is for RDDs only

Question 5

What is a Spark UDF and what are the performance implications of using one?

Accepted Answer

Python UDFs serialize row-by-row via Py4J causing 10-100x overhead; Pandas UDFs use Arrow for batch processing

Answer

UDFs are faster than built-in functions due to Python optimization

Answer

Pandas UDFs require GPU acceleration to outperform Python UDFs

Answer

UDFs execute natively in the JVM without serialization overhead

Question 6

What is the Spark job/stage/task hierarchy and how does it relate to shuffles?

Accepted Answer

Job triggered by action, divided into stages at shuffle boundaries, each stage has one task per partition

Answer

Job equals one stage; tasks run across multiple jobs

Answer

Stages and jobs are identical; tasks are partitioned differently

Answer

Tasks trigger stages; stages trigger jobs in reverse order

Question 7

What is Spark's dynamic partition overwrite mode and when is it critical?

Accepted Answer

Overwrites only matching partitions in the write DataFrame; default static mode deletes the entire table

Answer

A feature that auto-creates partitions based on data statistics

Answer

Required to write to encrypted Parquet files

Answer

Enables concurrent writes from multiple Spark jobs

Question 8

What is Z-ordering in Delta Lake and how does it improve query performance?

Accepted Answer

Reorders data within files to co-locate related values, improving Parquet file skipping for filtered queries

Answer

Compresses files using Z-standard encoding for smaller storage footprint

Answer

A partitioning scheme that uses Z-order keys as directory names

Answer

An index structure stored in the Delta transaction log

Question 9

What is checkpointing in Spark Structured Streaming and why is it essential?

Accepted Answer

Writes streaming progress and state to reliable storage for fault recovery and exactly-once semantics

Answer

A memory optimization that snapshots DataFrames to avoid recomputation

Answer

Required only when using Kafka as a source

Answer

An alternative to WAL (Write Ahead Log) for batch Spark jobs

Question 10

How do you use Spark's columnar processing with Apache Arrow for PySpark performance?

Accepted Answer

Enables columnar batch conversion between JVM and Python, making toPandas() and Pandas UDFs 10-100x faster

Answer

Arrow replaces Parquet as the file format for Spark tables

Answer

Arrow is required for Spark SQL window functions

Answer

Arrow enables GPU-accelerated joins in Spark

Question 11

What are the implications of using groupByKey vs reduceByKey on RDDs?

Accepted Answer

reduceByKey applies local combining before shuffle; groupByKey shuffles all values without pre-aggregation

Answer

They are identical for all aggregation functions

Answer

groupByKey is faster for numeric aggregations

Answer

reduceByKey only works with integer values

Question 12

What is a Spark accumulator and what are common misuse patterns to avoid?

Accepted Answer

Only for monitoring/debugging — double-counting occurs on stage retries; never use for business logic correctness

Answer

Accumulators are the safest way to aggregate data in Spark

Answer

Accumulators support both read and write from worker nodes

Answer

Accumulators replace groupBy aggregations for better performance

Question 13

What is Spark's external shuffle service and why is it required for dynamic allocation?

Accepted Answer

A persistent daemon on each worker that preserves shuffle data when executors are removed by dynamic allocation

Answer

A Kafka-based shuffle mechanism for cross-cluster data transfer

Answer

The HDFS NameNode's shuffle coordination service

Answer

An optional optimization that improves sort performance by 50%

Test yourself