CONCEPT of CHECKPOINTS in Spark

In this blog Concept of Checkpoint in Spark, you will learn about checkpoints in detail. First of all we will see what is Checkpoint, the need of Checkpoint followed by the Fault Tolerance feature of Checkpoint, then we will go through the difference between checkpoint and persist and finally, the different types of checkpoints in Apache Spark. So let’s start.

What is Checkpoint ?

Before diving into the topic “Checkpoint,” let’s understand the definition of some key words that we will be using in this blog.

• RDD – RDD stands for Resilient Distributed Dataset. It is a fundamental data structure in Apache Spark, an open-source distributed computing framework. RDD represents an immutable, fault-tolerant collection of elements that can be processed in parallel across a cluster of computers.
  Example – val seqOfNumber = sc.parallelize(Seq(1, 2, 3, 4, 5))
  Here create an RDD named seqOfNumber by parallelizing a sequence of numbers. The parallelize method distributes the data across the cluster and forms partitions.

• Fault Tolerance – Fault tolerance refers to the ability of a system or component to continue functioning properly in the presence of faults or failures.

• DStreams (Discretized Streams) – DStreams, short for Discretized Streams, are a fundamental unit in Apache Spark Streaming. They represent a continuous stream of data, which is divided into small, discrete batches.

• HDFS (Hadoop Distributed File System) – It is a distributed file system designed to store and manage large volumes of data across multiple machines in a Hadoop cluster.

Now, let’s see what Checkpoint is –

Checkpoint is a crucial mechanism in Apache Spark that allows for fault tolerance and recovery in distributed data processing. It involves saving the intermediate state of RDDs and DStreams to a reliable storage system, such as HDFS or a distributed file system.

In Spark, a Checkpoint represents a stable point in the execution of a job where the RDD/DStream data is written to storage. This data can be used for recovery in case of failures and also enables certain optimisations within Spark.

Need of Checkpoint

• Fault tolerance: Checkpointing allows programs to recover from failures or crashes by restoring their previous state.

• Incremental progress: In lengthy computations or simulations, it can be advantageous to save intermediate results at regular intervals.

• Resource management: Some computations require significant computational resources and may run for an extended period. Checkpointing allows users to temporarily release resources (such as compute nodes in a cluster) while saving the computation state.

• Experiment tracking: In machine learning, Checkpoints are often used to save the model’s parameters or intermediate results during training or experimentation.

• Optimization and debugging: Checkpointing can aid in performance optimization and debugging. By saving checkpoints at critical stages of a program, developers can analyze the state of variables, data structures, or internal states to identify bottlenecks, investigate issues, or verify program correctness.

Fault Tolerance Feature of Checkpoint

Let’s dive into a theoretical example to understand the fault-tolerance feature of Checkpoints in Spark.

Suppose we have a Spark application that performs a series of transformations on a large dataset. The application reads the data, applies multiple operations such as filtering, mapping, and aggregating, and finally generates some output.

Now, imagine that during the execution of this application, a node in the Spark cluster fails due to hardware issues or network problems. Without any fault tolerance mechanism, the entire computation would be lost, and the application would need to start from scratch once the failed node is back online.

However, by enabling checkpoints in Spark, the intermediate RDDs can be periodically persisted to a reliable storage system, such as HDFS or a distributed file system. This Checkpoint process occurs at specific intervals or stages defined by the application.

Checkpoint vs Persist

Checkpoint

• Refers to saving the current state of a program or model during execution.
• Typically used in long-running processes or complex computations.
• Allows for recovery and resuming from a specific point in case of errors.
• Used in distributed systems to ensure data integrity and fault tolerance.

Persist

• Means to store data in a durable or permanent manner.
• Involves preserving data beyond the duration of a program’s execution.
• Enables data retrieval and usage even after system restarts or fails.
• Often used in software development and databases for long-term data storage.

Types of Checkpoint in Apache Spark

Spark supports two types of checkpointing: RDD (Resilient Distributed Datasets) checkpointing and DataFrame/Dataset checkpointing.

• RDD Checkpointing:
  RDD checkpointing involves saving the RDD lineage information to a reliable storage system. Checkpointing RDDs helps in reducing the memory usage and improves fault tolerance. When you enable RDD checkpointing, Spark materializes the RDDs to disk, which breaks the lineage and allows Spark to recover the data from disk in case of failures. To enable RDD checkpointing, you can use the RDD.checkpoint() method on an RDD object. You also need to set a checkpoint directory using SparkContext.setCheckpointDir() to specify where the checkpoint data should be stored.

• DataFrame/Dataset Checkpointing:
  DataFrame/Dataset checkpointing allows you to checkpoint intermediate DataFrame or Dataset operations. It provides similar benefits as RDD checkpointing but is specifically designed for structured data processing using DataFrames or Datasets. To enable DataFrame/Dataset checkpointing, you can use the DataFrame/Dataset.checkpoint() method on a DataFrame or Dataset object. You also need to set a checkpoint directory using SparkSession.setCheckpointDir() to specify where the checkpoint data should be stored.

Conclusion

In conclusion, Spark checkpointing is a crucial technique for ensuring fault tolerance and data durability in distributed data processing applications. By periodically saving the RDD lineage and metadata to a reliable storage system, Spark can recover lost data or failures and resume computations from a consistent state. This provides resiliency to system failures, improves job reliability, and enables efficient recovery in case of errors. Overall, using checkpointing into Spark applications can significantly enhance their robustness and reliability in handling big data workloads.

References

Spark Official Documentation