high performance spark读书笔记

Spark SQL is a component that can be used in tandem with the Spark Core. Spark
SQL defines an interface for a semi-structured data type, called DataFrames and a
typed version called Dataset, with APIs in Scala, Java, and Python, as well as support for basic SQL queries

Spark MLLib is primarily built on top of RDDs, while ML is build on top of SparkSQL data frames. 4 Eventually the Spark community plans to move over to ML and deprecate MLlib.

Spark uses five main properties to represent an RDD internally. The three required properties are the list of partition objects, a function for computing an iterator of each partition, and a list of dependencies on other RDDs.Optionally, RDDs also include a partitioner (for RDDs of rows of key-value pairs represented as Scala tuples) and a list of preferred locations (for the HDFS file)

A Spark application consists of a driver process, which is where the high-level Spark logic is written, and a series of executor processes that can be scattered across the nodes of a cluster. The Spark program itself runs in the driver node and parts are sent to the executors.

A Spark application corresponds to a set of Spark jobs defined by one Spark Context in the driver program. A Spark application begins when a Spark Context is started. When the Spark Context is started, each worker node starts an executor (its own Java Virtual Machine, JVM). Spark applications can run multiple concurrent jobs. Jobs correspond to each action called on an RDD in a given application

fenb
spark的整体执行流程：
In some ways, the simplest way to think of the Spark execution model is that a Spark job is the set of RDD transformations needed to compute one final result. Each stage corresponds to a segment of work, which can be accomplished without involving the driver. In other words, one stage can be computed without moving data across the partitions. Within one stage, the tasks are the units of work done for each partition of the data.