Spark（2.3.1）集群安装【一篇就够】
操作系统：Linux（CenOS7）
Spark集群至少需要有JDK、Hadoop、Scala、Spark
一、安装Spark依赖的Scala（每个节点都安装）
1.1 下载Scala（2.11.12）
wget "https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz"
1.2 解压、配置环境变量
vi /etc/profile 编辑文件，追加如下内容：
# scala environment
export SCALA_HOME=/usr/scala-2.11.12
export PATH=$PATH:$SCALA_HOME/bin
1.3 验证Scala安装成功
[[email protected] usr]# source /etc/profile <== 使配置生效
[[email protected] usr]# scala -version
Scala code runner version 2.11.12 -- Copyright 2002-2017, LAMP/EPFL
二、安装Spark（每个节点都安装）
2.1 下载Spark（2.3.1）
wget "http://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz"
2.2 解压、配置环境变量
vi /etc/profile 编辑文件，追加如下内容：
# spark environment
export SPARK_HOME=/usr/spark-2.3.1-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin <==因为$SPARK_HOME/sbin目录下一些文件名和$HADOOP_HOME/bin目 录下一些文件名相同，为了避免文件名冲突，这里不添加 $SPARK_HOME/sbin目录
2.3 配置conf目录下文件
2.3.1 新建spark-env.sh文件
[[email protected] conf]# cp spark-env.sh.template spark-env.sh
[[email protected] conf]# vi spark-env.sh 编辑文件，增加如下内容：
JAVA_HOME=/usr/java/jdk1.8.0_162
HADOOP_HOME=/usr/hadoop/hadoop-2.9.0
HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
SCALA_HOME=/usr/scala-2.11.12
SPARK_HOME=/usr/spark-2.3.1-bin-hadoop2.7
SPARK_MASTER_IP=master
SPARK_EXECUTOR_MEMORY=1G
2.3.2 新建slaves文件
[[email protected] conf]# cp slaves.template slaves
[[email protected] conf]# vi slaves 编辑文件，增加如下内容：
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# A Spark Worker will be started on each of the machines listed below.
slave1
slave2
三、启动Spark集群
因为Spark是依赖于Hadoop提供的分布式文件系统的，所以在启动Spark之前，先确保Hadoop在正常运行。（Hadoop安装与启动）
在Hadoop正常运行的情况下，在master（Hadoop的NameNode，即Spark的master节点）上，
/usr/spark-2.3.1-bin-hadoop2.7/sbin目录下执行：
[[email protected] sbin]# ./start-all.sh <==前面必须有 ./ ，否则会执行$HADOOP_HOME/bin下的start-all.sh脚本
starting org.apache.spark.deploy.master.Master, logging to /usr/spark-2.3.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out
slave1: starting org.apache.spark.deploy.worker.Worker, logging to /usr/spark-2.3.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave1.out
slave2: starting org.apache.spark.deploy.worker.Worker, logging to /usr/spark-2.3.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave2.out
访问Spark集群URL：
四、运行Spark提供的计算圆周率的示例程序
4.1 使用单机local模式提交任务
local模式也就是本地模式，即在本地机器上单机执行程序。使用这个模式，并不需要启动Hadoop集群，也不需要启动Spark集群，只要有一台机器上安装了JDK、Scala、Spark即可运行。
[[email protected] sbin]# spark-submit --class org.apache.spark.examples.SparkPi --master local /usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar
2018-09-09 21:21:21 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-09 21:21:22 INFO SparkContext:54 - Running Spark version 2.3.1
2018-09-09 21:21:22 INFO SparkContext:54 - Submitted application: Spark Pi
2018-09-09 21:21:22 INFO SecurityManager:54 - Changing view acls to: root
2018-09-09 21:21:22 INFO SecurityManager:54 - Changing modify acls to: root
2018-09-09 21:21:22 INFO SecurityManager:54 - Changing view acls groups to:
2018-09-09 21:21:22 INFO SecurityManager:54 - Changing modify acls groups to:
2018-09-09 21:21:22 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
2018-09-09 21:21:23 INFO Utils:54 - Successfully started service 'sparkDriver' on port 44606.
2018-09-09 21:21:23 INFO SparkEnv:54 - Registering MapOutputTracker
2018-09-09 21:21:23 INFO SparkEnv:54 - Registering BlockManagerMaster
2018-09-09 21:21:23 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-09-09 21:21:23 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-09-09 21:21:23 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-d76c7c4a-0859-4c11-aba0-7d8e88efdec6
2018-09-09 21:21:24 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2018-09-09 21:21:24 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2018-09-09 21:21:24 INFO log:192 - Logging initialized @6994ms
2018-09-09 21:21:24 INFO Server:346 - jetty-9.3.z-SNAPSHOT
2018-09-09 21:21:24 INFO Server:414 - Started @7429ms
2018-09-09 21:21:25 INFO AbstractConnector:278 - Started [email protected]{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:21:25 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/jobs,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/jobs/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/jobs/job,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/jobs/job/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/stages,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/stages/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/stages/stage,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/stages/stage/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/stages/pool,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/stages/pool/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/storage,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/storage/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/storage/rdd,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/environment,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/environment/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/executors,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/executors/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/executors/threadDump,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/static,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/api,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO ContextHandler:781 - Started [email protected]{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-09-09 21:21:25 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
2018-09-09 21:21:25 INFO SparkContext:54 - Added JAR file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar at spark://master:44606/jars/spark-examples_2.11-2.3.1.jar with timestamp 1536499285929
2018-09-09 21:21:26 INFO Executor:54 - Starting executor ID driver on host localhost
2018-09-09 21:21:26 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46568.
2018-09-09 21:21:26 INFO NettyBlockTransferService:54 - Server created on master:46568
2018-09-09 21:21:26 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-09-09 21:21:26 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 46568, None)
2018-09-09 21:21:27 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:46568 with 413.9 MB RAM, BlockManagerId(driver, master, 46568, None)
2018-09-09 21:21:27 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 46568, None)
2018-09-09 21:21:27 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 46568, None)
2018-09-09 21:21:27 INFO ContextHandler:781 - Started [email protected]{/metrics/json,null,AVAILABLE,@Spark}
2018-09-09 21:21:30 INFO SparkContext:54 - Starting job: reduce at SparkPi.scala:38
2018-09-09 21:21:30 INFO DAGScheduler:54 - Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
2018-09-09 21:21:30 INFO DAGScheduler:54 - Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
2018-09-09 21:21:30 INFO DAGScheduler:54 - Parents of final stage: List()
2018-09-09 21:21:30 INFO DAGScheduler:54 - Missing parents: List()
2018-09-09 21:21:30 INFO DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
2018-09-09 21:21:31 INFO MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 1832.0 B, free 413.9 MB)
2018-09-09 21:21:32 INFO MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 1181.0 B, free 413.9 MB)
2018-09-09 21:21:32 INFO BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on master:46568 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:21:32 INFO SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1039
2018-09-09 21:21:32 INFO DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
2018-09-09 21:21:32 INFO TaskSchedulerImpl:54 - Adding task set 0.0 with 2 tasks
2018-09-09 21:21:32 INFO TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7853 bytes)
2018-09-09 21:21:32 INFO Executor:54 - Running task 0.0 in stage 0.0 (TID 0)
2018-09-09 21:21:32 INFO Executor:54 - Fetching spark://master:44606/jars/spark-examples_2.11-2.3.1.jar with timestamp 1536499285929
2018-09-09 21:21:33 INFO TransportClientFactory:267 - Successfully created connection to master/192.168.11.128:44606 after 146 ms (0 ms spent in bootstraps)
2018-09-09 21:21:33 INFO Utils:54 - Fetching spark://master:44606/jars/spark-examples_2.11-2.3.1.jar to /tmp/spark-3fae86fe-d3da-4b87-b0e1-c94efab840c5/userFiles-5292d290-2b31-4dc6-bfd6-c85e4cdaaae6/fetchFileTemp6719743492164655300.tmp
2018-09-09 21:21:34 INFO Executor:54 - Adding file:/tmp/spark-3fae86fe-d3da-4b87-b0e1-c94efab840c5/userFiles-5292d290-2b31-4dc6-bfd6-c85e4cdaaae6/spark-examples_2.11-2.3.1.jar to class loader
2018-09-09 21:21:34 INFO Executor:54 - Finished task 0.0 in stage 0.0 (TID 0). 867 bytes result sent to driver
2018-09-09 21:21:34 INFO TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 7853 bytes)
2018-09-09 21:21:34 INFO Executor:54 - Running task 1.0 in stage 0.0 (TID 1)
2018-09-09 21:21:34 INFO TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 2050 ms on localhost (executor driver) (1/2)
2018-09-09 21:21:34 INFO Executor:54 - Finished task 1.0 in stage 0.0 (TID 1). 867 bytes result sent to driver
2018-09-09 21:21:34 INFO TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 318 ms on localhost (executor driver) (2/2)
2018-09-09 21:21:34 INFO TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool
2018-09-09 21:21:34 INFO DAGScheduler:54 - ResultStage 0 (reduce at SparkPi.scala:38) finished in 3.901 s
2018-09-09 21:21:34 INFO DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 4.356085 s
Pi is roughly 3.146475732378662
2018-09-09 21:21:35 INFO AbstractConnector:318 - Stopped [email protected]{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:21:35 INFO SparkUI:54 - Stopped Spark web UI at http://master:4040
2018-09-09 21:21:35 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-09-09 21:21:35 INFO MemoryStore:54 - MemoryStore cleared
2018-09-09 21:21:35 INFO BlockManager:54 - BlockManager stopped
2018-09-09 21:21:35 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2018-09-09 21:21:35 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-09-09 21:21:35 INFO SparkContext:54 - Successfully stopped SparkContext
2018-09-09 21:21:35 INFO ShutdownHookManager:54 - Shutdown hook called
2018-09-09 21:21:35 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-3fae86fe-d3da-4b87-b0e1-c94efab840c5
2018-09-09 21:21:35 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-ba114487-f80c-4684-a962-870139ca248d
[[email protected] sbin]#
4.2 使用独立的Spark集群模式提交任务
这种模式也就是Standalone模式，使用独立的Spark集群模式提交任务，需要先启动Spark集群，但是不需要启动Hadoop集群。
启动Spark集群的方法是进入$SPARK_HOME/sbin目录下，执行start-all.sh脚本，启动成功后，可以访问下面的地址看是否成功：http://master:8080/
[[email protected] sbin]# spark-submit --class org.apache.spark.examples.SparkPi --master spark://master:7077 /usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar
2018-09-09 21:33:01 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-09 21:33:02 INFO SparkContext:54 - Running Spark version 2.3.1
2018-09-09 21:33:02 INFO SparkContext:54 - Submitted application: Spark Pi
2018-09-09 21:33:02 INFO SecurityManager:54 - Changing view acls to: root
2018-09-09 21:33:02 INFO SecurityManager:54 - Changing modify acls to: root
2018-09-09 21:33:02 INFO SecurityManager:54 - Changing view acls groups to:
2018-09-09 21:33:02 INFO SecurityManager:54 - Changing modify acls groups to:
2018-09-09 21:33:02 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
2018-09-09 21:33:03 INFO Utils:54 - Successfully started service 'sparkDriver' on port 44732.
2018-09-09 21:33:03 INFO SparkEnv:54 - Registering MapOutputTracker
2018-09-09 21:33:03 INFO SparkEnv:54 - Registering BlockManagerMaster
2018-09-09 21:33:03 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-09-09 21:33:03 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-09-09 21:33:03 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-8089bd51-572f-4c01-93a8-b0a9d5d78014
2018-09-09 21:33:03 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2018-09-09 21:33:03 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2018-09-09 21:33:04 INFO log:192 - Logging initialized @6786ms
2018-09-09 21:33:04 INFO Server:346 - jetty-9.3.z-SNAPSHOT
2018-09-09 21:33:04 INFO Server:414 - Started @7198ms
2018-09-09 21:33:04 INFO AbstractConnector:278 - Started [email protected]{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:33:04 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/jobs,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/jobs/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/jobs/job,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/jobs/job/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/stages,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/stages/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/stages/stage,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/stages/stage/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/stages/pool,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/stages/pool/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/storage,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/storage/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/storage/rdd,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/environment,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/environment/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/executors,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/executors/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/executors/threadDump,null,AVAILABLE,@Spark}
2018-09-09 21:33:04 INFO ContextHandler:781 - Started [email protected]{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO ContextHandler:781 - Started [email protected]{/static,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO ContextHandler:781 - Started [email protected]{/,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO ContextHandler:781 - Started [email protected]{/api,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO ContextHandler:781 - Started [email protected]{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO ContextHandler:781 - Started [email protected]{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-09-09 21:33:05 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
2018-09-09 21:33:05 INFO SparkContext:54 - Added JAR file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar at spark://master:44732/jars/spark-examples_2.11-2.3.1.jar with timestamp 1536499985512
2018-09-09 21:33:06 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://master:7077...
2018-09-09 21:33:06 INFO TransportClientFactory:267 - Successfully created connection to master/192.168.11.128:7077 after 139 ms (0 ms spent in bootstraps)
2018-09-09 21:33:07 INFO StandaloneSchedulerBackend:54 - Connected to Spark cluster with app ID app-20180909213306-0000
2018-09-09 21:33:07 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40542.
2018-09-09 21:33:07 INFO NettyBlockTransferService:54 - Server created on master:40542
2018-09-09 21:33:07 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-09-09 21:33:07 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180909213306-0000/0 on worker-20180909210806-192.168.11.129-41138 (192.168.11.129:41138) with 1 core(s)
2018-09-09 21:33:07 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180909213306-0000/0 on hostPort 192.168.11.129:41138 with 1 core(s), 1024.0 MB RAM
2018-09-09 21:33:07 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20180909213306-0000/1 on worker-20180909210808-192.168.11.130-41969 (192.168.11.130:41969) with 1 core(s)
2018-09-09 21:33:07 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20180909213306-0000/1 on hostPort 192.168.11.130:41969 with 1 core(s), 1024.0 MB RAM
2018-09-09 21:33:07 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180909213306-0000/0 is now RUNNING
2018-09-09 21:33:08 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20180909213306-0000/1 is now RUNNING
2018-09-09 21:33:08 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 40542, None)
2018-09-09 21:33:08 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:40542 with 413.9 MB RAM, BlockManagerId(driver, master, 40542, None)
2018-09-09 21:33:08 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 40542, None)
2018-09-09 21:33:08 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 40542, None)
2018-09-09 21:33:09 INFO ContextHandler:781 - Started [email protected]{/metrics/json,null,AVAILABLE,@Spark}
2018-09-09 21:33:10 INFO StandaloneSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
2018-09-09 21:33:13 INFO SparkContext:54 - Starting job: reduce at SparkPi.scala:38
2018-09-09 21:33:13 INFO DAGScheduler:54 - Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
2018-09-09 21:33:13 INFO DAGScheduler:54 - Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
2018-09-09 21:33:13 INFO DAGScheduler:54 - Parents of final stage: List()
2018-09-09 21:33:13 INFO DAGScheduler:54 - Missing parents: List()
2018-09-09 21:33:14 INFO DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
2018-09-09 21:33:15 INFO MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 1832.0 B, free 413.9 MB)
2018-09-09 21:33:16 INFO MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 1181.0 B, free 413.9 MB)
2018-09-09 21:33:16 INFO BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on master:40542 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:33:16 INFO SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1039
2018-09-09 21:33:16 INFO DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
2018-09-09 21:33:16 INFO TaskSchedulerImpl:54 - Adding task set 0.0 with 2 tasks
2018-09-09 21:33:19 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.11.129:49616) with ID 0
2018-09-09 21:33:19 INFO TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, 192.168.11.129, executor 0, partition 0, PROCESS_LOCAL, 7857 bytes)
2018-09-09 21:33:20 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.11.130:40952) with ID 1
2018-09-09 21:33:20 INFO TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, 192.168.11.130, executor 1, partition 1, PROCESS_LOCAL, 7857 bytes)
2018-09-09 21:33:20 INFO BlockManagerMasterEndpoint:54 - Registering block manager 192.168.11.129:34978 with 413.9 MB RAM, BlockManagerId(0, 192.168.11.129, 34978, None)
2018-09-09 21:33:20 INFO BlockManagerMasterEndpoint:54 - Registering block manager 192.168.11.130:44746 with 413.9 MB RAM, BlockManagerId(1, 192.168.11.130, 44746, None)
2018-09-09 21:33:23 INFO BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on 192.168.11.129:34978 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:33:23 INFO BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on 192.168.11.130:44746 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:33:24 INFO TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 4487 ms on 192.168.11.129 (executor 0) (1/2)
2018-09-09 21:33:25 INFO TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 5012 ms on 192.168.11.130 (executor 1) (2/2)
2018-09-09 21:33:25 INFO TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool
2018-09-09 21:33:25 INFO DAGScheduler:54 - ResultStage 0 (reduce at SparkPi.scala:38) finished in 10.560 s
2018-09-09 21:33:25 INFO DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 11.549968 s
Pi is roughly 3.1398956994784974
2018-09-09 21:33:25 INFO AbstractConnector:318 - Stopped [email protected]{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:33:25 INFO SparkUI:54 - Stopped Spark web UI at http://master:4040
2018-09-09 21:33:25 INFO StandaloneSchedulerBackend:54 - Shutting down all executors
2018-09-09 21:33:25 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Asking each executor to shut down
2018-09-09 21:33:25 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-09-09 21:33:25 INFO MemoryStore:54 - MemoryStore cleared
2018-09-09 21:33:25 INFO BlockManager:54 - BlockManager stopped
2018-09-09 21:33:25 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2018-09-09 21:33:25 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-09-09 21:33:25 INFO SparkContext:54 - Successfully stopped SparkContext
2018-09-09 21:33:25 INFO ShutdownHookManager:54 - Shutdown hook called
2018-09-09 21:33:25 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-ca417e29-82e1-4084-b518-46fde5110317
2018-09-09 21:33:25 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-bc263b74-64a0-426c-840c-74c6084d131a
[[email protected] sbin]#
4.3 用yarn-client模式执行计算程序
这种模式也叫On-Yarn模式，主要包括yarn-Client和yarn-Cluster两种模式。在这种模式下提交任务，需要先启动Hadoop集群，然后再启动Spark集群。
[[email protected] sbin]# spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client /usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar
2018-09-09 21:51:06 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-09 21:51:06 INFO SparkContext:54 - Running Spark version 2.3.1
2018-09-09 21:51:06 INFO SparkContext:54 - Submitted application: Spark Pi
2018-09-09 21:51:07 INFO SecurityManager:54 - Changing view acls to: root
2018-09-09 21:51:07 INFO SecurityManager:54 - Changing modify acls to: root
2018-09-09 21:51:07 INFO SecurityManager:54 - Changing view acls groups to:
2018-09-09 21:51:07 INFO SecurityManager:54 - Changing modify acls groups to:
2018-09-09 21:51:07 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
2018-09-09 21:51:07 INFO Utils:54 - Successfully started service 'sparkDriver' on port 37707.
2018-09-09 21:51:07 INFO SparkEnv:54 - Registering MapOutputTracker
2018-09-09 21:51:08 INFO SparkEnv:54 - Registering BlockManagerMaster
2018-09-09 21:51:08 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-09-09 21:51:08 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-09-09 21:51:08 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-f4bcc309-8d7a-4238-bcfa-5587921eec3b
2018-09-09 21:51:08 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2018-09-09 21:51:08 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2018-09-09 21:51:09 INFO log:192 - Logging initialized @7824ms
2018-09-09 21:51:09 INFO Server:346 - jetty-9.3.z-SNAPSHOT
2018-09-09 21:51:09 INFO Server:414 - Started @8272ms
2018-09-09 21:51:09 INFO AbstractConnector:278 - Started [email protected]{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:51:09 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/jobs,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/jobs/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/jobs/job,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/jobs/job/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/stages,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/stages/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]04c{/stages/stage,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/stages/stage/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/stages/pool,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/stages/pool/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/storage,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/storage/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/storage/rdd,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/environment,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/environment/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/executors,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/executors/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/executors/threadDump,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/static,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/api,null,AVAILABLE,@Spark}
2018-09-09 21:51:09 INFO ContextHandler:781 - Started [email protected]{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-09-09 21:51:10 INFO ContextHandler:781 - Started [email protected]{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-09-09 21:51:10 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
2018-09-09 21:51:10 INFO SparkContext:54 - Added JAR file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar at spark://master:37707/jars/spark-examples_2.11-2.3.1.jar with timestamp 1536501070145
2018-09-09 21:51:13 INFO RMProxy:98 - Connecting to ResourceManager at master/192.168.11.128:8032
2018-09-09 21:51:14 INFO Client:54 - Requesting a new application from cluster with 2 NodeManagers
2018-09-09 21:51:14 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2018-09-09 21:51:14 INFO Client:54 - Will allocate AM container, with 896 MB memory including 384 MB overhead
2018-09-09 21:51:14 INFO Client:54 - Setting up container launch context for our AM
2018-09-09 21:51:15 INFO Client:54 - Setting up the launch environment for our AM container
2018-09-09 21:51:15 INFO Client:54 - Preparing resources for our AM container
2018-09-09 21:51:20 WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2018-09-09 21:51:52 INFO Client:54 - Uploading resource file:/tmp/spark-4adf63c4-82a7-4cf7-bf6f-0abf78282f74/__spark_libs__7942925809039923414.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1536498065211_0002/__spark_libs__7942925809039923414.zip
2018-09-09 21:52:39 INFO Client:54 - Uploading resource file:/tmp/spark-4adf63c4-82a7-4cf7-bf6f-0abf78282f74/__spark_conf__6224445360548244890.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1536498065211_0002/__spark_conf__.zip
2018-09-09 21:52:40 INFO SecurityManager:54 - Changing view acls to: root
2018-09-09 21:52:40 INFO SecurityManager:54 - Changing modify acls to: root
2018-09-09 21:52:40 INFO SecurityManager:54 - Changing view acls groups to:
2018-09-09 21:52:40 INFO SecurityManager:54 - Changing modify acls groups to:
2018-09-09 21:52:40 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
2018-09-09 21:52:40 INFO Client:54 - Submitting application application_1536498065211_0002 to ResourceManager
2018-09-09 21:52:40 INFO YarnClientImpl:273 - Submitted application application_1536498065211_0002
2018-09-09 21:52:40 INFO SchedulerExtensionServices:54 - Starting Yarn extension services with app application_1536498065211_0002 and attemptId None
2018-09-09 21:52:41 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:41 INFO Client:54 -
client token: N/A
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1536501160345
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1536498065211_0002/
user: root
2018-09-09 21:52:42 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:43 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:44 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:45 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:46 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:47 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:48 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:49 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:50 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:51 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:52 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:53 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:54 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:55 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:56 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:57 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:58 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:52:59 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:00 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:01 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:02 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:03 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:04 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:05 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:06 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:07 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:08 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:09 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:10 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:11 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:12 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:13 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:14 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:15 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:16 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:17 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:18 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:19 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:20 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:21 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:22 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:23 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:24 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:25 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:26 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:27 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:28 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:29 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:30 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:31 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:32 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:33 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:34 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:35 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:36 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:37 INFO YarnClientSchedulerBackend:54 - Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> master, PROXY_URI_BASES -> http://master:8088/proxy/application_1536498065211_0002), /proxy/application_1536498065211_0002
2018-09-09 21:53:37 INFO JettyUtils:54 - Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2018-09-09 21:53:37 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:38 INFO Client:54 - Application report for application_1536498065211_0002 (state: ACCEPTED)
2018-09-09 21:53:39 INFO YarnSchedulerBackend$YarnSchedulerEndpoint:54 - ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
2018-09-09 21:53:39 INFO Client:54 - Application report for application_1536498065211_0002 (state: RUNNING)
2018-09-09 21:53:39 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.11.129
ApplicationMaster RPC port: 0
queue: default
start time: 1536501160345
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1536498065211_0002/
user: root
2018-09-09 21:53:39 INFO YarnClientSchedulerBackend:54 - Application application_1536498065211_0002 has started running.
2018-09-09 21:53:39 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39617.
2018-09-09 21:53:39 INFO NettyBlockTransferService:54 - Server created on master:39617
2018-09-09 21:53:39 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-09-09 21:53:40 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, master, 39617, None)
2018-09-09 21:53:40 INFO BlockManagerMasterEndpoint:54 - Registering block manager master:39617 with 413.9 MB RAM, BlockManagerId(driver, master, 39617, None)
2018-09-09 21:53:40 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, master, 39617, None)
2018-09-09 21:53:40 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, master, 39617, None)
2018-09-09 21:53:40 INFO ContextHandler:781 - Started [email protected]{/metrics/json,null,AVAILABLE,@Spark}
2018-09-09 21:53:41 INFO YarnClientSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
2018-09-09 21:54:01 INFO SparkContext:54 - Starting job: reduce at SparkPi.scala:38
2018-09-09 21:54:01 INFO DAGScheduler:54 - Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
2018-09-09 21:54:01 INFO DAGScheduler:54 - Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
2018-09-09 21:54:01 INFO DAGScheduler:54 - Parents of final stage: List()
2018-09-09 21:54:01 INFO DAGScheduler:54 - Missing parents: List()
2018-09-09 21:54:01 INFO DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
2018-09-09 21:54:14 INFO MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 1832.0 B, free 413.9 MB)
2018-09-09 21:54:14 INFO MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 1181.0 B, free 413.9 MB)
2018-09-09 21:54:14 INFO BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on master:39617 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:54:14 INFO SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1039
2018-09-09 21:54:14 INFO DAGScheduler:54 - Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
2018-09-09 21:54:14 INFO YarnScheduler:54 - Adding task set 0.0 with 2 tasks
2018-09-09 21:54:32 WARN YarnScheduler:66 - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2018-09-09 21:54:44 WARN YarnScheduler:66 - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2018-09-09 21:54:46 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.11.129:59958) with ID 2
2018-09-09 21:54:46 INFO TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, slave1, executor 2, partition 0, PROCESS_LOCAL, 7864 bytes)
2018-09-09 21:54:46 INFO BlockManagerMasterEndpoint:54 - Registering block manager slave1:39770 with 413.9 MB RAM, BlockManagerId(2, slave1, 39770, None)
2018-09-09 21:54:48 INFO BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on slave1:39770 (size: 1181.0 B, free: 413.9 MB)
2018-09-09 21:54:50 INFO TaskSetManager:54 - Starting task 1.0 in stage 0.0 (TID 1, slave1, executor 2, partition 1, PROCESS_LOCAL, 7864 bytes)
2018-09-09 21:54:50 INFO TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 3931 ms on slave1 (executor 2) (1/2)
2018-09-09 21:54:50 INFO TaskSetManager:54 - Finished task 1.0 in stage 0.0 (TID 1) in 338 ms on slave1 (executor 2) (2/2)
2018-09-09 21:54:50 INFO YarnScheduler:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool
2018-09-09 21:54:50 INFO DAGScheduler:54 - ResultStage 0 (reduce at SparkPi.scala:38) finished in 48.440 s
2018-09-09 21:54:50 INFO DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 48.819130 s
Pi is roughly 3.144115720578603
2018-09-09 21:54:50 INFO AbstractConnector:318 - Stopped [email protected]{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-09-09 21:54:50 INFO SparkUI:54 - Stopped Spark web UI at http://master:4040
2018-09-09 21:54:50 INFO YarnClientSchedulerBackend:54 - Interrupting monitor thread
2018-09-09 21:54:50 INFO YarnClientSchedulerBackend:54 - Shutting down all executors
2018-09-09 21:54:50 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Asking each executor to shut down
2018-09-09 21:54:51 INFO SchedulerExtensionServices:54 - Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
2018-09-09 21:54:51 INFO YarnClientSchedulerBackend:54 - Stopped
2018-09-09 21:54:51 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-09-09 21:54:51 INFO MemoryStore:54 - MemoryStore cleared
2018-09-09 21:54:51 INFO BlockManager:54 - BlockManager stopped
2018-09-09 21:54:51 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2018-09-09 21:54:51 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-09-09 21:54:51 INFO SparkContext:54 - Successfully stopped SparkContext
2018-09-09 21:54:51 INFO ShutdownHookManager:54 - Shutdown hook called
2018-09-09 21:54:51 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-4adf63c4-82a7-4cf7-bf6f-0abf78282f74
2018-09-09 21:54:51 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-f085e710-596c-4c23-b436-c879d61d4a01
[[email protected] sbin]#
4.4 用yarn-cluster模式执行计算程序
[[email protected] sbin]# spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar
2018-09-09 22:00:59 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-09 22:01:03 INFO RMProxy:98 - Connecting to ResourceManager at master/192.168.11.128:8032
2018-09-09 22:01:04 INFO Client:54 - Requesting a new application from cluster with 2 NodeManagers
2018-09-09 22:01:04 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2018-09-09 22:01:04 INFO Client:54 - Will allocate AM container, with 1408 MB memory including 384 MB overhead
2018-09-09 22:01:04 INFO Client:54 - Setting up container launch context for our AM
2018-09-09 22:01:04 INFO Client:54 - Setting up the launch environment for our AM container
2018-09-09 22:01:04 INFO Client:54 - Preparing resources for our AM container
2018-09-09 22:01:09 WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2018-09-09 22:01:21 INFO Client:54 - Uploading resource file:/tmp/spark-351d275a-b939-41da-ba44-35f2eba3ba25/__spark_libs__624052493228396690.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1536498065211_0003/__spark_libs__624052493228396690.zip
2018-09-09 22:02:25 INFO Client:54 - Uploading resource file:/usr/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar -> hdfs://master:9000/user/root/.sparkStaging/application_1536498065211_0003/spark-examples_2.11-2.3.1.jar
2018-09-09 22:02:40 INFO Client:54 - Uploading resource file:/tmp/spark-351d275a-b939-41da-ba44-35f2eba3ba25/__spark_conf__5330471617982538290.zip -> hdfs://master:9000/user/root/.sparkStaging/application_1536498065211_0003/__spark_conf__.zip
2018-09-09 22:02:40 INFO SecurityManager:54 - Changing view acls to: root
2018-09-09 22:02:40 INFO SecurityManager:54 - Changing modify acls to: root
2018-09-09 22:02:40 INFO SecurityManager:54 - Changing view acls groups to:
2018-09-09 22:02:40 INFO SecurityManager:54 - Changing modify acls groups to:
2018-09-09 22:02:40 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
2018-09-09 22:02:40 INFO Client:54 - Submitting application application_1536498065211_0003 to ResourceManager
2018-09-09 22:02:40 INFO YarnClientImpl:273 - Submitted application application_1536498065211_0003
2018-09-09 22:02:41 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:41 INFO Client:54 -
client token: N/A
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1536501760869
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1536498065211_0003/
user: root
2018-09-09 22:02:42 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:43 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:44 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:46 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:47 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:48 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:49 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:50 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:51 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:52 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:53 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:54 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:55 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:56 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:57 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:58 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:02:59 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:00 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:01 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:02 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:03 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:04 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:05 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:06 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:07 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:08 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:09 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:10 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:11 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:12 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:13 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:14 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:15 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:16 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:17 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:18 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:19 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:20 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:21 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:22 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:23 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:24 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:25 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:26 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:27 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:28 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:29 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:30 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:31 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:32 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:33 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:34 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:35 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:36 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:37 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:38 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:39 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:40 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:41 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:42 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:43 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:44 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:45 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:46 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:47 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:48 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:49 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:50 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:51 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:52 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:53 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:54 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:55 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:56 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:57 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:58 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:03:59 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:00 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:01 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:02 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:03 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:04 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:05 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:06 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:07 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:08 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:09 INFO Client:54 - Application report for application_1536498065211_0003 (state: ACCEPTED)
2018-09-09 22:04:10 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:10 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.11.130
ApplicationMaster RPC port: 0
queue: default
start time: 1536501760869
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1536498065211_0003/
user: root
2018-09-09 22:04:11 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:12 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:13 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:14 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:15 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:16 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:17 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:18 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:19 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:20 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:21 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:22 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:23 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:24 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:25 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:26 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:27 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:28 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:29 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:30 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:31 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:32 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:33 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:34 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:35 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:36 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:37 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:38 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:39 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:40 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:41 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:42 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:43 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:46 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:47 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:48 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:49 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:50 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:51 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:53 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:54 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:55 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:56 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:57 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:58 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:04:59 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:00 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:01 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:02 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:03 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:04 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:05 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:06 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:07 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:08 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:10 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:11 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:12 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:13 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:14 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:15 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:16 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:17 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:18 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:19 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:20 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:21 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:22 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:23 INFO Client:54 - Application report for application_1536498065211_0003 (state: RUNNING)
2018-09-09 22:05:24 INFO Client:54 - Application report for application_1536498065211_0003 (state: FINISHED)
2018-09-09 22:05:24 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.11.130
ApplicationMaster RPC port: 0
queue: default
start time: 1536501760869
final status: SUCCEEDED
tracking URL: http://master:8088/proxy/application_1536498065211_0003/
user: root
2018-09-09 22:05:24 INFO ShutdownHookManager:54 - Shutdown hook called
2018-09-09 22:05:24 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-0f174e46-416d-45dd-bb59-51c652b4a70d
2018-09-09 22:05:24 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-351d275a-b939-41da-ba44-35f2eba3ba25
[[email protected] sbin]#
使用yarn-cluster模式计算，结果没有输出在控制台，结果写在了Hadoop集群的日志中
点击红框application id
点击Logs
点击stdout