Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

1 安装Spark依赖的Scala

          1.1下载和解压缩Scala

          1.2 配置环境变量

          1.3 验证Scala

2下载和解压缩Spark

          2.1 下载Spark压缩包

          2.2 解压缩Spark

3 Spark相关的配置

          3.1 配置环境变量

          3.2 配置conf目录下的文件

                  3.2.1 新建spark-env.h文件

                  3.2.2 新建slaves文件

4 启动和测试Spark集群

         4.1 启动Spark

         4.2 测试和使用Spark集群

                 4.2.1访问Spark集群提供的URL

                 4.2.2运行Spark提供的计算圆周率的示例程序

 

关键字：Linux   CentOS  Hadoop  Spark   Scala   Java

版本号：CentOS7  Hadoop2.8.0  Spark2.1.1   Scala2.12.2   JDK1.8

 

          说明：Spark可以在只安装了JDK、Scala的机器上直接单机安装，但是这样的话只能使用单机模式运行不涉及分布式运算和分布式存储的代码，例如可以单机安装Spark，单机运行计算圆周率的Spark程序。但是我们要运行的是Spark集群，并且需要调用Hadoop的分布式文件系统，所以请你先安装Hadoop

 

在这里相信有许多想要学习大数据的同学，大家可以+下大数据学习裙：716加上【五8一】最后014，即可免费领取一整套系统的大数据学习教程

     

        Spark集群的最小化安装只需要安装这些东西：JDK  、Scala  、Hadoop  、Spark

1 安装Spark依赖的Scala

          Hadoop的安装请参考上面提到的博文，因为Spark依赖scala，所以在安装Spark之前，这里要先安装scala。在每个节点上都进行安装。

1.1  下载和解压缩Scala

        如图：

Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

 

如图：

      Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

 

 

 

在Linux服务器的opt目录下新建一个名为scala的文件夹，并将下载的压缩包上载上去

如图：

   Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

   

执行命令，进入到该目录：

cd    /opt/scala

执行命令进行解压缩：

tar   -xvf    scala-2.12.2

1.2  配置环境变量

       编辑/etc/profile这个文件，在文件中增加一行配置：

 

export    SCALA_HOME=/opt/scala/scala-2.12.2

 

      在该文件的PATH变量中增加下面的内容：

 

  ${SCALA_HOME}/bin

 

 

      添加完成后，我的/etc/profile的配置如下：

 

 

  1. export JAVA_HOME=/opt/java/jdk1.8.0_121

  2. export HADOOP_HOME=/opt/hadoop/hadoop-2.8.0

  3. export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

  4. export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native

  5. export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib"

  6. export HIVE_HOME=/opt/hive/apache-hive-2.1.1-bin

  7. export HIVE_CONF_DIR=${HIVE_HOME}/conf

  8. export SQOOP_HOME=/opt/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha

  9. export HBASE_HOME=/opt/hbase/hbase-1.2.5

  10. export ZK_HOME=/opt/zookeeper/zookeeper-3.4.10

  11. export SCALA_HOME=/opt/scala/scala-2.12.2

  12. export CLASS_PATH=.:${JAVA_HOME}/lib:${HIVE_HOME}/lib:$CLASS_PATH

  13. export PATH=.:${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${SPARK_HOME}/bin:${ZK_HOME}/bin:${HIVE_HOME}/bin:${SQOOP_HOME}/bin:${HBASE_HOME}/bin:${SCALA_HOME}/bin:$PATH

         说明：你可以只关注开头说的JDK  SCALA   Hadoop  Spark的环境变量，其余的诸如Zookeeper、Hbase、Hive、Sqoop都不用管。

        如图：

Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

 

      环境变量配置完成后，执行下面的命令：

 

source   /etc/profile

 

1.3 验证Scala

    执行命令：

 

scala     -version

 

   如图：

    Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

2 下载和解压缩Spark

      在每个节点上都安装Spark，也就是重复下面的步骤。

 

  如图：

        Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

 

     

2.2  解压缩Spark

      在Linux服务器的opt目录下新建一个名为spark的文件夹，把刚才下载的压缩包，上载上去。

如图：

Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

进入到该目录内，也就是执行下面的命令：

cd    /opt/spark

执行解压缩命令：

tar   -zxvf   spark-2.1.1-bin-hadoop2.7.tgz

3  Spark相关的配置

         说明：因为我们搭建的是基于hadoop集群的Spark集群，所以每个hadoop节点上我都安装了Spark，都需要按照下面的步骤做配置，启动的话只需要在Spark集群的Master机器上启动即可，我这里是在hserver1上启动。

3.1  配置环境变量

编辑/etc/profile文件，增加

 

export  SPARK_HOME=/opt/spark/spark-2.1.1-bin-hadoop2.7

 

      上面的变量添加完成后编辑该文件中的PATH变量，添加

 

${SPARK_HOME}/bin

 

      注意：因为$SPARK_HOME/sbin目录下有一些文件名称和$HADOOP_HOME/sbin目录下的文件同名，为了避免同名文件冲突，这里不在PATH变量里添加$SPARK_HOME/sbin只添加了$SPARK_HOME/bin。

修改完成后，我的/etc/profile文件内容是：

 

 

  1. export JAVA_HOME=/opt/java/jdk1.8.0_121

  2. export HADOOP_HOME=/opt/hadoop/hadoop-2.8.0

  3. export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

  4. export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native

  5. export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib"

  6. export HIVE_HOME=/opt/hive/apache-hive-2.1.1-bin

  7. export HIVE_CONF_DIR=${HIVE_HOME}/conf

  8. export SQOOP_HOME=/opt/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha

  9. export HBASE_HOME=/opt/hbase/hbase-1.2.5

  10. export ZK_HOME=/opt/zookeeper/zookeeper-3.4.10

  11. export SCALA_HOME=/opt/scala/scala-2.12.2

  12. export SPARK_HOME=/opt/spark/spark-2.1.1-bin-hadoop2.7

  13. export CLASS_PATH=.:${JAVA_HOME}/lib:${HIVE_HOME}/lib:$CLASS_PATH

  14. export PATH=.:${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${SPARK_HOME}/bin:${ZK_HOME}/bin:${HIVE_HOME}/bin:${SQOOP_HOME}/bin:${HBASE_HOME}:${SCALA_HOME}/bin:$PATH

 

      说明：你可以只关注开头说的JDK  SCALA   Hadoop  Spark的环境变量，其余的诸如Zookeeper、Hbase、Hive、Sqoop都不用管。 

      如图：

Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

编辑完成后，执行命令：

source    /etc/profile

3.2 配置conf目录下的文件

         对/opt/spark/spark-2.1.1-bin-hadoop2.7/conf目录下的文件进行配置。

3.2.1  新建spark-env.h文件

        执行命令，进入到/opt/spark/spark-2.1.1-bin-hadoop2.7/conf目录内：

cd    /opt/spark/spark-2.1.1-bin-hadoop2.7/conf

       以spark为我们创建好的模板创建一个spark-env.h文件，命令是：

cp    spark-env.sh.template   spark-env.sh

     如图：

Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

    编辑spark-env.h文件，在里面加入配置(具体路径以自己的为准)：

 

 

  1. export SCALA_HOME=/opt/scala/scala-2.12.2

  2. export JAVA_HOME=/opt/java/jdk1.8.0_121

  3. export HADOOP_HOME=/opt/hadoop/hadoop-2.8.0

  4. export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

  5. export SPARK_HOME=/opt/spark/spark-2.1.1-bin-hadoop2.7

  6. export SPARK_MASTER_IP=hserver1

  7. export SPARK_EXECUTOR_MEMORY=1G

 

3.2.2 新建slaves文件

执行命令，进入到/opt/spark/spark-2.1.1-bin-hadoop2.7/conf目录内：

cd   /opt/spark/spark-2.1.1-bin-hadoop2.7/conf

以spark为我们创建好的模板创建一个slaves文件，命令是：

cp    slaves.template   slaves

如图：

Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

编辑slaves文件，里面的内容为：

 

hserver2
hserver3

 

4 启动和测试Spark集群

4.1 启动Spark

          因为spark是依赖于hadoop提供的分布式文件系统的，所以在启动spark之前，先确保hadoop在正常运行。

        在hadoop正常运行的情况下，在hserver1（也就是hadoop的namenode，spark的marster节点）上执行命令：

   cd   /opt/spark/spark-2.1.1-bin-hadoop2.7/sbin

    执行启动脚本：

  ./start-all.sh

  如图：

Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

 

  完整内容是：

 

 

  1. [[email protected] sbin]# cd/opt/spark/spark-2.1.1-bin-hadoop2.7/sbin

  2. [[email protected] sbin]# ./start-all.sh

  3. starting org.apache.spark.deploy.master.Master,logging to/opt/spark/spark-2.1.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.master.Master-1-hserver1.out

  4. hserver2: startingorg.apache.spark.deploy.worker.Worker, logging to/opt/spark/spark-2.1.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hserver2.out

  5. hserver3: startingorg.apache.spark.deploy.worker.Worker, logging to/opt/spark/spark-2.1.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hserver3.out

  6. [[email protected] sbin]#

 

         注意：上面的命令中有./这个不能少，./的意思是执行当前目录下的start-all.sh脚本。

4.2  测试和使用Spark集群

4.2.1  访问Spark集群提供的URL

  

       如图：

Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

4.2.2  运行Spark提供的计算圆周率的示例程序

      这里只是简单的用local模式运行一个计算圆周率的Demo。按照下面的步骤来操作。

      第一步，进入到Spark的根目录，也就是执行下面的脚本：

cd     /opt/spark/spark-2.1.1-bin-hadoop2.7

如图：

        Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

第二步，调用Spark自带的计算圆周率的Demo，执行下面的命令：

 

./bin/spark-submit  --class  org.apache.spark.examples.SparkPi  --master local   examples/jars/spark-examples_2.11-2.1.1.jar

 

 

        命令执行后，spark示例程序已经开始执行

        如图：

Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

        很快执行结果出来了，执行结果我用红框标出来了

如图：

Linux安装Spark集群(CentOS7+Spark2.1.1+Hadoop2.8.0)

        完整的控制台输出是：

 

 

  1. [[email protected] bin]# cd /opt/spark/spark-2.1.1-bin-hadoop2.7

  2. [[email protected] spark-2.1.1-bin-hadoop2.7]# ./bin/spark-submit--class org.apache.spark.examples.SparkPi --master localexamples/jars/spark-examples_2.11-2.1.1.jar

  3. 17/05/16 14:26:23 INFO spark.SparkContext: Running Spark version2.1.1

  4. 17/05/16 14:26:24 WARN util.NativeCodeLoader: Unable to loadnative-hadoop library for your platform... using builtin-java classes whereapplicable

  5. 17/05/16 14:26:25 INFO spark.SecurityManager: Changing view acls to:root

  6. 17/05/16 14:26:25 INFO spark.SecurityManager: Changing modify aclsto: root

  7. 17/05/16 14:26:25 INFO spark.SecurityManager: Changing view aclsgroups to:

  8. 17/05/16 14:26:25 INFO spark.SecurityManager: Changing modify aclsgroups to:

  9. 17/05/16 14:26:25 INFO spark.SecurityManager: SecurityManager:authentication disabled; ui acls disabled; users with view permissions: Set(root); groups withview permissions: Set(); users withmodify permissions: Set(root); groups with modify permissions: Set()

  10. 17/05/16 14:26:25 INFO util.Utils: Successfully started service'sparkDriver' on port 40855.

  11. 17/05/16 14:26:26 INFO spark.SparkEnv: Registering MapOutputTracker

  12. 17/05/16 14:26:26 INFO spark.SparkEnv: RegisteringBlockManagerMaster

  13. 17/05/16 14:26:26 INFO storage.BlockManagerMasterEndpoint: Usingorg.apache.spark.storage.DefaultTopologyMapper for getting topology information

  14. 17/05/16 14:26:26 INFO storage.BlockManagerMasterEndpoint:BlockManagerMasterEndpoint up

  15. 17/05/16 14:26:26 INFO storage.DiskBlockManager: Created localdirectory at /tmp/blockmgr-cf8cbb42-95d2-4284-9a48-67592363976a

  16. 17/05/16 14:26:26 INFO memory.MemoryStore: MemoryStore started withcapacity 413.9 MB

  17. 17/05/16 14:26:26 INFO spark.SparkEnv: RegisteringOutputCommitCoordinator

  18. 17/05/16 14:26:26 INFO util.log: Logging initialized @5206ms

  19. 17/05/16 14:26:27 INFO server.Server: jetty-9.2.z-SNAPSHOT

  20. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/jobs,null,AVAILABLE,@Spark}

  21. 17/05/16 14:26:27 INFO handler.ContextHandler: Started [email protected]{/jobs/json,null,AVAILABLE,@Spark}

  22. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/jobs/job,null,AVAILABLE,@Spark}

  23. 17/05/16 14:26:27 INFO handler.ContextHandler: Started [email protected]{/jobs/job/json,null,AVAILABLE,@Spark}

  24. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/stages,null,AVAILABLE,@Spark}

  25. 17/05/16 14:26:27 INFO handler.ContextHandler: Started [email protected]{/stages/json,null,AVAILABLE,@Spark}

  26. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/stages/stage,null,AVAILABLE,@Spark}

  27. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/stages/stage/json,null,AVAILABLE,@Spark}

  28. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/stages/pool,null,AVAILABLE,@Spark}

  29. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/stages/pool/json,null,AVAILABLE,@Spark}

  30. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/storage,null,AVAILABLE,@Spark}

  31. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/storage/json,null,AVAILABLE,@Spark}

  32. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/storage/rdd,null,AVAILABLE,@Spark}

  33. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/storage/rdd/json,null,AVAILABLE,@Spark}

  34. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/environment,null,AVAILABLE,@Spark}

  35. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/environment/json,null,AVAILABLE,@Spark}

  36. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/executors,null,AVAILABLE,@Spark}

  37. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/executors/json,null,AVAILABLE,@Spark}

  38. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/executors/threadDump,null,AVAILABLE,@Spark}

  39. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/executors/threadDump/json,null,AVAILABLE,@Spark}

  40. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/static,null,AVAILABLE,@Spark}

  41. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/,null,AVAILABLE,@Spark}

  42. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/api,null,AVAILABLE,@Spark}

  43. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/jobs/job/kill,null,AVAILABLE,@Spark}

  44. 17/05/16 14:26:27 INFO handler.ContextHandler: [email protected]{/stages/stage/kill,null,AVAILABLE,@Spark}

  45. 17/05/16 14:26:27 INFO server.ServerConnector: [email protected]{HTTP/1.1}{0.0.0.0:4040}

  46. 17/05/16 14:26:27 INFO server.Server: Started @5838ms

  47. 17/05/16 14:26:27 INFO util.Utils: Successfully started service'SparkUI' on port 4040.

  48. 17/05/16 14:26:27 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, andstarted at http://192.168.27.143:4040

  49. 17/05/16 14:26:27 INFO spark.SparkContext: Added JARfile:/opt/spark/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jarat spark://192.168.27.143:40855/jars/spark-examples_2.11-2.1.1.jar withtimestamp 1494915987472

  50. 17/05/16 14:26:27 INFO executor.Executor: Starting executor IDdriver on host localhost

  51. 17/05/16 14:26:27 INFO util.Utils: Successfully started service'org.apache.spark.network.netty.NettyBlockTransferService' on port 41104.

  52. 17/05/16 14:26:27 INFO netty.NettyBlockTransferService: Servercreated on 192.168.27.143:41104

  53. 17/05/16 14:26:27 INFO storage.BlockManager: Usingorg.apache.spark.storage.RandomBlockReplicationPolicy for block replicationpolicy

  54. 17/05/16 14:26:27 INFO storage.BlockManagerMaster: RegisteringBlockManager BlockManagerId(driver, 192.168.27.143, 41104, None)

  55. 17/05/16 14:26:27 INFO storage.BlockManagerMasterEndpoint:Registering block manager 192.168.27.143:41104 with 413.9 MB RAM,BlockManagerId(driver, 192.168.27.143, 41104, None)

  56. 17/05/16 14:26:27 INFO storage.BlockManagerMaster: RegisteredBlockManager BlockManagerId(driver, 192.168.27.143, 41104, None)

  57. 17/05/16 14:26:27 INFO storage.BlockManager: InitializedBlockManager: BlockManagerId(driver, 192.168.27.143, 41104, None)

  58. 17/05/16 14:26:28 INFO handler.ContextHandler: [email protected]{/metrics/json,null,AVAILABLE,@Spark}

  59. 17/05/16 14:26:28 INFO internal.SharedState: Warehouse path is'file:/opt/spark/spark-2.1.1-bin-hadoop2.7/spark-warehouse'.

  60. 17/05/16 14:26:28 INFO handler.ContextHandler: [email protected]{/SQL,null,AVAILABLE,@Spark}

  61. 17/05/16 14:26:28 INFO handler.ContextHandler: Started [email protected]{/SQL/json,null,AVAILABLE,@Spark}

  62. 17/05/16 14:26:28 INFO handler.ContextHandler: [email protected]{/SQL/execution,null,AVAILABLE,@Spark}

  63. 17/05/16 14:26:28 INFO handler.ContextHandler: Started [email protected]{/SQL/execution/json,null,AVAILABLE,@Spark}

  64. 17/05/16 14:26:28 INFO handler.ContextHandler: [email protected]{/static/sql,null,AVAILABLE,@Spark}

  65. 17/05/16 14:26:30 INFO spark.SparkContext: Starting job: reduce atSparkPi.scala:38

  66. 17/05/16 14:26:30 INFO scheduler.DAGScheduler: Got job 0 (reduce atSparkPi.scala:38) with 2 output partitions

  67. 17/05/16 14:26:30 INFO scheduler.DAGScheduler: Final stage:ResultStage 0 (reduce at SparkPi.scala:38)

  68. 17/05/16 14:26:30 INFO scheduler.DAGScheduler: Parents of finalstage: List()

  69. 17/05/16 14:26:30 INFO scheduler.DAGScheduler: Missing parents:List()

  70. 17/05/16 14:26:30 INFO scheduler.DAGScheduler: SubmittingResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has nomissing parents

  71. 17/05/16 14:26:30 INFO memory.MemoryStore: Block broadcast_0 storedas values in memory (estimated size 1832.0 B, free 413.9 MB)

  72. 17/05/16 14:26:30 INFO memory.MemoryStore: Block broadcast_0_piece0stored as bytes in memory (estimated size 1167.0 B, free 413.9 MB)

  73. 17/05/16 14:26:31 INFO storage.BlockManagerInfo: Addedbroadcast_0_piece0 in memory on 192.168.27.143:41104 (size: 1167.0 B, free:413.9 MB)

  74. 17/05/16 14:26:31 INFO spark.SparkContext: Created broadcast 0 frombroadcast at DAGScheduler.scala:996

  75. 17/05/16 14:26:31 INFO scheduler.DAGScheduler: Submitting 2 missingtasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34)

  76. 17/05/16 14:26:31 INFO scheduler.TaskSchedulerImpl: Adding task set0.0 with 2 tasks

  77. 17/05/16 14:26:31 INFO scheduler.TaskSetManager: Starting task 0.0in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL,6026 bytes)

  78. 17/05/16 14:26:31 INFO executor.Executor: Running task 0.0 in stage0.0 (TID 0)

  79. 17/05/16 14:26:31 INFO executor.Executor: Fetchingspark://192.168.27.143:40855/jars/spark-examples_2.11-2.1.1.jar with timestamp1494915987472

  80. 17/05/16 14:26:31 INFO client.TransportClientFactory: Successfullycreated connection to /192.168.27.143:40855 after 145 ms (0 ms spent inbootstraps)

  81. 17/05/16 14:26:31 INFO util.Utils: Fetchingspark://192.168.27.143:40855/jars/spark-examples_2.11-2.1.1.jar to/tmp/spark-702c8654-489f-47f2-85e0-8b658ebb2988/userFiles-0a07fa86-4d14-4939-ad2b-95ac8488e187/fetchFileTemp3302336691796081023.tmp

  82. 17/05/16 14:26:33 INFO executor.Executor: Addingfile:/tmp/spark-702c8654-489f-47f2-85e0-8b658ebb2988/userFiles-0a07fa86-4d14-4939-ad2b-95ac8488e187/spark-examples_2.11-2.1.1.jarto class loader

  83. 17/05/16 14:26:34 INFO executor.Executor: Finished task 0.0 in stage0.0 (TID 0). 1114 bytes result sent to driver

  84. 17/05/16 14:26:34 INFO scheduler.TaskSetManager: Starting task 1.0in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL,6026 bytes)

  85. 17/05/16 14:26:34 INFO executor.Executor: Running task 1.0 in stage0.0 (TID 1)

  86. 17/05/16 14:26:34 INFO scheduler.TaskSetManager: Finished task 0.0in stage 0.0 (TID 0) in 2815 ms on localhost (executor driver) (1/2)

  87. 17/05/16 14:26:34 INFO executor.Executor: Finished task 1.0 in stage0.0 (TID 1). 1114 bytes result sent to driver

  88. 17/05/16 14:26:34 INFO scheduler.TaskSetManager: Finished task 1.0in stage 0.0 (TID 1) in 416 ms on localhost (executor driver) (2/2)

  89. 17/05/16 14:26:34 INFO scheduler.TaskSchedulerImpl: Removed TaskSet0.0, whose tasks have all completed, from pool

  90. 17/05/16 14:26:34 INFO scheduler.DAGScheduler: ResultStage 0 (reduceat SparkPi.scala:38) finished in 3.269 s

  91. 17/05/16 14:26:34 INFO scheduler.DAGScheduler: Job 0 finished:reduce at SparkPi.scala:38, took 4.404894 s

  92. Pi is roughly 3.1434157170785855

  93. 17/05/16 14:26:34 INFO server.ServerConnector: [email protected]{HTTP/1.1}{0.0.0.0:4040}

  94. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/stages/stage/kill,null,UNAVAILABLE,@Spark}

  95. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/jobs/job/kill,null,UNAVAILABLE,@Spark}

  96. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/api,null,UNAVAILABLE,@Spark}

  97. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/,null,UNAVAILABLE,@Spark}

  98. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/static,null,UNAVAILABLE,@Spark}

  99. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/executors/threadDump/json,null,UNAVAILABLE,@Spark}

  100. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/executors/threadDump,null,UNAVAILABLE,@Spark}

  101. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/executors/json,null,UNAVAILABLE,@Spark}

  102. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/executors,null,UNAVAILABLE,@Spark}

  103. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/environment/json,null,UNAVAILABLE,@Spark}

  104. 17/05/16 14:26:34 INFO handler.ContextHandler: Stopped [email protected]{/environment,null,UNAVAILABLE,@Spark}

  105. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/storage/rdd/json,null,UNAVAILABLE,@Spark}

  106. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/storage/rdd,null,UNAVAILABLE,@Spark}

  107. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/storage/json,null,UNAVAILABLE,@Spark}

  108. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/storage,null,UNAVAILABLE,@Spark}

  109. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/stages/pool/json,null,UNAVAILABLE,@Spark}

  110. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/stages/pool,null,UNAVAILABLE,@Spark}

  111. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/stages/stage/json,null,UNAVAILABLE,@Spark}

  112. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/stages/stage,null,UNAVAILABLE,@Spark}

  113. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/stages/json,null,UNAVAILABLE,@Spark}

  114. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/stages,null,UNAVAILABLE,@Spark}

  115. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/jobs/job/json,null,UNAVAILABLE,@Spark}

  116. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/jobs/job,null,UNAVAILABLE,@Spark}

  117. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/jobs/json,null,UNAVAILABLE,@Spark}

  118. 17/05/16 14:26:34 INFO handler.ContextHandler: [email protected]{/jobs,null,UNAVAILABLE,@Spark}

  119. 17/05/16 14:26:34 INFO ui.SparkUI: Stopped Spark web UI athttp://192.168.27.143:4040

  120. 17/05/16 14:26:34 INFO spark.MapOutputTrackerMasterEndpoint:MapOutputTrackerMasterEndpoint stopped!

  121. 17/05/16 14:26:34 INFO memory.MemoryStore: MemoryStore cleared

  122. 17/05/16 14:26:34 INFO storage.BlockManager: BlockManager stopped

  123. 17/05/16 14:26:34 INFO storage.BlockManagerMaster: BlockManagerMasterstopped

  124. 17/05/16 14:26:34 INFOscheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!

  125. 17/05/16 14:26:34 INFO spark.SparkContext: Successfully stoppedSparkContext

  126. 17/05/16 14:26:34 INFO util.ShutdownHookManager: Shutdown hookcalled

  127. 17/05/16 14:26:34 INFO util.ShutdownHookManager: Deleting directory/tmp/spark-702c8654-489f-47f2-85e0-8b658ebb2988

  128. [[email protected] spark-2.1.1-bin-hadoop2.7]#

注意：上面只是使用了单机本地模式调用Demo，使用集群模式运行Demo

相关文章：

猜你喜欢
相关资源
相似解决方案
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode