如何将 Spark scala jar 部署/运行/提交到 HDInsight Azure 集群？答案

【问题标题】：How to deploy/run/submit Spark scala jars to HDInsight Azure cluster?如何将 Spark scala jar 部署/运行/提交到 HDInsight Azure 集群？
【发布时间】：2018-08-27 18:39:25
【问题描述】：

我想创建一个按需集群并将 spark-scala jar 提交到该集群，并在完成后终止该集群。在 AWS 中使用 command-runner.jar 非常简单如何为 Azure HDInsight 实现这一目标？

我不想要任何手动或通过 Azure UI 的东西。

【问题讨论】：

标签： apache-spark azure-hdinsight ambari

【解决方案1】：

几个月前我有同样的需求，并制作了以下 PowerShell 脚本，它调用 PuTTY 上传 JAR 和配置文件，然后发送 spark-submit 命令。

cls

$PuttyBasePath = "C:\Program Files\PuTTY\"
$PlinkPath = $PuttyBasePath + "plink.exe"
$PscpPath = $PuttyBasePath + "pscp.exe"

$SshHost = "someCluster-ssh.azurehdinsight.net"
$SshUserName = "sshuser"
$SshPassword = "somePassword"

$MainPath = "/home/sshuser/"
$BaseDirectoryToUploadFiles = "/home/sshuser/testPath"
$DirectoryToUploadFiles = $BaseDirectoryToUploadFiles + "20180505235534"

# Files to load
$ConfigFilePath = "C:\config.json"
$JarFilePath = "C:\somejar.jar"    

function UploadFiles 
{     
    &$PlinkPath -v -pw $SshPassword $SshUserName@$SshHost mkdir -p $DirectoryToUploadFiles

    $FilesToUpload = @($JarFilePath, $ConfigFilePath)

    foreach ($FileToUpload in $FilesToUpload) 
    {   
        &$PscpPath -v -pw $SshPassword -r $FileToUpload $SshUserName@${SshHost}:$DirectoryToUploadFiles
    }
}


function RunSparkJob
{
    cls

    echo y | &$PlinkPath -v -pw $SshPassword $SshUserName@$SshHost ("/usr/hdp/current/spark2-client/bin/spark-submit" + 
    " --master yarn" +
    " --deploy-mode cluster" +
    " --conf spark.yarn.maxAppAttempts=8" +
    " --conf spark.yarn.am.attemptFailuresValidityInterval=1h" +
    " --conf spark.yarn.max.executor.failures=240" +
    " --conf spark.yarn.executor.failuresValidityInterval=1h" +
    " --conf spark.task.maxFailures=4" +
    " --conf spark.speculation=true" +
    " --conf spark.speculation.multiplier=40" + 
    " --conf spark.speculation.quantile=0.85" +
    " --num-executors 30" +
    " --executor-cores 10" +
    " --files ${DirectoryToUploadFiles}/config.json" +
    " --packages org.apache.spark:spark-streaming_2.11:2.2.0,com.microsoft.azure:azure-eventhubs-spark_2.11:2.2.0" +
    " --class com.test.someClass ${DirectoryToUploadFiles}/somejar.jar")
}

UploadFiles
RunSparkJob

【讨论】：