无法创建 hive 连接 jdbc:hive2://localhost:10000。集群模式下的 spark-submit答案

【问题标题】：Cannot create hive connection jdbc:hive2://localhost:10000. spark-submit in cluster mode无法创建 hive 连接 jdbc:hive2://localhost:10000。集群模式下的 spark-submit
【发布时间】：2021-05-19 02:44:48
【问题描述】：

我正在 Apache Spark 上运行 Apache Hudi 应用程序。当我在客户端模式下提交应用程序时，它工作正常，但是当我在集群模式下提交应用程序时，出现错误

py4j.protocol.Py4JJavaError: An error occurred while calling o196.save.
: org.apache.hudi.hive.HoodieHiveSyncException: Cannot create hive connection jdbc:hive2://localhost:10000/
    at org.apache.hudi.hive.HoodieHiveClient.createHiveConnection(HoodieHiveClient.java:422)
    at org.apache.hudi.hive.HoodieHiveClient.<init>(HoodieHiveClient.java:95)
    at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:66)
    at org.apache.hudi.HoodieSparkSqlWriter$.org$apache$hudi$HoodieSparkSqlWriter$$syncHive(HoodieSparkSqlWriter.scala:321)
    at org.apache.hudi.HoodieSparkSqlWriter$$anonfun$metaSync$2.apply(HoodieSparkSqlWriter.scala:363)
    at org.apache.hudi.HoodieSparkSqlWriter$$anonfun$metaSync$2.apply(HoodieSparkSqlWriter.scala:359)
    at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
    at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:359)
    at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:417)
    at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:205)
    at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:125)
    at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:173)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:169)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:197)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:194)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:169)
    at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:114)
    at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:112)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:696)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:696)
    at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$executeQuery$1(SQLExecution.scala:83)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1$$anonfun$apply$1.apply(SQLExecution.scala:94)
    at org.apache.spark.sql.execution.QueryExecutionMetrics$.withMetrics(QueryExecutionMetrics.scala:141)
    at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$withMetrics(SQLExecution.scala:178)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:93)

【问题讨论】：

标签： apache-spark hive spark-streaming hadoop-yarn apache-hudi

【解决方案1】：

修改 hudi 配置后“hoodie.datasource.hive_sync.jdbcurl”开始工作。

【讨论】：

hoodie.datasource.hive_sync.jdbcurl 在使用 EMR 并在集群模式下使用 spark-submit @PradeepSaini 时，应指定主 DNS 主机名
@VikasChitturi 尝试使用主 DNS 主机名 - 同样的问题。
好的，你能分享一下纱线日志中的异常吗？主 DNS 名称应如下所示：jdbc:hive2://ip-XX-XXX-XX-XX.ec2.internal:10000 @dytyniak
@VikasChitturi 调用 o107.save 时出错。 : org.apache.hudi.hive.HoodieHiveSyncException: 无法创建配置单元连接 jdbc:hive2://ip-172-31-35-94.ec2.internal:10000/
也许检查您的集群配置？它在子网中运行吗？可能是防火墙问题？不确定，因为它对我有用，我的 EMR 为主实例和工作实例启用了适当的安全组

【解决方案2】：

以下是我使用的 hudi 写入选项，只要 EMR 集群正确配置了正确的安全组和子网设置

hudi_write_table_options = {
        "hoodie.table.name": "hudi_data_test",
        "hoodie.datasource.write.table.type": "MERGE_ON_READ",
        "hoodie.datasource.write.storage.type": "MERGE_ON_READ",
        "hoodie.datasource.write.recordkey.field": ['a','b'],
        "hoodie.datasource.write.partitionpath.field": ['a','b'],
        "hoodie.datasource.write.precombine.field": 'c',
        "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
        "hoodie.datasource.write.operation": "bulk_insert",
        "hoodie.consistency.check.enabled": "true",
        "hoodie.datasource.write.hive_style_partitioning": "true",
        "hoodie.datasource.hive_sync.enable": "true",
        "hoodie.datasource.hive_sync.auto_create_database":"true",
        "hoodie.datasource.hive_sync.database":"hudidatabase",
        "hoodie.datasource.hive_sync.table": "hudi_data_test",
        "hoodie.datasource.hive_sync.partition_fields": ['a','b'],
        'hoodie.datasource.hive_sync.jdbcurl':"jdbc:hive2://ip-XXX-XX-XX-XX.ec2.internal:10000/",
        "hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor"
    }

【讨论】：