Python 与 HDInsight 中的 Hive 数据库的连接答案

【问题标题】：Python connection with Hive database in HDInsightPython 与 HDInsight 中的 Hive 数据库的连接
【发布时间】：2020-08-07 12:51:07
【问题描述】：

我正在尝试通过我的 python 脚本创建与托管在 HDInsight 群集中的 Hive 的连接并出现以下错误-

    Traceback (most recent call last):
  File "ClassLoader.java", line 357, in java.lang.ClassLoader.loadClass
  File "Launcher.java", line 349, in sun.misc.Launcher$AppClassLoader.loadClass
  File "ClassLoader.java", line 424, in java.lang.ClassLoader.loadClass
  File "URLClassLoader.java", line 382, in java.net.URLClassLoader.findClass
java.lang.ClassNotFoundException: java.lang.ClassNotFoundException: org.apache.thrift.transport.TTransportException
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "org.jpype.JPypeContext.java", line 330, in org.jpype.JPypeContext.callMethod
  File "Method.java", line 498, in java.lang.reflect.Method.invoke
  File "DelegatingMethodAccessorImpl.java", line 43, in sun.reflect.DelegatingMethodAccessorImpl.invoke
  File "NativeMethodAccessorImpl.java", line 62, in sun.reflect.NativeMethodAccessorImpl.invoke
  File "NativeMethodAccessorImpl.java", line -2, in sun.reflect.NativeMethodAccessorImpl.invoke0
  File "DriverManager.java", line 247, in java.sql.DriverManager.getConnection
  File "DriverManager.java", line 664, in java.sql.DriverManager.getConnection
  File "HiveDriver.java", line 105, in org.apache.hive.jdbc.HiveDriver.connect
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "s.py", line 5, in <module>
    "/root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar")
  File "/usr/local/lib64/python3.6/site-packages/jaydebeapi/__init__.py", line 412, in connect
    jconn = _jdbc_connect(jclassname, url, driver_args, jars, libs)
  File "/usr/local/lib64/python3.6/site-packages/jaydebeapi/__init__.py", line 230, in _jdbc_connect_jpype
    return jpype.java.sql.DriverManager.getConnection(url, *dargs)
java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: org/apache/thrift/transport/TTransportException

我的脚本是 -

import jaydebeapi
conn = jaydebeapi.connect("org.apache.hive.jdbc.HiveDriver",
       "jdbc:hive2://10.20.30.40:10001/default;transportMode=http;ssl=false;httpPath=/hive2",
       ["username", "password"],
       "/root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar")

我已经导出了 CLASSPATH 和所有的 jar 文件。

【问题讨论】：

你能分享你的 CLASSPATH jars 列表吗？
使用https://repo1.maven.org/maven2/org/apache/hive/hive-jdbc/1.2.1/hive-jdbc-1.2.1.jar jar 并分享结果？
Traceback（最近一次调用最后）：文件“s.py”，第 5 行，在 "/root/jdbc/hive-jdbc-1.2.1.jar") 文件 "/ usr/local/lib64/python3.6/site-packages/jaydebeapi/__init__.py”，第 412 行，在 connect jconn = jdbc_connect(jclassname, url, driver_args, jars, libs) 文件“/usr/local/ lib64/python3.6/site-packages/jaydebeapi/__init_.py”，第 230 行，在 _jdbc_connect_jpype 返回 jpype.java.sql.DriverManager.getConnection(url, *dargs) java.lang.NoClassDefFoundError: java. lang.NoClassDefFoundError: org/apache/thrift/transport/TTransportException
CLASSPATH- [root@hostname]# echo $CLASSPATH /root/jdbc/hive-jdbc-1.2.1.jar:/root/jdbc/httpclient-4.4.jar:/root/jdbc/ httpcore-4.4.4.jar
替换 hive-jdbc-1.2.1.jar 代替 /root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar

标签： python python-3.x hive azure-hdinsight

【解决方案1】：

错误为java.lang.ClassNotFoundException: java.lang.ClassNotFoundException，表示执行找不到jar/root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar.I相信它只放置在您执行代码的主机中。我建议将 jar 文件放在集群中所有节点的相同目录结构中，并检查权限，以便执行作业的用户可以访问该路径。

【讨论】：