执行作业时如何将 OracleDriver 添加到 hadoop 类路径？

Question

我正在使用 hadoop 处理来自数据库的大量数据。我正在使用 oracle 的 jdbc 驱动程序连接到 Oracle DB 并进行处理。但是当我尝试通过 bin/hadoop 使用打包的 JAR 文件执行 hadoop 作业时，它显示找不到 OracleDriver 类。我该如何解决？

$ bin/hadoop jar ~/hadoop1.jar name.hadoop.Hadoop1 ~/output

Exception in thread "main" java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:575) at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1051) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1043) at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:959) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:886) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323) at name.shahalpk.poc.hadoop.Hadoop1.main(Hadoop1.java:73) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) ... 20 more Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver at org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271) ... 25 more Caused by: java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:186) at org.apache.hadoop.mapred.lib.db.DBConfiguration.getConnection(DBConfiguration.java:123) at org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:266) ... 25 more

Answer 1

将 ojdbc5.jar 添加到类路径中：

${JRE_HOME}\jre\lib\ext

注意：

${JRE_HOME} means JRE(Java Runtime Environment) Installed Directory; Like below
${JRE_HOME}=C:\Program Files\Java\jre6\

Answer 2

您可以使用-libjars 通用选项更轻松地实现此目的，它还可以将您的 jar 分发到集群节点：

$ bin/hadoop jar ~/hadoop1.jar name.hadoop.Hadoop1 -libjars ojdbc5.jar ~/output

这确实假设您的主类 (name.hadoop.Hadoop1) 正在使用 ToolRunner.run() 方法来启动您的工作：

public class Hadoop1 extends Configured implements Tool {
  public static void main(String args[]) throws Exception {
    ToolRunner.run(new Hadoop1(), args);
  }

  public int run(String args[]) {
    JobConf job = new JobConf(getConf());

    // rest of your job init code...

    RunningJob rj = JobClient.runJob(job);
    rj.waitForCompletion();
    return rj.isSuccessful() ? 0 : 1;
  }
}

（手写代码，如有错别字或编译错误，敬请见谅）