【发布时间】：2015-09-04 13:56:24
【问题描述】：
我正在尝试在 Windows 8.1 上使用 Hadoop 运行此 Map Reduce 程序。经过一番努力，我已经接近工作了。我有 Java 1.8.0_45 和 Hadoop-2.7.0。我还有 winutils.exe 和 hadoop.dll，它们给很多人带来了问题。
代码如下：
public class OSProject {
public static class Map extends MapReduceBase implements
Mapper<LongWritable, Text, Text, IntWritable> {
@Override
public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter)
throws IOException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
value.set(tokenizer.nextToken());
output.collect(value, new IntWritable(1));
}
}
}
public static class Reduce extends MapReduceBase implements
Reducer<Text, IntWritable, Text, IntWritable> {
@Override
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output, Reporter reporter)
throws IOException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
BasicConfigurator.configure();
JobConf conf = new JobConf(OSProject.class);
conf.setJobName("wordcount");
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(Map.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
FileInputFormat.setInputPaths(conf, new Path("c:/hwork/input"));
FileOutputFormat.setOutputPath(conf, new Path("c:/hwork/output"));
//FileInputFormat.setInputPaths(conf, new Path(args[0]));
//FileOutputFormat.setOutputPath(conf, new Path(args[1]));
JobClient.runJob(conf);
}
}
问题是程序在我运行时抛出这个错误：
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:473)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:526)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:504)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:305)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:147)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:833)
at osproject.OSProject.main(OSProject.java:86)
出错的行是：
JobClient.runJob(conf);
看起来它由于某种原因无法创建输出文件。任何关于为什么会发生这种情况以及如何解决它的答案将不胜感激。
【问题讨论】：
-
我猜你用 JDK 1.8 构建了这个应用程序。您应该检查您是否使用正确的 JRE 运行应用程序，也许您的默认 JRE 来自 Java 6
-
这似乎不是问题。它们都是 Java 8。
-
你的路径还好吗？请仔细检查“c:/hwork/input”或现在使用相对路径
-
我没有看到输入路径有任何问题。使用输出路径，如果文件夹存在，则会给出错误提示“输出目录文件：/hwork/output 已存在”。所以我假设库中的一种方法应该是创建目录本身。如果目录“C:/hwork/output”不存在，它会给出问题中指定的错误。我尝试将其设置为许多不同的路径，但没有任何效果。