【问题标题】:Nutch Crawler error: Premission deniedNutch Crawler 错误:权限被拒绝
【发布时间】:2014-11-29 09:40:54
【问题描述】:

我正在尝试运行一个基本的爬虫。从 NutchTutorial 得到命令: bin/crawl urls -dir crawl -depth 3 -topN 5

(完成所有预设后)

我是从 windows 运行的,所以我安装了 cygwin64 作为运行环境

当我从 nutch 主目录运行 bin/nutch 时,我没有看到任何问题,但是当我尝试像上面那样运行爬虫时,我收到以下错误:

Injector: starting at 2014-11-29 11:31:35
Injector: crawlDb: -dir/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: java.io.IOException: Failed to set permissions of path: \tmp\hadoop-Er
an\mapred\staging\Eran996102549\.staging to 0700
        at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSys
tem.java:514)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.jav
a:349)
        at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:19
3)
        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmi
ssionFiles.java:126)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Unknown Source)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1190)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:9
36)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
        at org.apache.nutch.crawl.Injector.inject(Injector.java:324)
        at org.apache.nutch.crawl.Injector.run(Injector.java:380)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.crawl.Injector.main(Injector.java:370)

他们的教程中没有提到该错误。 我该怎么办?

【问题讨论】:

    标签: java hadoop web-crawler nutch


    【解决方案1】:

    这是一个权限问题。您应该设置文件夹的读取、写入和执行权限(Hadoop 配置文件 core-site.xml 中的<name>hadoop.tmp.dir</name> 值)。

    希望对您有所帮助,

    乐国岛

    【讨论】:

      猜你喜欢
      • 2018-07-19
      • 1970-01-01
      • 2016-09-25
      • 2015-09-16
      • 2012-02-11
      • 2016-11-14
      • 2016-12-28
      • 2016-10-18
      • 2013-05-25
      相关资源
      最近更新 更多