【发布时间】：2015-09-23 09:22:13
【问题描述】：
我正在运行一个简单的 Python 映射器和缩减器，并得到 13 permission denied error。需要帮助。
我不确定这里发生了什么，需要帮助。 Hadoop 世界的新手。
我正在运行简单的 map reduce 来计算单词。 mapper 和 reducer 在 linus 或 windows powershell 上独立运行
======================================================================
hadoop@ubuntu:~/hadoop-1.2.1$ bin/hadoop jar contrib/streaming/hadoop-streaming-1.2.1.jar -file /home/hadoop/mapper.py -mapper mapper.py -file /home/hadoop/reducer.py -reducer reducer.py -input /deepw/pg4300.txt -output /deepw/pg3055
Warning: $HADOOP_HOME is deprecated.
packageJobJar: [/home/hadoop/mapper.py, /home/hadoop/reducer.py, /tmp/hadoop-hadoop/hadoop-unjar2961168567699201508/] [] /tmp/streamjob4125164474101219622.jar tmpDir=null
15/09/23 14:39:16 INFO util.NativeCodeLoader: Loaded the native-hadoop library
15/09/23 14:39:16 WARN snappy.LoadSnappy: Snappy native library not loaded
15/09/23 14:39:16 INFO mapred.FileInputFormat: Total input paths to process : 1
15/09/23 14:39:16 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-hadoop/mapred/local]
15/09/23 14:39:16 INFO streaming.StreamJob: Running job: job_201509231312_0003
15/09/23 14:39:16 INFO streaming.StreamJob: To kill this job, run:
15/09/23 14:39:16 INFO streaming.StreamJob: /home/hadoop/hadoop-1.2.1/libexec/../bin/hadoop job -Dmapred.job.tracker=192.168.56.102:9001 -kill job_201509231312_0003
15/09/23 14:39:16 INFO streaming.StreamJob: Tracking URL: http://192.168.56.102:50030/jobdetails.jsp?jobid=job_201509231312_0003
15/09/23 14:39:17 INFO streaming.StreamJob: map 0% reduce 0%
15/09/23 14:39:41 INFO streaming.StreamJob: map 100% reduce 100%
15/09/23 14:39:41 INFO streaming.StreamJob: To kill this job, run:
15/09/23 14:39:41 INFO streaming.StreamJob: /home/hadoop/hadoop-1.2.1/libexec/../bin/hadoop job -Dmapred.job.tracker=192.168.56.102:9001 -kill job_201509231312_0003
15/09/23 14:39:41 INFO streaming.StreamJob: Tracking URL: http://192.168.56.102:50030/jobdetails.jsp?jobid=job_201509231312_0003
15/09/23 14:39:41 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201509231312_0003_m_000000
15/09/23 14:39:41 INFO streaming.StreamJob: killJob...
Streaming Command Failed!
================================================================
java.io.IOException: Cannot run program "/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201509231312_0003/attempt_201509231312_0003_m_000001_3/work/./mapper.py": error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException: error=13, Permission denied
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
... 24 more
【问题讨论】：
-
尝试在您的命令前添加
sudo -
添加 sudo 也不起作用。java.io.IOException: Cannot run program "/tmp/hadoop-hadoop/mapred/local/taskTracker/root/jobcache/job_201509231626_0007/attempt_201509231626_0007_m_000000_3/work/./ mapper.py": error=13, 权限被拒绝
-
整个路径“/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201509231312_0003/attempt_201509231312_0003_m_000001_3/work/./mapper.py”的权限是否适合您能够执行该程序并确保您的文件系统都没有满。
-
如何为整个路径提供权限，任何指针都会有所帮助
-
@SwadeepMishra 我的问题是你检查权限了吗？
标签： python-2.7 hadoop-streaming