【问题标题】:Cannot run program "-files" (in directory "."): error=2, No such file or directory无法运行程序“-files”(在目录“.”中):错误=2,没有这样的文件或目录
【发布时间】:2016-04-27 19:38:54
【问题描述】:

我正在尝试使用 aws sdk ruby​​ 为使用 hadoop 的 Amazon ElasticMapReduce 服务运行一个步骤,虽然我可以创建集群和步骤,但该步骤总是失败,但在使用 Web 界面手动设置时不会失败

emr = Aws::EMR::Client.new
    cluster_id = "*******"
    resp = emr.add_job_flow_steps({
      job_flow_id: cluster_id, # required
      steps: [ # required
        {
          name: "TestStep", # required
          action_on_failure: "CANCEL_AND_WAIT", # accepts TERMINATE_JOB_FLOW, TERMINATE_CLUSTER, CANCEL_AND_WAIT, CONTINUE
          hadoop_jar_step: { # required
            jar: 'command-runner.jar',
            args:[
                "-files",
                "s3://source123/mapper.py,s3://source123/source_reducer.py",
                "-mapper",
                "mapper.py",
                "-reducer",
                "source_reducer.py",
                "-input",
                "s3://source123/input/",
                "-output",
                "s3://source123/output/"
            ]
          },
        },
      ],
    })

我得到的错误是这样的

Cannot run program "-files" (in directory "."): error=2, No such file or directory

有什么线索吗?

【问题讨论】:

    标签: ruby-on-rails ruby hadoop amazon-web-services emr


    【解决方案1】:

    似乎添加hadoop-streaming的工作如下

    emr = Aws::EMR::Client.new
        cluster_id = "*******"
        resp = emr.add_job_flow_steps({
          job_flow_id: cluster_id, # required
          steps: [ # required
            {
              name: "TestStep", # required
              action_on_failure: "CANCEL_AND_WAIT", # accepts TERMINATE_JOB_FLOW, TERMINATE_CLUSTER, CANCEL_AND_WAIT, CONTINUE
              hadoop_jar_step: { # required
                jar: 'command-runner.jar',
                args:[
                    "hadoop-streaming",
                    "-files",
                    "s3://source123/mapper.py,s3://source123/source_reducer.py",
                    "-mapper",
                    "mapper.py",
                    "-reducer",
                    "source_reducer.py",
                    "-input",
                    "s3://source123/input/",
                    "-output",
                    "s3://source123/output/"
                ]
              },
            },
          ],
        })
    

    【讨论】:

      猜你喜欢
      • 2018-01-30
      • 2016-08-22
      • 1970-01-01
      • 2020-11-23
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2015-04-22
      • 1970-01-01
      相关资源
      最近更新 更多