【问题标题】:Errors when using Sqoop action in Oozie editor (Hue)在 Oozie 编辑器 (Hue) 中使用 Sqoop 操作时出错
【发布时间】:2016-01-06 04:14:04
【问题描述】:

我正在尝试在 Hue 的 Oozie 编辑器中使用 Sqoop 操作,但我无法让它工作。

这是我迄今为止尝试过的。

我将所有内容都放在参数中,而不是命令 (http://alvincjin.blogspot.com.au/2014/06/create-sqoop-action-in-oozie-using-hue.html)

此外,我正在尝试连接到 Teradata,因此我已将 jdbc jar 放在 HDFS 中,并将它们添加到文件中。 这是当前工作流程在编辑器中的样子: Sqoop Action .

工作流定义为:

<workflow-app name="Sqoop_test" xmlns="uri:oozie:workflow:0.5">
    <start to="sqoop-b20d"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="sqoop-b20d">
        <sqoop xmlns="uri:oozie:sqoop-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>development</value>
                </property>
                <property>
                    <name>mapred.job.name</name>
                    <value>test_sqoop</value>
                </property>
                <property>
                    <name>mapred.task.timeout</name>
                    <value>0</value>
                </property>
            </configuration>
              <arg>import</arg>
              <arg>--connect</arg>
              <arg>jdbc:teradata://XXXXX</arg>
              <arg>--query</arg>
              <arg>select count(*) from XXXXX</arg>
              <arg>--fetch-size</arg>
              <arg>10000</arg>
              <arg>--num-mappers</arg>
              <arg>1</arg>
              <arg>--hive-table-name</arg>
              <arg>XXXXX.tmp_sqoop_test</arg>
              <arg>--hive-import</arg>
              <arg>--hive-overwrite</arg>
              <arg>--target-dir</arg>
              <arg>/user/dXXXXX/digital/test/tmp_sqoop_test</arg>
              <arg>--username</arg>
              <arg>XXXXX</arg>
              <arg>--password</arg>
              <arg>XXXXX</arg>
            <file>/user/hue/oozie/workspaces/digital/lib/terajdbc4.jar#terajdbc4.jar</file>
            <file>/user/hue/oozie/workspaces/digital/lib/teradata-connector-1.3.4-hadoop220.jar#teradata-connector-1.3.4-hadoop220.jar</file>
        </sqoop>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

但是,我收到此错误:

2016-01-06 14:13:52,114 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(296)) - Error parsing arguments for import:
2786 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: --hive-table-name
2016-01-06 14:13:52,114 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(299)) - Unrecognized argument: --hive-table-name
2786 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: XXXXX.tmp_sqoop_test
2016-01-06 14:13:52,114 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(299)) - Unrecognized argument: tdcprdr_app_digital.tmp_sqoop_test
2786 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: --hive-import
2016-01-06 14:13:52,114 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(299)) - Unrecognized argument: --hive-import
2786 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: --hive-overwrite
2016-01-06 14:13:52,114 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(299)) - Unrecognized argument: --hive-overwrite
2787 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool  - Unrecognized argument: --target-dir
2016-01-06 14:13:52,115 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(299)) - Unrecognized argument: --target-dir
...

我的印象是可以通过将所有内容放在参数中来解决此错误。 通过 shell 脚本运行相同的代码。我尝试将导入命令和连接字符串放在命令部分,但这甚至没有运行。我也尝试过创建一个简约的 sqoop 操作,只使用如下查询和连接语句:

<workflow-app name="Sqoop_minimal" xmlns="uri:oozie:workflow:0.5">
    <start to="sqoop-eeeb"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="sqoop-eeeb">
        <sqoop xmlns="uri:oozie:sqoop-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
              <arg>import</arg>
              <arg>--connect</arg>
              <arg>jdbc:teradata://tdXXXXX</arg>
              <arg>--query</arg>
              <arg>select count(*) from XXXXX</arg>
              <arg>--target-dir</arg>
              <arg>/user/dXXXXX/digital/test/tmp_sqoop_test</arg>
              <arg>--username</arg>
              <arg>XXXXX</arg>
              <arg>--password</arg>
              <arg>XXXXX</arg>
            <file>/user/hue/oozie/workspaces/digital/lib/teradata-connector-1.3.4-hadoop220.jar#teradata-connector-1.3.4-hadoop220.jar</file>
            <file>/user/hue/oozie/workspaces/digital/lib/terajdbc4.jar#terajdbc4.jar</file>
        </sqoop>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

使用此工作流程,我得到一个非常模糊的错误,如下所示:

>>> Invoking Sqoop command line now >>>

2287 [main] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2016-01-06 14:57:48,381 WARN  [main] tool.SqoopTool (SqoopTool.java:loadPluginsFromConfDir(175)) - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2324 [main] INFO  org.apache.sqoop.Sqoop  - Running Sqoop version: 1.4.5.3.0.0.0-249
2016-01-06 14:57:48,418 INFO  [main] sqoop.Sqoop (Sqoop.java:<init>(92)) - Running Sqoop version: 1.4.5.3.0.0.0-249
2339 [main] WARN  org.apache.sqoop.tool.BaseSqoopTool  - Setting your password on the command-line is insecure. Consider using -P instead.
2016-01-06 14:57:48,433 WARN  [main] tool.BaseSqoopTool (BaseSqoopTool.java:applyCredentialsOptions(1014)) - Setting your password on the command-line is insecure. Consider using -P instead.
Intercepting System.exit(1)

<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

Oozie Launcher failed, finishing Hadoop job gracefully

Oozie 版本是 4.1.0.3.0.0.0-249。

我尝试在网上搜索解决方案,但没有成功。 任何帮助,将不胜感激。谢谢!

已经看过并尝试过链接:

https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Sqoop-fails-with-quot-Error-parsing-arguments-for-import-quot/td-p/31930

http://stackoverflow.com/questions/25770698/sqoop-free-form-query-causing-unrecognized-arguments-in-hue-oozie

【问题讨论】:

    标签: hadoop oozie sqoop hue


    【解决方案1】:

    sqoop 没有这样的参数 --hive 表名 采用 --蜂巢表。它现在不应该显示 Unrecognized 参数

    【讨论】:

    • 谢谢!我将选项更改为“hive-table”,它起作用了,即在某种程度上。它现在显示错误“2417 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - 创建数据库管理器时出错:java.io.IOException: No manager for connect string: jdbc:teradata://XXXXX/LOGMECH=TD2 ”。我已经将 teradata 连接器 jar 放在 Oozie 的共享库目录中,并且还在 sqoop 操作中指定了此属性:sqoop.connection.factories=com.cloudera.sqoop.manager.DefaultManagerFactory,com.cloudera.sqoop.manager.TeradataManagerFactory。
    • 在job.properties中需要提到oozie.use.system.libpath=true
    猜你喜欢
    • 1970-01-01
    • 2015-06-07
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2017-04-05
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多