hdfs 显示本地文件列表

Question

我在 OS X 中安装了 Hadoop，它运行良好。我的经验是最近的，正在努力学习更多关于使用 Hadoop 进行应用程序开发的知识。

昨天，当我需要在 Hadoop 中查找目录和/或文件列表时，我可以输入

$ hadoop fs -ls 

而且，它会显示集群中的所有内容。

今天，它显示了文件系统中的所有本地内容。我必须提供 hdfs 的确切地址才能获取内容列表，

$ hadoop fs -ls hdfs://localhost:8020/user/myName

我的core-site.xml文件和以前一样，

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<!-- Put site-specific property overrides in this file. -->
 <configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:8020</value>
    </property>
</configuration>

在启动 hadoop 守护进程之前，我停止了集群并再次使用以下命令重新格式化分布式文件系统，以便我们可以在执行 map-reduce 作业时将数据源放入 hdfs 文件系统

$ hdfs namenode -format

我收到管理员报告通知 FileSystem file:/// 不是 HDFS 文件系统，

$ hadoop dfsadmin -report
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.

2018-10-18 18:01:27,316 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
report: FileSystem file:/// is not an HDFS file system
Usage: hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning] [-enteringmaintenance] [-inmaintenance]

在core-site.xml文件中，我也将配置更新为如下，

<property>
    <!-- <name>fs.default.name</name> -->
    <!-- <value>hdfs://localhost:8020</value> -->
    <name>fs.defaultFS</name>
    <value>hdfs://localhost.localdomain:8020/</value>
</property>

那时我已经改过自新，这并没有改变任何事情。正如提到的另一个答案，haddop 主页已经在~/.bashrc 文件中提供，

export HADOOP_HOME=/Users/chaklader/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

如何切换到 HDFS 文件系统？任何形式的建议将不胜感激。

Answer 1

您需要确保已添加一个名为 HADOOP_CONF_DIR 的环境变量，以将其设置为包含来自 Hadoop 的 XML 文件的目录。

您可以在主文件夹中的.bashrc 中执行此操作

---

否则，您将获得默认文件系统file://，它仍然有效并且仍然可以正常运行 MapReduce 作业

---

FWIW，这是我的核心站点

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:///tmp/hadoop/hdfs/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

还有 hdfs 网站

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
  <property>
      <name>dfs.namenode.name.dir</name>
      <value>file:///tmp/hadoop/hdfs/names</value>
  </property>
  <property>
    <name>fs.checkpoint.dir</name>
    <value>file:///tmp/hadoop/hdfs/checkpoint</value>
  </property>
  <property>
    <name>fs.checkpoint.edits.dir</name>
    <value>file:///tmp/hadoop/hdfs/checkpoint-edits</value>
  </property>
  <property>
      <name>dfs.datanode.data.dir</name>
      <value>file:///tmp/hadoop/hdfs/data</value>
  </property>
</configuration>

Answer 2

在您的 core-site.xml 文件中进行如下编辑。

<value>hdfs://localhost.localdomain:8020/</value>

我相信缺少斜线 (8020/) 会造成问题。 试试看。

Answer 3

我们在 Cloudera Data Platform 7.1.5 边缘节点（没有主节点和从节点的节点：只有客户端和 Cloudera 管理器）上遇到了同样的问题。 HDFS 文件在每个集群节点上正常显示，除了在显示本地文件系统的边缘节点上。解决方案是在我们的边缘节点上安装网关角色，如 https://community.cloudera.com/t5/Cloudera-Manager-Installation/quot-Hadoop-fs-ls-quot-Produces-the-Local-Filesystem-s-quot/td-p/4743 所述（感谢链接，Jim Todd）