Hadoop put 和 copyfromlocal 的区别答案

【问题标题】：Hadoop difference between put and copyfromlocalHadoop put 和 copyfromlocal 的区别
【发布时间】：2018-09-13 04:00:21
【问题描述】：

CopyFromLocal 将从本地文件系统上传数据。

Do put 将从任何 fs 上传数据，例如。本地 fs，亚马逊 s3 还是仅来自本地 fs ???

【问题讨论】：

Difference between hadoop fs -put and hadoop fs -copyFromLocal的可能重复
如果你使用 S3 或其他 Hadoop 文件系统，你会使用 DistCp
put和copyfromlocal只有一个区别，就是。 Copyfromlocal 仅从本地和复制数据。 Put 将从不同的文件系统复制，我正在寻找答案，这些文件系统是什么。请有人给出所有可以使用“hadoop fs-put”命令获取/放置数据的文件系统名称
您不能将 get 数据与 put。那没有意义。 put 和 copyFromLocal 的源代码使用完全相同的 Java 方法。正如我所提到的，DistCp 在不同的文件系统之间进行复制

标签： hadoop hdfs

【解决方案1】：

请找出这两个命令的用法。

put
=======
Usage: hadoop fs -put [-f] [-p] [-l] [-d] [ - | <localsrc1> .. ]. <dst>

Copy single src, or multiple srcs from local file system to the destination file system. Also reads input from stdin and writes to destination file system if the source is set to “-”

Copying fails if the file already exists, unless the -f flag is given.

Options:

-p : Preserves access and modification times, ownership and the permissions. (assuming the permissions can be propagated across filesystems)
-f : Overwrites the destination if it already exists.
-l : Allow DataNode to lazily persist the file to disk, Forces a replication factor of 1. This flag will result in reduced durability. Use with care.
-d : Skip creation of temporary file with the suffix ._COPYING_.
Examples:

hadoop fs -put localfile /user/hadoop/hadoopfile
hadoop fs -put -f localfile1 localfile2 /user/hadoop/hadoopdir
hadoop fs -put -d localfile hdfs://nn.example.com/hadoop/hadoopfile
hadoop fs -put - hdfs://nn.example.com/hadoop/hadoopfile Reads the input from stdin.
Exit Code:

Returns 0 on success and -1 on error.

copyFromLocal
=============

Usage: hadoop fs -copyFromLocal <localsrc> URI

Similar to the fs -put command, except that the source is restricted to a local file reference.

Options:

-p : Preserves access and modification times, ownership and the permissions. (assuming the permissions can be propagated across filesystems)
-f : Overwrites the destination if it already exists.
-l : Allow DataNode to lazily persist the file to disk, Forces a replication factor of 1. This flag will result in reduced durability. Use with care.
-d : Skip creation of temporary file with the suffix ._COPYING_.

【讨论】：

这是文档所说的，但源代码另有说明

【解决方案2】：

它们是identical（根据extends Put，除此之外没有其他方法或条件）。在这个答案中，CopyCommands.java 类自 2014 年以来没有新的提交。

  public static class CopyFromLocal extends Put {
        public static final String NAME = "copyFromLocal";
        public static final String USAGE = Put.USAGE;
        public static final String DESCRIPTION = "Identical to the -put command.";
  }

请有人给出所有可以使用“hadoop fs-put”命令获取/放置数据的文件系统名称

使用hadoop fs 命令支持任何Hadoop Compatible FileSystem (HCFS)。 hdfs dfs仅支持HDFS

【讨论】：