Hadoop：是否可以避免某些文件的复制？

Question

据我所知，在 hdfs 中，所有文件都被复制，但我们会在工作的情况下进行某些日志记录，我们不希望复制的文件可能会不必要地维护复制的副本，是否可以这样做？即避免只复制日志文件。？

Answer 1

您可以使用 -setrep 标志和 hadoop fs shell 命令来设置复制。

Usage: hadoop fs -setrep [-R] [-w] <numReplicas> <path>

Changes the replication factor of a file. If path is a directory then the command recursively changes the replication factor of all files under the directory tree rooted at path.

Options:

The -w flag requests that the command wait for the replication to complete. This can potentially take a very long time.
The -R flag is accepted for backwards compatibility. It has no effect.
Example:

hadoop fs -setrep -w 3 /user/hadoop/dir1

为避免复制，您可以将 numReplicas 设置为 1。