使用scala API，如何将一个HDFS位置的所有文件复制到另一个HDFS位置[重复]答案

【问题标题】：Using scala API, How do I copy all files of one HDFS location to another HDFS location [duplicate]使用scala API，如何将一个HDFS位置的所有文件复制到另一个HDFS位置[重复]
【发布时间】：2021-07-29 11:40:46
【问题描述】：

使用 scala ，我想将 srcFilePath 中的所有文件复制到 destFilePath，但是下面的代码会抛出错误有人可以帮我解决这个错误和复制文件的解决方案

 scala> val srcFilePath = "/development/staging/b8baf3f4-abce-11eb-8592-0242ac110032/"
 srcFilePath: String = /development/staging/b8baf3f4-abce-11eb-8592-0242ac110032/

 scala> val destFilePath = "/development/staging/dest_b8baf3f4-abce-11eb-8592-0242ac110032/"
 destFilePath: String = /development/staging/dest_b8baf3f4-abce-11eb-8592-0242ac110032/

 scala> val hadoopConf = new Configuration()
  hadoopConf: org.apache.hadoop.conf.Configuration = Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml

  scala> val hdfs = FileSystem.get(hadoopConf)
  hdfs: org.apache.hadoop.fs.FileSystem = DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1792011619_1, ugi=be9dusr@INTERNAL.IMSGLOBAL.COM (auth:KERBEROS)]]

 scala>

  scala> val srcPath = new Path(srcFilePath)
 srcPath: org.apache.hadoop.fs.Path = /development/staging/b8baf3f4-abce-11eb-8592-0242ac110032

  scala> val destPath = new Path(destFilePath)
  destPath: org.apache.hadoop.fs.Path = /development/staging/dest_b8baf3f4-abce-11eb-8592-0242ac110032

 scala>

 scala> hdfs.copy(srcPath, destPath)
 <console>:52: error: value move is not a member of org.apache.hadoop.fs.FileSystem
   hdfs.copy(srcPath, destPath)

【问题讨论】：

标签： scala apache-spark hadoop

【解决方案1】：

你可能想看看this SO post上的答案

Try Hadoop's FileUtil.copy() command, as described here: https://hadoop.apache.org/docs/r2.8.5/api/org/apache/hadoop/fs/FileUtil.html#copy(org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20boolean,%20org.apache.hadoop.conf.Configuration)

val conf = new org.apache.hadoop.conf.Configuration()
val srcPath = new org.apache.hadoop.fs.Path("hdfs://my/src/path")
val dstPath = new org.apache.hadoop.fs.Path("hdfs://my/dst/path")

org.apache.hadoop.fs.FileUtil.copy(
srcPath.getFileSystem(conf), 
srcPath, 
dstPath.getFileSystem(conf), 
dstPath, 
true, 
conf
)

【讨论】：

是的，这段代码正在运行，我会解压缩文件并将其放入 dstPath。基本上我需要从 srcPath 读取一个 zip 文件并将其解压缩并将解压缩的文件放在 dstPath 中。有没有办法做到这一点