来自不同线程的顺序文件 IO 因 FileSystemException 而失败答案

【问题标题】：Sequential file IO from different threads fail with FileSystemException来自不同线程的顺序文件 IO 因 FileSystemException 而失败
【发布时间】：2015-06-17 09:55:35
【问题描述】：

我想看看我能做到多快：

写一个小文件
重命名
删除它

这基本上看起来像：

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;


public class QuickIO {

  public static void main(String[] args) throws IOException {
    Path fileToWriteTo = Paths.get("C:\\Temp\\somefile.txt");
    Path fileToMoveTo = Paths.get("C:\\Temp\\someotherfile.txt");
    Files.write(fileToWriteTo, "sometext".getBytes());
    Files.move(fileToWriteTo, fileToMoveTo);
    Files.delete(fileToMoveTo);
  }

}

以上工作绝对没问题。我想对大量文件做同样的事情以获得相关数据；每个操作都应该由不同的线程完成（类似于我的最终架构）。它们将通过阻塞队列进行通信（因此文件上的每个操作都将在没有其他线程在处理文件时完成）。我在 Windows 7 上运行它，但失败并显示以下内容：

10:21:39.756 [Thread-1] WARN  FileMover - IOException
java.nio.file.FileSystemException: C:\Temp\file_0.in -> C:\Temp\file_0.in\file_0.out: The process cannot access the file because it is being used by another process.

    at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) ~[na:1.7.0_67]
    at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[na:1.7.0_67]
    at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) ~[na:1.7.0_67]
    at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) ~[na:1.7.0_67]
    at java.nio.file.Files.move(Files.java:1347) ~[na:1.7.0_67]
    at FileMover.run(FileMover.java:40) ~[classes/:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]

所以基本上写操作工作正常，将 Path 对象移交给 FileMover 尝试移动文件，但这会失败，因为文件被另一个进程“使用”。没有其他东西在使用该文件，因为它是刚刚创建的，并且创建它的进程按照 Files.write() 的确保关闭了该文件。

知道为什么文件系统会在不应该看到文件仍在使用的情况下看到该文件吗？以及如何解决？ :)

这是整个代码（使用“java App C:\Temp 1”调用一个文件，但失败了）

App.java

import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;

public class App {

  public static void main(String[] args) {
    if(args.length != 2 ) {
      System.exit(1);
    }
    Path workDir = Paths.get(args[0]);
    int numFilesToWrite = Integer.parseInt(args[1]);

    if(!Files.isDirectory(workDir)) {
      System.exit(1);
    }

    if(numFilesToWrite < 1) {
      System.exit(1);
    }

    BlockingQueue<Path> fromFileWriterToFileMover = new LinkedBlockingQueue<Path>();
    BlockingQueue<Path> fromFileMoverToFileRemover = new LinkedBlockingQueue<Path>();

    Thread writer = new Thread(new FileWriter(workDir, fromFileWriterToFileMover, numFilesToWrite));
    Thread mover = new Thread(new FileMover(fromFileWriterToFileMover, fromFileMoverToFileRemover));
    Thread remover = new Thread(new FileRemover(fromFileMoverToFileRemover));

    remover.start();
    mover.start();
    writer.start();

  }

}

FileWrite.java

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.concurrent.BlockingQueue;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class FileWriter implements Runnable {
  private static final Logger log = LoggerFactory.getLogger(FileWriter.class);

  Path workDirectory;
  BlockingQueue<Path> toFileMover;
  int numFiles;

  public FileWriter(
      Path workDirectory,
      BlockingQueue<Path> toFileMover,
      int numFiles) {
    this.workDirectory = workDirectory;
    this.toFileMover = toFileMover;
    this.numFiles = numFiles;
  }

  public void run() {
    RandomString rs = new RandomString(2345);
    long msSpentWritingFiles = 0L;

    for(int i = 0 ; i < numFiles ; i++) {
      try {
        Path file = workDirectory.resolve(Paths.get("file_" + i + ".in"));

        byte [] bytes = rs.nextString().getBytes();
        long beforeWrite = System.currentTimeMillis();
        Files.write(file, bytes);
        long afterWrite = System.currentTimeMillis();
        msSpentWritingFiles = msSpentWritingFiles + (afterWrite - beforeWrite);

        toFileMover.put(file);
      } catch (IOException e) {
        log.warn("IO Exception", e);
      }
      catch (InterruptedException e) {
        log.warn("InterruptedException", e);
      }
    }

    Path endPath = Paths.get("/THEEND");
    try {
      toFileMover.put(endPath);
    } catch (InterruptedException e) {
      log.warn("InterruptedException", e);
    }

    log.info("Time spent writing files: " + msSpentWritingFiles + "ms");
  }

}

FileMover.java

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.concurrent.BlockingQueue;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class FileMover implements Runnable {
  private static final Logger log = LoggerFactory.getLogger(FileMover.class);

  BlockingQueue<Path> fromFileWriter;
  BlockingQueue<Path> toFileRemover;

  public FileMover(
      BlockingQueue<Path> fromFileWriter,
      BlockingQueue<Path> toFileRemover) {
    this.fromFileWriter = fromFileWriter;
    this.toFileRemover = toFileRemover;
  }

  @Override
  public void run() {
    boolean carryOn = true;
    long msSpentMovingFiles = 0L;

    while(carryOn) {
      try {
        Path origin = fromFileWriter.take();

        if("THEEND".equals(origin.getFileName().toString())){
          carryOn = false;
          toFileRemover.put(origin);
        } else {

          Path destination = origin.resolve(origin.getFileName().toString().replace(".in", ".out"));
          long beforeMove = System.currentTimeMillis();
          Files.move(origin, destination);
          long afterMove = System.currentTimeMillis();

          msSpentMovingFiles = msSpentMovingFiles + (afterMove - beforeMove);

          toFileRemover.put(destination);
        }

      } catch (InterruptedException e) {
        log.warn("InterruptedException", e);
      } catch (IOException e) {
        log.warn("IOException", e);
      }

    }
    log.info("Time spent moving files: " + msSpentMovingFiles + "ms");
  }
}

FileRemover.java

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.concurrent.BlockingQueue;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class FileRemover implements Runnable {
private static final Logger log = LoggerFactory.getLogger(FileRemover.class);

  BlockingQueue<Path> fromFileMover;

  public FileRemover(BlockingQueue<Path> fromFileMover) {
    this.fromFileMover = fromFileMover;
  }

  @Override
  public void run() {
    boolean carryOn = true;
    long msSpentDeletingFiles = 0L;

    while(carryOn) {
      try {
        Path fileToDelete = fromFileMover.take();

        if("THEEND".equals(fileToDelete.getFileName().toString())){
          carryOn = false;
        } else {


          long beforeDelete = System.currentTimeMillis();
          Files.delete(fileToDelete);
          long afterDelete = System.currentTimeMillis();

          msSpentDeletingFiles = msSpentDeletingFiles + (afterDelete - beforeDelete);

        }
      } catch (InterruptedException e) {
        log.warn("InterruptedException", e);
      } catch (IOException e) {
        log.warn("IOException", e);
      }

    }
    log.info("Time spent deleting files: " + msSpentDeletingFiles + "ms");
  }

}

RandomString.java

import java.util.Random;

public class RandomString {

  private static final char[] symbols;

  static {
    StringBuilder tmp = new StringBuilder();
    for (char ch = '0'; ch <= '9'; ++ch)
      tmp.append(ch);
    for (char ch = 'a'; ch <= 'z'; ++ch)
      tmp.append(ch);
    symbols = tmp.toString().toCharArray();
  }   

  private final Random random = new Random();

  private final char[] buf;

  public RandomString(int length) {
    if (length < 1)
      throw new IllegalArgumentException("length < 1: " + length);
    buf = new char[length];
  }

  public String nextString() {
    for (int idx = 0; idx < buf.length; ++idx) 
      buf[idx] = symbols[random.nextInt(symbols.length)];
    return new String(buf);
  }
}

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.company</groupId>
  <artifactId>small-file-writer</artifactId>
  <version>0.0.1-SNAPSHOT</version>

  <properties>
    <java.version>1.7</java.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <logback.version>1.1.2</logback.version>
    <slf4j.version>1.7.10</slf4j.version>
  </properties>

  <dependencies>
      <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>jcl-over-slf4j</artifactId>
      <version>${slf4j.version}</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>jul-to-slf4j</artifactId>
      <version>${slf4j.version}</version>
    </dependency>
    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
      <version>${logback.version}</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
          <source>${java.version}</source>
          <target>${java.version}</target>
          <encoding>UTF-8</encoding>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

【问题讨论】：

在 BlockingQueue 中放置路径对象而不是文件对象。路径不是文件！将路径想象为指向文件的指针，因此当路径被添加到队列中时，后台的文件可能根本没有准备好。尝试使用带有文件的 BlockingQueue 来完成。添加之前，请确保文件已完全写入或移动。这也可能是 Path 的解决方法。
文件也不是文件！ :) 文件是“文件和目录路径名的抽象表示”。 Path 只是 File 的一个更新、更好的版本。每个文件操作都确保在返回之前关闭底层文件。只有在操作完成后，Path才会通过队列传递给下一个线程，因此对单个文件的操作是顺序的。
你是对的，但是路径对象不会被注意到何时关闭/完成将发生。在文件对象上也是如此。您和 GC 负责执行此操作。线程可能还有更多的事情，比如调用 join 来等待线程完成？
您无法关闭文件对象。只有一个 OutputStream，由 Files.write(Path path, byte[] bytes, OpenOption... options) 方法完成，该方法在其 JavaDoc 中明确表示“该方法确保在写入所有字节后关闭文件”。如果有问题，我使用 QuickIO 的第一个示例应该会失败。
现在我们知道了

标签： java multithreading windows-7 io

【解决方案1】：

您只是在 FileMover 运行方法中有一些错字：

Path destination = origin.resolve(origin.getFileName().toString().replace(".in", ".out"));

目的地将像 C:\temp\file_0.in\file_0.out 这不起作用，因为它是一个文件而不是一个目录 :-)

换成这样的：

String now = origin.toString().replace(".in", ".out");
Path destination = Paths.get(now);

【讨论】：

不客气！我的坚持可能来自时不时喝一杯好艾莱岛:-)