使用 FileSystemWatcher 和 C# 处理具有多个文件的文件夹

Question

我创建了一个相对简单的 Windows 应用程序，用于监视文件夹中的文件。当在文件夹中创建新文件时，应用程序（通过 FileSystemWatcher）将打开文件并处理内容。长话短说，这些内容与 Selenium 一起用于通过 IE11 自动化网页。这个处理每个文件大约需要 20 秒。

问题是，如果大致同时在文件夹中创建了多个文件，或者当应用程序正在处理一个文件时，FileSystemWatcher onCreated 不会看到下一个文件。因此，当第一个文件的处理完成时，应用程序就会停止。同时文件夹中有一个文件没有得到处理。如果在 onCreated 处理完成后添加文件，它可以正常工作并处理下一个文件。

有人可以指导我解决这个问题吗？过多的细节是非常受欢迎的。

Answer 1

FileSystemWatcher（正如您已经注意到的）不可靠，您将始终必须为丢失的文件添加“自定义”/手动逻辑（另外，请注意，您可能会看到多个相同的事件文件）

您可以在下面看到一个简单的示例，其中包含未处理文件的“背景”检查。
您可以通过使用并发集合来避免锁定，例如 BlockingCollection
您还可以选择并行处理文件
我正在根据计时器处理文件，但您可以使用自己的策略。
如果您不想实时处理文件，可能您甚至不需要 FileSystemWatcher

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;

namespace ConsoleAppDemo
{
    class Program
    {
        private static object lockIbj = new object();
        private static List<string> _proccessedFiles = new List<string>();
        private static readonly List<string> toProccessFiles = new List<string>();
        private static List<string> _proccessingFiles = new List<string>();
        private const string directory = @"C:\Path";
        private const string extension = @"*.txt";
        static void Main(string[] args)
        {
            FileSystemWatcher f = new FileSystemWatcher();
            f.Path = directory;
            f.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
                             | NotifyFilters.FileName | NotifyFilters.DirectoryName;
            f.Filter = extension ;
            f.Created += F_Created;
            f.EnableRaisingEvents = true;

            Timer manualWatcher = new Timer(ManuallWatcherCallback, null, 0, 3000);

            Timer manualTaskRunner = new Timer(ManuallRunnerCallback, null, 0, 10000);

            Console.ReadLine();
        }

        private static void F_Created(object sender, FileSystemEventArgs e)
        {
            lock (lockIbj)
            {
                toProccessFiles.Add(e.FullPath);
                Console.WriteLine("Adding new File from watcher: " + e.FullPath);
            }

        }

        private static void ManuallWatcherCallback(Object o)
        {
            var files = Directory.GetFiles(directory, extension);
            lock (lockIbj)
            {
                foreach (var file in files)
                {
                    if (!_proccessedFiles.Contains(file) && !toProccessFiles.Contains(file) && !_proccessingFiles.Contains(file))
                    {
                        toProccessFiles.Add(file);
                        Console.WriteLine("Adding new File from manuall timer: " + file);
                    }
                }

            }
        }

        private static bool processing;
        private static void ManuallRunnerCallback(Object o)
        {
            if (processing)
                return;

            while (true)
            {
                //you could proccess file in parallel
                string fileToProcces = null;

                lock (lockIbj)
                {
                    fileToProcces = toProccessFiles.FirstOrDefault();
                    if (fileToProcces != null)
                    {
                        processing = true;
                        toProccessFiles.Remove(fileToProcces);
                        _proccessingFiles.Add(fileToProcces);
                    }
                    else
                    {
                        processing = false;
                        break;


                    }
                }

                if (fileToProcces == null)
                    return;

                //Must add error handling
                ProccessFile(fileToProcces);
            }
        }

        private static void ProccessFile(string fileToProcces)
        {
            Console.WriteLine("Processing:" + fileToProcces);
            lock (lockIbj)
            {
                _proccessingFiles.Remove(fileToProcces);
                _proccessedFiles.Add(fileToProcces);
            }
        }
    }
}

Answer 2

您可以使用 P/Invoke 来运行 Win32 文件系统更改通知功能，而不是使用 FileSystemWatcher，并在文件系统更改发生时循环它们：

[DllImport("kernel32.dll", EntryPoint = "FindFirstChangeNotification")]
static extern System.IntPtr FindFirstChangeNotification (string lpPathName, bool bWatchSubtree, uint dwNotifyFilter);

[DllImport("kernel32.dll", EntryPoint = "FindNextChangeNotification")]
static extern bool FindNextChangeNotification (System.IntPtr hChangedHandle);

[DllImport("kernel32.dll", EntryPoint = "FindCloseChangeNotification")]
static extern bool FindCloseChangeNotification (System.IntPtr hChangedHandle);

[DllImport("kernel32.dll", EntryPoint = "WaitForSingleObject")]
static extern uint WaitForSingleObject (System.IntPtr handle, uint dwMilliseconds);

[DllImport("kernel32.dll", EntryPoint = "ReadDirectoryChangesW")]
static extern bool ReadDirectoryChangesW(System.IntPtr hDirectory, System.IntPtr lpBuffer, uint nBufferLength, bool bWatchSubtree, uint dwNotifyFilter, out uint lpBytesReturned, System.IntPtr lpOverlapped, ReadDirectoryChangesDelegate lpCompletionRoutine);

基本上，您使用要监视的目录调用FindFirstChangeNotification，它会为您提供等待句柄。然后，您使用句柄调用WaitForSingleObject，当它返回时，您就知道发生了一项或多项更改。然后，您致电ReadDirectoryChangesW 以了解发生了什么变化，并处理这些变化。调用FindNextChangeNotification 为您提供了等待文件系统下一次更改的句柄，因此您可能会调用它，然后调用WaitForSingleObject，然后循环调用ReadDirectoryChangesW。完成后，您可以致电FindCloseChangeNotification 停止跟踪更改。

编辑：这是一个更完整的例子：

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices;

[DllImport("kernel32.dll", EntryPoint = "FindFirstChangeNotification")]
static extern System.IntPtr FindFirstChangeNotification(string lpPathName, bool bWatchSubtree, uint dwNotifyFilter);

[DllImport("kernel32.dll", EntryPoint = "FindNextChangeNotification")]
static extern bool FindNextChangeNotification(System.IntPtr hChangedHandle);

[DllImport("kernel32.dll", EntryPoint = "FindCloseChangeNotification")]
static extern bool FindCloseChangeNotification(System.IntPtr hChangedHandle);

[DllImport("kernel32.dll", EntryPoint = "WaitForSingleObject")]
static extern uint WaitForSingleObject(System.IntPtr handle, uint dwMilliseconds);

[DllImport("kernel32.dll", EntryPoint = "ReadDirectoryChangesW")]
static extern bool ReadDirectoryChangesW(System.IntPtr hDirectory, System.IntPtr lpBuffer, uint nBufferLength, bool bWatchSubtree, uint dwNotifyFilter, out uint lpBytesReturned, System.IntPtr lpOverlapped, IntPtr lpCompletionRoutine);

[DllImport("kernel32.dll", EntryPoint = "CreateFile")]
public static extern IntPtr CreateFile(string lpFileName, uint dwDesiredAccess, uint dwShareMode, IntPtr SecurityAttributes, uint dwCreationDisposition, uint dwFlagsAndAttributes, IntPtr hTemplateFile);

enum FileSystemNotifications
{
    FileNameChanged = 0x00000001,
    DirectoryNameChanged = 0x00000002,
    FileAttributesChanged = 0x00000004,
    FileSizeChanged = 0x00000008,
    FileModified = 0x00000010,
    FileSecurityChanged = 0x00000100,
}

enum FileActions
{
    FileAdded = 0x00000001,
    FileRemoved = 0x00000002,
    FileModified = 0x00000003,
    FileRenamedOld = 0x00000004,
    FileRenamedNew = 0x00000005
}

enum FileEventType
{
    FileAdded,
    FileChanged,
    FileDeleted,
    FileRenamed
}

class FileEvent
{
    private readonly FileEventType eventType;
    private readonly FileInfo file;

    public FileEvent(string fileName, FileEventType eventType)
    {
        this.file = new FileInfo(fileName);
        this.eventType = eventType;
    }

    public FileEventType EventType => eventType;
    public FileInfo File => file;
}

[StructLayout(LayoutKind.Sequential)]
struct FileNotifyInformation
{
    public int NextEntryOffset;
    public int Action;
    public int FileNameLength;
    public IntPtr FileName;
}

class DirectoryWatcher
{
    private const int MaxChanges = 4096;
    private readonly DirectoryInfo directory;

    public DirectoryWatcher(string dirPath)
    {
        this.directory = new DirectoryInfo(dirPath);
    }

    public IEnumerable<FileEvent> Watch(bool watchSubFolders = false)
    {
        var directoryHandle = CreateFile(directory.FullName, 0x80000000, 0x00000007, IntPtr.Zero, 3, 0x02000000, IntPtr.Zero);    
        var fileCreatedDeletedOrUpdated = FileSystemNotifications.FileNameChanged | FileSystemNotifications.FileModified;
        var waitable = FindFirstChangeNotification(directory.FullName, watchSubFolders, (uint)fileCreatedDeletedOrUpdated);
        var notifySize = Marshal.SizeOf(typeof(FileNotifyInformation));

        do
        {
            WaitForSingleObject(waitable, 0xFFFFFFFF); // Infinite wait
            var changes = new FileNotifyInformation[MaxChanges];
            var pinnedArray = GCHandle.Alloc(changes, GCHandleType.Pinned);
            var buffer = pinnedArray.AddrOfPinnedObject();
            uint bytesReturned;            

            ReadDirectoryChangesW(directoryHandle, buffer, (uint)(notifySize * MaxChanges), watchSubFolders, (uint)fileCreatedDeletedOrUpdated, out bytesReturned, IntPtr.Zero, IntPtr.Zero);

            for (var i = 0; i < bytesReturned / notifySize; i += 1)
            {
                var change = Marshal.PtrToStructure<FileNotifyInformation>(new IntPtr(buffer.ToInt64() + i * notifySize));

                if ((change.Action & (int)FileActions.FileAdded) == (int)FileActions.FileAdded)
                {
                    yield return new FileEvent(Marshal.PtrToStringAuto(change.FileName, change.FileNameLength), FileEventType.FileAdded);
                }
                else if ((change.Action & (int)FileActions.FileModified) == (int)FileActions.FileModified)
                {
                    yield return new FileEvent(Marshal.PtrToStringAuto(change.FileName, change.FileNameLength), FileEventType.FileChanged);
                }
                else if ((change.Action & (int)FileActions.FileRemoved) == (int)FileActions.FileRemoved)
                {
                    yield return new FileEvent(Marshal.PtrToStringAuto(change.FileName, change.FileNameLength), FileEventType.FileDeleted);
                }
                else if ((change.Action & (int)FileActions.FileRenamedNew) == (int)FileActions.FileRenamedNew)
                {
                    yield return new FileEvent(Marshal.PtrToStringAuto(change.FileName, change.FileNameLength), FileEventType.FileRenamed);
                }
            }

            pinnedArray.Free();
        } while (FindNextChangeNotification(waitable));

        FindCloseChangeNotification(waitable);
    }
}

var watcher = new DirectoryWatcher(@"C:\Temp");

foreach (var change in watcher.Watch())
{
    Console.WriteLine("File {0} was {1}", change.File.Name, change.EventType);
}

Answer 3

我以前做过，但没有源代码（以前的工作），我遇到了同样的问题。我最终创建了一个 BackgroundWorker 实例，该实例将检查文件夹中的新文件。我会处理这些文件，然后将它们归档到一个子文件夹中。不确定这是否可能。

如果移动文件不是一个选项，BackgroundWorker 可能仍然是答案。跟踪文件的 LastModifiedDate 或 CreatedDate 并处理任何更新的。在您的 onCreated 中，您将创建一个 BackgroundWorker 实例并将其 DoWork 放在文件上。由于您的处理需要 20 秒，我假设您在 onCreated 事件逻辑中直接调用了所有逻辑。通过将其转移到另一个线程，您可以进行近乎即时的处理并完成，而另一个线程则不断搅拌直到完成。