【问题标题】:Optimize Speed of Recursive File Search in Subdirectories?优化子目录中递归文件搜索的速度?
【发布时间】:2015-05-28 15:41:49
【问题描述】:

我正在寻找使用 Excel 宏递归搜索子目录以查找文件模式的最快执行方法。 Excel VBA 在这方面似乎相当缓慢。

到目前为止我尝试过的事情(一些基于其他 stackoverflow 建议):

  • 专门使用 Dir 递归遍历子目录并在每个文件夹中搜索文件模式。 (最慢)
  • 使用 Folder.Files 集合遍历 FileSystemObject 文件夹,根据文件模式检查每个文件。 (更好,但仍然很慢)
  • 遍历 FileSystemObject 文件夹,然后使用 Dir 检查每个文件夹的文件模式(目前最快,但每个文件仍需要几秒钟,如果可能,我想优化)

我查看了 My.Computer.FileSystem.GetFiles,它看起来很完美(允许您指定通配符模式并使用单个命令搜索子文件夹) - 但 Excel 似乎不支持它据我所知,VBA 仅在 VB 中。

我目前正在使用下面的 FindFile Sub,它是迄今为止性能最好的。如果有人对如何进一步改进这一点有任何建议,我将非常感激!

Option Explicit
Private Declare Function GetTickCount Lib "kernel32" () As Long


Function Recurse(sPath As String, targetName As String) As String

    Dim FSO As New FileSystemObject
    Dim myFolder As Folder
    Dim mySubFolder As Folder
    Dim myFile As File

    On Error Resume Next
    Set myFolder = FSO.GetFolder(sPath)
    If Err.Number <> 0 Then
        MsgBox "Error accessing " & sPath & ". The macro will abort."
        Err.Clear
        Exit Function
    End If
    On Error GoTo 0

    Dim foundFolderPath As String
    Dim foundFileName As String
    foundFolderPath = ""
    foundFileName = ""

    For Each mySubFolder In myFolder.SubFolders

        foundFileName = Dir(mySubFolder.Path & "\" & targetName & "*")
        If foundFileName <> vbNullString Then
            foundFolderPath = mySubFolder.Path & "\" & foundFileName
        End If

        If foundFolderPath <> vbNullString Then
            Recurse = foundFolderPath
            Exit Function
        End If

        foundFolderPath = Recurse(mySubFolder.Path, targetName)

        If foundFolderPath <> vbNullString Then
            Recurse = foundFolderPath
            Exit Function
        End If
    Next

End Function


Sub FindFile()

    Dim start As Long
    start = GetTickCount()

    Dim targetName As String
    Dim targetPath As String
    targetName = Range("A1").Value 'Target file name without extension
    targetPath = "C:\Example\" & Range("B1").Value 'Subfolder name

    Dim target As String
    target = Recurse(targetPath, targetName)

    Dim finish As Long
    finish = GetTickCount()

    MsgBox "found: " & target & vbNewLine & vbNewLine & (finish - start) & " milliseconds"

End Sub

根据接受的答案更新文件搜索功能

这个版本的 FindFile() 的执行速度大约是我最初粘贴在上述问题中的方法的两倍。正如以下帖子中所讨论的,这应该适用于 32 位或 64 位版本的 Excel 2010 及更高版本。

Option Explicit

Private Declare PtrSafe Function FindClose Lib "kernel32" (ByVal hFindFile As LongPtr) As Long
Private Declare PtrSafe Function FindFirstFileW Lib "kernel32" (ByVal lpFileName As LongPtr, ByVal lpFindFileData As LongPtr) As LongPtr
Private Declare PtrSafe Function FindNextFileW Lib "kernel32" (ByVal hFindFile As LongPtr, ByVal lpFindFileData As LongPtr) As LongPtr

Private Type FILETIME
  dwLowDateTime  As Long
  dwHighDateTime As Long
End Type

Const MAX_PATH  As Long = 260
Const ALTERNATE As Long = 14

' Can be used with either W or A functions
' Pass VarPtr(wfd) to W or simply wfd to A
Private Type WIN32_FIND_DATA
  dwFileAttributes As Long
  ftCreationTime   As FILETIME
  ftLastAccessTime As FILETIME
  ftLastWriteTime  As FILETIME
  nFileSizeHigh    As Long
  nFileSizeLow     As Long
  dwReserved0      As Long
  dwReserved1      As Long
  cFileName        As String * MAX_PATH
  cAlternate       As String * ALTERNATE
End Type

Private Const FILE_ATTRIBUTE_DIRECTORY As Long = 16 '0x10
Private Const INVALID_HANDLE_VALUE As LongPtr = -1

Function Recurse(folderPath As String, fileName As String)
    Dim fileHandle    As LongPtr
    Dim searchPattern As String
    Dim foundPath     As String
    Dim foundItem     As String
    Dim fileData      As WIN32_FIND_DATA

    searchPattern = folderPath & "\*"

    foundPath = vbNullString
    fileHandle = FindFirstFileW(StrPtr(searchPattern), VarPtr(fileData))
    If fileHandle <> INVALID_HANDLE_VALUE Then
        Do
            foundItem = Left$(fileData.cFileName, InStr(fileData.cFileName, vbNullChar) - 1)

            If foundItem = "." Or foundItem = ".." Then 'Skip metadirectories
            'Found Directory
            ElseIf fileData.dwFileAttributes And FILE_ATTRIBUTE_DIRECTORY Then
                foundPath = Recurse(folderPath & "\" & foundItem, fileName)
            'Found File
            'ElseIf StrComp(foundItem, fileName, vbTextCompare) = 0 Then 'these seem about equal
            ElseIf InStr(1, foundItem, fileName, vbTextCompare) > 0 Then 'for performance
                foundPath = folderPath & "\" & foundItem
            End If

            If foundPath <> vbNullString Then
                Recurse = foundPath
                Exit Function
            End If

        Loop While FindNextFileW(fileHandle, VarPtr(fileData))
    End If

    'No Match Found
    Recurse = vbNullString
End Function

Sub FindFile()

    Dim targetName As String
    Dim targetPath As String
    targetName = Range("A4").Value
    targetPath = "C:\Example\" & Range("B4").Value

    Dim target As String
    target = Recurse(targetPath, targetName)

    MsgBox "found: " & target

End Sub

【问题讨论】:

  • 您已经在使用来自kernel32 的函数来为您的日常工作计时,我建议您在某个地方有更快的方法来搜索驱动器。我不确定你需要什么函数调用,但我认为通过 Win API 看一下就可以了。
  • 如果这部分的速度很关键,你能把它移到 Excel 之外吗?不确定速度,但这是related post on that front
  • +1 表示问题。不确定这是否很快,但你看过FindFirstFileFindNextFile 吗?有一个例子here。 API 调用应该会影响性能,但总体而言它们可能会更快。
  • 为了将来参考,如果您发布有关 64 位版本 Excel 的问题,您应该指定。出于兼容性原因,Microsoft 建议使用 32 位版本。如果您没有使用推荐的兼容版本,您想告诉人们,以便他们可以为您提供适当的帮助。
  • 我发现使用 windows shell 比原生 VBA 或 FileSystemObject 方法快得多。

标签: performance vba excel


【解决方案1】:

使用 FindFirstFile 或 FindFirstFileEx。内置的原生 API 的执行速度将比 VBA 快得多。

答案在 stackoverflow 上:https://stackoverflow.com/a/3865850/2250183
如该答案所述,您可以在此处找到示例代码:http://www.xtremevbtalk.com/showpost.php?p=1157418&postcount=4

更新了 64 位支持的示例

此代码应适用于 Excel 2010 及更高版本的 64 位和 32 位。它不适用于早期版本的 Excel。如果您打算使用 64 位版本,我建议您阅读 documentation on 64 bit support in VBA。该文档还解释了如何添加对早期版本的 Excel 的支持。

Option Explicit

Private Declare PtrSafe Function FindClose Lib "kernel32" (ByVal hFindFile As LongPtr) As Long
Private Declare PtrSafe Function FindFirstFileW Lib "kernel32" (ByVal lpFileName As LongPtr, ByVal lpFindFileData As LongPtr) As LongPtr
Private Declare PtrSafe Function FindNextFileW Lib "kernel32" (ByVal hFindFile As LongPtr, ByVal lpFindFileData As LongPtr) As LongPtr

Private Type FILETIME
  dwLowDateTime  As Long
  dwHighDateTime As Long
End Type

Const MAX_PATH  As Long = 260
Const ALTERNATE As Long = 14

' Can be used with either W or A functions
' Pass VarPtr(wfd) to W or simply wfd to A
Private Type WIN32_FIND_DATA
  dwFileAttributes As Long
  ftCreationTime   As FILETIME
  ftLastAccessTime As FILETIME
  ftLastWriteTime  As FILETIME
  nFileSizeHigh    As Long
  nFileSizeLow     As Long
  dwReserved0      As Long
  dwReserved1      As Long
  cFileName        As String * MAX_PATH
  cAlternate       As String * ALTERNATE
End Type

Private Const INVALID_HANDLE_VALUE As LongPtr = -1

Private Sub Form_Load()
  Dim hFile     As LongPtr
  Dim sFileName As String
  Dim wfd       As WIN32_FIND_DATA

  sFileName = "c:\*.*" ' Can be up to 32,767 chars

  hFile = FindFirstFileW(StrPtr(sFileName), VarPtr(wfd))

  If hFile <> INVALID_HANDLE_VALUE Then
    Do While FindNextFileW(hFile, VarPtr(wfd))
      Debug.Print Left$(wfd.cFileName, InStr(wfd.cFileName, vbNullChar) - 1)
    Loop

    FindClose hFile
  End If
End Sub

【讨论】:

  • 我在尝试使用此解决方案时遇到了问题,因为 FindFirstFile 函数似乎需要 32 位指针,但我使用的是 64 位系统。特别是,在链接的示例代码中使用 StrPtr 会引发错误。 Excel 还要求我将 FindFirstFile 函数声明为 PtrSafe(不确定这是否会导致问题)。 Microsoft 似乎已经发布了一个“修补程序”来解决这个问题 (support.microsoft.com/en-us/kb/983246) 但这对我来说是一个死胡同,因为我需要能够在不附加任何条件的情况下分发这个宏。
  • 您不需要修补程序,它不是这样做的。您应该阅读有关 64 位 VBA 的文档。我已经为您更新了示例,但同样,如果您要使用 64 位版本,您应该阅读文档。您会经常遇到兼容性问题,它们很容易修复,但您需要了解它们是什么以及它们发生的原因。
【解决方案2】:

我遇到了类似的性能问题,我使用上面建议的 win API 函数解决了问题,我的问题与您的问题略有不同,因为我不需要递归搜索目录树,我只是将文件名从给定文件夹中提取到一个收藏,但您可能可以修改我的代码:

'for windows API call to FindFirstFileEx
Private Const INVALID_HANDLE_VALUE = -1
Private Const MAX_PATH = 260

Private Type FILETIME
    dwLowDateTime   As Long
    dwHighDateTime  As Long
End Type

Private Type WIN32_FIND_DATA
    dwFileAttributes    As Long
    ftCreationTime      As FILETIME
    ftLastAccessTime    As FILETIME
    ftLastWriteTime     As FILETIME
    nFileSizeHigh       As Long
    nFileSizeLow        As Long
    dwReserved0         As Long
    dwReserved1         As Long
    cFileName           As String * MAX_PATH
    cAlternate          As String * 14
End Type

Private Const FIND_FIRST_EX_CASE_SENSITIVE  As Long = 1
'MSDN: "This value is not supported until Windows Server 2008 R2 and Windows 7."
Private Const FIND_FIRST_EX_LARGE_FETCH     As Long = 2

Private Enum FINDEX_SEARCH_OPS
    FindExSearchNameMatch
    FindExSearchLimitToDirectories
    FindExSearchLimitToDevices
End Enum

Private Enum FINDEX_INFO_LEVELS
    FindExInfoStandard
    FindExInfoBasic 'MSDN: "This value is not supported until Windows Server 2008 R2 and Windows 7."
    FindExInfoMaxInfoLevel
End Enum

Private Declare Function FindFirstFileEx Lib "kernel32" Alias "FindFirstFileExA" ( _
ByVal lpFileName As String, ByVal fInfoLevelId As Long, lpFindFileData As WIN32_FIND_DATA, _
    ByVal fSearchOp As Long, ByVal lpSearchFilter As Long, ByVal dwAdditionalFlags As Long) As Long
Private Declare Function FindNextFile Lib "kernel32" Alias "FindNextFileA" ( _
    ByVal hFindFile As Long, lpFindFileData As WIN32_FIND_DATA) As Long
Private Declare Function FindClose Lib "kernel32" (ByVal hFindFile As Long) As Long


Private Function GetFileNames(ByVal sPath As String) As Collection

    Dim fileInfo    As WIN32_FIND_DATA  'buffer for file info
    Dim hFile       As Long             'file handle
    Dim colFiles    As New Collection

    sPath = sPath & "*.*"

    hFile = FindFirstFileEx(sPath & vbNullChar, FindExInfoBasic, fileInfo, FindExSearchNameMatch, 0&, FIND_FIRST_EX_LARGE_FETCH)

    If hFile <> INVALID_HANDLE_VALUE Then
        Do While FindNextFile(hFile, fileInfo)
            colFiles.Add Left(fileInfo.cFileName, InStr(fileInfo.cFileName, vbNullChar) - 1)
        Loop

        FindClose hFile
    End If

    Set GetFileNames = colFiles

End Function

【讨论】:

    【解决方案3】:

    我花了几天时间解决这个问题并想出了这段代码。从 reddit 帖子中获取第一部分(感谢)并对其进行了一些修改。我插入了几个不同的目录情况,搜索出现在相似的时间。

    在 223 个文件夹中找到。 FindFast:23.42 秒,递归:24.14 秒。 如果我在最后一个文件夹中选择一个文件来签入 FindFast,我们会在 387 个文件夹中找到。 FindFast:62.82 秒,递归:0.3 秒。所以检查文件夹的顺序是不一样的。

    有一些差异需要注意。我的代码的最初目的是获取所有基于通配符的 xl 文件,例如“*_ThisName.xlsx”。它最终在 9 秒内给了我全部 41 个。我能够为我的多个文件搜索节省 10 秒是因为我可以指定我要查找的文件位于名为“Working”的子目录中,并且我将目录计数限制为 10 深.我注释掉了这个测试的这些限制,它增加了 10 秒才能找到一个文件。

    我仍然希望我们可以进一步缩短搜索时间。

        Function FindFast(TargetFolder As String, Patt As String)
           
            Dim Folder As Object, SubFolder As Object, File As Object
            Dim FQueue As New Collection
    
           'Test view all folders:
    '       Dim FolderColl As New Collection
            
           Dim Count As Integer
    
            Dim fl As String
               
            With CreateObject("Scripting.FileSystemObject")
    
                FQueue.Add .GetFolder(TargetFolder)
                Do While FQueue.Count > 0
                   Set Folder = FQueue(1)
                   FQueue.Remove 1
                    'Code for individual folder
                    For Each SubFolder In Folder.subFolders
    
                        'Test view all folders:
                        FolderColl.Add SubFolder
    
                        'Only 10 folders deep
         '               Count = Len(SubFolder) - Len(Replace(SubFolder, "\", ""))
         '               If Count < 13 Then
                            FQueue.Add SubFolder
                       
                 '           ' Only look for the file in Working folder
                   '        If InStr(1, SubFolder, "Working") > 1 Then
        
                                fl = Dir(SubFolder & "\" & Patt)
                    ' Added as exact match return.  Otherwise will find all with pattern match
                    If fl <> "" Then
                        FindFast = SubFolder & "\" & fl
                        Exit Function
                    End If
                                   
                     '       End If
            '             End If
                    Next SubFolder
                Loop
    
    '   Test view all folders:
    ' Dim i As Long
    'For i = 1 To FolderColl.Count
    '                                Range("A" & i).value = FolderColl(i)
    'Next i           
    
            End With
             FindFast = vbNullString
        End Function
        
        
        Sub FindFile()
            Dim StartTime As Double
            Dim SecondsElapsed As Double
            Dim target As String
        
            Dim targetName As String
            Dim targetPath As String
            targetName = "5-3-21_Order_Sent.xlsx"
           '    Patt = "*_Order_Sent.xlsx"
            ' or wild extension   Patt = "*_ThisName.*"
        
            targetPath = "\\Fulfill\Company\Orders\Completed"
    
            StartTime = Timer    
            target = FindFast(targetPath, targetName)
            Debug.Print target
            SecondsElapsed = Round(Timer - StartTime, 2)
            Debug.Print "FindFast: " & SecondsElapsed & " Secs"
        
            MsgBox "found FindFast: " & target & " - " & SecondsElapsed & " Secs"       
    
            StartTime = Timer   
            target = Recurse(targetPath, targetName)
            Debug.Print target
        
             SecondsElapsed = Round(Timer - StartTime, 2)
            Debug.Print "Recurse: " & SecondsElapsed & " Secs"
        
             MsgBox "found Recurse: " & target & " - " & SecondsElapsed & " Secs"
        
        End Sub
    

    所以在上面的这个表单中,它正在加载所有文件夹,按子文件夹筛选它们,子文件夹从最旧到最新。在一种形式中,我希望在最新文件夹中的文件中找到一条记录,目录越旧的可能性越小。在另一种用法中,我希望获取所有文件的列表,并在可能是最新的文件管理器中寻找记录,但可能更旧,文件越旧的可能性越小。

    【讨论】:

      猜你喜欢
      • 2012-04-07
      • 1970-01-01
      • 2014-01-10
      • 2011-04-21
      • 2018-08-26
      • 2014-07-07
      • 1970-01-01
      • 2021-05-25
      相关资源
      最近更新 更多