解析数千个日志文件

Question

我目前必须解析大约 35000 到 50000 个日志文件来提取感兴趣的行。 由于限制和政策，我必须在没有任何外部库的情况下在 Powershell 中完成。

日志大小介于 100 kB 和 1000 kB 之间。

我将结果写入一个文件，该文件末尾大约有 500 万到 700 万行。

性能令人毛骨悚然……解析 50000 条日志并将结果写入输出文件大约需要 1 小时 15 分钟。

我只知道这对性能不利：

if (($result[1..8] -join "").Trim() -ne "") {

还有一个复杂度为 O(V*E) 的嵌套循环如果我错了，请纠正我

foreach ($file in $fileList) {
   ...
   while (($line = $reader.ReadLine()) -ne $null) {
   ... 

$search 变量保存字符串“自定义日志条目：”

根据您的要求，这里是日志文件内容的示例：

2021 年 10 月 2 日星期六 00:20:12 信息：带有一些信息的字符串： mail.address@domain.com
2021 年 10 月 2 日星期六 00:20:12 信息：第二 带有一些信息的字符串
2021 年 10 月 2 日星期六 00:20:12 信息：XZY 000000000 关于当前线路的一些信息
Sat Oct 2 00:20:12 2021 信息：XZY 000000000 发生了一些动作：动作
10 月 2 日星期六 2021 年 00:20:12 信息：XZY 000000000 使用了某些东西：使用过的对象
2021 年 10 月 2 日星期六 00:20:12 信息：XZY 000000000 有关的一些信息 当前线路
Sat Oct 2 00:20:12 2021 信息：XZY 000000000 一些 当前线路信息
Sat Oct 2 00:20:12 2021 信息： XZY 000000000 部分数据：判决否定
2021 年 10 月 2 日星期六 00:20:12 信息：XZY 000000000 自定义日志条目：重要线路
10 月 2 日星期六 2021 年 00:20:12 信息：XZY 000000000 一些信息
10 月 2 日星期六 2021 年 00:20:12 自定义日志条目：重要行

我查看了foreach -parallel (...)，但工作流的限制实在是太可怕了......

也许只是打开每个文件，将其写入 MemoryStream，然后处理所有文件（RAM 不是问题）？

你们能给我一些关于如何加快速度的建议吗？

下面是对代码的更全面的了解：

try {
    # Output stream in which we write.
    $outStream = New-Object System.IO.FileStream( `
        "C:\Users\anon\outfile.csv", `
        [System.IO.FileMode]::Create, `
        [System.IO.FileAccess]::Write, `
        [System.IO.FileAccess]::Read)
    # Writer object which is used to write to stream.
    $outWrite = New-Object System.IO.StreamWriter($outStream)
    # Iterate through files.
    foreach ($file in $fileList) {       

        try {
            # Create reader stream for log.
            $reader = New-Object System.IO.StreamReader($file)    
            # Length of time stamp
            $fileNameDateLen = fileNameDateFormat.Length 
            $fileNameDate = $file.Substring($file.Length - 2 - $fileNameDateLen, $fileNameDateLen)
            # Convert to usable DateTime object.
            $fileNameDateConverted = ([System.DateTime]::ParseExact(`
                $fileNameDate, `
                $fileNameDateFormat, `
                [System.Globalization.CultureInfo]::InvariantCulture))
            # Change format of extracted file name date.
            $fileNameDateConverted = $fileNameDateConverted.Date.ToString("yyyy-MM-dd")                        
            # StringBuilder for storing row values.
            $rowBuffer = New-Object System.Text.StringBuilder
            # Iterate through files.
            while (($line = $reader.ReadLine()) -ne $null) {
                # Validate line.
                if ($line -Match $search) {       
                    # Calc position of relevant data.
                    $pos = $line.IndexOf($search) + $searchLength
                    # Actual length of relevant data.
                    $relLength = $line.Length - $pos
                    # Extract relevant data.
                    $result = $line.Substring($pos, $relLength).Trim().Split(';')      
                    # Check if line is empty.
                    if (($result[1..8] -join "").Trim() -ne "") {
                        # Get timestamp from line.
                        #$timeValue = $timeRegex.Match($line).Value
                        $timeValue = $line.Substring(12, 8)
                        # Combine date from file name with time.
                        $dateString = "$fileNameDateConverted $timeValue"
                        # Format timestamp.
                        $timeStamp = Get-Date $dateString -Format "yyyy-MM-dd HH:mm:ss"
                        # Format last result.
                        $result[8] = $result[8] -Replace "^""|""$"
                        # Create CSV row.
                        [void] $rowBuffer.AppendLine("$timeStamp;$($result[1]);$($result[2]);" `
                            + "$($result[3]);$($result[4]);$($result[5]);" `
                            + "$($result[6]);$($result[7]);$($result[8])")          
                    }
                }
            }
            # Write results to file.
            $outWrite.Write($rowBuffer.ToString())
            # Clear buffer.
            [void] $rowBuffer.Clear()
            # Close input.
            [void] $reader.Close()
            # Free input memory.
            [void] $reader.Dispose()
        }   
        catch {
            if ($rowBuffer -ne $null) {
                [void] $rowBuffer.Clear()
            }
            if ($reader -ne $null) {
                [void] $reader.Close()
                [void] $reader.Dispose()
            }
        }
    }
    $sp.Stop()
    Write-Host "Finished after $($sp.Elapsed)"
} 
catch {
    if ($outWrite -ne $null) { 
        [void] $outWrite.Dispose()
    }
    if ($outStream -ne $null) {
        [void] $outStream.Dispose()
    }
}
finally {
    # Close and free output.
    [void] $outWrite.Close()
    [void] $outStream.Close()
    [void] $outWrite.Dispose()
    [void] $outStream.Dispose()
}

Answer 1

我遵循thepip3r 的建议并采用了多线程，这是迄今为止最相关的事情。

我没有将解析结果写入单个文件，因为在这种情况下内存对我来说不是问题。我所有的返回值都存储在 ConcurrentBag 中，这是一个线程安全的集合。
对于内存有限的每个人，您可能希望使用这样的原子写入（它会减慢您的速度）：

    # Atomic write stream object for thread-safe file append.
    $stream = New-Object System.IO.FileStream( `
        $destFile, `
        [System.IO.FileMode]::OpenOrCreate, `
        [System.Security.AccessControl.FileSystemRights]::AppendData, `
        [System.IO.FileAccess]::Write, `
        0x2000, 
        [System.IO.FileOptions]::None)

    $writer = New-Object System.IO.StreamWriter($stream)
    # Enable AutoFlush or FileStream will keep on buffering.
    $writer.AutoFlush = $true
    # Get thread results and dispose them.
    foreach ($thread in $parserThreads) {        
        $writer.WriteLine(($thread.Thread.EndInvoke($thread.Status) | Out-String))
        $thread.Thread.Dispose()
    }
    $parserThreads = $null

使用多线程后，时间从 70-83 分钟减少到 20-33 分钟（取决于加载了多少大 (r) 输入文件。

不使用 Export-CSV 或管道会更快。 管道相当慢，我认为它们是不必要的（大部分时间）。使用诸如 ConvertTo-Csv 或 Select-String 之类的东西，或者任何只是增加开销的东西，我不想要也不需要像这样的简单任务。收集输入文件已经在开始时使用管道完成，类似于post。具有单个 IF 条件的两个嵌套 foreach 循环一起比管道版本快约 200 倍（管道版本为 50 秒，嵌套循环为

感谢大家的 cmets、帮助和建议。

这是结果代码：

Clear

try {
    
    $ParseLog = {

        Param (
            # File to parse
            [string]$targetFile,
            # Search string
            [string]$searchValue
        )

        try {
            # Length of search string. 
            $svLength = $searchValue.Length            
            # Date time format found within file name.
            $fileNameDateFormat = "yyyyMMddTHHmmss"
            # Init reader object.
            $reader = New-Object System.IO.StreamReader($targetFile)    
            # Get date from file name.
            $fileNameTimeLen = $fileNameDateFormat.Length
            # Extract the date time value from the file name. 
            $fileNameTime = $targetFile.Substring($targetFile.Length - 2 - $fileNameTimeLen, $fileNameTimeLen)
            # Convert to usable DateTime object.
            $fileDateConverted = ([System.DateTime]::ParseExact(`
                $fileNameTime, `
                $fileNameDateFormat, `
                [System.Globalization.CultureInfo]::InvariantCulture))
            # Change format of extracted file name date.
            $fileDateConverted = $fileDateConverted.Date.ToString("yyyy-MM-dd")
            # StringBuilder for storing row values.
            $rowBuffer = [System.Text.StringBuilder]::new()
            # Iterate through files.
            while (($line = $reader.ReadLine()) -ne $null) {
                # Validate line.
                if ($line -Match $searchValue) {       
                    # Calc position of relevant data.
                    $pos = $line.IndexOf($searchValue) + $svLength
                    # Actual length of relevant data.
                    $relLength = $line.Length - $pos
                    # Extract relevant data.
                    $result = $line.Substring($pos, $relLength).Trim().Split(';')  
                    # Check if line is empty.
                    if (($result[1..8] -join "").Trim() -ne "") {
                        # Get timestamp from line (position and length is fixed).
                        $timeValue = $line.Substring(12, 8)
                        # Combine date from file name with time.
                        $dateString = "$fileDateConverted $timeValue"
                        # Format timestamp.
                        $timeStamp = Get-Date $dateString -Format "yyyy-MM-dd HH:mm:ss"
                        # Format last result.
                        $result[8] = $result[8] -Replace "^""|""$" -Replace "\\xc3\\xbc", "ue"
                        # Create CSV row.
                        [void] $rowBuffer.AppendLine("$timeStamp;$($result[1]);$($result[2]);" `
                            + "$($result[3]);$($result[4]);$($result[5]);" `
                            + "$($result[6]);$($result[7]);$($result[8])")          
                    }
                }
            }
            # Close input.
            [void] $reader.Close()
            # Free input memory.
            [void] $reader.Dispose()
            # Return the parser result.
            return $rowBuffer.ToString()
        }   
        catch {
            # Free memory
            if ($rowBuffer -ne $null) {
                [void] $rowBuffer.Clear()
            }
            # Free StreamReader memory.
            if ($reader -ne $null) {
                [void] $reader.Close()
                [void] $reader.Dispose()
            }
            # Return error for debugging.
            #return $error[0].Exception()
        }
    }

    # StopWatch object to measure performance.
    $sp = [System.Diagnostics.Stopwatch]::StartNew()
    # Root path of log directories
    $basePath = "\\some\network\path"
    # Test directories
    $directories = @("dir0", "dir1")
    # Text we need to look for within the the logs files.
    $search = "Custom Log Entry: "
    # Cache length of search as we need it quite often.
    $searchLength = $search.Length
    # Test start date
    $Start = Get-Date -Date "01.07.2021" 
    # Test end date
    $End = Get-Date -Date "11.10.2021"
    # List object storing file paths.
    $fileList = New-Object System.Collections.Generic.List[string]
    # List object storing directory paths. 
    $dirList = New-Object System.Collections.Generic.List[string]

    # Collect all directories and sub-directories.
    foreach ($dir in $directories) {    
        # Combine directories
        $path = "$basePath\$dir\subdir"
        # Get directory info.
        $dirInfo = New-Object System.IO.DirectoryInfo($path) 
        # iterate through sub-directories.
        foreach ($subDir in $dirInfo.GetDirectories()) {
            # Cache creation time.
            $ct = $subDir.CreationTime.Date
            # Check if folder meets requirements.
            if ($ct -ge $Start.Date -and $ct -le $End.Date) {
                $dirList.Add($subDir.FullName)
            }        
        }
    }
    $sp.Stop()
    Write-Host "Received all directories after: $($sp.Elapsed)"

    $sp.Restart()
    # Temp array for files.
    $files = @()
    # Iterate through files in folder list.
    foreach ($dir in $dirList) {
        # Get directory info.
        $dirInfo = New-Object System.IO.DirectoryInfo($dir)
        # Get all log files.
        $files = $dirInfo.GetFiles("*.log", [System.IO.SearchOption]::AllDirectories)
        # Iterate through files.
        foreach ($file in $files) {
            # Cache last write time.
            $lwt = $file.LastWriteTime
            # Check if file is applicable.
            if ($lwt -ge $Start.Date -and $lwt -le $End.Date) {
                $fileList.Add($file.FullName)
            }
        }    
    }
    $sp.Stop()
    Write-Host "Received all files after: $($sp.Elapsed)"

    # CSV header string.
    $header = "h0;h1;h2;h3;h4;h5;h6;h7;h8"
    
    # Use a threadsafe collection object to store thread results. 
    $threadSafeBuffer = [System.Collections.Concurrent.ConcurrentBag[string]]::new()
    # Create new thread pool.
    $parserThreadPool = [RunspaceFactory]::CreateRunspacePool(1, [int]$env:NUMBER_OF_PROCESSORS)
    # Set multithreaded apartment state.
    $parserThreadPool.ApartmentState = "MTA"
    $parserThreadPool.Open()
    $parserThreads = @()

    $sp.Restart()

    # Iterate through files.
    foreach ($file in $fileList) {     
        # Create thread for file.
        $parserThread = [PowerShell]::Create()
        # Load parser script.
        [void] $parserThread.AddScript($ParseLog)
        # Pass parameters.
        [void] $parserThread.AddArgument($file) 
        [void] $parserThread.AddArgument($search) 
        # Add to thread pool
        $parserThread.RunspacePool = $parserThreadPool
        # Thread callback
        $parserThreads += [PSCustomObject]@{ `
            Thread = $parserThread; `
            Status = $parserThread.BeginInvoke() `
        }                
    }

    # Wait for threads to finish.
    while ($parserThreads.Status.IsCompleted -notcontains $true) {}
    # Get thread results and dispose them.
    foreach ($thread in $parserThreads) {                
        [void] $threadSafeBuffer.TryAdd($thread.Thread.EndInvoke($thread.Status))
        $thread.Thread.Dispose()
    }
    $sp.Stop()
    Write-Host "Finished parsing after: $($sp.Elapsed)"

    $sp.Restart()
    # Create stream and writer for output.
    $stream = New-Object System.IO.FileStream( `
        "c:\users\anon\documents\parser_result\result.csv", `
        [System.IO.FileMode]::Create, `
        [System.IO.FileAccess]::Write, `
        [System.IO.FileAccess]::Read)
    $writer = New-Object System.IO.StreamWriter($stream)
    
    # Write results into file.         
    foreach ($item in $threadSafeBuffer) {
        $writer.WriteLine($item)
    }

    # Free memory.
    $threadSafeBuffer = $null
    $writer.Close()
    $writer.Dispose()
    $stream.Close()
    $stream.Dispose()
    $parserThreadPool.Close()
    $parserThreadPool.Dispose()

    $sp.Stop()
    Write-Host "Finished writing parser results after $($sp.Elapsed)"
} 
catch {

    $threadSafeBuffer = $null

    if ($parserThreadPool -ne $null) {
        $parserThreadPool.Close() 
        $parserThreadPool.Dispose() 
    }

    if ($writer -ne $null) {
        $writer.Close()
        $writer.Dispose()
    }   
    
    if ($stream -ne $null) {     
        $stream.Close()
        $stream.Dispose()
    }
}