Powershell：从大文件中获取内容（服务器列表）[重复]

Question

我有来自文本文件 (serverlist.txt) 的 100,000 个服务器列表

当我一次运行时，它会破坏我的内存和 cpu，并且需要更长的时间（大约 3 天）才能完成对 DNSlookup 的扫描。

我尝试拆分包含以下 20k 服务器列表的文件，并且可以完成扫描长达 10 分钟的每个文件。

serverlist1.txt
serverlist2.txt
serverlist3.txt
serverlist4.txt
serverlist5.txt

---

$objContainer = @()
$values = @()
$domains = Get-Content -path "serverlist1.txt"
$named = 0
$timestamp= get-date

$domains | ForEach-Object {
    $domain = $_
    nslookup $domain 2>&1 | ForEach-Object {
        if ($_ -match '^Name:\s*(.*)$') {
            $values += $matches[1]
            $named = 1;
        } elseif (($_ -match '^.*?(\d*\.\d*\.\d*\.\d*)$') -and ($named -eq 1)) {
            $values += $matches[1]
        } elseif ($_ -match '^Aliases:\s*(.*)$') {
            $values += $matches[1]
        }
    }

    $obj = New-Object -TypeName PSObject
    #$obj | Add-Member -MemberType NoteProperty -name 'Domain' -value $domain
    $obj | Add-Member -MemberType NoteProperty -name 'Name' -value $values[0]
    $obj | Add-Member -MemberType NoteProperty -name 'IP Address' -value $values[1]
    $obj | Add-Member -MemberType NoteProperty -name 'Alias' -value $values[2]
    $obj | Add-Member -MemberType NoteProperty -name 'Timestamp' -value $timestamp
    $objContainer += $obj

    $values = @()
    $named = 0
}

Write-Output $objContainer
$objContainer | Export-csv "dnslog_$((Get-Date).ToString('MM-dd-yyyy_hh-mm-ss')).csv" -NoTypeInformation

我的问题是，如何在生成 dnslog(datetime).csv 后立即执行并循环输入文本文件

例如：

1. 运行 powershell 脚本 .\filename.ps1
2. 来自 serverlist1.txt 的输入
3. 输出 dnslog(datetime).csv
4. 来自 serverlist2.txt 的输入
5. 输出 dnslog(datetime).csv
6. 来自 serverlist3.txt 的输入
7. 输出 dnslog(datetime).csv
8. 来自 serverlist4.txt 的输入
9. 输出 dnslog(datetime).csv
10. 来自 serverlist5.txt 的输入
11. 输出 dnslog(datetime).csv

完成！

如果我有超过 5 个文本文件列表，它将继续从输入文件循环直到完成。

Answer 1

除了 Chris 的回答之外，我还将向 Get-Content 添加一个 ReadCount 标志，如下所示：

Get-Content -path "serverlist1.txt" -ReadCount 1 | % {

这将省去将整个文件读入内存的麻烦。

Answer 2

您应该考虑运行此并行批处理作业。您是否已经尝试过这样做？

您可以通过删除所有对内存的提交（变量分配和使用 += 重写数组）来解决 RAM 破坏问题。

$timestamp = get-date

Get-Content -path "serverlist1.txt" | ForEach-Object {
    $domain = $_

    # You can clear this here.
    $values = @()
    $named = 0

    # There are potentially better options than nslookup.
    # Needs a bit of care to understand what's an alias here though.
    # [System.Net.Dns]::GetHostEntry($domain)
    # And if you don't like that, quite a few of us have written equivalent tools in PowerShell.
    nslookup $domain 2>&1 | ForEach-Object {
        if ($_ -match '^Name:\s*(.*)$') {
            $values += $matches[1]
            $named = 1;
        } elseif (($_ -match '^.*?(\d*\.\d*\.\d*\.\d*)$') -and ($named -eq 1)) {
            $values += $matches[1]
        } elseif ($_ -match '^Aliases:\s*(.*)$') {
            $values += $matches[1]
        }
    }

    # Leave the output object in the output pipeline
    # If you're running PowerShell 3 or better:
    [PSCustomObject]@{
        Domain       = $domain
        Name         = $values[0]
        'IP Address' = $values[1]
        Alias        = $values[2]
        TimeStamp    = $timestamp
    }
    # PowerShell 2 is less flexible. This or Select-Object.
    #$obj = New-Object -TypeName PSObject
    ##$obj | Add-Member -MemberType NoteProperty -name 'Domain' -value $domain
    #$obj | Add-Member -MemberType NoteProperty -name 'Name' -value $values[0]
    #$obj | Add-Member -MemberType NoteProperty -name 'IP Address' -value $values[1]
    #$obj | Add-Member -MemberType NoteProperty -name 'Alias' -value $values[2]
    #$obj | Add-Member -MemberType NoteProperty -name 'Timestamp' -value $timestamp
    # To leave this in the output pipeline, uncomment this
    # $obj

    # No version of PowerShell needs you to do this. It's a good way to ramp up memory usage 
    # for large data sets.
    # $objContainer += $obj
} | Export-Csv "dnslog_$(Get-Date -Format 'MM-dd-yyyy_hh-mm-ss').csv" -NoTypeInformation