Powershell FTP 发送大文件 System.OutOfMemoryException

Question

我有一个部分基于此处的脚本：Upload files with FTP using PowerShell

这一切都适用于小文件，但我正在尝试使用它来使我们用于将访问 mdb 文件导出到只有 ftp 的客户端的过程更加健壮。

我的第一个测试涉及一个 10MB 的文件，我在 Get-Content 阶段遇到了 System.OutOfMemoryException

在获取尝试期间，powershell ISE 的使用量接近 2GIG。

这是一个完整的脚本示例（请保持温和。我对它相当陌生）：

#####
# User variables to control the script
#####

# How many times connection will be re-tried
$connectionTries = 5
#time between tries in seconds
$connectionTryInterval = 300
#Where to log the output
$logFile = "D:\MyPath\ftplog.txt"
#maximum log file size in KB before it is archived
$logFileMaxSize = 500

#formatted date part for the specific file to transfer
#This is appended to the filename base. Leave as "" for none
$datePart = ""
#base part of the file name
$fileNameBase = "Myfile"
#file extension
$fileExtension = ".mdb"
#location of the source file (please include trailing backslash)
$sourceLocation = "D:\MyPath\"

#location and credentials of the target ftp server    
$userName = "iamafish"
$password = "ihavenofingers"
$ftpServer = "10.0.1.100"

######
# Main Script
#####

#If there is a log file and it is longer than the declared limit then archive it with  the current timestamp
if (test-path $logfile)
{
    if( $((get-item $logFile).Length/1kb) -gt $logFileMaxSize)
    {
        write-host $("archiving log to ftplog_" + (get-date -format yyyyMMddhhmmss) +     ".txt")
        rename-item $logFile $("ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt")
    }
}

#start new log entry
#Add-Content $logFile "___________________________________________________________"
#write-host $logEntry

#contruct source file and destination uri
$fileName = $fileNameBase + $datePart + $fileExtension
$sourceFile = $sourceLocation + $fileName
$sourceuri = "ftp://" + $ftpServer + "/" + $fileName


# Create a FTPWebRequest object to handle the connection to the ftp server
$ftprequest = [System.Net.FtpWebRequest]::create($sourceuri)

# set the request's network credentials for an authenticated connection
$ftprequest.Credentials = New-Object System.Net.NetworkCredential($username,$password)

$ftprequest.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftprequest.UseBinary = $true
$ftprequest.KeepAlive = $false

$succeeded = $true
$errorMessage = ""

# read in the file to upload as a byte array
trap [exception]{
    $script:succeeded = $false
    $script:errorMessage = $_.Exception.Message
    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|" +    $_.Exception.Message)
    #write-host $logEntry
    #write-host $("TRAPPED: " + $_.Exception.GetType().FullName)
    #write-host $("TRAPPED: " + $_.Exception.Message)
    exit
}
#The -ea 1 forces the error to be trappable
$content = gc -en byte $sourceFile -ea 1


$try = 0

do{
    trap [System.Net.WebException]{
        $script:succeeded = $false
        $script:errorMessage = $_.Exception.Message
        Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|" +    $_.Exception.Message)
        #write-host $logEntry
        #write-host $("TRAPPED: " + $_.Exception.GetType().FullName)
        $script:try++
        start-sleep -s $connectionTryInterval
        continue
        }
        $ftpresponse = $ftprequest.GetResponse()

} while(($try -le $connectionTries) -and (-not $succeeded))

if ($succeeded) { 

    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|0|" +    "Starting file transfer.")
    # get the request stream, and write the bytes into it
    $rs = $ftprequest.GetRequestStream()
    $rs.Write($content, 0, $content.Length)
    # be sure to clean up after ourselves
    $rs.Close()
    $rs.Dispose()
    $content.Close()
    $content.Dispose()
    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|0|" +    "Transfer complete.")
    #write-host $logEntry
}

我不能在 cmets 中放入代码，多亏了来自 keith 的指针，我已将文件访问位向下移动到底部，以便像这样将它与另一个链接..

trap [Exception]{
    $script:succeeded = $false
    $script:errorMessage = $_.Exception.Message
    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|Check File Connection|" + $_.Exception.Message)
    $sourceStream.Close()
    $sourceStream.Dispose()
    #write-host $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|Attempt to open file|" + $_.Exception.Message)
    #write-host $("TRAPPED: " + $_.Exception.GetType().FullName)
    exit
}
$sourceStream = New-Object IO.FileStream ($(New-Object System.IO.FileInfo $sourceFile),[IO.FileMode]::Open)
[byte[]]$readbuffer = New-Object byte[] 1024

# get the request stream, and write the bytes into it
$rs = $ftprequest.GetRequestStream()
do{
    $readlength = $sourceStream.Read($readbuffer,0,1024)
    $rs.Write($readbuffer,0,$readlength)
} while ($readlength -ne 0)

我只需要弄清楚为什么会得到：使用“0”参数调用“GetResponse”的异常：“无法访问已处置的对象。 每隔一次我运行它。这是在 ISE 中运行它的怪癖，还是我在初始声明或最终处置方面做了一些严重错误的事情？

完成后我会发布完整的最终脚本，因为我认为它会成为一个带有错误捕获和日志记录的坚固的 ftp 导出示例。

---

好的，这是完整的脚本。 Dispose 已被编辑，但无论是否在 5 分钟内运行脚本，都会给我一条消息，指出我无法使用已处理的对象，或者告诉我 getResponse() 产生了错误 (226) 文件传输（在 ISE 中运行） .虽然这在正常操作期间不会出现问题，但我想正确注销 FTP 会话并在脚本末尾清理资源，并确保我根据需要正确声明了它们。

#####
# User variables to control the script
#####

# How many times connection will be re-tried
$connectionTries = 5
#time between tries in seconds
$connectionTryInterval = 1
#Where to log the output
$logFile = "D:\MyPath\ftplog.txt"
#maximum log file size in KB before it is archived
$logFileMaxSize = 500
#log to file or console - #true=log to file, #false = log to console
$logToFile=$false

#formatted date part for the specific file to transfer
#This is appended to the filename base. Leave as "" for none
$datePart = ""
#base part of the file name
$fileNameBase = "MyFile"
#file extension
$fileExtension = ".mdb"
#location of the source file (please include trailing backslash)
$sourceLocation = "D:\MyPath\"

#location and credentials of the target ftp server
$userName = "iamafish"
$password = "ihavenofingers"
$ftpServer = "10.0.1.100"

######
# Main Script
#####

function logEntry($entryType, $section, $message)
{
    #just to make a one point switch for logging to console for testing
    # $entryType: 0 = success, 1 = Error
    # $section: The section of the script the log entry was generated from
    # $message: the log message

    #This is pipe separated to fit in with my standard MSSQL linked flat file schema for easy querying
    $logString = "$(get-Date -format "yyyy-MM-dd hh:mm:ss")|$entryType|$section|$message"

    if($script:logtoFile)
    {
        Add-Content $logFile $logString
    }
    else
    {
        write-host $logString
    }
}

#If there is a log file and it is longer than the declared limit then archive it with the current timestamp
if (test-path $logfile)
{
    if( $((get-item $logFile).Length/1kb) -gt $logFileMaxSize)
    {
        write-host $("archiving log to ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt")
        rename-item $logFile $("ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt")
        New-Item $logFile -type file
    }
}
else
{
    New-Item $logFile -type file
}


#contruct source file and destination uri
$fileName = $fileNameBase + $datePart + $fileExtension
$sourceFile = $sourceLocation + $fileName
$destination = "ftp://" + $ftpServer + "/" + $fileName


#Check if the source file exists
if ((test-path $sourceFile) -eq $false)
{
    logEntry 1 "Check Source File" $("File not found: " + $sourceFile)
    Exit
}


# Create a FTPWebRequest object to handle the connection to the ftp server
$ftpRequest = [System.Net.FtpWebRequest]::create($destination)

# set the request's network credentials for an authenticated connection
$ftpRequest.Credentials = New-Object System.Net.NetworkCredential($username,$password)
$ftpRequest.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftpRequest.UseBinary = $true
$ftpRequest.KeepAlive = $false

$succeeded = $true
$try = 1

do{
    trap [Exception]{
        $script:succeeded = $false
        logEntry 1 "Check FTP Connection" $_.Exception.Message
        $script:try++
        start-sleep -s $connectionTryInterval
        continue
        }
        $ftpResponse = $ftpRequest.GetResponse()

} while(($try -le $connectionTries) -and (-not $succeeded))

if ($succeeded) {
    logEntry 0 "Connection to FTP" "Success"


    # Open a filestream to the source file
    trap [Exception]{
        logEntry 1 "Check File Connection" $_.Exception.Message
        $sourceStream.Close()
        $ftpResponse.Close()
        exit
    }
    $sourceStream = New-Object IO.FileStream ($(New-Object System.IO.FileInfo $sourceFile),[IO.FileMode]::Open)
    [byte[]]$readbuffer = New-Object byte[] 1024

    logEntry 0 "Starting file transfer" "Success"
    # get the request stream, and write the bytes into it
    $rs = $ftpRequest.GetRequestStream()
    do{
        $readlength = $sourceStream.Read($readbuffer,0,1024)
        $rs.Write($readbuffer,0,$readlength)
    } while ($readlength -ne 0)

    logEntry 0 "Transfer complete" "Success"
    # be sure to clean up after ourselves
    $rs.Close()
    #$rs.Dispose()
    $sourceStream.Close()
    #$sourceStream.Dispose()

}
$ftpResponse.Close()

---

尝试在末尾捕获 Transfer OK 响应的示例：

logEntry 0 "Starting file transfer" "Success"
# get the request stream, and write the bytes into it
$rs = $ftpRequest.GetRequestStream()
do{
    $readlength = $sourceStream.Read($readbuffer,0,1024)
    $rs.Write($readbuffer,0,$readlength)
} while ($readlength -ne 0)
$rs.Close()
#start-sleep -s 2

trap [Exception]{
    $script:succeeded = $false
    logEntry 1 "Check FTP Connection" $_.Exception.Message
    continue
}
$ftpResponse = $ftpRequest.GetResponse()

Answer 1

我自己遇到了类似的问题，RAM 使用量达到 GB 上传 3MB 文件，我发现替换：

 $content = gc -en byte $sourceFile

与：

 $content = [System.IO.File]::ReadAllBytes($sourceFile)

提供更好的性能。正如其他地方所提到的，对于 真正 大文件来说，分块将是一个更好的解决方案，因为这样你就不会一次将整个文件保存在内存中，但上面的代码至少只消耗 ~(文件大小) 字节的 RAM，这意味着它应该可以达到大约 10 秒的 MB 范围。

Answer 2

与其使用 Get-Content 将整个文件读入内存，不如尝试一次读取一个块并将其写入 FTP 请求流。我会使用一种较低级别的 .NET 文件流 API 来进行读取。诚然，您不会认为 10MB 会造成内存问题。

另外，请确保您在获取请求流并写入后得到响应。响应流的获取是上传数据的内容。来自文档：

当使用 FtpWebRequest 对象 将文件上传到服务器，您必须 将文件内容写入请求 通过调用获得的流 GetRequestStream 方法或其 异步对应物， BeginGetRequestStream 和 EndGetRequestStream 方法。你必须 写入流并关闭 在发送请求之前进行流式传输。

请求通过以下方式发送到服务器 调用 GetResponse 方法或其 异步对应物， BeginGetResponse 和 EndGetResponse 方法。当请求的操作 完成后，一个 FtpWebResponse 对象是 回来。 FtpWebResponse 对象 提供操作状态 以及从 服务器。