【发布时间】:2018-07-12 18:46:18
【问题描述】:
所以我有一个脚本,旨在将一些相当大的 (700mb) txt 文件发送到我们使用的服务的 FTP,该服务会自动设置我们在一系列域中的价格。
我正在使用 bitarray(如在此处找到)将其上传到站点,并且我有一些非常基本的内置错误处理以及一些数据库连接。
如果我们单独运行它们,它们会移动得非常快。无论出于何种原因,在一个区块完成后,直到开始另一个区块的时间都非常不稳定。有时是 2 分钟,有时脚本在移动到下一个块之前只停留了 40 分钟。
我有点假设问题是我给的东西比我应该做的更多?还值得注意的是,即使停止脚本有时也需要 15 分钟。 (比如,如果我只是在脚本运行的中间点击中断,可能需要 15-20 分钟才能停止)
此外,就其价值而言,过去几天脚本在运行时方面变得更糟了。我不知道我们可以进行哪些更改以使其开始花费更长的时间,但我们到了。
无论如何,任何见解都会受到赞赏。
注意:我在清除变量时不会清除内容变量,是吗? 我应该保持rs打开吗?我不认为我可以,因为我使用不同的用户名连接到 FTP。
这是代码(我实际上有大约 12 个 *.txt 块,但它们是相同的,所以我在这里将其保留为三个):
#================================== SETUP BLOCK ==================================#
#get dat email ready
$strpasswd = Get-Content "PASSWORDFILE" | ConvertTo-SecureString
$mycreds = New-Object System.Management.Automation.PSCredential ("EXCHANGEUSER",$strpasswd)
$EmailTo = 'some@email','goes@here'
$EmailFrom = 'EXCHANGEUSER'
$EmailSubject = "CA Feed Issue Undefined Subject"
$emailbody = "Body Not Yet Defined"
$SmtpServer = 'MUHSERVER'
#Opens up database session so we can send queries
$strserver = "Server\MUHDB"
$strdatabase = "logs"
$strusername = "EXCHANGEUSER"
#createsdatabaseconnection
$sqlConnection = new-object System.Data.SqlClient.SqlConnection "server='$strserver';database='$strdatabase';Integrated Security=SSPI; User ID='$strusername'; password='$strpassword'"
$sqlConnection.Open()
#define the defaultquery
$strQuery =
"
INSERT INTO [logs].[dbo].[EventLog] (SourceID, Started, Completed, Result, Context, Machine)
values (50,1500,1500,'NOTEDEFINED','NOTDEFINED','Server\MUHCLIENTMACHINE-CAFeed')
"
#this is how I execute the command
#$sqlCommand = $sqlConnection.CreateCommand()
#$sqlCommand.CommandText = $strquery
#$sqlCommand.ExecuteReader()
#==================================Luna.txt ==================================#
##DEFINE THESE TO CREATE NEW FEEDS
$strFilename = "\\PATH\Luna.txt"
$ftp = [System.Net.FtpWebRequest]::Create("FTPLINK1")
$user = "USERNAME1"
$password = "PASSWORDREDACTED"
# create the FtpWebRequest and configure it
$ftp = [System.Net.FtpWebRequest]$ftp
# build authentication and connection
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user,$password)
$ftp.UseBinary = $true
$ftp.UsePassive = $true
$ftp.timeout = -1
#start a timer and error handling
$starttime = (get-date).ToString()
$error.Clear()
# read in the file to upload as a byte array
$content = [System.IO.File]::ReadAllBytes("$strfilename")
$ftp.ContentLength = $content.Length
# get the request stream, and write the bytes into it
$rs = $ftp.GetRequestStream()
$rs.Write($content, 0, $content.Length)
$endtime = (get-date).ToString()
#error handle
if ($error)
{
#Assemble the Query
$sqlresult = "THERE IS AN ERROR, Check the error email for details"
$sqlcontext = ($strfilename + '|' + $content.length + ' bytes')
$strquery =
"INSERT INTO [logs].[dbo].[EventLog] (SourceID, Started, Completed, Result, Context, Machine)
values (50,'$starttime','$endtime','$sqlresult','$sqlcontext','Server\MUHCLIENTMACHINE-CAFEEDSCRIPT')"
#Create Command and Execute.
$sqlCommand = $sqlConnection.CreateCommand()
$sqlCommand.CommandText = $strQuery
$sqlCommand.ExecuteNonQuery()
#Send dem emails
$emailbody = "A file for the CA Feed failed on $strfilename at " + (get-date).ToString() + " with the error '$error[0]'"
$emailsubject = "CA Feed Failed File"
Send-MailMessage -SmtpServer $SmtpServer -to $EmailTo -from $EmailFrom -subject $EmailSubject -Body $emailbody
}
else
{
write-host ("$strfilename" + ' Ran Without Errors')
$sqlresult = "RAN WITHOUT ERRORS"
$sqlcontext = ($strfilename + '|' + $content.length + ' bytes')
$strquery =
"INSERT INTO [logs].[dbo].[EventLog] (SourceID, Started, Completed, Result, Context, Machine)
values (50,'$starttime','$endtime','$sqlresult','$sqlcontext','Server\MUHCLIENTMACHINE-CAFEEDSCRIPT')"
#Create a command object.
$sqlCommand = $sqlConnection.CreateCommand()
$sqlCommand.CommandText = $strQuery
$sqlCommand.ExecuteNonQuery()
}
# be sure to clean up after ourselves and get ready for next block
Clear-Variable -Name starttime,endtime,strfilename,sqlresult,sqlcontext,ftp
$rs.Close()
$rs.Dispose()
#==================================LDE.txt ==================================#
##DEFINE THESE TO CREATE NEW FEEDS
$strFilename = "\\PATH\LDE.txt"
$ftp = [System.Net.FtpWebRequest]::Create("FTPLINK2")
$user = "USERNAME2"
$password = "PASSWORDREDACTED"
# create the FtpWebRequest and configure it
$ftp = [System.Net.FtpWebRequest]$ftp
# build authentication and connection
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user,$password)
$ftp.UseBinary = $true
$ftp.UsePassive = $true
$ftp.timeout = -1
#start a timer and error handling
$starttime = (get-date).ToString()
$error.Clear()
# read in the file to upload as a byte array
$content = [System.IO.File]::ReadAllBytes("$strfilename")
$ftp.ContentLength = $content.Length
# get the request stream, and write the bytes into it
$rs = $ftp.GetRequestStream()
$rs.Write($content, 0, $content.Length)
$endtime = (get-date).ToString()
#error handle
if ($error)
{
#Assemble the Query
$sqlresult = "THERE IS AN ERROR, Check the error email for details"
$sqlcontext = ($strfilename + '|' + $content.length + ' bytes')
$strquery =
"INSERT INTO [logs].[dbo].[EventLog] (SourceID, Started, Completed, Result, Context, Machine)
values (50,'$starttime','$endtime','$sqlresult','$sqlcontext','Server\MUHCLIENTMACHINE-CAFEEDSCRIPT')"
#Create Command and Execute.
$sqlCommand = $sqlConnection.CreateCommand()
$sqlCommand.CommandText = $strQuery
$sqlCommand.ExecuteNonQuery()
#Send dem emails
$emailbody = "A file for the CA Feed failed on $strfilename at " + (get-date).ToString() + " with the error '$error[0]'"
$emailsubject = "CA Feed Failed File"
Send-MailMessage -SmtpServer $SmtpServer -to $EmailTo -from $EmailFrom -subject $EmailSubject -Body $emailbody
}
else
{
write-host ("$strfilename" + ' Ran Without Errors')
$sqlresult = "RAN WITHOUT ERRORS"
$sqlcontext = ($strfilename + '|' + $content.length + ' bytes')
$strquery =
"INSERT INTO [logs].[dbo].[EventLog] (SourceID, Started, Completed, Result, Context, Machine)
values (50,'$starttime','$endtime','$sqlresult','$sqlcontext','Server\MUHCLIENTMACHINE-CAFEEDSCRIPT')"
#Create a command object.
$sqlCommand = $sqlConnection.CreateCommand()
$sqlCommand.CommandText = $strQuery
$sqlCommand.ExecuteNonQuery()
}
# be sure to clean up after ourselves and get ready for next block
Clear-Variable -Name starttime,endtime,strfilename,sqlresult,sqlcontext,ftp
$rs.Close()
$rs.Dispose()
【问题讨论】:
-
抱歉,我不打算调试那堵代码墙。如果您遇到性能问题,您需要找到实际的瓶颈。一种非常简单的方法是将语句添加到代码中的关键点,将时间戳(可能还有一些其他相关信息)写入日志文件。这应该可以让您缩小比预期花费更多时间的部分。
-
如果我不得不猜测,这是 sql 查询需要很长时间,或者您看到上传到 FTP 的时间很长。但是请参阅@rohinsidharth 关于使用 StopWatch 检查代码中各个位置的经过时间的答案,以便了解实际瓶颈在哪里。您可能还需要考虑在问题中添加
sql-server或ftp标签。
标签: arrays powershell optimization