【发布时间】:2018-01-03 23:41:13
【问题描述】:
好的,Stack Overflow,据我所知,这个太棒了。
我创建了一个启用宏的 Excel 文件,该文件在运行时执行以下操作(高级):
- 用户通过文件对话框选择模板文件(它本身已启用宏)
- 用户也可以通过文件对话框选择数据文件(未启用宏)
- 宏逐个遍历数据文件,并逐个打开它们,格式化数据,将数据迁移到中间工作簿中的新工作表中,然后关闭数据文件而不保存它
- 循环完所有文件后,中间工作簿也会保存,但保持打开状态
- 循环完所有数据文件后,循环遍历中间工作簿的每张工作表,将当前工作表中的数据转移到模板文件中,并将模板文件另存为一个新的、唯一标记的文件。此包含现在数据的文件中的一行数据被复制到摘要表中
(比这要复杂一些,但据我所知,这些是重要的方面)
这就是问题所在;被选择的数据文件数以千计(到目前为止,我们尝试的最大运行是 4000 个文件)。随着宏的进行,保存这些文件所需的时间会缓慢但稳定地变长。它从大约五秒钟开始,但到最后一些文件需要大约五分钟才能保存。
我唯一的线索是我添加了一个迭代功能,一旦所有数据文件循环通过,它就会完全关闭模板文件并打开一个具有不同设置的新实例,并且然后重新开始该过程。这会导致保存时间恢复正常,然后再次开始增长。在此步骤中还会保存并关闭摘要文件,并为新的迭代打开一个新文件。
我考虑过每隔一百个左右的数据文件就关闭和重新打开模板文件,如果必须的话,我会实施,但我宁愿找到一个适当的解决方案来解决这个问题,而不是创可贴的方法。如果我每次都打开和关闭模板文件,我可以避免时间问题,但随后宏变得非常不稳定,在运行过程中它有时会在完全随机的点崩溃(但只是有时)。
这是在与互联网或任何类型的网络隔离并保存到固态驱动器的计算机上(我们已尝试控制很多变量)。
无论如何,我很困惑,所以欢迎任何建议!
Option Explicit
Public Sub Example()
Dim Trial As Integer, Trials As Integer, DataSet As Integer
Dim TrialChecker As Boolean
Dim StartTime As Double, WaitTime As Double
Dim StartDate As Date
Dim FileSaveName As String
Dim CopiedDataRange As Range
Dim SummaryRunTimes As Worksheet, Calcs As Worksheet, CutoffsShifts As Worksheet
Dim SheetObjects() As Worksheet
Dim IntermediaryWorkbook As Workbook, Summary As Workbook, Template As Workbook
Application.ScreenUpdating = False
Application.Calculation = xlCalculationManual
'The 1 and Trials are actually set by Lbound and Ubound funcitons, but the premise is the same
For Trial = 1 To Trials
Workbooks.Add
Set Summary = ActiveWorkbook
'I use this one sheet to keep track of how long different parts of the code take to run
Set SummaryRunTimes = Summary.Worksheets(1)
SummaryRunTimes.Name = "Run Times"
SummaryRunTimes.Cells(1, 1).Value = "ID"
SummaryRunTimes.Cells(1, 2).Value = "Data Copy Time (s)"
SummaryRunTimes.Cells(1, 3).Value = "Formula Copy and Calc Time (s)"
SummaryRunTimes.Cells(1, 4).Value = "Summary Copy Time (s)"
SummaryRunTimes.Cells(1, 5).Value = "Save and Cleanup Time (s)"
'sheetnames is defined elsewhere in the code (it's a global variable right now. I intend to change that later).
'It's simply an array of strings with six elements.
For Counter = LBound(sheetnames) To UBound(sheetnames)
Summary.Worksheets.Add
Summary.ActiveSheet.Name = sheetnames(Counter)
Next Counter
'Again, TemplateLocation is defined elsewhere. It's just a string grabbed from a filedialog
Workbooks.Open (TemplateLocation)
Set Template = ActiveWorkbook
Set Calcs = Template.Sheets("Calcs")
Set CutoffsShifts = Template.Sheets("Log Cutoffs & Shifts")
'SheetObjects is simply used as a convenient reference for various sheets in the template file. I found
'it cleaned up the code a bit. Some might say it's unnecessary.
For Counter = LBound(sheetnames) To UBound(sheetnames)
Set SheetObjects(Counter) = Template.Sheets(sheetnames(Counter))
Next Counter
'This is where the parameters for the given trial are set in the template file. Trialchecker is set elsewhere
'(it checks a yes/no dropdown in the original spreadsheet). ParameterAddresses is a range that's grabbed from a
'table object in the original spreadsheet and contains where these parameters go in the template file. These
'will not change depending on the trial, thus column = 1. TrialParameters is in the same table, and are the
'parameters themselves. These DO depend on the trial, and thus the column is equal to the trial number
If TrialChecker = True Then
For Counter = LBound(ParameterAddresses) To UBound(ParameterAddresses)
CutoffsShifts.Range(ParameterAddresses(Counter, 1)).Value = TrialParameters(Counter, Trial)
Next Counter
End If
For DataSet = 1 To IntermediaryWorkbook.Worksheets.Count - 1
'This is where I start my timers
StartTime = Timer
StartDate = Date
'This is where the data is actually copied from the intermediary file into the template. It's always five
'columns wide, but can be any number of rows. the SummaryRunTimes statement is merely grabbing the unique
'identifier of that given worksheet
With IntermediaryWorkbook
Set CopiedDataRange = Calcs.Range("$A$3:$E$" & .Worksheets(Counter).UsedRange.Rows.Count + 1)
CopiedDataRange.Value = IntermediaryWorkbook.Worksheets(Counter).Range("$A$2:$E$" & .Worksheets(Counter).UsedRange.Rows.Count).Value
SummaryRunTimes.Cells(Counter + 1, 1) = Calcs.Cells(3, 1).Value
End With
'First timestamp
SummaryRunTimes.Cells(Counter + 1, 2) = CStr(Round(86400 * (Date - StartDate) + Timer - StartTime, 1))
StartTime = Timer
StartDate = Date
'This statement copies down the formulas that go with the data (which is aobut 100 columsn worth of formuals).
'Throughout this process, calculation is set to manual, so calculation is manually triggered here (Don't ask
'me why I do it twice. If I recall, it's because pivot tables are weird)
Set CopiedFormulaRange = Calcs.Range("$F$3:$KL$" & Calcs.UsedRange.Rows.Count)
CopiedFormulaRange.FillDown
Application.Calculate
Template.RefreshAll
Application.Calculate
'Second timestamp
SummaryRunTimes.Cells(Counter + 1, 3) = CStr(Round(86400 * (Date - StartDate) + Timer - StartTime, 1))
StartTime = Timer
StartDate = Date
'This is a separate function that copies data from the template file into the summary sheet.
'I know you can't see the code, but just know that it only copies six sets of seven cells, so
'as far as I can tell, it's not what is causing the problem. The timestamp supports this idea, as
'it's consistent and short
Call SummaryPopulate(Summary, sheetnames, SheetObjects, r)
r = r + 1
'Third timestamp
SummaryRunTimes.Cells(Counter + 1, 4) = CStr(Round(86400 * (Date - StartDate) + Timer - StartTime, 1))
StartTime = Timer
StartDate = Date
'These following few lines are meant to save the template file as a new file. As I mentioned, this is where
'things get bogged down. FileNameSuffix is a string set via a InputBox. TrialNames is set via the table object
'mentioned above, and is an array of strings.
Application.DisplayAlerts = False
If TrialChecker = True Then
FileSaveName = FolderLocation & "\" & Replace(Calcs.Cells(3, 1).Value, "/", " ") & " OOIP " & FileNameSuffix & " - " & TrialNames(1, Trial) & ".xlsm"
Else
FileSaveName = FolderLocation & "\" & Replace(Calcs.Cells(3, 1).Value, "/", " ") & " OOIP " & FileNameSuffix & ".xlsm"
End If
Template.SaveAs Filename:=FileSaveName, ConflictResolution:=xlLocalSessionChanges
Application.DisplayAlerts = True
'This part clears the copied data and formulas. I added the two Set Nothing lines in the hopes that it would
'solve my problem, but it doesn't seem to do anything
CopiedDataRange.ClearContents
CopiedDataRange.Offset(1, 0).Rows.Delete
Set CopiedDataRange = Nothing
Set CopiedFormulaRange = Nothing
'Fourth and final timestamp
SummaryRunTimes.Cells(Counter + 1, 5) = CStr(Round(86400 * (Date - StartDate) + Timer - StartTime, 1))
'It seems to run a bit better if there's this Wait line here, but I'm not sure why. The WaitTime
'is grabbed from the original worksheet, and is a Double
Application.Wait (TimeSerial(Hour(Now()), Minute(Now()), Second(Now()) + WaitTime))
Next DataSet
'This but simply saves the summary file and then closes that and the template file. Then the process starts anew.
'This seems to be the key for resetting something that reduces the run times.
If TrialChecker = True Then
Summary.SaveAs Filename:=FolderLocation & "\" & "OOIP Summary " & FileNameSuffix & " - " & TrialNames(1, Trial) & ".xlsx"
Else
Summary.SaveAs Filename:=FolderLocation & "\" & "OOIP Summary " & FileNameSuffix & ".xlsx"
End If
Template.Close False
Summary.Close False
Next Trial
Application.ScreenUpdating = True
Application.Calculation = xlCalculationAutomatic
IntermediaryWorkbook.Close False
End Sub
【问题讨论】:
-
在您的工作流程中,您没有保存数据文件的步骤。您实际上是否将数据文件中的数据复制到模板文件中(一个模板文件,许多数据文件)?如果是这样,则在导入每个数据文件之后,唯一的保存将是模板文件。导入后是否关闭数据文件?也许您可以查看上面的问题以使其更清楚。我也认为不看代码就很难做出猜测。
-
在处理这么多文件时,内存看起来如何?您的任务管理器锁定如何?打开的 Excel 文件超过数百个,还是您的 Excel 模板增加了他的记忆力?
-
@Variatus 你说得对,我很抱歉。我编辑了我的问题以尝试提供更多细节。我可以包含我的代码,但如果可能的话,我宁愿避免它。为了在这种情况下有意义,我必须剪掉很多东西。在进入该步骤之前,我可以说可能有用的是我不断设置和重置范围和工作表变量。我不确定这是否会有所不同,但我读过它会使内存有点混乱(我尝试在每次使用后将它们设置为 Nothing,但这似乎不起作用)。
-
您怀疑导致问题的最后迭代不清楚。为什么要重复“整个过程”?当然,此时不应需要对数据文件的引用,因为您(应该)在中间工作簿中拥有您想要的所有内容。
-
@Variatus 我试图在不遗漏任何重要内容的情况下尽可能地总结我的代码,并包含 cmets 以帮助使其更容易阅读。希望这能帮助我解释自己。如果不清楚,请说出没有意义的地方,我一定会修复它。感谢您的耐心等待!
标签: excel large-data-volumes save-as vba