使用 python pandas 解析日志文件

Question

问题 1

我正在尝试从模拟中读取和解析日志文件。理想情况下，我想用熊猫来做这件事，但我遇到了问题。有问题的文件名为log，下面包含一个示例。现在我尝试做

     import pandas as pd
     import numpy as np
     import csv
    
     # This method works:
     results = []
     with open('log') as inputfile:
             for row in csv.reader(inputfile):
                 results.append(row)
    
   
      # This method doesn't work:
      pd.read_csv('log') 

错误是

pandas.parser.CParserError: Error tokenizing data. C error: Expected 1 fields in line 77, saw 4

为什么会这样？

问题 2

解析日志文件最有效的方法是什么？请记住，该文件实际上会非常大。基本上我想将每个时间步的残差（每个时间步由Time= x指定）解析为一个numpy数组进行绘图。例如

smoothSolver:  Solving for gamma, Initial residual = 0.813857557031, Final residual = 6.74558898691e-06, No Iterations 2

我想将0.813857557031 附加到数组res_gamma_init

最有效的方法是什么？

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.3.x                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 2.3.x-590d57f32fed
Exec   : simpleFoam
Date   : Oct 30 2015
Time   : 17:24:08
Host   : "jack"
PID    : 4034
Case   : /home/jack/OpenFOAM/jack-2.3.x/run/sim
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Reading field p

Reading field U

Reading/calculating face flux field phi

Selecting incompressible transport model Newtonian
Selecting RAS turbulence model kOmegaSST
Using gammaReThetat-correlations by Langtry and Menter (2009)
gammaReThetatSSTCoeffs
{
    ca1             2;
    ce1             1;
    ca2             0.06;
    ce2             50;
    cThetat         0.03;
    sigmaf          1;
    sigmaThetat     2;
    s1              2;
    dUds            false;
    alphaK1         0.85034;
    alphaK2         1;
    alphaOmega1     0.5;
    alphaOmega2     0.85616;
    gamma1          0.5532;
    gamma2          0.4403;
    beta1           0.075;
    beta2           0.0828;
    betaStar        0.09;
    a1              0.31;
    c1              10;
    kInf            0;
    omegaInf        0;
}

No finite volume options present


SIMPLE: convergence criteria
    field p  tolerance 1e-06
    field U  tolerance 1e-06
    field "(k|omega|ReThetaTilda|nut|gamma)"     tolerance 1e-06


Starting time loop


forceCoeffs forceCoeffs:
    Not including porosity effects

Time = 1

smoothSolver:  Solving for Ux, Initial residual = 0.999999999388, Final residual = 0.00692443749034, No Iterations 2
smoothSolver:  Solving for Uy, Initial residual = 0.999999994742, Final residual = 0.0027182321684, No Iterations 2
DICPCG:  Solving for p, Initial residual = 1, Final residual = 0.00976261323556, No Iterations 326
time step continuity errors : sum local = 8.88353600919, global = -0.154492041821, cumulative = -0.154492041821
smoothSolver:  Solving for omega, Initial residual = 1, Final residual = 0.0110458877993, No Iterations 2
smoothSolver:  Solving for k, Initial residual = 0.999999999804, Final residual = 0.022014322402, No Iterations 2
bounding k, min: 0 max: 0.415754458684 average: 0.410526729282
smoothSolver:  Solving for ReThetatTilda, Initial residual = 1, Final residual = 0.0259070897862, No Iterations 2
smoothSolver:  Solving for gamma, Initial residual = 1, Final residual = 1.51565684699e-05, No Iterations 2
ExecutionTime = 1.41 s  ClockTime = 1 s

forceCoeffs forceCoeffs output:
    Cm    = 0
    Cd    = -19811.1501384
    Cl    = 4.01861806726e-15
    Cl(f) = 2.00930903363e-15
    Cl(r) = 2.00930903363e-15

Time = 2

smoothSolver:  Solving for Ux, Initial residual = 0.510606164486, Final residual = 0.0151751146151, No Iterations 2
smoothSolver:  Solving for Uy, Initial residual = 0.570639175137, Final residual = 0.0164524779792, No Iterations 2
DICPCG:  Solving for p, Initial residual = 0.160667758287, Final residual = 0.00156013775625, No Iterations 143
time step continuity errors : sum local = 88.8245306208, global = 11.6476210117, cumulative = 11.4931289699
smoothSolver:  Solving for omega, Initial residual = 0.531389189498, Final residual = 0.00899532326598, No Iterations 2
smoothSolver:  Solving for k, Initial residual = 0.562103650426, Final residual = 0.0138022250836, No Iterations 2
bounding k, min: 0 max: 0.425403200237 average: 0.411310758596
smoothSolver:  Solving for ReThetatTilda, Initial residual = 0.301017401945, Final residual = 0.00693946297375, No Iterations 2
smoothSolver:  Solving for gamma, Initial residual = 0.813857557031, Final residual = 6.74558898691e-06, No Iterations 2
ExecutionTime = 2.13 s  ClockTime = 2 s

forceCoeffs forceCoeffs output:
    Cm    = 0
    Cd    = -20283.0256988
    Cl    = 3.99364280461e-15
    Cl(f) = 1.9968214023e-15
    Cl(r) = 1.9968214023e-15

Time = 3

Answer 1

回答你的第二个问题。这可能会有所帮助，尽管正则表达式可能会更好。

line1 = 'smoothSolver:  Solving for Ux, Initial residual = 0.999999999388, Final residual = 0.00692443749034, No Iterations 2'
line2 = 'smoothSolver:  Solving for Uy, Initial residual = 0.999999994742, Final residual = 0.0027182321684, No Iterations 2'
test = [line1, line2]
import re
for line in test:
    if line.startswith('smoothSolver'):
        print re.findall(r'[residual = ][\d][.][\d]+', line)

控制台中的结果是：

[' 0.999999999388', ' 0.00692443749034']
[' 0.999999994742', ' 0.0027182321684']