使用 gnuplot 和 awk 计算应用程序加速

【问题标题】：Calculation of application speedup using gnuplot and awk使用 gnuplot 和 awk 计算应用程序加速
【发布时间】：2015-06-16 18:53:01
【问题描述】：

问题来了：

加速公式：S(p) = T(1)/T(p) = (一个进程的平均时间/p个进程的平均时间)

有 5 个日志，希望从中提取信息。 cg.B.1.log 包含一个进程的执行时间，因此我们计算平均时间以获得 T(1)。其他日志文件包含 2、4、8 和 16 个进程的执行时间。还必须计算这些时间的平均值，因为它们是 T(p)。

这是计算平均值的代码：

tavg(n) = "awk 'BEGIN { FS = \"[ \\t]*=[ \\t]*\" }  /Time in seconds/ { s += $2; c++ } /Total processes/ { if (! CP) CP = $2 } END { print s/c }' cg.B.".n.".log ".(n == 1 ? ">" : ">>")." tavg.dat;"

以及计算加速比的代码：

system "awk 'NR==1{n=$0} {print n/$0}' tavg.dat > speedup.dat;"

如何组合这两个命令，以便直接生成输出“speedup.dat”而不使用文件tavg.dat？

这里是文件的内容，所有日志文件的结构都是一样的。为了缩写，我只附上了前两个处决。

cg.B.1.log

-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-
Start in 16:45:15--25/12/2014


 NAS Parallel Benchmarks 3.3 -- CG Benchmark

 Size:      75000
 Iterations:    75
 Number of active processes:     1
 Number of nonzeroes per row:       13
 Eigenvalue shift: .600E+02

   iteration           ||r||                 zeta
    1       0.30354859861452E-12    59.9994751578754
    2       0.11186435488267E-14    21.7627846142536
    3       0.11312258511928E-14    22.2876617043224
    4       0.11222160585284E-14    22.5230738188346
    5       0.11244234177219E-14    22.6275390653892
    6       0.11330434819384E-14    22.6740259189533
    7       0.11334259623050E-14    22.6949056826251
    8       0.11374839313647E-14    22.7044023166872
    9       0.11424877443039E-14    22.7087834345620
   10       0.11329475190566E-14    22.7108351397177
   11       0.11337364093482E-14    22.7118107121341
   12       0.11379928308864E-14    22.7122816240971
   13       0.11369453681794E-14    22.7125122663243
   14       0.11430390337015E-14    22.7126268007594
   15       0.11400318886400E-14    22.7126844161819
   16       0.11352091331197E-14    22.7127137461755
   17       0.11350923439124E-14    22.7127288402000
   18       0.11475378864565E-14    22.7127366848296
   19       0.11366777929028E-14    22.7127407981217
   20       0.11274243312504E-14    22.7127429721364
   21       0.11353930792856E-14    22.7127441294025
   22       0.11299685800278E-14    22.7127447493900
   23       0.11296405041170E-14    22.7127450834533
   24       0.11381975597887E-14    22.7127452643881
   25       0.11328127301663E-14    22.7127453628451
   26       0.11367332658939E-14    22.7127454166517
   27       0.11283372178605E-14    22.7127454461696
   28       0.11384734158863E-14    22.7127454624211
   29       0.11394011989719E-14    22.7127454713974
   30       0.11354294067640E-14    22.7127454763703
   31       0.11412988029103E-14    22.7127454791343
   32       0.11358088407717E-14    22.7127454806740
   33       0.11263266152515E-14    22.7127454815316
   34       0.11275183080286E-14    22.7127454820131
   35       0.11328306951409E-14    22.7127454822840
   36       0.11357880314891E-14    22.7127454824349
   37       0.11332687790488E-14    22.7127454825202
   38       0.11324108818137E-14    22.7127454825684
   39       0.11365065523777E-14    22.7127454825967
   40       0.11361185361321E-14    22.7127454826116
   41       0.11276519820716E-14    22.7127454826202
   42       0.11317183424878E-14    22.7127454826253
   43       0.11236007481770E-14    22.7127454826276
   44       0.11304065564684E-14    22.7127454826296
   45       0.11287791356431E-14    22.7127454826310
   46       0.11297028000133E-14    22.7127454826310
   47       0.11281236869666E-14    22.7127454826314
   48       0.11277254075548E-14    22.7127454826317
   49       0.11320327289847E-14    22.7127454826309
   50       0.11287655285563E-14    22.7127454826321
   51       0.11230503422400E-14    22.7127454826324
   52       0.11292089094944E-14    22.7127454826313
   53       0.11366728396408E-14    22.7127454826315
   54       0.11222618466968E-14    22.7127454826310
   55       0.11278193276516E-14    22.7127454826315
   56       0.11244624896030E-14    22.7127454826316
   57       0.11264508872685E-14    22.7127454826318
   58       0.11255583774760E-14    22.7127454826314
   59       0.11227129146723E-14    22.7127454826314
   60       0.11189480800173E-14    22.7127454826318
   61       0.11163241472678E-14    22.7127454826315
   62       0.11278839424218E-14    22.7127454826318
   63       0.11226804133008E-14    22.7127454826313
   64       0.11222456601361E-14    22.7127454826317
   65       0.11270879524310E-14    22.7127454826308
   66       0.11303771390006E-14    22.7127454826319
   67       0.11240101357287E-14    22.7127454826319
   68       0.11240278884391E-14    22.7127454826321
   69       0.11207748067718E-14    22.7127454826317
   70       0.11178755187571E-14    22.7127454826327
   71       0.11195935245649E-14    22.7127454826313
   72       0.11260715126337E-14    22.7127454826322
   73       0.11281677964997E-14    22.7127454826316
   74       0.11162340034815E-14    22.7127454826318
   75       0.11208709203921E-14    22.7127454826310
 Benchmark completed 
 VERIFICATION SUCCESSFUL 
 Zeta is     0.2271274548263E+02
 Error is    0.3128387698896E-15


 CG Benchmark Completed.
 Class           =                        B
 Size            =                    75000
 Iterations      =                       75
 Time in seconds =                    88.72
 Total processes =                        1
 Compiled procs  =                        1
 Mop/s total     =                   616.64
 Mop/s/process   =                   616.64
 Operation type  =           floating point
 Verification    =               SUCCESSFUL
 Version         =                      3.3
 Compile date    =              25 Dec 2014

 Compile options:
MPIF77       = mpif77
FLINK        = $(MPIF77)
FMPI_LIB     = -L/usr/lib/openmpi/lib -lmpi -lopen-rte -lo...
FMPI_INC     = -I/usr/lib/openmpi/include -I/usr/lib/openm...
FFLAGS       = -O
FLINKFLAGS   = -O
RAND         = randi8


 Please send the results of this run to:

 NPB Development Team 
 Internet: npb@nas.nasa.gov

 If email is not available, send this to:

 MS T27A-1
 NASA Ames Research Center
 Moffett Field, CA  94035-1000

 Fax: 650-604-3957


Finish in 16:46:46--25/12/2014
-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-
-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-
Start in 17:03:13--25/12/2014


 NAS Parallel Benchmarks 3.3 -- CG Benchmark

 Size:      75000
 Iterations:    75
 Number of active processes:     1
 Number of nonzeroes per row:       13
 Eigenvalue shift: .600E+02

   iteration           ||r||                 zeta
    1       0.30354859861452E-12    59.9994751578754
    2       0.11186435488267E-14    21.7627846142536
    3       0.11312258511928E-14    22.2876617043224
    4       0.11222160585284E-14    22.5230738188346
    5       0.11244234177219E-14    22.6275390653892
    6       0.11330434819384E-14    22.6740259189533
    7       0.11334259623050E-14    22.6949056826251
    8       0.11374839313647E-14    22.7044023166872
    9       0.11424877443039E-14    22.7087834345620
   10       0.11329475190566E-14    22.7108351397177
   11       0.11337364093482E-14    22.7118107121341
   12       0.11379928308864E-14    22.7122816240971
   13       0.11369453681794E-14    22.7125122663243
   14       0.11430390337015E-14    22.7126268007594
   15       0.11400318886400E-14    22.7126844161819
   16       0.11352091331197E-14    22.7127137461755
   17       0.11350923439124E-14    22.7127288402000
   18       0.11475378864565E-14    22.7127366848296
   19       0.11366777929028E-14    22.7127407981217
   20       0.11274243312504E-14    22.7127429721364
   21       0.11353930792856E-14    22.7127441294025
   22       0.11299685800278E-14    22.7127447493900
   23       0.11296405041170E-14    22.7127450834533
   24       0.11381975597887E-14    22.7127452643881
   25       0.11328127301663E-14    22.7127453628451
   26       0.11367332658939E-14    22.7127454166517
   27       0.11283372178605E-14    22.7127454461696
   28       0.11384734158863E-14    22.7127454624211
   29       0.11394011989719E-14    22.7127454713974
   30       0.11354294067640E-14    22.7127454763703
   31       0.11412988029103E-14    22.7127454791343
   32       0.11358088407717E-14    22.7127454806740
   33       0.11263266152515E-14    22.7127454815316
   34       0.11275183080286E-14    22.7127454820131
   35       0.11328306951409E-14    22.7127454822840
   36       0.11357880314891E-14    22.7127454824349
   37       0.11332687790488E-14    22.7127454825202
   38       0.11324108818137E-14    22.7127454825684
   39       0.11365065523777E-14    22.7127454825967
   40       0.11361185361321E-14    22.7127454826116
   41       0.11276519820716E-14    22.7127454826202
   42       0.11317183424878E-14    22.7127454826253
   43       0.11236007481770E-14    22.7127454826276
   44       0.11304065564684E-14    22.7127454826296
   45       0.11287791356431E-14    22.7127454826310
   46       0.11297028000133E-14    22.7127454826310
   47       0.11281236869666E-14    22.7127454826314
   48       0.11277254075548E-14    22.7127454826317
   49       0.11320327289847E-14    22.7127454826309
   50       0.11287655285563E-14    22.7127454826321
   51       0.11230503422400E-14    22.7127454826324
   52       0.11292089094944E-14    22.7127454826313
   53       0.11366728396408E-14    22.7127454826315
   54       0.11222618466968E-14    22.7127454826310
   55       0.11278193276516E-14    22.7127454826315
   56       0.11244624896030E-14    22.7127454826316
   57       0.11264508872685E-14    22.7127454826318
   58       0.11255583774760E-14    22.7127454826314
   59       0.11227129146723E-14    22.7127454826314
   60       0.11189480800173E-14    22.7127454826318
   61       0.11163241472678E-14    22.7127454826315
   62       0.11278839424218E-14    22.7127454826318
   63       0.11226804133008E-14    22.7127454826313
   64       0.11222456601361E-14    22.7127454826317
   65       0.11270879524310E-14    22.7127454826308
   66       0.11303771390006E-14    22.7127454826319
   67       0.11240101357287E-14    22.7127454826319
   68       0.11240278884391E-14    22.7127454826321
   69       0.11207748067718E-14    22.7127454826317
   70       0.11178755187571E-14    22.7127454826327
   71       0.11195935245649E-14    22.7127454826313
   72       0.11260715126337E-14    22.7127454826322
   73       0.11281677964997E-14    22.7127454826316
   74       0.11162340034815E-14    22.7127454826318
   75       0.11208709203921E-14    22.7127454826310
 Benchmark completed 
 VERIFICATION SUCCESSFUL 
 Zeta is     0.2271274548263E+02
 Error is    0.3128387698896E-15


 CG Benchmark Completed.
 Class           =                        B
 Size            =                    75000
 Iterations      =                       75
 Time in seconds =                    87.47
 Total processes =                        1
 Compiled procs  =                        1
 Mop/s total     =                   625.43
 Mop/s/process   =                   625.43
 Operation type  =           floating point
 Verification    =               SUCCESSFUL
 Version         =                      3.3
 Compile date    =              25 Dec 2014

 Compile options:
MPIF77       = mpif77
FLINK        = $(MPIF77)
FMPI_LIB     = -L/usr/lib/openmpi/lib -lmpi -lopen-rte -lo...
FMPI_INC     = -I/usr/lib/openmpi/include -I/usr/lib/openm...
FFLAGS       = -O
FLINKFLAGS   = -O
RAND         = randi8


 Please send the results of this run to:

 NPB Development Team 
 Internet: npb@nas.nasa.gov

 If email is not available, send this to:

     MS T27A-1
     NASA Ames Research Center
     Moffett Field, CA  94035-1000

     Fax: 650-604-3957


Finish in 17:04:43--25/12/2014
-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-

tavg.dat

88.3055
45.1482
37.7202
37.4035
53.777

speedup.dat

1
1.9559
2.34107
2.36089
1.64207

【问题讨论】：

    标签： performance awk gnuplot


    【解决方案1】：

    您可以在一个处理所有日志文件的 awk 脚本中完成所有操作：

    #!/usr/bin/awk -f

BEGIN { FS="=" }

lfname != FILENAME { lfname = FILENAME; split(FILENAME, a, "."); fnum=a[3] }

/Time in seconds/ { tsecs[fnum] += $2; tcnt[fnum]++ }
/Total processes/ { cp[fnum] = int($2) }

END {
    tavg1 = tsecs[1]/tcnt[1]
    for( k in tsecs ) {
        tavgk = tsecs[k]/tcnt[k]
        if( tavgk > 0 ) {
            print k OFS cp[k] OFS tavgk OFS tavg1/tavgk
        }
    }
}

    如果你将它放在一个名为 awk.script 的文件中并使用 chmod +x awk.script 使其可执行，你可以在 bash 中运行它，如下所示：

     ./awk.script cg.B.*.log

    如果您使用的是 GNU awk，则会对输出进行排序（可能需要额外的步骤来确保使用其他 awk 风格对输出进行排序）。

    我生成了第二个和第三个文件，输出如下：

    1 1 88.095 1
2 2 68.095 1.29371
3 4 49.595 1.77629

    其中未命名的列如下：file number、# processes、avg per file、speedup。您可以通过将END 块中的打印更改为print tavg1/tavgk 来获得加速。

    以下是脚本的细分：

    • 在BEGIN 中使用更简单的字段分隔符
    • lfname != FILENAME - 从文件名中解析出文件号为fnum，但仅当FILENAME 更改时。
    • /Time in seconds/ - 使用 fnum 键将值存储在 tsecs 和 tcnt 数组中。使用int() 函数从进程值中去除空格。
    • /Total processes/ - 使用 fnum 键将进程存储在 cp 数组中
    • END - 计算 fnum 1 的平均值为 tavg1，循环遍历 tsecs 中的键并通过 fnum 键计算平均值为 tavgk。当tavgk > 0 如上所述打印输出时。

    【讨论】：

      【解决方案2】：

      你已经弄清楚了所有困难的部分。您根本不需要tavg.dat 文件。直接创建您的tavg(n) 函数作为系统调用：

      tavg(n) = system("awk 'BEGIN { FS = \"[ \\t]*=[ \\t]*\" }  \
/Time in seconds/ { s += $2; c++ } /Total processes/ { \
if (! CP) CP = $2 } END { print s/c }' cg.B.".n.".log")

      还有一个speedup(n) 函数

      speedup(n)=tavg(n)/tavg(1)

      现在您可以set print 写入文件：

      set print "speedup.dat"
do for [i=1:5] {
print speedup(i)
}
unset print

      【讨论】：

        猜你喜欢
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 2017-08-10
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        相关资源
        最近更新 更多
        热门标签
        Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode