如何在 Julia 中进行正确的微基准测试？

Question

Julia 1.0.0 documentation 提供一般提示。

它还建议不要使用@time 宏：

对于更严格的基准测试，请考虑 BenchmarkTools.jl 包，其中包括多次评估函数以减少噪音。

它们在使用方面如何比较？使用“基础”Julia 之外的东西是否值得麻烦？

Answer 1

从统计上看，@benchmark 比@time 好很多

TL;DR BenchmarkTools @benchmark 宏是一个很棒的微基准测试工具。 谨慎使用@time 宏，不要认真对待第一次运行。

这个简单的例子说明了用途和区别：

julia> # Fresh Julia 1.0.0 REPL

julia> # Add BenchmarkTools package using ] key package manager

(v1.0) pkg> add BenchmarkTools  
julia> # Press backspace key to get back to Julia REPL

# Load BenchmarkTools package into current REPL
julia> using BenchmarkTools

julia> # Definine a function with a known elapsed time
julia> f(n) = sleep(n)  # n is in seconds
f (generic function with 1 method)

# Expect just over 500 ms for elapsed time
julia> @benchmark f(0.5)
BenchmarkTools.Trial:
  memory estimate:  192 bytes
  allocs estimate:  5
  --------------
  minimum time:     501.825 ms (0.00% GC)
  median time:      507.386 ms (0.00% GC)
  mean time:        508.069 ms (0.00% GC)
  maximum time:     514.496 ms (0.00% GC)
  --------------
  samples:          10
  evals/sample:     1

julia> # Try second run to compare consistency
julia> # Note the very close consistency in ms for both median and mean times

julia> @benchmark f(0.5)
BenchmarkTools.Trial:
  memory estimate:  192 bytes
  allocs estimate:  5
  --------------
  minimum time:     502.603 ms (0.00% GC)
  median time:      508.716 ms (0.00% GC)
  mean time:        508.619 ms (0.00% GC)
  maximum time:     515.602 ms (0.00% GC)
  --------------
  samples:          10
  evals/sample:     1


julia> # Define the same function with new name for @time macro tests
julia> g(n) = sleep(n)
g (generic function with 1 method)

# First run suffers from compilation time, so 518 ms
julia> @time sleep(0.5)
  0.517897 seconds (83 allocations: 5.813 KiB)

# Second run drops to 502 ms, 16 ms drop
julia> @time sleep(0.5)
  0.502038 seconds (9 allocations: 352 bytes)

# Third run similar to second
julia> @time sleep(0.5)
  0.503606 seconds (9 allocations: 352 bytes)

# Fourth run increases over second by about 13 ms
julia> @time sleep(0.5)
  0.514629 seconds (9 allocations: 352 bytes)

这个简单的例子说明了使用@benchmark 宏是多么容易，以及应该谨慎对待@time 宏结果。

是的，使用@benchmark 宏是值得的。