Background
I've self-taught myself machine learning and have recently started delving into the Julia Machine Learning Ecosystem.
Coming from a python background and having some Tensorflow and OpenCV/
skimage experience, I want to benchmark Julia ML libraries (Flux/JuliaImages) against its counterparts to see how fast or slow it really performs CV(any) task(s) and to decide if I should shift to using Julia. I know how to get the time taken to execute a function in python using timeit module like this :
#Loading an Image using OpenCVs = """\img = cv2.imread('sample_image.png', 1)"""setup = """\import timeit"""print(str(round((timeit.timeit(stmt = s, setup = setup, number = 1))*1000, 2)) + " ms")#printing the time taken in ms rounded to 2 digitsHow does one compare the execution time of a function performing the same task in Julia using the appropriate library (in this case, JuliaImages).
Does Julia provide any function/macro to time/benchmark ?
Best Answer
using BenchmarkTools is the recommended way to benchmark Julia functions. Unless you are timing something that takes quite a while, use either @benchmark or the less verbose @btime macros exported from it. Because the machinery behind these macros evaluates the target function many times, @time is useful for benchmarking things that run slowly (e.g. where disk access or very time-consuming calculations are involved).
It is important to use @btime or @benchmark correctly, this avoids misleading results. Usually, you are benchmarking a function that takes one or more arguments. When benchmarking, all arguments should be external variables: (without the benchmark macro)
x = 1f(x)# do not use f(1)The function will be evaluated many times. To prevent the function arguments from being re-evaluated whenever the function is evaluated, we must mark each argument by prefixing a $ to the name of each variable that is used as an argument. The benchmarking macros use this to indicate that the variable should be evaluated (resolved) once, at the start of the benchmarking process and then the result is to be reused directly as is:
julia> using BenchmarkToolsjulia> a = 1/2;julia> b = 1/4;julia> c = 1/8;julia> a, b, c(0.5, 0.25, 0.125)julia> function sum_cosines(x, y, z)return cos(x) + cos(y) + cos(z)end;julia> @btime sum_cosines($a, $b, $c); # the `;` suppresses printing the returned value11.899 ns (0 allocations: 0 bytes) # calling the function takes ~12 ns (nanoseconds)# the function does not allocate any memory# if we omit the '$', what we see is misleadingjulia> @btime sum_cosines(a, b, c); # the function appears more than twice slower 28.441 ns (1 allocation: 16 bytes) # the function appears to be allocating memory# @benchmark can be used the same way that @btime is usedjulia> @benchmark sum_cosines($a,$b,$c) # do not use a ';' hereBenchmarkTools.Trial:memory estimate: 0 bytesallocs estimate: 0--------------minimum time: 12.111 ns (0.00% GC)median time: 12.213 ns (0.00% GC)mean time: 12.500 ns (0.00% GC)maximum time: 39.741 ns (0.00% GC)--------------samples: 1500evals/sample: 999While there are parameters than can be adjusted, the default values usually work well. For additional information about BenchmarkTools for experienced ursers, see the manual.
Julia provides two macros for timing/benchmarking code runtime. These are :
- @time
- @benchmark : external, install by
Pkg.add("BenchmarkTools")
Using BenchmarkTools' @benchmark is very easy and would be helpful to you in comparing the speed of the two languages. Example of using @benchark against the python bench you provided.
using Images, FileIO, BenchmarkTools@benchmark img = load("sample_image.png")Output :
BenchmarkTools.Trial: memory estimate: 3.39 MiBallocs estimate: 322--------------minimum time: 76.631 ms (0.00% GC)median time: 105.579 ms (0.00% GC)mean time: 110.319 ms (0.41% GC)maximum time: 209.470 ms (0.00% GC)--------------samples: 46evals/sample: 1Now to compare for the mean time, you should put samples (46) as the number in your python timeit code and divide it by the same number to get the mean time of execution.
print(str(round((timeit.timeit(stmt = s, setup = setup, number = 46)/46)*1000, 2)) + " ms")You can follow this process for benchmarking any function in both Julia and Python.I hope you're doubt has been cleared.
Note : From a statistical point of view, @benchmark is much better than @time.