Create benchmark suite infrastructure

benchmarks/Project.toml

@@ -0,0 +1,16 @@
+[deps]
+BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
+CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
+DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
+Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
+DifferentialEquations = "0c46a032-eb83-5123-abaf-570d42b7fbaa"
+Distributed = "8ba89e20-285c-5b6f-9357-94700520ee1b"
+Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
+Graphs = "86223c79-3864-5bf0-83f7-82e725a168b6"
+LibGit2 = "76f85450-5226-5b5a-8eaa-529ad045b433"
+LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
+MacroTools = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
+MetaGraphs = "626554b9-1ddb-594c-aa3c-2596fe9399a5"
+Neuroblox = "769b91e5-4c60-41ee-bfae-153c84203cb2"
+Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
+Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

benchmarks/README.md

@@ -0,0 +1,167 @@
+# Neuroblox Benchmark suite
+
+Tools for benchmarking Neuroblox.jl
+
+The standard way to use this would be to navigate to this directory, `Pkg.activate` it, then do
+
+```julia
+include("benchmarks.jl")
+run_and_save_benchmarks()
+```
+
+## Docstrings: 
+
+<details><summary> run_and_save_benchmarks </summary>
+<p>
+
+```
+run_and_save_benchmarks(; benches=neuroblox_benchmark_suite,
+                          filename="benchmark_history/benchmarks-$(now()).csv")
+```
+
+Run the specified list of benchmarks `benches` (default to the full benchmark suite), and save the outcomes in CSV form to the file path specified by `filename`.
+
+The tabular benchmark data will include the following columns:
+
+  * `benchmark_name`: a list of strings containing the names of each benchmark
+  * `initial_runtime_ns`: a floating point list containing how long the first call to the benchmark function took in nanoseconds. This should be representative of a users experience in a fresh julia session.
+  * `initial_compiletime_percent`: a floating point list containing percentages (0 to 100) of how much of the initial runtime was taken up by compilation.
+  * `warmed_up_minimum_runtim_ns`: a floating point list describing the minimum amount of nanoseconds each benchmark took to run in a `BenchmarkTools.@benchmark` loop. This should be representative of the actual performance of the package once compilation is out of the way.
+  * `warmed_up_median_runtim_ns`: a floating point list describing the median amount of nanoseconds each benchmark took to run in a `BenchmarkTools.@benchmark` loop. This should be representative of the actual performance of the package once compilation is out of the way.
+  * `initial_alloced_bytes`: an integer list describing how many bytes were allocated while running the initial benchmark function (including compilation).
+  * `initial_gc_time_ns`: a floating point list describing how long the initial benchmark function spent performing garbage collection (in nanoseconds).
+  * `intial_gc_alloc_count`: an integer list containing the number of separate allocations in the initial call to the benchmark function, including allocations from compilation.
+  * `warmed_up_allocated_bytes`: an integer list describing how many bytes were allocated while running the benchmark function in a `BenchmarkTools.@benchmark` loop. Due to how BenchmarkTools.jl works, this currently only reports the minimum number of bytes which may be misleading for non-uniform benchmarks.
+  * `warmed_up_median_gc_time_ns`: a floating point list describing the median time spent (in nanoseconds) performing garbage collection during the `BenchmarkTools.@benchmark` loop.
+  * `warmed_up_gc_alloc_count`: an integer list describing how many bytes were allocated while running the benchmark function during the `BenchmarkTools.@benchmark` loop. Due to how BenchmarkTools.jl works, this currently only reports the minimum number of allocations which may be misleading for non-uniform benchmarks.
+  * `metadata` a list of metadata about the benchmark process. All empty except for the first entry which contains information about the latest commit of Neuroblox.jl when the benchmark was run, and detailed information about the computer which ran the benchmark.
+
+
+</details>
+</p>
+
+____________________________
+
+<details><summary> BenchSetup </summary>
+<p>
+
+```
+BenchSetup(;
+    name::String,
+    setup::Expr = quote end,
+    args::Vector{Symbol} = Symbol[],
+    sample_setup::Expr = quote end,
+    bench::Expr,
+    sample_teardown::Expr = quote end,
+    samples::Int = 100,
+    evals::Int = 5,
+    seconds::Int = 1000,
+)
+```
+
+Create a benchmark object to be evaluated with `run`.
+
+Required arguments:
+
+  * `name`: the name of the benchmark to be run.
+  * `bench`: an `Expr` object containing the quoted code to be benchmarked. This will be run once in a `let` block to collect compilation information, and then again inside a `BenchmarkTools.@benchmark` loop to collect warmed up runtime information.
+
+Optional arguments:
+
+  * `setup`: any initial, top-level setup code that needs to be run before the benchmarking begins. This code is only executed once and is intended for things like loading packages and defining functions and datastructures.
+  * `args`: arguments to be passed to the `bench` expression in the benchmark loop, rather than treating them as global variables, they will be passed like function arguments to the `sample_setup` code.
+  * `sample_setup`: This code is run before each benchmark sample and is intended for things like reseting mutable state before a benchmark occurs (this corresponds to the `setup` argument to `BenchmarkTools.@benchmark`).
+  * `sample_teardown`: This code is run after each benchmark sample and is intended for things like reseting mutable state after a benchmark occurs (this corresponds to the `teardown` argument to `BenchmarkTools.@benchmark`).
+  * `samples`: The number of samples of the runtime of `bench` to be collected by `BenchmarkTools.@benchmark`.
+  * `evals`: The number of times the `bench` code should be run per-sample.
+  * `seconds`: a budget of time given to `BenchmarkTools.@benchmark`. The benchmark will run either until it collects the requested number of samples, or until the time budget is exceeded.
+
+
+</details>
+</p>
+
+____________________________
+
+<details><summary> neuroblox_benchmark_suite </summary>
+<p>
+
+```
+neuroblox_benchmark_suite :: Vector{BenchSetup}
+```
+
+Vector of the default Neuroblox benchmarks, see `benchmarks/benchmarkable_expamples.jl` for details.
+
+
+</details>
+</p>
+
+____________________________
+
+<details><summary> run </summary>
+<p>
+
+```
+run(command, args...; wait::Bool = true)
+```
+
+Run a command object, constructed with backticks (see the [Running External Programs](@ref) section in the manual). Throws an error if anything goes wrong, including the process exiting with a non-zero status (when `wait` is true).
+
+The `args...` allow you to pass through file descriptors to the command, and are ordered like regular unix file descriptors (eg `stdin, stdout, stderr, FD(3), FD(4)...`).
+
+If `wait` is false, the process runs asynchronously. You can later wait for it and check its exit status by calling `success` on the returned process object.
+
+When `wait` is false, the process' I/O streams are directed to `devnull`. When `wait` is true, I/O streams are shared with the parent process. Use [`pipeline`](@ref) to control I/O redirection.
+
+```
+run(b::BenchSetup) :: @NamedTuple{name::String, ttfx_data::TimeData, runtime_data::BenchmarkTools.Trial}
+```
+
+Take a given `BenchSetup` and run it in a distributed process generating information describing both the compile times associated with the benchmark, and the warmed-up runtimes.
+
+This returns a `NamedTuple` of low-level data collected during the benchmark. It can be processed into something more digestable by `summarize_benchmar_data`.
+
+```
+run(b::Benchmark[, p::Parameters = b.params]; kwargs...)
+```
+
+Run the benchmark defined by [`@benchmarkable`](@ref).
+
+```
+run(group::BenchmarkGroup[, args...]; verbose::Bool = false, pad = "", kwargs...)
+```
+
+Run the benchmark group, with benchmark parameters set to `group`'s by default.
+
+
+</details>
+</p>
+
+____________________________
+
+<details><summary> summarize_benchmark_data </summary>
+<p>
+
+```
+summarize_benchmark_data(bench_result::@NamedTuple{name::String, ttfx_data::TimeData, runtime_data::Trial})
+```
+
+Take low-level data returned by `run(b::BenchSetup)` and summarize it into a named tuple containing
+
+  * `benchmark_name`: a list of strings containing the names of each benchmark
+  * `initial_runtime_ns`: a floating point list containing how long the first call to the benchmark function took in nanoseconds. This should be representative of a users experience in a fresh julia session.
+  * `initial_compiletime_percent`: a floating point list containing percentages (0 to 100) of how much of the initial runtime was taken up by compilation.
+  * `warmed_up_minimum_runtim_ns`: a floating point list describing the minimum amount of nanoseconds each benchmark took to run in a `BenchmarkTools.@benchmark` loop. This should be representative of the actual performance of the package once compilation is out of the way.
+  * `warmed_up_median_runtim_ns`: a floating point list describing the median amount of nanoseconds each benchmark took to run in a `BenchmarkTools.@benchmark` loop. This should be representative of the actual performance of the package once compilation is out of the way.
+  * `initial_alloced_bytes`: an integer list describing how many bytes were allocated while running the initial benchmark function (including compilation).
+  * `initial_gc_time_ns`: a floating point list describing how long the initial benchmark function spent performing garbage collection (in nanoseconds).
+  * `intial_gc_alloc_count`: an integer list containing the number of separate allocations in the initial call to the benchmark function, including allocations from compilation.
+  * `warmed_up_allocated_bytes`: an integer list describing how many bytes were allocated while running the benchmark function in a `BenchmarkTools.@benchmark` loop. Due to how BenchmarkTools.jl works, this currently only reports the minimum number of bytes which may be misleading for non-uniform benchmarks.
+  * `warmed_up_median_gc_time_ns`: a floating point list describing the median time spent (in nanoseconds) performing garbage collection during the `BenchmarkTools.@benchmark` loop.
+  * `warmed_up_gc_alloc_count`: an integer list describing how many bytes were allocated while running the benchmark function during the `BenchmarkTools.@benchmark` loop. Due to how BenchmarkTools.jl works, this currently only reports the minimum number of allocations which may be misleading for non-uniform benchmarks.
+
+
+</details>
+</p>
+
+____________________________
+

benchmarks/benchmark_history/benchmarks-2024-02-23T16:50:05.343.csv

@@ -0,0 +1,14 @@
+benchmark_name	initial_runtime_ns	initial_compiletime_percent	warmed_up_minimum_runtime_ns	initial_alloced_bytes	initial_gc_time_ns	initial_gc_alloc_count	warmed_up_allocated_bytes	warmed_up_mean_gc_time_ns	warmed_up_gc_alloc_count
+Load NeuroBlox	7.369476917e9	1.4214368425302444	37625.0	734850107	6.79702999e8	10318708	16064	0.0	169
+Create LinearNeuralMass	4.99576791e8	99.75219084987476	57083.4	121260952	3.8999125e7	1769692	33800	0.0	602
+Harmonic Oscillator system creation	2.915693667e9	99.69101808938422	1.31335e6	625046264	1.18453583e8	8957166	773248	30214.0	14025
+Harmonic Oscillator structural simplify	2.720329333e9	99.31451009354683	1.91855e6	483187136	8.6839083e7	6912307	1468456	55464.834	23131
+Harmonic Oscillator solve	1.878186875e9	97.01073856135855	5.62715416e7	381869760	7.8092124e7	5163962	36118160	9.23291192e6	261615
+Jansen-Ritt system creation	3.528524834e9	99.43399264736478	9.2312084e6	747459240	1.54588374e8	10744894	4996656	195979.84	91539
+Jansen-Ritt structural simplify	2.685369042e9	99.51134124231108	5.642375e6	500089312	8.2470417e7	7169517	4949912	199222.91199999998	69411
+Jansen-Ritt solve	2.385998708e9	54.78274479434462	6.239073918e8	1707037304	5.85118835e8	14103713	1500800000	1.4001589530200002e8	11151973
+80 randomly connected neurons system creation	7.177286458e9	53.91559634193996	3.2914240582e9	2697485592	3.05394377e8	42882470	1877312888	2.5646218270666665e8	31164365
+80 randomly connected neurons structural simplify	3.190426425e10	8.616895808841603	2.8860887775e10	47401816984	2.815831364e9	387670661	46913880496	2.4955725992571435e9	380694603
+80 randomly connected neurons solve	5.9224711625e10	99.97451493542836	1.32897916e7	1951777427	5.77902916e8	36997038	1797280	34229.164	964
+RF_learning_simple agent and envionment creation	5.41550956916e11	3.069243824192552	1.662084073082e11	396358930512	3.82809568071e11	2114706979	393777212206	2.42064097534e10	2068528421
+RF_learning_simple run_experiment (three trials)	4.83825805792e11	42.700792927696305	2.72608228125e11	436918259851	1.1921961085e11	258622395	425552597088	1.4670334617882352e11	61636174

benchmarks/benchmark_history/benchmarks-2024-02-24T11:23:43.585.csv

@@ -0,0 +1,56 @@
+benchmark_name,initial_runtime_ns,initial_compiletime_percent,warmed_up_minimum_runtime_ns,warmed_up_median_runtime_ns,initial_alloced_bytes,initial_gc_time_ns,initial_gc_alloc_count,warmed_up_allocated_bytes,warmed_up_median_gc_time_ns,warmed_up_gc_alloc_count,metadata
+Load NeuroBlox,2.3689213282e10,0.9605418985057539,49708.4,57840.4,708067611,8.4616221e8,9865576,14712,0.0,146,"------------------------------
+Neuroblox Commit:
+
+Git Commit:
+Commit Author: Name: Mason Protter, Email: mason.protter@icloud.com, Time: 2024-02-24 17:22:59+01:00
+Committer: Name: Mason Protter, Email: mason.protter@icloud.com, Time: 2024-02-24 17:22:59+01:00
+SHA: 8d39ad99177d01b7fec501c875b724eb44def36c
+Message:
+fix sample setup
+
+
+------------------------------
+------------------------------
+Benchmark run on:
+
+Julia Version 1.10.1
+Commit 7790d6f0641 (2024-02-13 20:41 UTC)
+Build Info:
+  Official https://julialang.org/ release
+Platform Info:
+  OS: Linux (x86_64-linux-gnu)
+      Ubuntu 22.04.3 LTS
+  uname: Linux 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64
+  CPU: AMD EPYC 7532 32-Core Processor: 
+                 speed         user         nice          sys         idle          irq
+       #1-64  1500 MHz   36831493 s      20687 s    1488026 s  610210021 s          0 s
+  Memory: 62.646976470947266 GB (30179.05859375 MB free)
+  Uptime: 1.01382693e6 sec
+  Load Avg:  23.09  19.09  18.11
+  WORD_SIZE: 64
+  LIBM: libopenlibm
+  LLVM: libLLVM-15.0.7 (ORCJIT, znver2)
+Threads: 16 default, 0 interactive, 8 GC (on 64 virtual cores)
+Environment:
+  JULIA_NUM_THREADS = 16
+  MODULESHOME = /usr/share/modules
+  MANPATH = :
+  HOME = /shared/home/mason
+  TERM = screen
+  PATH = /shared/home/mason/.juliaup/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+  MODULEPATH = /gpfs/shared/modulefiles:/usr/share/modules/modulefiles
+
+"
+Create LinearNeuralMass,1.341931212e9,99.70725026999372,136454.8,141049.7,121010280,1.38613665e8,1767459,33800,0.0,602,
+Harmonic Oscillator system creation,7.627280553e9,99.71800202115891,3.2645118e6,3.3476793e6,629220056,2.80532331e8,9020746,773248,0.0,14025,
+Harmonic Oscillator structural simplify,6.328241536e9,99.6353857249484,4.5152328e6,4.6711626e6,484370976,1.91971592e8,6930668,1468456,0.0,23131,
+Harmonic Oscillator solve,4.426762989e9,97.31985307786262,1.015780046e8,1.105760651e8,380593208,1.03119086e8,5166826,36117328,8.4045782e6,261615,
+Jansen-Ritt system creation,8.629570402e9,99.46279314217941,2.26080786e7,2.4186183299999997e7,751370136,6.14411505e8,10806664,4996656,0.0,91539,
+Jansen-Ritt structural simplify,6.534040501e9,99.47871538300402,1.30600202e7,1.46482039e7,501144352,1.72191768e8,7186954,4948888,0.0,69411,
+Jansen-Ritt solve,4.494750214e9,73.90513289600123,1.1920581394e9,1.2825090629e9,1706325792,2.47025326e8,14100853,1500799168,2.493041477e8,11151973,
+40 randomly connected neurons system creation,1.5438616888e10,88.22216346716046,1.6309225704e9,1.6880669004e9,1205756944,8.05258818e8,18341269,375690560,7.84903415e7,6479060,
+40 randomly connected neurons structural simplify,1.0763618282e10,60.512716870425336,3.9282565634e9,4.0224453687e9,3363512680,4.86392699e8,31750486,2874459600,2.346931983e8,24756368,
+40 randomly connected neurons solve,2.0764418816e10,99.86013018106907,2.81746992e7,2.95260945e7,511097716,1.32650335e8,9185143,5267248,0.0,8397,
+RF_learning_simple agent and envionment creation,3.93546498608e11,10.615794321070537,3.547249751278e11,3.547249751278e11,389967141616,3.007737433e10,2102480891,390868456628,3.11829260584e10,2071661808,
+RF_learning_simple run_experiment (three trials),1.463582102466e12,18.443110551583878,1.181199463249e12,1.190139770507e12,438096672434,8.14319374759e11,258818535,426681803520,8.27850111267e11,61756677,