Making benchmarks run on non-Windows and refresh the benchmarking gui…

…delines (#16628) * Refresh the benchmarking guidelines * Update DEVGUIDE.md * Make this run on other platforms * up
dotnet · Feb 5, 2024 · b0d0062 · b0d0062
1 parent 589d350
commit b0d0062
Show file tree

Hide file tree

Showing 10 changed files with 235 additions and 193 deletions.
diff --git a/DEVGUIDE.md b/DEVGUIDE.md
@@ -275,9 +275,25 @@ Where `<version>` corresponds to the latest Visual Studio version on your machin
 
 Use the `Debug` configuration to test your changes locally. It is the default. Do not use the `Release` configuration! Local development and testing of Visual Studio tooling is not designed for the `Release` configuration.
 
-### Writing and running benchmarks
+### Benchmarking
 
-Existing compiler benchmarks can be found in `tests\benchmarks\`.
+Existing compiler benchmarks can be found in `tests\benchmarks\`. The folder contains READMEs describing specific benchmark projects as well as guidelines for creating new benchmarks. There is also `FSharp.Benchmarks.sln` solution containing all the benchmark project and their dependencies.
+
+To exercise the benchmarking infrastructure locally, run:
+
+(Windows)
+```cmd
+build.cmd -configuration Release -testBenchmarks
+```
+
+(Linux/Mac)
+```shell
+./build.sh --configuration Release --testBenchmarks
+```
+
+This is executed in CI as well. It does the following:
+- builds all the benchmarking projects
+- does smoke testing for fast benchmarks (executes them once to check they don't fail in the runtime)
 
 ### Benchmarking and profiling the compiler
 
@@ -286,151 +302,6 @@ Existing compiler benchmarks can be found in `tests\benchmarks\`.
 * Always build both versions of compiler/FCS from source and not use pre-built binaries from SDK (SDK binaries are crossgen'd, which can affect performance).
 * To run `Release` build of compiler/FCS.
 
-### Example benchmark setup using [BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet)
-
-1. Perform a clean build of the compiler and FCS from source (as described in this document, build can be done with `-noVisualStudio` in case if FCS/FSharp.Core is being benchmarked/profiled).
-
-2. Create a benchmark project (in this example, the project will be created in `tests\benchmarks\FCSBenchmarks`).
-
-      ```shell
-      cd tests\benchmarks\FCSBenchmarks
-      dotnet new console -o FcsBench --name FcsBench -lang F#
-      ```
-
-3. Add needed packages and project references.
-
-    ```shell
-    cd FcsBench
-    dotnet add package BenchmarkDotNet
-    dotnet add reference ..\..\..\src\Compiler\FSharp.Compiler.Service.fsproj
-    ```
-
-4. Additionally, if you want to test changes to the FSharp.Core (note that the relative path can be different)
-
-     ```shell
-     dotnet add reference ..\..\..\src\FSharp.Core\FSharp.Core.fsproj
-     ```
-
-    > as well as the following property have to be added to `FcsBench.fsproj`:
-
-    ```xml
-    <PropertyGroup>
-        <DisableImplicitFSharpCoreReference>true</DisableImplicitFSharpCoreReference>
-    </PropertyGroup>
-    ```
-
-5. Add a new benchmark for FCS/FSharp.Core by editing `Program.fs`.
-
-      ```fsharp
-      open System.IO
-      open FSharp.Compiler.CodeAnalysis
-      open FSharp.Compiler.Diagnostics
-      open FSharp.Compiler.Text
-      open BenchmarkDotNet.Attributes
-      open BenchmarkDotNet.Running
-
-      [<MemoryDiagnoser>]
-      type CompilerService() =
-          let mutable checkerOpt = None
-          let mutable sourceOpt = None
-
-          let parsingOptions =
-              {
-                  SourceFiles = [|"CheckExpressions.fs"|]
-                  ConditionalDefines = []
-                  DiagnosticOptions = FSharpDiagnosticOptions.Default
-                  LangVersionText = "default"
-                  IsInteractive = false
-                  LightSyntax = None
-                  CompilingFsLib = false
-                  IsExe = false
-              }
-
-          [<GlobalSetup>]
-          member _.Setup() =
-              match checkerOpt with
-              | None ->
-                  checkerOpt <- Some(FSharpChecker.Create(projectCacheSize = 200))
-              | _ -> ()
-
-              match sourceOpt with
-              | None ->
-                  sourceOpt <- Some <| SourceText.ofString(File.ReadAllText("""C:\Users\vlza\code\fsharp\src\Compiler\Checking\CheckExpressions.fs"""))
-              | _ -> ()
-
-
-          [<Benchmark>]
-          member _.ParsingTypeCheckerFs() =
-              match checkerOpt, sourceOpt with
-              | None, _ -> failwith "no checker"
-              | _, None -> failwith "no source"
-              | Some(checker), Some(source) ->
-                  let results = checker.ParseFile("CheckExpressions.fs",  source, parsingOptions) |> Async.RunSynchronously
-                  if results.ParseHadErrors then failwithf "parse had errors: %A" results.Diagnostics
-
-          [<IterationCleanup(Target = "ParsingTypeCheckerFs")>]
-          member _.ParsingTypeCheckerFsSetup() =
-              match checkerOpt with
-              | None -> failwith "no checker"
-              | Some(checker) ->
-                  checker.InvalidateAll()
-                  checker.ClearLanguageServiceRootCachesAndCollectAndFinalizeAllTransients()
-                  checker.ParseFile("dummy.fs", SourceText.ofString "dummy", parsingOptions) |> Async.RunSynchronously |> ignore
-
-      [<EntryPoint>]
-      let main _ =
-          BenchmarkRunner.Run<CompilerService>() |> ignore
-          0
-      ```
-
-      > For more detailed information about available BenchmarkDotNet options, please refer to [BenchmarkDotNet Documentation](https://benchmarkdotnet.org/articles/overview.html).
-
-6. Build and run the benchmark.
-
-      ```shell
-      dotnet build -c Release
-      dotnet run -c Release
-      ```
-
-7. You can find results in `.\BenchmarkDotNet.Artifacts\results\` in the current benchmark project directory.
-
-    ```shell
-    > ls .\BenchmarkDotNet.Artifacts\results\
-
-        Directory: C:\Users\vlza\code\fsharp\tests\benchmarks\FCSBenchmarks\FcsBench\BenchmarkDotNet.Artifacts\results
-
-    Mode                 LastWriteTime         Length Name
-    ----                 -------------         ------ ----
-    -a---           4/25/2022  1:42 PM            638 Program.CompilerService-report-github.md
-    -a---           4/25/2022  1:42 PM           1050 Program.CompilerService-report.csv
-    -a---           4/25/2022  1:42 PM           1169 Program.CompilerService-report.html
-    ```
-
-    > *-report-github.md can be used to post benchmark results to GitHub issue/PR/discussion or RFC.
-    >
-    >*-report.csv can be used for comparison purposes.
-
-    **Example output:**
-
-    ``` ini
-
-    BenchmarkDotNet=v0.13.1, OS=Windows 10.0.25102
-    Intel Core i7-8750H CPU 2.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
-    .NET SDK=6.0.200
-      [Host]     : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT DEBUG
-      Job-GDIBXX : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT
-
-    InvocationCount=1  UnrollFactor=1
-
-    ```
-
-    |               Method |     Mean |   Error |  StdDev |   Median |     Gen 0 |     Gen 1 | Allocated |
-    |--------------------- |---------:|--------:|--------:|---------:|----------:|----------:|----------:|
-    | ParsingTypeCheckerFs | 199.4 ms | 3.84 ms | 9.78 ms | 195.5 ms | 4000.0000 | 1000.0000 |     28 MB |
-
-8. Repeat for any number of changes you would like to test.
-9. **Optionally:** benchmark code and results can be included as part of the PR for future reference.
-
 ## Additional resources
 
 The primary technical guide to the core compiler code is [The F# Compiler Technical Guide](https://github.com/dotnet/fsharp/blob/main/docs/index.md). Please read and contribute to that guide.

diff --git a/eng/build-utils.ps1 b/eng/build-utils.ps1
@@ -179,7 +179,7 @@ function Get-PackageDir([string]$name, [string]$version = "") {
 }
 
 function Run-MSBuild([string]$projectFilePath, [string]$buildArgs = "", [string]$logFileName = "", [switch]$parallel = $true, [switch]$summary = $true, [switch]$warnAsError = $true, [string]$configuration = $script:configuration, [string]$verbosity = $script:verbosity) {
-    # Because we override the C#/VB toolset to build against our LKG package, it is important
+    # Because we override the C#/VB toolset to build against our LKG (Last Known Good) package, it is important
     # that we do not reuse MSBuild nodes from other jobs/builds on the machine. Otherwise,
     # we'll run into issues such as https://github.com/dotnet/roslyn/issues/6211.
     # MSBuildAdditionalCommandLineArgs=

diff --git a/eng/build.sh b/eng/build.sh
@@ -25,6 +25,7 @@ usage()
   echo "Test actions:"
   echo "  --testcoreclr                  Run unit tests on .NET Core (short: --test, -t)"
   echo "  --testCompilerComponentTests   Run FSharp.Compiler.ComponentTests on .NET Core"
+  echo "  --testBenchmarks               Build and Run Benchmark suite"
   echo ""
   echo "Advanced settings:"
   echo "  --ci                           Building in CI"
@@ -56,6 +57,7 @@ pack=false
 publish=false
 test_core_clr=false
 test_compilercomponent_tests=false
+test_benchmarks=false
 configuration="Debug"
 verbosity='minimal'
 binary_log=false
@@ -126,6 +128,9 @@ while [[ $# > 0 ]]; do
     --testcompilercomponenttests)
       test_compilercomponent_tests=true
       ;;
+      --testbenchmarks)
+      test_benchmarks=true
+      ;;
     --ci)
       ci=true
       ;;
@@ -330,4 +335,10 @@ if [[ "$test_compilercomponent_tests" == true ]]; then
   TestUsingNUnit --testproject "$repo_root/tests/FSharp.Compiler.ComponentTests/FSharp.Compiler.ComponentTests.fsproj" --targetframework $coreclrtestframework  --notestfilter 
 fi
 
+if [[ "$test_benchmarks" == true ]]; then
+  pushd "$repo_root/tests/benchmarks"
+  ./SmokeTestBenchmarks.sh
+  popd
+fi
+
 ExitWithExitCode 0
diff --git a/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/README.md b/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/README.md
@@ -12,7 +12,7 @@ To run a benchmark for a local FCS in the current codebase you can run the bench
 
 ```dotnet run --project HistoricalBenchmark.fsproj -c Release --filter *```
 
-To run a comparison use the `runner.ipynb` .NET notebook
+To run a comparison use the `runner.ipynb` .NET notebook.
 
 ## How it works
 
@@ -35,9 +35,4 @@ As of now the minimum supported version of FCS is 13.0.0
 
 ## Sample results
 
-Below is a sample result of running the notebook locally with a selection of versions:
-![a](./sample_result.png?raw=true)
-
-## Other
-
-You can find this document under 'tests/benchmarks/FCSBenchmarks/BenchmarkComparison/README.md'.
+See sample results in the dedicated [sample_results](./sample_results/) folder.
diff --git a/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/sample_results/README.md b/tests/benchmarks/FCSBenchmarks/BenchmarkComparison/sample_results/README.md
@@ -1,13 +1,16 @@
 # Sample results
-This folder contains a selection of results obtained by running the notebook located in `../runner.ipynb`
+
+This folder contains a selection of results obtained by running the notebook located in `../runner.ipynb`.
 
 ## Timings are not accurate
+
 The results were gathered on a busy machine without much care taken to provide a reliable performance environment.
 While this means the timing metrics are not very useful, the results can still be useful for two reasons:
 * allocation data is quite accurate as it doesn't tend to depend much on the environment
 * they work as examples that can make using the benchmarks easier
 
 ## Structure
+
 Each directory contains 3 files output by `HistoricalBenchmark.Runner.runAll` function for a given selection of versions.
 
 The three different version sets are:
@@ -16,10 +19,12 @@ The three different version sets are:
 - `10_latest_nuget_versions` - 10 FCS NuGet versions between `v41.0.2` and ``v41.0.5-preview.22327.2`
 
 ## Observations
-One thing that can be observed by looking at the results in `between_2_nuget_versions` is the noticable increase of allocations in https://github.com/dotnet/fsharp/pull/11517
+
+One thing that can be observed by looking at the results in `between_2_nuget_versions` is the noticeable increase of allocations in https://github.com/dotnet/fsharp/pull/11517.
 
 While this isn't necessarily something worth addressing, partly because later revisions show reduced allocations, it shows how running a historical benchmark can be potentially useful.
 
 ## Notes
+
 - The metrics gathered here are very limited - much more data can be gathered from each benchmark.
 - Such historical benchmarks run locally might be mostly deprecated once CI setup exists for performance tests that will provide the necessary historical information
diff --git a/tests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/README.md b/tests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/README.md
@@ -2,21 +2,24 @@
 
 ## What is it
 
-* A selection of BDN benchmarks analysing FCS performance
-* Uses BDN's commandline API
+* A selection of BDN benchmarks analyzing FCS performance
+* Uses BDN's command line API
 
 ## How to run it
 
 Running all benchmarks:
 ```dotnet run -c Release --filter *```
 
 Running a specific benchmark:
-```dotnet run -c Release --filter *ParsingTypeCheckerFs*```
+```dotnet run -c Release --filter *ParsingCheckExpressionsFs*```
 
 ## Sample results
 
-*TODO*
-
-## Other
-
-You can find this document under 'tests/benchmarks/FCSBenchmarks/BenchmarkComparison/README.md'.
+|                                 Method |        Job | UnrollFactor |           Mean |        Error |        StdDev |         Median |       Gen 0 |      Gen 1 |     Gen 2 |  Allocated |
+|--------------------------------------- |----------- |------------- |---------------:|-------------:|--------------:|---------------:|------------:|-----------:|----------:|-----------:|
+|                          SimplifyNames | DefaultJob |           16 |    17,221.4 us |    378.14 us |   1,097.04 us |    17,164.1 us |   1875.0000 |    31.2500 |         - |  11,654 KB |
+|                            UnusedOpens | DefaultJob |           16 |       852.7 us |     16.96 us |      36.87 us |       852.0 us |    120.1172 |    37.1094 |         - |     736 KB |
+|                     UnusedDeclarations | DefaultJob |           16 |       208.2 us |      6.65 us |      19.09 us |       202.7 us |     71.5332 |     3.6621 |         - |     438 KB |
+|              ParsingCheckExpressionsFs | Job-CXFNSP |            1 |   255,107.0 us | 39,778.24 us | 117,287.03 us |   186,340.7 us |   4000.0000 |  1000.0000 |         - |  30,082 KB |
+|                              ILReading | Job-CXFNSP |            1 | 1,256,653.6 us | 24,802.85 us |  48,958.41 us | 1,249,170.3 us | 102000.0000 | 31000.0000 | 2000.0000 | 671,507 KB |
+| TypeCheckFileWith100ReferencedProjects | Job-CXFNSP |            1 |     6,541.1 us |    242.62 us |     700.00 us |     6,614.2 us |           - |          - |         - |   3,547 KB |
diff --git a/tests/benchmarks/FCSBenchmarks/FCSSourceFiles/README.md b/tests/benchmarks/FCSBenchmarks/FCSSourceFiles/README.md
@@ -9,12 +9,18 @@
 ## How to run it
 
 1. Run `dotnet run -c release`
-2. Output available on the commandline and in `BenchmarkDotNet.Artifacts/`
+2. Output available on the command line and in `BenchmarkDotNet.Artifacts/`
 
 ## Sample results
 
-*TODO*
-
-## Other
-
-You can find this document under 'tests/benchmarks/FCSBenchmarks/FCSSourceFiles/README.md'.
+```
+BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22621
+11th Gen Intel Core i7-1185G7 3.00GHz, 1 CPU, 8 logical and 4 physical cores
+.NET SDK=6.0.320
+  [Host]     : .NET 6.0.25 (6.0.2523.51912), X64 RyuJIT DEBUG
+  DefaultJob : .NET 6.0.25 (6.0.2523.51912), X64 RyuJIT
+```
+
+|                     Method |    Mean |   Error |  StdDev |        Gen 0 |       Gen 1 |     Gen 2 | Allocated |
+|--------------------------- |--------:|--------:|--------:|-------------:|------------:|----------:|----------:|
+| ParseAndCheckFileInProject | 22.14 s | 0.543 s | 1.522 s | 1645000.0000 | 307000.0000 | 6000.0000 |     10 GB |
diff --git a/tests/benchmarks/FCSBenchmarks/README.md b/tests/benchmarks/FCSBenchmarks/README.md
@@ -2,26 +2,19 @@
 
 ## What can be found here
 
-Benchmarks that exercise performance of `FSharp.Compiler.Service`
+Benchmarks that exercise performance of `FSharp.Compiler.Service`.
+
+## Testing performance of FSharp.Compiler.Service
 
-## Testing performance of FSharp.Compiler.Service 
 Performance of the compiler service is crucial for having good developer experience.
 This includes compilation, type checking and any other parts of the API used by IDEs.
 
 When making changes to the FCS source code, consider running some of these to assess impact of the changes on performance.
 
 ## Benchmark list
+
 * `BenchmarkComparison/` - a Notebook-based benchmark that analyses performance of `FSharpChecker.ParseAndCheckFileInProject` on a single-file F# project. Supports comparing different revisions of the FCS codebase and fetching the code automatically.
-* `CompilerServiceBenchmarks/` - 
+* `CompilerServiceBenchmarks/` - a selection of BDN benchmarks analyzing FCS performance
 * `FCSSourceFiles/` - analyses performance of `FSharpChecker.ParseAndCheckFileInProject` for the `FSharp.Core` project. Uses locally available source code.
 
 All the above benchmarks use BenchmarkDotNet.
-
-## Quickly validating that the benchmarks work
-`SmokeTestAllBenchmarks.ps1` allows to run all BDN benchmarks in this directory with a minimum number of iterations, as a way to verify that the benchmarks still work.
-
-This doesn't validate the notebook-based meta-benchmarks.
-
-## Other
-
-You can find this document under 'tests/benchmarks/FCSBenchmarks/README.md'.