Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x64: 25 Regressions on 3/10/2023 8:23:08 AM #14000

Closed
performanceautofiler bot opened this issue Mar 14, 2023 · 2 comments
Closed

[Perf] Windows/x64: 25 Regressions on 3/10/2023 8:23:08 AM #14000

performanceautofiler bot opened this issue Mar 14, 2023 · 2 comments
Assignees
@AndyAyersMS
Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 3db965d1c9fcc8d9c4bc4c2dc9b644f61e7bb6ce
Compare a923c644067c4a37f0bb7c7a331a791d2e51c183
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HeapSort - Duration of single invocation 2.79 μs 2.96 μs 1.06 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_PriorityQueue<String, String>.HeapSort(Size: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9576690780619113 > 2.9383173093405683.
IsChangePoint: Marked as a change because one of 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -17.360229880248127 (T) = (0 -2967.871813274792) / Math.Sqrt((9813.457382327486 / (44)) + (820.2249742602418 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.10726897693139238 = (2680.353080513232 - 2967.871813274792) / 2680.353080513232 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 3db965d1c9fcc8d9c4bc4c2dc9b644f61e7bb6ce
Compare a923c644067c4a37f0bb7c7a331a791d2e51c183
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 31.56 μs 42.11 μs 1.33 0.04 False
Count - Duration of single invocation 42.89 μs 55.22 μs 1.29 0.04 False
Count - Duration of single invocation 41.44 ms 45.90 ms 1.11 0.29 False
Count - Duration of single invocation 34.85 μs 52.86 μs 1.52 0.08 False
Count - Duration of single invocation 46.07 μs 56.00 μs 1.22 0.06 False
Count - Duration of single invocation 29.70 μs 78.11 μs 2.63 0.17 False
Count - Duration of single invocation 23.23 μs 42.00 μs 1.81 0.03 False
Count - Duration of single invocation 23.21 μs 41.98 μs 1.81 0.03 False
Count - Duration of single invocation 242.87 μs 256.51 μs 1.06 0.09 False
Count - Duration of single invocation 29.19 μs 47.58 μs 1.63 0.28 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aqj", Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 42.10845687331536 > 33.191861016414144.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 2/9/2023 4:27:10 AM, 2/16/2023 9:36:08 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -1347.8086981968615 (T) = (0 -42072.14502803651) / Math.Sqrt((686.3818791209757 / (44)) + (715.7135838783402 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.33126022406956734 = (31603.246508353543 - 42072.14502803651) / 31603.246508353543 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aei", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.22360718492345 > 45.11800525388567.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 2/9/2023 4:27:10 AM, 2/16/2023 9:36:08 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -50.96978180903732 (T) = (0 -55357.9421702336) / Math.Sqrt((2058875.9613437243 / (44)) + (13410.977992567374 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.2514780688411035 = (44234.04896059927 - 55357.9421702336) / 44234.04896059927 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\\p{L}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.89966 > 43.734725999999995.
IsChangePoint: Marked as a change because one of 1/19/2023 4:24:48 PM, 3/8/2023 9:36:36 PM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -5.35469458782003 (T) = (0 -45177587.99071542) / Math.Sqrt((2909175628053.5327 / (38)) + (2294219403088.0537 / (22))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (22) - 2, .025) and -0.05307862396842545 = (42900489.06364468 - 45177587.99071542) / 42900489.06364468 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock Holmes", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 52.861717171717174 > 36.54963223428607.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/18/2023 3:20:52 PM, 2/9/2023 4:27:10 AM, 2/16/2023 9:36:08 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -11.030602299426423 (T) = (0 -46663.08355286457) / Math.Sqrt((1737414.0645207271 / (24)) + (34884311.1701462 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.31792575491794495 = (35406.45850400719 - 46663.08355286457) / 35406.45850400719 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aei", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 56.0008784562212 > 48.41459218289086.
IsChangePoint: Marked as a change because one of 1/5/2023 12:24:13 PM, 2/9/2023 4:27:10 AM, 2/16/2023 9:36:08 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -350.8317918177498 (T) = (0 -55943.88647016258) / Math.Sqrt((17053.61167121431 / (43)) + (6431.976897569224 / (16))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (16) - 2, .025) and -0.21538969833617586 = (46029.58750329028 - 55943.88647016258) / 46029.58750329028 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock Holmes", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 78.11363184079602 > 31.072296756791463.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/18/2023 3:20:52 PM, 2/9/2023 4:27:10 AM, 2/16/2023 9:36:08 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -7.0363563954930415 (T) = (0 -51716.48556696194) / Math.Sqrt((14482653.001323603 / (44)) + (105979168.39073879 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.5594709060520947 = (33162.84091370816 - 51716.48556696194) / 33162.84091370816 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aqj", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 41.9993817204301 > 24.315575023148153.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 2/9/2023 4:27:10 AM, 2/16/2023 9:36:08 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -23.52967465260281 (T) = (0 -42005.646866533505) / Math.Sqrt((17893370.38018 / (44)) + (848.7292719647381 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.5557831363367535 = (26999.68002316826 - 42005.646866533505) / 26999.68002316826 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aqj", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 41.98148570151922 > 24.38228420595726.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/9/2023 4:27:10 AM, 2/16/2023 9:36:08 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -23.68777882348211 (T) = (0 -41967.89273271779) / Math.Sqrt((17555442.755086664 / (44)) + (1386.996628331775 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.5541490097390989 = (27003.77664543447 - 41967.89273271779) / 27003.77664543447 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 256.51440188172046 > 252.64857307692307.
IsChangePoint: Marked as a change because one of 2/17/2023 5:12:50 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -13.936335922401263 (T) = (0 -259340.19974920046) / Math.Sqrt((13122775.20556015 / (24)) + (56526569.23292118 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.08578966040023935 = (238849.39156042758 - 259340.19974920046) / 238849.39156042758 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 47.576506605097045 > 30.5618039208936.
IsChangePoint: Marked as a change because one of 1/18/2023 11:30:09 AM, 2/9/2023 4:27:10 AM, 2/16/2023 9:36:08 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -26.403388423117672 (T) = (0 -47379.45298146063) / Math.Sqrt((14044638.499533728 / (44)) + (9784.602292157204 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.4601689373651937 = (32447.925557815677 - 47379.45298146063) / 32447.925557815677 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 3db965d1c9fcc8d9c4bc4c2dc9b644f61e7bb6ce
Compare a923c644067c4a37f0bb7c7a331a791d2e51c183
Diff Diff

Regressions in System.Collections.ContainsKeyFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedDictionary - Duration of single invocation 306.76 μs 331.90 μs 1.08 0.04 False
ImmutableSortedDictionary - Duration of single invocation 281.95 μs 320.49 μs 1.14 0.04 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<String, String>.SortedDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 331.90107269503545 > 320.218867965215.
IsChangePoint: Marked as a change because one of 1/3/2023 3:01:43 PM, 1/19/2023 8:10:32 PM, 1/26/2023 1:49:07 PM, 2/17/2023 9:53:52 PM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -20.902728335299493 (T) = (0 -331583.3966153089) / Math.Sqrt((56603522.26521541 / (43)) + (1264011.7726643789 / (16))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (16) - 2, .025) and -0.08045645051369799 = (306891.95891019865 - 331583.3966153089) / 306891.95891019865 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsKeyFalse&lt;String, String&gt;.ImmutableSortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 320.48834821428574 > 295.716971970943.
IsChangePoint: Marked as a change because one of 1/20/2023 1:51:38 AM, 3/6/2023 4:48:38 PM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -10.638843104472334 (T) = (0 -313800.5998067116) / Math.Sqrt((65884801.18315772 / (43)) + (64115923.458917566 / (16))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (16) - 2, .025) and -0.08671399913764334 = (288760.98040121555 - 313800.5998067116) / 288760.98040121555 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 3db965d1c9fcc8d9c4bc4c2dc9b644f61e7bb6ce
Compare a923c644067c4a37f0bb7c7a331a791d2e51c183
Diff Diff

Regressions in System.Xml.Linq.Perf_XElement

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetValue - Duration of single invocation 56.80 ns 77.15 ns 1.36 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Xml.Linq.Perf_XElement*'

Payloads

Baseline
Compare

Histogram

System.Xml.Linq.Perf_XElement.GetValue

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 77.15336331129251 > 60.196612921049145.
IsChangePoint: Marked as a change because one of 1/5/2023 11:35:29 PM, 1/13/2023 4:37:08 PM, 2/9/2023 4:10:34 PM, 2/20/2023 10:31:48 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -20.39594113405279 (T) = (0 -69.05708769543413) / Math.Sqrt((0.9522471660034456 / (44)) + (4.706275270162643 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.19899823407604186 = (57.5956542159965 - 69.05708769543413) / 57.5956542159965 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 3db965d1c9fcc8d9c4bc4c2dc9b644f61e7bb6ce
Compare a923c644067c4a37f0bb7c7a331a791d2e51c183
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Clear - Duration of single invocation 4.51 ns 5.57 ns 1.24 0.25 False
Fill - Duration of single invocation 3.56 ns 4.78 ns 1.34 0.22 False
IndexOfAnyTwoValues - Duration of single invocation 7.25 ns 8.36 ns 1.15 0.03 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.Clear(Size: 33)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.571250182340605 > 4.797357764276794.
IsChangePoint: Marked as a change because one of 1/26/2023 7:55:58 PM, 1/27/2023 7:12:54 PM, 2/8/2023 7:09:09 PM, 2/16/2023 4:50:02 PM, 2/28/2023 10:21:25 PM, 3/4/2023 1:07:20 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -10.513950879401998 (T) = (0 -5.605083391113334) / Math.Sqrt((0.3429131545629304 / (43)) + (0.008368584917275037 / (16))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (16) - 2, .025) and -0.2090675844043878 = (4.63587268686433 - 5.605083391113334) / 4.63587268686433 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.Fill(Size: 33)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.779768954738696 > 3.9400281495517824.
IsChangePoint: Marked as a change because one of 2/9/2023 12:40:40 PM, 2/13/2023 6:54:14 PM, 2/28/2023 10:21:25 PM, 3/4/2023 1:07:20 AM, 3/8/2023 9:36:36 PM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -4.909196791737205 (T) = (0 -4.6888546666373365) / Math.Sqrt((0.45237456790804503 / (38)) + (0.04980733660066975 / (22))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (22) - 2, .025) and -0.14236808580712482 = (4.10450425295669 - 4.6888546666373365) / 4.10450425295669 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyTwoValues(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.356226692691582 > 7.604048457824599.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -168.55581435748834 (T) = (0 -8.362388067121671) / Math.Sqrt((0.0010121293880814963 / (24)) + (0.0001500034740954831 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.15903064882942777 = (7.214984414404685 - 8.362388067121671) / 7.214984414404685 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 3db965d1c9fcc8d9c4bc4c2dc9b644f61e7bb6ce
Compare a923c644067c4a37f0bb7c7a331a791d2e51c183
Diff Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LastIndexOf_Word_NotFound - Duration of single invocation 21.00 ns 25.97 ns 1.24 0.11 False
IsSuffix_SecondHalf - Duration of single invocation 172.67 ns 203.06 ns 1.18 0.10 False
IsSuffix_SecondHalf - Duration of single invocation 172.73 ns 202.78 ns 1.17 0.07 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, Ordinal, False))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.97246253097995 > 22.131990728920563.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/16/2023 9:36:08 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -28.61814161088012 (T) = (0 -25.668722534878356) / Math.Sqrt((0.27846001754446037 / (44)) + (0.22012401504402132 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.18766436794983246 = (21.61277481043585 - 25.668722534878356) / 21.61277481043585 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, IgnoreCase, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 203.05856980206693 > 181.46865024969247.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/5/2023 12:24:13 PM, 1/6/2023 5:32:27 AM, 1/13/2023 4:18:46 AM, 1/18/2023 3:20:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -664.9004358744993 (T) = (0 -202.6012809188048) / Math.Sqrt((0.017305511655362534 / (24)) + (0.045020955733093226 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.1723848023838373 = (172.81124807047217 - 202.6012809188048) / 172.81124807047217 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, IgnoreCase, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 202.78384174256664 > 181.2946761945911.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 2/16/2023 9:36:08 AM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -358.58137174017713 (T) = (0 -202.57607370393686) / Math.Sqrt((0.03451414832488343 / (24)) + (0.19664560387101904 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.1723894717006282 = (172.78905909150387 - 202.57607370393686) / 172.78905909150387 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 3db965d1c9fcc8d9c4bc4c2dc9b644f61e7bb6ce
Compare a923c644067c4a37f0bb7c7a331a791d2e51c183
Diff Diff

Regressions in System.Collections.TryGetValueFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableSortedDictionary - Duration of single invocation 278.65 μs 310.94 μs 1.12 0.04 False
SortedDictionary - Duration of single invocation 299.29 μs 334.64 μs 1.12 0.06 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueFalse<String, String>.ImmutableSortedDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 310.9368928571428 > 292.1992626096491.
IsChangePoint: Marked as a change because one of 1/7/2023 12:45:06 AM, 1/10/2023 1:28:02 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -10.64909586972241 (T) = (0 -316438.02218770963) / Math.Sqrt((46377835.395175576 / (44)) + (141146587.45079333 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.11826396201032534 = (282972.5654565875 - 316438.02218770963) / 282972.5654565875 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.TryGetValueFalse&lt;String, String&gt;.SortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 334.63622562056736 > 317.43223084860307.
IsChangePoint: Marked as a change because one of 1/27/2023 7:12:54 PM, 3/6/2023 11:36:59 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -7.7702672435563835 (T) = (0 -341483.0335640237) / Math.Sqrt((283037831.1299954 / (44)) + (154052034.81629285 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.10034126500711774 = (310342.8403749038 - 341483.0335640237) / 310342.8403749038 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 3db965d1c9fcc8d9c4bc4c2dc9b644f61e7bb6ce
Compare a923c644067c4a37f0bb7c7a331a791d2e51c183
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Replace_String - Duration of single invocation 48.92 ns 54.74 ns 1.12 0.02 False
Replace_Char - Duration of single invocation 23.83 ns 28.43 ns 1.19 0.04 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 54.737843792140985 > 51.3744031300299.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -23.01162562902453 (T) = (0 -54.48628974007603) / Math.Sqrt((1.43999745885594 / (44)) + (0.11170784089574597 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.0918934494696286 = (49.900738727338236 - 54.48628974007603) / 49.900738727338236 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58l", oldChar: 'b', newChar: '+')

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.429553145568168 > 25.108693885277397.
IsChangePoint: Marked as a change because one of 1/13/2023 4:18:46 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -71.35702194797511 (T) = (0 -29.178256216939808) / Math.Sqrt((0.03000133379114158 / (24)) + (0.1440007970198561 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.21536008759202302 = (24.00791050720639 - 29.178256216939808) / 24.00791050720639 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 3db965d1c9fcc8d9c4bc4c2dc9b644f61e7bb6ce
Compare a923c644067c4a37f0bb7c7a331a791d2e51c183
Diff Diff

Regressions in System.Reflection.Invoke

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Field_Get_int - Duration of single invocation 55.01 ns 61.95 ns 1.13 0.10 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Invoke*'

Payloads

Baseline
Compare

Histogram

System.Reflection.Invoke.Field_Get_int

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.95133390705679 > 58.204005910818104.
IsChangePoint: Marked as a change because one of 1/13/2023 4:18:46 AM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -27.060397524082454 (T) = (0 -62.563818835535706) / Math.Sqrt((3.465918660800388 / (44)) + (0.5479737952178403 / (16))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (16) - 2, .025) and -0.17014848661387952 = (53.46656390299656 - 62.563818835535706) / 53.46656390299656 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@EgorBo
Copy link
Member

EgorBo commented Mar 14, 2023

Some of these e.g. Field_Get_int could be dotnet/runtime#80963

@EgorBo EgorBo closed this as completed Mar 14, 2023
@EgorBo EgorBo removed the untriaged label Mar 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AndyAyersMS AndyAyersMS

Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@EgorBo @AndyAyersMS