Adding the new fastest mandelbrot implementation to benchmarks-game.

[tannergooding]

FYI. @JosephTremoulet, @AndyAyersMS, @ViktorHofer, @danmosemsft

New fastest implementation. At this point, I'm pretty certain the only reason it isn't even faster is because the Q6600 processor the official benches use likely doesn't support the optimization where movups is as fast as movaps (it looks like that optimization was added in the subsequent microarchitecture).

[tannergooding]

Also FYI. @mellinoe

[tannergooding]

@dotnet-bot test Windows_NT x64 perf
@dotnet-bot test Windows_NT x86 perf
@dotnet-bot test linux perf flow

[JosephTremoulet]

Thanks for updating. You should remove mandelbrot-4 in this change, we generally don't want to keep more than two variants of each to avoid cluttering the reporting tracking these. You'll also, once this gets merged, want to port it to the release/1.1.0 and release/2.0.0 branches, like #14094 and #14095. Since you're just touching files in these directories, simple cherry-picks should work.

/cc @jorive

[benaadams]

There is an issue with the benchmark games in that they also include Jit time; so vectorizing isn't as fast because the Jit startup for Vectors is longer :-/

I was hoping #14244 would elevate this

[tannergooding]

Will fix up the build errors shortly.

[tannergooding]

You should remove mandelbrot-4 in this change

@JosephTremoulet, any particular reason why mandelbrot-4 over mandelbrot-2?

[JosephTremoulet]

any particular reason why mandelbrot-4 over mandelbrot-2?

coreclr/tests/src/JIT/Performance/CodeQuality/BenchmarksGame/README.TXT

Lines 32 to 37 in 8eb9adb

	These benchmarks are just a subset of the benchmarks available in C# from
	the Benchmarks Game site. The highest-scoring C# .NET Core variant of each
	benchmark is included, and in the (common) case of benchmarks where the
	best-scoring variant uses multiple threads, we've also selected variants
	that do not rely on multiple threads, to ensure relative benchmark stability
	across a variety of machines.

[tannergooding]

@dotnet-bot test Windows_NT x64 perf
@dotnet-bot test Windows_NT x86 perf
@dotnet-bot test linux perf flow

[JosephTremoulet]

LGTM. It would be good to verify that an InnerIterationCount of 7 is still reasonable (our rule of thumb has been to try to make the duration reported by run-xunit-performance roughly 1000ns).

[jorive]

@JosephTremoulet I think you meant to say 1000ms, right?

[JosephTremoulet]

@JosephTremoulet I think you meant to say 1000ms, right?

Whoops! Yes, ms, good catch, thanks.

[tannergooding]

@JosephTremoulet, what is the correct way to validate that? Still not quite sure how to properly navigate bench view.

[ViktorHofer]

Still not quite sure how to properly navigate bench view.

+1

[tannergooding]

That is, I can see the numbers for the jobs, etc. I just don't know how to properly compare them for improvement/etc, since Mandelbrot 4 and Mandelbrot 7 are "separate" scenarios.

I see 5619.01ms for Mandelbrot 4 and 6196.77ms for Mandelbrot 7 (duration), which seems backwards since 7 is measurably faster.

[JosephTremoulet]

What we've been doing to validate the iteration count is just run the benchmark via run-xunit-performance.cmd locally. The point is just to make sure it's neither too fast-running to produce good measurements and profiles nor so long-running that it wastes lab resources.

The different variants of each test had their iteration counts set independently, to make sure we're getting usable measurements for each. Yes, this means that comparing them to each other in benchview is meaningless, but the goal there is to have usable measurements and look independently at their improvements/regressions over time, not to rank them against each other (which, of course, is what happens over at BenchmarksGame).

[tannergooding]

run the benchmark via run-xunit-performance.cmd locally

@JosephTremoulet, what hardware are the actual jobs run against?

If you have a 8-core, 16-thread machine (or higher) this can seriously skew the results as compared to a 4-core, 8-thread machine.

[JosephTremoulet]

@JosephTremoulet, what hardware are the actual jobs run against?

Not sure... @jorive?

[jorive]

Haswell machines with: 4 cores, 8 threads, 3.6GHz

[tannergooding]

7 seems to be fine for the iteration count.

[tannergooding]

Am I good to merge?

[JosephTremoulet]

Am I good to merge?

Yes. And please port to release/1.1.0 and release/2.0.0 afterward.