Performance regression makes function unusable on 1.10.0-beta1

[nathanrboyer]

Sample code for testing is at the following repository: https://github.com/nathanrboyer/JuliaNightlyTestCode

polyfit function takes 5 minutes on 1.9 and over 1 day on 1.10. I get similar performance to 1.10.0-beta1 using the Nightly build from August 8th, 2023.

[nathanrboyer]

My timings using the trimmed data are below.

julia> versioninfo()
Julia Version 1.9.2
Commit e4ee485e90 (2023-07-05 09:39 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 1 on 8 virtual cores
Environment:
  JULIA_EDITOR = code.cmd
  JULIA_NUM_THREADS =

julia> include("test.jl")
  2.359726 seconds (6.76 M allocations: 3.041 GiB, 16.36% gc time, 72.71% compilation time)
  0.601131 seconds (2.38 M allocations: 2.750 GiB, 32.01% gc time)
  2.536775 seconds (8.09 M allocations: 5.714 GiB, 17.44% gc time, 54.94% compilation time)

julia> include("test.jl")
  0.613573 seconds (2.38 M allocations: 2.750 GiB, 33.90% gc time)
  0.572166 seconds (2.38 M allocations: 2.750 GiB, 29.21% gc time)
  1.119766 seconds (4.84 M allocations: 5.502 GiB, 30.87% gc time)

julia> versioninfo()
Julia Version 1.11.0-DEV.239
Commit c40ecd3f03 (2023-08-08 03:06 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, skylake)
  Threads: 1 on 8 virtual cores

julia> include("test.jl")
  1.996888 seconds (6.10 M allocations: 1.759 GiB, 11.57% gc time, 74.28% compilation time)
  0.489526 seconds (2.06 M allocations: 1.490 GiB, 37.59% gc time)
1119.263458 seconds (665.26 M allocations: 50.574 GiB, 0.50% gc time, 0.09% compilation time)

julia> include("test.jl")
  0.533068 seconds (2.13 M allocations: 1.494 GiB, 25.27% gc time, 18.15% compilation time)
  0.427734 seconds (2.06 M allocations: 1.490 GiB, 22.92% gc time)
1096.128067 seconds (663.10 M allocations: 50.433 GiB, 0.38% gc time, 0.00% compilation time)

Looks like 1.10 is allocating too much.

[gbaraldi]

So this seems to have regressed a couple of times actually.
I'm doing the bisec right now, but one big regression happened in
35e4a1f @vtjnash

[Seelengrab]

That's a 288x regression 😮

[gbaraldi]

It actually has exponential behaviour here which is very fun.

[Seelengrab]

Seems like a good candidate for GC benchmarks then!

[gbaraldi]

It's not a GC regression at all, 35e4a1f made it super bad and fbbe9ed made it superbad 2x

[brenhinkeller]

Is there an intuitive explanation for why making Tuple{Union{}} "unconstructable" caused this?

[gbaraldi]

I believe it's the change to the broadcasting code that caused this.

[gbaraldi]

So what I think is going on here is that this code is somewhat unstable but return_type moved from a compile time thing to a runtime one. I'm not sure why however.

[brenhinkeller]

ahh, interesting

[nathanrboyer]

I'm open to just making my code more type-stable if you have suggestions for that and think it unlikely that others will run into this issue. I thought that I had defined types well enough for the compiler but apparently not.

[brenhinkeller]

That's always good too (sometimes easier said than done, though I find Cthulhu.jl is nice for tracking down where the instabilities start) -- but I think this is also something we'll want to try to fix! Looks like it's been assigned to the 1.10 milestone, meaning 1.10 won't be released until it's addressed at least in some way

[vtjnash]

Sounds probably the same as #51129