Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add preference to disable LoopVectorization #2295

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
32 changes: 25 additions & 7 deletions src/Trixi.jl
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ See also: [trixi-framework/Trixi.jl](https://github.com/trixi-framework/Trixi.jl
"""
module Trixi

using Preferences: @load_preference, set_preferences!
const _PREFERENCE_SQRT = @load_preference("sqrt", "sqrt_Trixi_NaN")
const _PREFERENCE_LOG = @load_preference("log", "log_Trixi_NaN")
const _PREFERENCE_POLYESTER = @load_preference("polyester", true)
const _PREFERENCE_LOOPVECTORIZATION = @load_preference("loop_vectorization", true)

# Include other packages that are used in Trixi.jl
# (standard library packages first, other packages next, all of them sorted alphabetically)

Expand Down Expand Up @@ -52,8 +58,25 @@ using DiffEqCallbacks: PeriodicCallback, PeriodicCallbackAffect
using FillArrays: Ones, Zeros
using ForwardDiff: ForwardDiff
using HDF5: HDF5, h5open, attributes, create_dataset, datatype, dataspace
using LinearMaps: LinearMap
using LoopVectorization: LoopVectorization, @turbo, indices
if _PREFERENCE_LOOPVECTORIZATION
using LoopVectorization: LoopVectorization, @turbo, indices
else
using LoopVectorization: LoopVectorization, indices
macro turbo(exprs...)
body = nothing
for expr in exprs
if expr.head == :for
body = expr
end
end
@assert body !== nothing
# TODO: We should insert !loopinfo !julia.ivdep !julia.simd
# but SimdLoop.compile doesn't deal with nested for loops.
# esc(Base.SimdLoop.compile(body, Symbol("julia.ivdep")))
return esc(body)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe @turbo also implies @inbounds. Right now this slows down simulations quite a bit.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it does.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we go ahead with this PR as it is or would you like to change something?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What kind of simulations do you look at to observe the significant slowdown?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

end
end

using StaticArrayInterface: static_length # used by LoopVectorization
using MuladdMacro: @muladd
using Octavian: Octavian, matmul!
Expand Down Expand Up @@ -81,11 +104,6 @@ using SimpleUnPack: @pack!
using DataStructures: BinaryHeap, FasterForward, extract_all!

using UUIDs: UUID
using Preferences: @load_preference, set_preferences!

const _PREFERENCE_SQRT = @load_preference("sqrt", "sqrt_Trixi_NaN")
const _PREFERENCE_LOG = @load_preference("log", "log_Trixi_NaN")
const _PREFERENCE_POLYESTER = @load_preference("polyester", true)

# finite difference SBP operators
using SummationByPartsOperators: AbstractDerivativeOperator,
Expand Down
15 changes: 15 additions & 0 deletions src/auxiliary/math.jl
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,21 @@ function set_polyester!(toggle::Bool; force = true)
@info "Please restart Julia and reload Trixi.jl for the `polyester` change to take effect"
end

"""
Trixi.set_loop_vectorization!(toggle::Bool; force = true)

Toggle the usage of [LoopVectorization.jl](https://github.com/JuliaSIMD/LoopVectorization.jl).
By default, LoopVectorization.jl is enabled, but it can
be useful for performance comparisons to switch to the Julia core backend.

This does not fully disable LoopVectorization.jl,
but only its internal use as part of Trixi.jl.
"""
function set_loop_vectorization!(toggle::Bool; force = true)
set_preferences!(TRIXI_UUID, "loop_vectorization" => toggle, force = force)
@info "Please restart Julia and reload Trixi.jl for the `loop_vectorization` change to take effect"
end

"""
Trixi.set_sqrt_type!(type; force = true)

Expand Down
3 changes: 3 additions & 0 deletions src/callbacks_step/summary.jl
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,9 @@ function initialize_summary_callback(cb::DiscreteCallback, u, t, integrator;
if !_PREFERENCE_POLYESTER
push!(setup, "Polyester" => "disabled")
end
if !_PREFERENCE_LOOPVECTORIZATION
push!(setup, "LoopVectorization" => "disabled")
end
if mpi_isparallel()
push!(setup,
"#MPI ranks" => mpi_nranks())
Expand Down
2 changes: 1 addition & 1 deletion src/solvers/dg.jl
Original file line number Diff line number Diff line change
Expand Up @@ -672,7 +672,7 @@ end
nvariables(equations) * nnodes(dg)^ndims(mesh) * nelements(dg, cache)
end
# See comments on the DGSEM version above
if LoopVectorization.check_args(u_ode)
if _PREFERENCE_POLYESTER && LoopVectorization.check_args(u_ode)
# Here, we do not specialize on the number of nodes using `StaticInt` since
# - it will not be type stable (SBP operators just store it as a runtime value)
# - FD methods tend to use high node counts
Expand Down
Loading