Allow StructArray arguments in CUDAnative kernels.

[lcw]

These changes allow StructArrays to be use as arguments to CUDAnative kernels and allows getindex and setindex! to be called. For example the following code now works

using CUDAnative, CuArrays, StaticArrays, StructArrays

c = [SHermitianCompact(@SVector(rand(3))) for i=1:5]
d = StructArray(c, unwrap = t -> t <: Union{SHermitianCompact,SVector,Tuple})

dd = replace_storage(CuArray, d)
de = similar(dd)

@show typeof(dd)

function kernel!(dest, src)
    i = (blockIdx().x-1)*blockDim().x + threadIdx().x
    if i <= length(dest)
        dest[i] = src[i]
    end
    return nothing
end

threads = 1024
blocks = cld(length(dd),threads)

@cuda threads=threads blocks=blocks kernel!(de, dd)

[vchuravy]

Might need to be

ntuple(Val(N))` do i
   Base.@_inline_meta
  @inbounds res = getfield(cols, i)[I...]
 end

But the generated function works as well.

[lcw]

I tried

function get_ith(cols::NTuple{N, Any}, I...) where N
    ntuple(Val(N)) do i
        Base.@_inline_meta
        @inbounds res = getfield(cols, i)[I...]
    end
end

but still got a dynamic function

ERROR: LoadError: InvalidIRError: compiling kernel!(StructArray{SHermitianCompact{2,Float64,3},1,NamedTuple{(:lowertriangle,),Tuple{StructArray{SArray{Tuple{3},Float64,1,3},1,NamedTuple{(:data,),Tuple{StructArray{Tuple{Float64,Float64,Float64},1,Tuple{CuDeviceArray{Float64,1,CUDAnative.AS.Global},CuDeviceArray{Float64,1,CUDAnative.AS.Global},CuDeviceArray{Float64,1,CUDAnative.AS.Global}},Int64}}},Int64}}},Int64}, StructArray{SHermitianCompact{2,Float64,3},1,NamedTuple{(:lowertriangle,),Tuple{StructArray{SArray{Tuple{3},Float64,1,3},1,NamedTuple{(:data,),Tuple{StructArray{Tuple{Float64,Float64,Float64},1,Tuple{CuDeviceArray{Float64,1,CUDAnative.AS.Global},CuDeviceArray{Float64,1,CUDAnative.AS.Global},CuDeviceArray{Float64,1,CUDAnative.AS.Global}},Int64}}},Int64}}},Int64}) resulted in invalid LLVM IR
Reason: unsupported dynamic function invocation (call to ntuple(f, ::Val{1}) in Base at ntuple.jl:41)
Stacktrace:
 [1] get_ith at /home/lucas/julia/dev/StructArrays/src/structarray.jl:138
 [2] get_ith at /home/lucas/julia/dev/StructArrays/src/structarray.jl:136
 [3] getindex at /home/lucas/julia/dev/StructArrays/src/structarray.jl:153
 [4] kernel! at /home/lucas/research/code/Heptapus.jl/examples/structarrays/try.jl:14
Reason: unsupported call to the Julia runtime (call to jl_f_getfield)
Stacktrace:
 [1] getindex at /home/lucas/julia/dev/StructArrays/src/structarray.jl:153
 [2] kernel! at /home/lucas/research/code/Heptapus.jl/examples/structarrays/try.jl:14
Reason: unsupported dynamic function invocation (call to foreachfield)
Stacktrace:
 [1] getindex at /home/lucas/julia/dev/StructArrays/src/structarray.jl:153
 [2] kernel! at /home/lucas/research/code/Heptapus.jl/examples/structarrays/try.jl:14
Reason: unsupported dynamic function invocation (call to foreachfield)
Stacktrace:
 [1] setindex! at /home/lucas/julia/dev/StructArrays/src/structarray.jl:168
 [2] kernel! at /home/lucas/research/code/Heptapus.jl/examples/structarrays/try.jl:14

[piever]

Generally, I would prefer a normal function over a generated function. What is the cause of "dynamic function invocation"? Is it due to the I... splatting? Never mind, splatting seems fine.

[lcw]

I agree it would be nice to avoid generated functions. I am not sure why this has to be generated but there is definitely something wrong with what I did for foreachfield as the test suite doesn't pass. Maybe that is causing issues. If you get a chance, could you take a peak at what I did and see if you can spot something astray?

[vchuravy]

Looks good from my end

[piever]

This introduces a lot of code complexity, while the original implementation is just a few lines of code: I'm not completely sure I understand conceptually what this refactor is trying to do. Is there an intuitive explanation as to way the original foreachfield is not suitable for CUDAnative while this implementation works?

[lcw]

This introduces a lot of code complexity, while the original implementation is just a few lines of code: I'm not completely sure I understand conceptually what this refactor is trying to do. Is there an intuitive explanation as to way the original `foreachfield` is not suitable for CUDAnative while this implementation works?

This is trying to identify all of the fields that have arrays associated with them at compile time. This is avoiding the current run time recursion that is causing dynamic function calls and thus the code not compile on the GPU.

[lcw]

Closing this in favor of #114.