need way to convert jl_value_t* back to Any for callbacks

[stevengj]

ccall allows you to pass Any for an argument type, which causes the raw jl_value_t* to the boxed argument to be passed through to the C routine. This is very useful for calling jl_ functions and other C functions that are aware of the Julia API.

It would also be very useful for passing true closures (issue #1096) and other Julia objects through to callbacks (many C callback APIs allow one to pass an arbitrary void* through to the callback). However, there needs to be a way to convert the jl_value_t* back to Any in Julia.

e.g. an unsafe_pointer_to_any(p) function, or unbox(Any, p::Ptr{Void}), or something like that.

It would be trivial to code this by adding a C function like:

  JL_CALLABLE(jl_identity);
  DLLEXPORT jl_value_t *jl_identity(jl_value_t *o) {
     return o;
  }

to jlapi.c, and then doing

unsafe_pointer_to_any(p) = ccall(:jl_identity, Any, (Ptr{Void},), p)

but this seems like a slightly silly way to implement it. However, I'm not sure where to put in a lower-level hook to do the conversion (or if this is already possible).

[stevengj]

@vtjnash, Doesn't seem to work quite properly. e.g.:

julia> p = ccall(:jl_call1, Ptr{Any}, (Any,Any), x -> x+1, 314158)       
Ptr{Any} @0x0000000002e37110
julia> unsafe_ref(p)
Int64

when I would have expected 314159, which is what ccall(:jl_call1, Any, (Any,Any), x -> x+1, 314158) gives.

[vtjnash]

hmm, true. I treated Ptr{Any} like an array of pointers to Any. but it does seem more correct to treat it as a literal pointer to Any and have Ptr{Ptr{Any}} give the (new) current behavior. i'll push a fix for this in a few minutes

[JeffBezanson]

See my comment in e8d6ba3.
Ptr{Any} corresponds to jl_value_t**, Any corresponds to jl_value_t*.

[vtjnash]

I'm still not convinced that patch was wrong. Although when you add support to cfunction for Any, the importance of that point should be sufficiently diminished to consider this resolved.

[JeffBezanson]

It seems obvious to me that cfunction should support Any the same way ccall does, and that that is the real fix.

[stevengj]

@JeffBezanson, allowing Any in cfunction arguments is great, but it only solves the problem if the jl_value_t* is passed as an argument to the callback. This isn't always the case.

For example, in Python a jl_value_t* might be passed back not as an argument but boxed inside some ctypes variable or another object. For these kinds of applications, it would still be good to be able to convert a raw jl_value_t* back into Any from Julia, without having to pass through a C function first.

I can understand that it seems inconsistent to call a jl_value_t* a Ptr{Any} for unsafe_unref; in this case maybe a different function name like unsafe_unref_any or unsafe_pointer_to_any (or whatever) might be called for.

[vtjnash]

@stevengj If it is boxed, does that imply you have the necessary indirection container to just use unsafe_ref() anyways?

@JeffBezanson, you specified that unsafe_ref for the Any type was initially an error so that it could be defined in the most useful way later. An Array of Any's is boxed to contain pointers. The special case in base is just to make it more convenient to users to express them as values, but still flexible on the backend for alternative behaviors. I think this may be a better solution: https://github.com/JuliaLang/julia/tree/jn/ref_any

[JeffBezanson]

Ok, then what is needed is a way to convert Ptr{Void} to an object reference.

There is no way the Ptr{Ptr{Any}} thing is correct. When you have a "typed location" (such as an array element) whose type is Any (or any abstract type), what you find there is a jl_value_t*. There is no such thing as an "Any object" whose representation is jl_value_t. That struct is just a placeholder type, it is not the representation of anything.

It seems to me pointerref should perform an indirection (load). Having a special case where it does not is really strange. That can't be right either.

As I said, don't read too much into the "special case" of Any in ccall. The only thing special about that is that we allow it, instead of giving an error. But it still does the right thing. That is not the same as Any having some randomly different behavior like not actually doing a load in an operation that's supposed to do a load.