Provide diagnostics mode to pinpoint source of NaNs

[davidanthoff]

This continues a discussion started in #318.

This model now solves, but we now know that there are iterations where it returns NaNs. I would like to be able to pinpoint the exact expression where the NaN is introduced, so that I can double check whether this is due to an error in my model, or whether I'm ok with this.

This is obviously not trivial at all, because the NaN could also be generated in a derivative, so in those cases ideally the diagnostic would say something like "the NaN was introduced in part x of the gradient of expression y" or something like that.

[mlubin]

Does GAMS (or AMPL) provide diagnostics like this?

[davidanthoff]

I don't know

[tkelman]

Depending on the solver, this is the kind of thing where I would just turn up the logging level, output to a file and start digging through it. Hard to handle in a solver-independent way, unless you want to start getting into adding optional diagnostic features to generated callback functions. Somewhat related to jump-dev/NLopt.jl#16

[davidanthoff]

I think at some point it would be good to have diagnostic features in the generated callback functions, so that is really the point of this issue. Right now it is almost impossible to debug those, because one can also not just insert println or anything like that. I'm not even sure what the right kind of diagnostics would be, I just feel it is a topic we might think about. Certainly not high priority.

[tkelman]

Julia really really badly needs a debugger. That much is obvious. I don't know if there's any good solution to make that happen any faster aside from locking Keno in a room.

At the JuMP level of solver independence, what diagnostics would even make sense beyond "print the callback function inputs" and "print the callback function outputs"? Any decent solver, assuming it was designed with optimization modeling languages in mind as most of them are, can already do that for you.

[davidanthoff]

I guess my main trouble is that even if I knew which input combination of values caused a NaN, it would still be a fair amount of work to trace that back to the precise expression that caused the NaN. I might have a complicated expression as my objective or constraint, and then I would have to code that expression as normal julia code, run that under a debugger (once it is there) and then figure out which exact operation causes the NaN. I'm sure it would work, but I know it would be painful. If there was some way for JuMP to just automatically tell me "this particular operation in constraint X returned NaN for these precise input parameters" it would just be super handy and safe an awful lot of time.

[tkelman]

Maybe something to the effect of -

1. use solver logging to identify nan's and save vector of inputs that caused them
2. request JuMP/ReverseDiffSparse to generate a debug version of the callback that doesn't use NaNMath, and instead causes errors
3. execute that callback interactively from Julia with a loaded copy of the same input vector, hope you get a backtrace that helps?

On part 3, I wonder if this is something where JuMP could insert extra compiler metadata into the debug version of the callback to help tie backtraces back to the corresponding @addNLConstraint expression? I don't know how much of the compiler metadata stuff is fully hooked up yet in base Julia, but this could make an interesting use case for it.

[davidanthoff]

Yes, Tony's idea would be perfect.

[mlubin]

This discussion is very old and I don't expect any progress on providing NaN diagnostics unless someone is willing to do the work. I'm tempted to close.