Priors/Hessian method corrections

[robbietuk]

This PR was going to simply add the accumulate_Hessian_times_input method to the RDP but while implementing it it was discovered that the implementations in the Quadratic and LogCosh priors were also incorrect. Additionally improves priors testing (#153).

This PR will attempt to do the following for each of the aforementioned priors:

• Correct implementation issues with accumulate_Hessian_times_input to compute the correct Hessian vector product
• Clearly document what is being done
• Add a variable is_convex = false to generalised prior. Set is_convex = true for all convex priors.
• Add test for convexity, much like Add a Hessian test to PoissonLogLikelihoodWithLinearModelForMeanAndProjData (and fix Hessian) #902 by testing x^T Hx >0
• Add compute_Hessian() to all convex methods
• Add test for convex priors by checking grad(x) - grad(x+eps) ~= H(x), where eps is a small perturbation.
• Add diagonal_second_derivative and off_diagonal_second_derivative methods for use in all Hessian methods

Relative Difference Prior accumulate_Hessian_times_input

• RDP accumulate_Hessian_times_input
• Added RDP Hessian documentation
• Add div by 0 safety
• Add epsilon to Hessian function

[robbietuk]

These latest commits do not fix a possible mistake with the RDP's accumulate_Hessian_times_input but it does indicate convexity.

[robbietuk]

Implemented compute_Hessian for QP, logcosh, and RDP all the same style. This is complete code duplication for each of these methods in preperation for #694 changes.

I have rewritten compute_Hessian for the QP and Logcosh. Their implementations were not incorrect but required the assumption that d^2/dx^2 = - d^2/dxdy. The RDP does not abide by this constraint. I have also added the diagonal_second_derivative and off-diagonal_second_derivative methods (name may be changed) to each of the classes.

Added accumulate_Hessian_times_input to RDP and modernised the QP and logcosh priors to use the same methodology (code duplication again in prep for changes #694).

Added _is_convex and get _is_convex() to Generalised Prior. Default is false and each derived prior sets to true (if is the case)

• Potential issue is the PLS prior (which is still broken PLS gradient has edge artefacts #705 and disabled in testing)

Regarding testing:

• Added Verbosity::set(0) calls to suppress compute_gradient info calls. When make test is run, this stuff is already suppressed but when executing test_priors the terminal output is flooded with gradient info calls without these verbosity changes.
• Added test_gradient , test_Hessian_convexity,test_Hessian_convexity_configuration and test_Hessian_against_numerical methods. Hessian tests are only conducted if _is_convex=true

[robbietuk]

Currently test_Hessian_against_numerical will fail for both RDP checks.

This is because of an inconsistancy between off-diagonal elements in the numerical-Hessian and the Hessian. The RDP is very sensitivie if eps (the perturbation) is too large relative to the comparison voxel values. Comparing the numerical-Hessian and the Hessian, the error is usually small (~10%), but can be large (>~60%), and occurs when both voxels values (x_j,x_k) are small w.r.t. eps, ergo effectively a large change in the numerical gradient.

I am fairly convinced that the implementation of the RDP's Hessian is correct and it is the test that is the issue.

Edit: Interesting that one of the travis jobs actually passed on test_priors. I guess that means that the numerical Hessian test for the RDP is only failing in certain input/target configurations. However, this is no stable. A constant eps in the test is not suitable for this prior.

[robbietuk]

Create bools to toggle the use of various tests for different penalties. RDP with RDP_eps = 0 will likely allways fail the numerical Hessian test.

[robbietuk]

Ready for review. There is overlap with #902.

[KrisThielemans]

some comments. I didn't check the tests yet

[robbietuk]

some comments. I didn't check the tests yet

I have addressed all pertinent suggestions

[robbietuk]

Future progress on this PR should include an intermediate class for the impacted priors. This class should handle the for loops for each of the Hessian methods and create the virtual functions for the derivative_20 and derivative_11.

[robbietuk]

I think this is ready to go, assuming the tests pass.

[KrisThielemans]

looks good. One tiny comment. Also, can you add something to release_5.1.htm? You probably will have to merge master into here first to get that file.