Issue 63: Implementing SBC

[SamuelBrand1]

This is an ongoing PR for implementing SBC.

• plots.
• prior predictive sampling over a range.
• Posterior sample for each prior predictive.
• Unit tests.
• Example usage.

This PR is ready for review now. It implements SBC methods very closely modelled on https://github.com/arviz-devs/simulation_based_calibration but for numpyro models.

For more extensive explanation see the committed notebook which covers the theoretical basis for SBC as well as an example implementation.

NB: the problem with pre-commit which #67 was aimed at is obviously not resolved.

This PR closes #63

[AFg6K7h4fhy2]

Putting these images for others to see, if interested:

Histogram plot of SBC results

ECDF/PIT plot of SBC results

[AFg6K7h4fhy2]

• Thank you very much for this PR @SamuelBrand1 !
• I have left some initial comments.
• I would like to critically go through some more of the code in sbc_model_checking.qmd and the tests.
• Most of the comments are lower priority but nice to have.
• Rank-uniformity checks + variables constitute the only moderately impactful suggestion, in my mind.
• The notebook is quite nice (as mentioned in private communication).
• I will try to respond promptly to an return remarks you have to mine.
• Some of my comments can be resolved after reading, others can be resolved once you've stated your preference and you believe I've read the comment.

[dylanhmorris]

A few minor comments/questions, but this is looking very good! Thanks, @SamuelBrand1!

[SamuelBrand1]

A few minor comments/questions, but this is looking very good! Thanks, @SamuelBrand1!

Thanks!

All of your comments are leftovers from me grabbing a pymc/bambi implementation https://github.com/arviz-devs/simuk . I've put an issue up over there arviz-devs/simuk#19 so in the future we might just be importing their package but I hope this implementation works for now.

Note that: a quick implementation in julia I did https://github.com/SamuelBrand1/SBC returns the p-value of the ChiSq uniformity test on the rank statistics. This could be a handy issue/next-step to go along with the viz.

[SamuelBrand1]

Following @dylanhmorris 's comments:

• Refactored the inner loop of the SBC. This is nicer, I've left it as a set of method calls because I find that easier to read but YMMV. Having a method to gather rank statistics makes this much cleaner to read.
• Added an inspection_mode boolean to the SBC class. If True then running SBC also saves the prior draws/preds and the idata objects of the fit (e.g. MCMC chains). The idea is that you can use this for inspection on a small number of SBC trials to get a sense of whats going on.
• @dylanhmorris spotted a hand rolled discrete uniform was not needed so I've removed and added the numpyro.distributions one.

The new method for the SBC class has a new unit test.

[AFg6K7h4fhy2]

• Thank you @dylanhmorris for your comments on this PR!
• Thank you again @SamuelBrand1 for this work!
• The tests and notebook both run successfully for me locally.
• There is nothing else within scope that I believe needs to be added for this PR to be merged.
• I approve this PR.