Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add simulation based calibration tooling #79

Open
SamuelBrand1 opened this issue Feb 6, 2025 · 0 comments
Open

Add simulation based calibration tooling #79

SamuelBrand1 opened this issue Feb 6, 2025 · 0 comments

Comments

@SamuelBrand1
Copy link

Goal

The goal here is to add simulation-based calibration model checking to the forecasttools toolbox. This then provides a set of checks for the self-consistency/correctness of a Bayesian model + a posterior sampling method. This is aimed as acting as a unit test for models.

Proposed solution

My proposal is to smooth any issue with using SBC to work with tidybayes (this is likely already available) objects that are structured over multiple sbc trials. In reality, most models we will try outputing an Arviz object so the idea will be to leverage Arviz to tidybayes conversions we already have in place.

At least as a first pass, the user will be responsible for generating the sbc trials.

As a diagram the expected eventual workflow is as follows:

graph TD
    Generate_data[Generate datasets from model] -->|inference| posterior_draws[Posterior draws for each dataset]
    posterior_draws -->|Arviz| make_data_set[Make InferenceData object over datasets]
    make_data_set -->|Serialise| serialise[Serialize to netCDF]
    make_data_set -->|direct convert| convert[Convert to tidybayes]
    serialise -->|deserialise and convert| convert
    convert -->|SBC| sbc[Run SBC checks and return report]
Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant