Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Test statistic specified by string instead of using qtilde kwarg #1231

Merged
merged 20 commits into from
Dec 19, 2020

Conversation

kratsg
Copy link
Contributor

@kratsg kratsg commented Dec 18, 2020

Description

This replaces qtilde=True and qtilde=False with test_stat="qtilde" and test_stat="q" everywhere. At the same time, we keep around qtilde for backwards-compatibility and mark it as deprecated in v0.6.0 . Likely we will remove this in v0.6.1.

This is more work towards getting PR #520 in.

ReadTheDocs build: https://pyhf.readthedocs.io/en/feat-flexibleteststatistic/api.html#inference

Checklist Before Requesting Reviewer

  • Tests are passing
  • "WIP" removed from the title of the pull request
  • Selected an Assignee for the PR to be responsible for the log summary

Before Merging

For the PR Assignees:

  • Summarize commit messages into a comprehensive review of the PR
* Deprecate qtilde in favor of test_stat keyword argument
* Test statistic now specified as a string, instead of relying just on qtilde=True/False

@kratsg kratsg added feat/enhancement New feature or request API Changes the public API refactor A code change that neither fixes a bug nor adds a feature labels Dec 18, 2020
@kratsg kratsg self-assigned this Dec 18, 2020
@codecov
Copy link

codecov bot commented Dec 18, 2020

Codecov Report

Merging #1231 (738f9b6) into master (f4df777) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1231   +/-   ##
=======================================
  Coverage   97.45%   97.45%           
=======================================
  Files          63       63           
  Lines        3688     3700   +12     
  Branches      524      524           
=======================================
+ Hits         3594     3606   +12     
  Misses         55       55           
  Partials       39       39           
Flag Coverage Δ
unittests 97.45% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/pyhf/infer/__init__.py 100.00% <ø> (ø)
src/pyhf/infer/intervals.py 100.00% <ø> (ø)
src/pyhf/cli/infer.py 97.61% <100.00%> (-0.03%) ⬇️
src/pyhf/exceptions/__init__.py 100.00% <100.00%> (ø)
src/pyhf/infer/calculators.py 100.00% <100.00%> (ø)
src/pyhf/infer/utils.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f4df777...738f9b6. Read the comment docs.

@kratsg kratsg force-pushed the feat/flexibleTestStatistic branch from a7bc38f to 8010510 Compare December 18, 2020 21:26
@kratsg kratsg requested a review from lukasheinrich December 18, 2020 21:52
Copy link
Member

@matthewfeickert matthewfeickert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kratsg I really like this PR overall a lot! Thanks for it and nice idea. :) I do have a suggestion on the API though, so let me know your thoughts.

kratsg and others added 2 commits December 18, 2020 21:14
Co-authored-by: Matthew Feickert <matthew.feickert@cern.ch>
Copy link
Member

@matthewfeickert matthewfeickert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also grab the notebooks and README in this PR as well

f$ git grep "qtilde=True"
README.rst:   >>> CLs_obs, CLs_exp = pyhf.infer.hypotest(test_mu, data, model, qtilde=True, return_expected=True)
README.rst:   >>> CLs_obs, CLs_exp = pyhf.infer.hypotest(test_mu, data, model, qtilde=True, return_expected=True)
README.rst:       pyhf.infer.hypotest(test_poi, data, model, qtilde=True, return_expected_set=True)
README.rst:       pyhf.infer.hypotest(test_poi, data, model, qtilde=True, return_expected_set=True)
docs/examples/notebooks/hello-world.ipynb:    "    test_mu, data, model, qtilde=True, return_expected=True\n",
docs/examples/notebooks/hello-world.ipynb:    "    test_mu, data, model, qtilde=True, return_tail_probs=True\n",
docs/examples/notebooks/hello-world.ipynb:    "    test_mu, data, model, qtilde=True, return_expected_set=True\n",
docs/examples/notebooks/toys.ipynb:    "    qtilde=True,\n",
src/pyhf/infer/calculators.py:            ``qtilde=True`` has been deprecated in favor of ``test_stat="qtilde"``.
src/pyhf/infer/calculators.py:            ``qtilde=True`` has been deprecated in favor of ``test_stat="qtilde"``.

@matthewfeickert matthewfeickert added the docs Documentation related label Dec 19, 2020
Copy link
Member

@matthewfeickert matthewfeickert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the beautiful PR @kratsg. I've made my final changes to also bring this to the README and notebooks. Let me know your thoughts but looks good on my side. :shipit:

@matthewfeickert
Copy link
Member

cc @alexander-held RE: a heads up on API shifts in v0.6.0.

@kratsg kratsg merged commit fd7930c into master Dec 19, 2020
@kratsg kratsg deleted the feat/flexibleTestStatistic branch December 19, 2020 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API Changes the public API docs Documentation related feat/enhancement New feature or request refactor A code change that neither fixes a bug nor adds a feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants