Add a script to supervise template fitting in pycbc live #4813
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maxtrevor
reviewed
Jul 15, 2024
maxtrevor
reviewed
Jul 15, 2024
1 task
16668e3 to
9e55544
Compare
|
Big changes in the last batch of commits: moving to use one supervision script is the main one. Fixed a couple of bugs now that we are dealing with fit files which may not contain triggers from the past, which may not have information for templates which triggered today. I think there isn't too much code duplication and have tried to log / comment / make things obvious at each step |
…e 'safe' ranges should be for the various different stages
| if found_files == 0: | ||
| raise RuntimeError("No files found") | ||
|
|
||
| if missed_files == 10: |
There was a problem hiding this comment.
If not too difficult, could we rather take this value from the config file? If you do, make sure you update the comment and warning message too.
There was a problem hiding this comment.
I have done, but I've made it so that if it isnt given, then 10 is used as a default, as I think that is fairly sensible
maxtrevor
approved these changes
Jul 23, 2024
prayush
pushed a commit
to prayush/pycbc
that referenced
this pull request
Nov 21, 2024
* Move code which will be shared into a module * getting more things ready for the trigger fits / dq supervision * start work on fit_by, fit_over and DQ trigger supervision * some work - bring in latest trigger file finding * updates to use latest trigger file finding * pass cuts through to the collation script * symlink fit_over_multiparam output to variable file * remove unused imports, argument ordering, plot titles * CC * Fix docstring, formatting * Move the trigger collatiojn/fitting superviosor to use configparsert * start moving significance fit supervision to configparser * Start puttinf template fits into supervision code shared with significance fits * Deal properly with whene there are no trigers, so we can't calculate the median_sigma * Remove no-longer-valid efficiency savings * Some minor fixes for when sending mail from supervision * Mve to a single supervision file for both trigger and significance fits * Remove FIXMEs * Use check=True with subprocess.run * Remove function checking fit coefficients - it is now unclear what the 'safe' ranges should be for the various different stages * Fix minor bug * typo * Some minor fixes and TDC comments * CC
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding a script, similar to the one for significance fits, to collate triggers and run fit_by_template and fit_over_multiparam on them
Standard information about the request
This is a new feature
This change affects the live search
This change changes scientific output
This change: follows style guidelines (See e.g. PEP8), has been proposed using the contribution guidelines
Motivation
This is a code so that the ranking statistics which use trigger fit files can be used in the pycbc live search. The triggers will be collected, and fit_over_multiparam files generated using a rolling window of daily fit_by_template
Contents
(Didn't realise I'd missed this originally - added in an edit)
A general overview of the supervision script is as follows:
A lot of this is similar to the significance fits code, so I could go through and remove duplicated effort, but it works for now
Links to any issues or associated PRs
#4527, #4670, #4689, #4695, #4697
Testing performed
Run daily (manually) so that plots are produced - see https://ldas-jobs.ligo.caltech.edu/~gareth.cabourndavies/pycbclive/supervise_fit_coeff_dq_files/testing/output_plots/2024/07/ for latest plots