Set up Monitor #71
Merged
Set up Monitor #71
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
852c6c7 to
a162665
Compare
da663e8 to
a522503
Compare
This was
linked to
issues
Aug 30, 2022
Closed
…e cal images only
* Really it's a gmail sender * Add base watchdog * Actually send summary by email * Default to last night * Add option for email attachment * Log to file * Fix typo in summary * Go for local time, giving symmetry with log * Watchdog logs with time * Set log in watchdog * Change email defaults * Add watchdog fields * Option for watchdog from main * Avoid watchdog name conflict with more... colourful... 'Scrutineer' * Remove concept of self.processors entirely, clearing path to multi-processing * Add monitor, redo cache/hash so cache names are based on hashes of the cal images only * Require load_raw_image again for pipeline * Catch duplicate uids for postgres * Catch duplicate uids for postgres * Deverboseify postgres logging * No output to terminal * Move ImageNotFoundError. * Raise ImageNotFoundError for batch * Properly filter out summer biases, flats (and darks, apparently) * Catch all errors * Require watchdog: * Add CalHunter * Use CalHunter for summer * Move cal requirement to cal hunter * Update import * Wrapper out cal finding function for recycling in Monitor * add option for default calibrations, used by monitor * actually use load_image for calhunter * New parsing error * Handle directory creation, fix errorstack instantiation * Skip but log non-critical errors * Deactivate y raw input * More check for weird filters * Integrate emails with working monitor * Less hardcoded cal requirements * Can't send email after monitor stops * u can do it via main... * Properly parse all options for monitor * Write full error stack to separate file * Count images processed and failed * Recycle standard reduction for realtime, fix #111 * Add extra line at end of email * Use integers to count errors, not floats * Write error file even for no errors * Don't reinvent wheel on error file writing * Default to last night for offline, tonight for monitor * Pep-compliant removal of raw 'Except:' clause * Remove redundant file * Spinoff load_raw_summer_image * Spinoff load_raw_summer_image * Spin off generators to different script * Spin off generators to different script * Remove duplicated files * Shared psfex config * De-hardcode imsub test * Don't save test reduction images * Reorgainse pipeline into blocks * Split up imsub into three pieces * Reorganise to put processor blocks in blocks, and recycle for tests * Add constants.py * Fix summer test * Option to specify raw sub dir * Use local time for date * Be more clever with subdirectories for cal hunter * Add in export to raw database for summer * Continually update error log with each error, rather than only at the end * CSV log into load images block * Check that fits files are correct length, otherwise wait a bit Co-authored-by: virajkaram <karambelkarvraj21197@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Monitor class can act as a watchdog, and run on new images in realtime as they appear. The Monitor will first check for a pre-built list of
CalRequirements, and go sequentially backwards until it has at least one image of each type (e.g a flat in u, g, r and i respectively). It starts with the current night.