Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up Monitor #71

Merged
merged 67 commits into from
Sep 1, 2022
Merged
Changes from 1 commit
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
57ee81c
Really it's a gmail sender
robertdstein Jul 31, 2022
699ac10
Add base watchdog
robertdstein Jul 31, 2022
e5f9d26
Actually send summary by email
robertdstein Jul 31, 2022
acd8f96
Default to last night
robertdstein Jul 31, 2022
96aa6ce
Add option for email attachment
robertdstein Aug 2, 2022
382c241
Log to file
robertdstein Aug 2, 2022
8fb476b
Fix typo in summary
robertdstein Aug 2, 2022
903778d
Go for local time, giving symmetry with log
robertdstein Aug 2, 2022
23fd656
Watchdog logs with time
robertdstein Aug 2, 2022
9a78d66
Set log in watchdog
robertdstein Aug 2, 2022
2d89667
Change email defaults
robertdstein Aug 2, 2022
49029a7
Add watchdog fields
robertdstein Aug 2, 2022
4fc967d
Option for watchdog from main
robertdstein Aug 2, 2022
0b4e1e7
Avoid watchdog name conflict with more... colourful... 'Scrutineer'
robertdstein Aug 2, 2022
6c85831
Remove concept of self.processors entirely, clearing path to multi-pr…
robertdstein Aug 2, 2022
41be8ae
Add monitor, redo cache/hash so cache names are based on hashes of th…
robertdstein Aug 3, 2022
5e83272
Require load_raw_image again for pipeline
robertdstein Aug 3, 2022
7e637ab
Catch duplicate uids for postgres
robertdstein Aug 4, 2022
d008301
Catch duplicate uids for postgres
robertdstein Aug 4, 2022
2034767
Deverboseify postgres logging
robertdstein Aug 4, 2022
c834437
No output to terminal
robertdstein Aug 4, 2022
caca005
Move ImageNotFoundError.
robertdstein Aug 4, 2022
3470db3
Raise ImageNotFoundError for batch
robertdstein Aug 4, 2022
ad008ba
Properly filter out summer biases, flats (and darks, apparently)
robertdstein Aug 4, 2022
9ec33f5
Catch all errors
robertdstein Aug 4, 2022
0d50076
Require watchdog:
robertdstein Aug 4, 2022
e4f5c68
Add CalHunter
robertdstein Aug 4, 2022
af88cc5
Use CalHunter for summer
robertdstein Aug 4, 2022
89fa033
Move cal requirement to cal hunter
robertdstein Aug 4, 2022
161e1ee
Update import
robertdstein Aug 4, 2022
62d995d
Wrapper out cal finding function for recycling in Monitor
robertdstein Aug 4, 2022
1339c42
add option for default calibrations, used by monitor
robertdstein Aug 4, 2022
7eb59c5
actually use load_image for calhunter
robertdstein Aug 5, 2022
8677598
New parsing error
robertdstein Aug 5, 2022
1931a23
Handle directory creation, fix errorstack instantiation
robertdstein Aug 5, 2022
45fff72
Skip but log non-critical errors
robertdstein Aug 5, 2022
217e199
Deactivate y raw input
robertdstein Aug 29, 2022
51218ae
More check for weird filters
robertdstein Aug 30, 2022
c5135d2
Integrate emails with working monitor
robertdstein Aug 30, 2022
574d185
Less hardcoded cal requirements
robertdstein Aug 30, 2022
31b7dab
Can't send email after monitor stops
robertdstein Aug 30, 2022
2426704
u can do it via main...
robertdstein Aug 30, 2022
3fe9a7d
Properly parse all options for monitor
robertdstein Aug 30, 2022
68ae8e2
Write full error stack to separate file
robertdstein Aug 30, 2022
4bebea3
Count images processed and failed
robertdstein Aug 30, 2022
126e9a1
Recycle standard reduction for realtime, fix #111
robertdstein Aug 30, 2022
13a8027
Add extra line at end of email
robertdstein Aug 30, 2022
1043318
Use integers to count errors, not floats
robertdstein Aug 30, 2022
b7637de
Write error file even for no errors
robertdstein Aug 30, 2022
c00d9a1
Don't reinvent wheel on error file writing
robertdstein Aug 30, 2022
7163bbd
Default to last night for offline, tonight for monitor
robertdstein Aug 30, 2022
87b4b7f
Pep-compliant removal of raw 'Except:' clause
robertdstein Aug 30, 2022
c03c733
Remove redundant file
robertdstein Aug 30, 2022
e408095
Spinoff load_raw_summer_image
robertdstein Aug 30, 2022
fa120a6
Spinoff load_raw_summer_image
robertdstein Aug 30, 2022
359ece6
Spin off generators to different script
robertdstein Aug 30, 2022
aba5eb5
Spin off generators to different script
robertdstein Aug 30, 2022
a445e10
Remove duplicated files
robertdstein Aug 31, 2022
33bc60a
Shared psfex config
robertdstein Aug 31, 2022
22efbaa
De-hardcode imsub test
robertdstein Aug 31, 2022
0d8f413
Don't save test reduction images
robertdstein Aug 31, 2022
3397508
Reorgainse pipeline into blocks
robertdstein Aug 31, 2022
37286eb
Split up imsub into three pieces
robertdstein Aug 31, 2022
d1b761e
Reorganise to put processor blocks in blocks, and recycle for tests
robertdstein Aug 31, 2022
2813b09
Add constants.py
robertdstein Aug 31, 2022
ee99bff
Fix summer test
robertdstein Aug 31, 2022
a98cab5
Summer (#115)
robertdstein Sep 1, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Default to last night for offline, tonight for monitor
  • Loading branch information
robertdstein committed Aug 31, 2022
commit 7163bbd676cae6511a5f3359fbef2c7a65321030
15 changes: 11 additions & 4 deletions winterdrp/__main__.py
Original file line number Diff line number Diff line change
@@ -16,11 +16,10 @@
)

ln = Time.now() - 1. * u.day
last_night = str(ln).split(" ")[0].replace("-", "")
parser.add_argument(
"-n",
"--night",
default=last_night,
default=None,
help="Sub-directory to use in the data directory"
)
parser.add_argument(
@@ -102,9 +101,13 @@
else:
email_recipients = None

night = args.night
if night is None:
night = str(Time.now()).split(" ")[0].replace("-", "")

monitor = Monitor(
pipeline=args.pipeline,
night=args.night,
night=night,
realtime_configurations=["realtime"],
log_level=args.level,
max_wait_hours=args.maxwaithours,
@@ -130,10 +133,14 @@
log.addHandler(handler)
log.setLevel(args.level)

night = args.night
if night is None:
night = str(ln).split(" ")[0].replace("-", "")

pipe = get_pipeline(
args.pipeline,
selected_configurations=args.config,
night=args.night,
night=night,
)

pipe.reduce_images([[[], []]], catch_all_errors=True)