Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue 137: unify argument patterns #138

Merged
merged 6 commits into from
Nov 19, 2024
Merged

Issue 137: unify argument patterns #138

merged 6 commits into from
Nov 19, 2024

Conversation

SamuelBrand1
Copy link
Collaborator

@SamuelBrand1 SamuelBrand1 commented Nov 19, 2024 •
edited
Loading

This draft PR ultimately aims to close #137.

Lists of scripts

These are the scripts to bring into common pattern as described in #137. In this case "script" means a file that is designed to have its code executed with a combination of positional and named args.

I've restricted to scripts where there is an associated bash shell script. Other scripts have patterns like input and output, or seem to have positional disease arguments, and these are unchanged in this PR.

Python scripts

  • fit_model.py
  • generate_predictive.py

R scripts

  • postprocess_state_forecast.R
  • score_forecast.R

Bash scripts

  • loop_fit.sh
  • loop_generate_predictive.sh
  • loop_postprocess.sh
  • loop_score.sh

Testing

Where possible I've adapted the experimental data generation from #117 and #131 to check that scripts still run via shell script calls on test data. But only for functions in loop_fit.sh and loop_generate_predictive.sh so far.

@SamuelBrand1 SamuelBrand1 linked an issue Nov 19, 2024 that may be closed by this pull request
Unify argument pattern in /pipelines for scripts aimed at loop inference/postprocessing #137
Closed
@SamuelBrand1
Copy link
Collaborator Author

The positional form for fit_model script was causing an error in its call down to build_model_from_dir because runpy.run_path(prior_path) was failing

AttributeError: 'PosixPath' object has no attribute 'startswith'

An easy fix committed here was to convert to string at this point.

@SamuelBrand1
Copy link
Collaborator Author

Last push gave error

Error: buildx failed with: ERROR: failed to solve: cfaprdbatchcr.azurecr.io/pyrenew-hew-dependencies:137-unify-argument-pattern-in-pipelines: failed to resolve source metadata for cfaprdbatchcr.azurecr.io/pyrenew-hew-dependencies:137-unify-argument-pattern-in-pipelines: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://cfaprdbatchcr.azurecr.io/oauth2/token?scope=repository%3Apyrenew-hew-dependencies%3Apull&service=cfaprdbatchcr.azurecr.io: 401 Unauthorized

This seems unconnected to last commit (which fixed an error).

@SamuelBrand1
Copy link
Collaborator Author

Last push gave error

Error: buildx failed with: ERROR: failed to solve: cfaprdbatchcr.azurecr.io/pyrenew-hew-dependencies:137-unify-argument-pattern-in-pipelines: failed to resolve source metadata for cfaprdbatchcr.azurecr.io/pyrenew-hew-dependencies:137-unify-argument-pattern-in-pipelines: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to cfaprdbatchcr.azurecr.io/oauth2/token?scope=repository%3Apyrenew-hew-dependencies%3Apull&service=cfaprdbatchcr.azurecr.io: 401 Unauthorized

This seems unconnected to last commit (which fixed an error).

After some inconsequential changes to trigger CI this resolved.

@gvegayon should we expect stochastic failure in the container build CI?

@SamuelBrand1 SamuelBrand1 marked this pull request as ready for review November 19, 2024 14:13
@SamuelBrand1 SamuelBrand1 mentioned this pull request Nov 19, 2024
Closed
@SamuelBrand1
Copy link
Collaborator Author

This should be sequenced ahead of #117 now.

damonbayer
damonbayer approved these changes Nov 19, 2024
View reviewed changes
Copy link
Collaborator

@damonbayer damonbayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @SamuelBrand1!

@damonbayer damonbayer mentioned this pull request Nov 19, 2024
Closed
@dylanhmorris dylanhmorris merged commit ed43ad5 into main Nov 19, 2024
3 checks passed
@dylanhmorris dylanhmorris deleted the 137-unify-argument-pattern-in-pipelines branch November 19, 2024 15:44
@dylanhmorris dylanhmorris mentioned this pull request Nov 19, 2024
Open
damonbayer pushed a commit that referenced this pull request Nov 19, 2024
@SamuelBrand1 @damonbayer
Issue 137: unify argument patterns (#138)
915284e
damonbayer added a commit that referenced this pull request Nov 19, 2024
@damonbayer @SamuelBrand1 @dylanhmorris
merge main into model_runs_2 (#146)
eae81c4 
* Issue 137: unify argument patterns (#138)

* add check to all subprocess commands (#143)

* Organize helper functions / utilities (#141)

---------

Co-authored-by: Samuel Brand <48288458+SamuelBrand1@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
damonbayer added a commit that referenced this pull request Nov 20, 2024
@damonbayer @SamuelBrand1 @dylanhmorris
Add model_runs directory (#136)
ba9e617 
* add model_runs as additional hierarchy

* fix default tag

* fix path in postprocess_state_forecast.R

* remove unused base_dir definition

* correct_path in timeseries_forecasts

* Issue 137: unify argument patterns (#138)

* add check to all subprocess commands (#143)

* Organize helper functions / utilities (#141)

* provide missing namespace

* merge main into model_runs_2 (#146)

* Issue 137: unify argument patterns (#138)

* add check to all subprocess commands (#143)

* Organize helper functions / utilities (#141)

---------

Co-authored-by: Samuel Brand <48288458+SamuelBrand1@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>

* load required packages

* more namespace fixes

* use hewr functionality

---------

Co-authored-by: Samuel Brand <48288458+SamuelBrand1@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
damonbayer added a commit that referenced this pull request Nov 20, 2024
@damonbayer @SamuelBrand1 @dylanhmorris
Add model_runs directory (3rd attempt) (#152)
b90dd9f 
* add model_runs as additional hierarchy

* fix default tag

* fix path in postprocess_state_forecast.R

* remove unused base_dir definition

* correct_path in timeseries_forecasts

* Issue 137: unify argument patterns (#138)

* add check to all subprocess commands (#143)

* Organize helper functions / utilities (#141)

* provide missing namespace

* merge main into model_runs_2 (#146)

* Issue 137: unify argument patterns (#138)

* add check to all subprocess commands (#143)

* Organize helper functions / utilities (#141)

---------

Co-authored-by: Samuel Brand <48288458+SamuelBrand1@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>

* load required packages

* more namespace fixes

* use hewr functionality

* first collate plots changes

* use parent dir for saving by default

* put figures in figures dir

* update score tables collation

---------

Co-authored-by: Samuel Brand <48288458+SamuelBrand1@users.noreply.github.com>
Co-authored-by: Dylan H. Morris <dylanhmorris@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dylanhmorris dylanhmorris dylanhmorris left review comments

@damonbayer damonbayer damonbayer approved these changes

@sbidari sbidari Awaiting requested review from sbidari sbidari is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Unify argument pattern in /pipelines for scripts aimed at loop inference/postprocessing
3 participants
@SamuelBrand1 @dylanhmorris @damonbayer