Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remove backoff, add manual retry #8328

Merged
merged 4 commits into from
Sep 20, 2022
Merged

remove backoff, add manual retry #8328

merged 4 commits into from
Sep 20, 2022

Conversation

leahecole
Copy link
Collaborator

@leahecole leahecole commented Sep 16, 2022 •
edited
Loading

Description

Fixes #8295

in a chat with @kurtisvg, he pointed out that previous backoff use to look for the subnet not ready error could be causing the aleradyexists. I removed the backoff and instead did a manual try/except for that error that includes resource teardown for the errored out dataproc batch job. I also moved the actual call into the dataproc fixture.

Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.

Checklist

@product-auto-label product-auto-label bot added the samples Issues that are directly related to samples. label Sep 16, 2022
@leahecole leahecole requested a review from bradmiro September 16, 2022 18:14
@leahecole leahecole marked this pull request as ready for review September 16, 2022 18:23
@leahecole leahecole merged commit ff545c2 into main Sep 20, 2022
@leahecole leahecole deleted the fix-8295 branch September 20, 2022 15:31
leahecole added a commit that referenced this pull request Sep 20, 2022
@leahecole @parthea
remove backoff, add manual retry (#8328)
0af547f 
* remove backoff, add manual retry

* fix lint

* remove unused import

Co-authored-by: Anthonios Partheniou <partheniou@google.com>
leahecole added a commit that referenced this pull request Sep 20, 2022
@leahecole
refactor test to match #8328
dba937c
rogerthatdev pushed a commit that referenced this pull request Sep 27, 2022
@leahecole @parthea @rogerthatdev
remove backoff, add manual retry (#8328)
3cdc785 
* remove backoff, add manual retry

* fix lint

* remove unused import

Co-authored-by: Anthonios Partheniou <partheniou@google.com>
leahecole added a commit that referenced this pull request Sep 28, 2022
@leahecole @kaiyang-code
@engelke @m-strzelczyk @dandhlee @davidcavazos @renovate-bot @parthea @averikitsch @mhenc @bradmiro
data analytics tutorial expansion project (#8290)
5310f9e 
* Kaiyang expansion project 2022 (#8224)

* chenged the dag to load ghcn dataset

* data preprocessing done

* modified preprocessing

* dataproc file added

* code runs great

* modifyed code based on Brad, still buggy

* finished modifying, haven't sync wit hDAG

* finished modifying DAG codes

* ready for draft PR

* pass lint

* addressed Brad and Leah's comments

* pass nox lint

* pass nox lint

* Fix: Retry CLI launch if needed (#8221)

* Fix: add region tags

* Fix: region tag typos

* Fix: urlpatterns moved to end

* Fix: typo

* Fix: cli retries to fix flakiness

* Fix: remove duplicate tags

* Fix: use backoff for retries

* Fix: lint import order error

* address Leah's comments about typo and comments

Co-authored-by: Charles Engelke <engelke@google.com>

* run blacken on dag and dataproc code

* WIP: not working test for process job

* working test for expansion dataproc script

* move dataproc expansion files to separate directory

* add readme

* update readme

* run black

* ignore data file

* fix import order

* ignore one line of lint because it's being silly

* add check for Notfound for test

* add requirements files

* add noxfile config

* update try/except

* experiment - fully qualify path

* update filepath

* update path

* try different path

* remove the directory that was causing test problems

* fix typo in header checker

* tell folks to skip cleanup of prereq

* clean up hyperlinks for distance weighting and arithmetic mean

* fix math links again

* remove debug statements

* remove commented out variables

* Update composer/2022_airflow_summit/data_analytics_dag_expansion_test.py

Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com>

* Apply suggestions from code review

* update apache-beam version (#8302)

Bumping the `apache-beam[gcp]` version to (indirectly) bump the `google-cloud-pubsub` version to accept the keyword argument `request` on `create_topic()`

* dataflow: replace job name underscores with hyphens (#8303)

* dataflow: replace job name underscores with hyphens

It looks like Dataflow no longer accepts underscores in the job names. Replacing them with hyphens should work.

* fix test checks

* improve error reporting

* fix test name for exception handling

* chore(deps): update dependency datalab to v1.2.1 (#8309)

* fix: unsanitized output (#8316)

* fix: unsanitized output

* fix: add license to template

* chore(deps): update dependency cryptography to v38 (#8317)

* chore(deps): update dependency cryptography to v38

* lint

Co-authored-by: Anthonios Partheniou <partheniou@google.com>

* Remove region tags to be consistent with other languages (#8322)

* fix lint in conftest (#8324)

* Pin perl version to 5.34.0 as latest doesn't work with the example. (#8319)

Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>

* refactor fixtures

* revert last change

* revert last change

* chore(deps): update dependency tensorflow to v2.7.2 [security] (#8329)

* remove backoff, add manual retry (#8328)

* remove backoff, add manual retry

* fix lint

* remove unused import

Co-authored-by: Anthonios Partheniou <partheniou@google.com>

* refactor test to match #8328

* update most write methods, fix test issue with comparing to exception

* Bmiro kaiyang edit (#8350)

* modified code to more closely adhere to Spark best practices

* remove unnecessary import

* improved explanation of Inverse Distance Weighting

* Apply suggestions from code review

Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>

Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>

* run black on process files

* fix relative import issue

* fixed jvm error (#8360)

* Add UDF type hinting (#8361)

* fixed jvm error

* add type hinting to UDF

* Update composer/2022_airflow_summit/data_analytics_process_expansion.py

* fix comment alignment

* change dataproc region to northamerica-northeast1

* refactor import

* switch other test to also use northamerica-northeast1

Co-authored-by: kaiyang-code <57576013+kaiyang-code@users.noreply.github.com>
Co-authored-by: Charles Engelke <engelke@google.com>
Co-authored-by: Maciej Strzelczyk <strzelczyk@google.com>
Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com>
Co-authored-by: David Cavazos <dcavazos@google.com>
Co-authored-by: WhiteSource Renovate <bot@renovateapp.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
Co-authored-by: Averi Kitsch <akitsch@google.com>
Co-authored-by: mhenc <mhenc@google.com>
Co-authored-by: Brad Miro <bmiro@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bradmiro bradmiro bradmiro approved these changes

@kurtisvg kurtisvg kurtisvg approved these changes

@rachael-ds rachael-ds Awaiting requested review from rachael-ds

@rafalbiegacz rafalbiegacz Awaiting requested review from rafalbiegacz

Assignees

@kurtisvg kurtisvg

Labels
samples Issues that are directly related to samples.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

composer.2022_airflow_summit.data_analytics_process_test: test_process failed
4 participants
@leahecole @bradmiro @kurtisvg @parthea