Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

switch to follow_redirects on httpx.get call in CloudSQL provider #20239

Merged
merged 3 commits into from
Dec 28, 2021

Conversation

philipherrmann
Copy link
Contributor

Also:

  • set min httpx version to 0.21.1 to match follow_redirects API
  • fix weakness in _get_bool which prevented to explicitly set parameters to 'false' in extras

Trying to solve #20088 (comment)


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Dec 12, 2021
@boring-cyborg
Copy link

boring-cyborg bot commented Dec 12, 2021

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@philipherrmann philipherrmann marked this pull request as ready for review December 14, 2021 08:58
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked a bit closer and thought about consequences. I really think we shoudl do it in backwards-compatible way and run a a conditional method. We would like to install new google provider with both - older and newer installations where httpx is potentially limited by Airlfow installation (and especially airflow constraints).

I think the right solution will be to sense with 'inspect signature' https://docs.python.org/3/library/inspect.html#introspecting-callables-with-the-signature-object and use the right parameter. Unfortunately httpx developer did not provide us with backwards-compatibility path, so we have to do it ourselves I am afraid.

@potiuk
Copy link
Member

potiuk commented Dec 27, 2021

ANy chance to get it thsi way @philipherrmann ?

@philipherrmann
Copy link
Contributor Author

Sure. I can't oversee the consequences nearly as good, so I would just follow your suggestion. Sensing is a nice idea, but just try/except is more "pythonic" (EAFP), isn't it? I would go for that later.

@potiuk
Copy link
Member

potiuk commented Dec 27, 2021

Sure. I can't oversee the consequences nearly as good, so I would just follow your suggestion. Sensing is a nice idea, but just try/except is more "pythonic" (EAFP), isn't it? I would go for that later.

Both would work, I guess, but I think checking for attr is more explicit IMHO and better expresses the intent. Also same exception can be thrown if you happen to pass another argument which is unexpected which might hide another error/typo

Also:
- set min httpx version to 0.21.1 to match follow_redirects API
- fix weakness in _get_bool which prevented to explicitly set parameters to 'false' in extras
Comment on lines +778 to 780
if val == 'False' or val is False:
return False
return True
Copy link
Member

@potiuk potiuk Dec 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if val == 'False' or val is False:
return False
return True
return val not in ("False", False)

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit with simpler function.

@github-actions
Copy link

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the okay to merge It's ok to merge this PR as it does not require more tests label Dec 28, 2021
@potiuk potiuk merged commit bfd6d45 into apache:main Dec 28, 2021
@boring-cyborg
Copy link

boring-cyborg bot commented Dec 28, 2021

Awesome work, congrats on your first merged pull request!

@potiuk potiuk added this to the Airflow 2.2.4 milestone Jan 22, 2022
@potiuk potiuk added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jan 22, 2022
potiuk pushed a commit that referenced this pull request Jan 22, 2022
…0239)

* switch to follow_redirects on httpx.get call in CloudSQL provider
* sense for parameter as suggested in review

(cherry picked from commit bfd6d45)
jedcunningham pushed a commit that referenced this pull request Jan 27, 2022
…0239)

* switch to follow_redirects on httpx.get call in CloudSQL provider
* sense for parameter as suggested in review

(cherry picked from commit bfd6d45)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) okay to merge It's ok to merge this PR as it does not require more tests provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants