Skip to content

Commit

Permalink
Update Pinned CI Packages (#11586)
Browse files Browse the repository at this point in the history
* update pinned CI deps

* fix custom checkers - rename msg_id -> msgid

* pylint updates for text analytics and form recognizer

* pylint updates for azure-identity and azure-keyvault-keys

* disable import-outside-toplevel

* Pylint for azure-core

* Mypy + pylint are friends

* Fix errors for pylint 2.5.2

* disable azure-identity warnings

* Refactor ReceivedMessageBase off such that the AIO version of ReceivedMessage does not violate pylint by overriding sync with async methods.  Now the sync-specific/async-specific versions imply implement their respective settlement methods, with the shared scaffolding in base.

* Temporarily disable pylint checks for AsyncServiceBusSharedKeyCredentialPolicy.
In the mid-term we need to deprecate the Async versions of this policy and credential to unify on a single model in the primary namespace, see issue here: #12533

* Fix deferred settlement tests by reverting the mgmt link settlement await.

* [Storage][Pylint]Fix Pylint

* resolve conflicts

Co-authored-by: Krista Pratico <krpratic@microsoft.com>
Co-authored-by: Charles Lowell <chlowe@microsoft.com>
Co-authored-by: Laurent Mazuel <laurent.mazuel@gmail.com>
Co-authored-by: yijxie <yijun.xie@micrsofot.com>
Co-authored-by: Kieran Brantner-Magee <kibrantn@microsoft.com>
Co-authored-by: xiafu <xiafu@microsoft.com>
  • Loading branch information
7 people authored Jul 14, 2020
1 parent 4bbcf39 commit 0f9c3be
Show file tree
Hide file tree
Showing 61 changed files with 181 additions and 131 deletions.
21 changes: 10 additions & 11 deletions eng/ci_tools.txt
Original file line number Diff line number Diff line change
@@ -1,36 +1,35 @@
# requirements leveraged by ci tools
setuptools==44.1.0; python_version == '2.7'
setuptools==45.1.0; python_version >= '3.5'
setuptools==46.4.0; python_version >= '3.5'
virtualenv==20.0.23
wheel==0.34.2
Jinja2==2.11.1
Jinja2==2.11.2
packaging==20.4
tox==3.14.6
tox==3.15.0
tox-monorepo==0.1.2
twine==1.15.0
twine==1.15.0; python_version == '2.7' or python_version == '3.5'
twine==3.1.1; python_version >= '3.6'
pathlib2==2.3.5
readme-renderer[md]==25.0
doc-warden==0.7.1
# we pin coverage to 4.5.4 because there is an bug with `pytest-cov`. the generated coverage files cannot be `coverage combine`ed
coverage==4.5.4
codecov==2.0.22
beautifulsoup4==4.8.2
codecov==2.1.0
beautifulsoup4==4.9.1
pkginfo==1.5.0.1

# locking packages defined as deps from azure-sdk-tools or azure-devtools
pytoml==0.1.21
pyOpenSSL==19.1.0
json-delta==2.0
ConfigArgParse==1.1
ConfigArgParse==1.2.3
six==1.14.0
vcrpy==3.0.0
pyyaml==5.3.1
pytest==5.4.1; python_version >= '3.5'
pytest==5.4.2; python_version >= '3.5'
pytest==4.6.9; python_version == '2.7'
pytest-cov==2.8.1

# local dev packages
./tools/azure-devtools
./tools/azure-sdk-tools



14 changes: 7 additions & 7 deletions eng/test_tools.txt
Original file line number Diff line number Diff line change
@@ -1,29 +1,29 @@
# requirements leveraged by ci for testing
pytest==5.4.1; python_version >= '3.5'
pytest==4.6.9; python_version == '2.7'
pytest-asyncio==0.10.0; python_version >= '3.5'
pytest==5.4.2; python_version >= '3.5'
pytest-asyncio==0.12.0; python_version >= '3.5'
pytest-cov==2.8.1
pytest-custom-exit-code==0.3.0
pytest-xdist==1.31.0
pytest-xdist==1.32.0
# we pin coverage to 4.5.4 because there is an bug with `pytest-cov`. the generated coverage files cannot be `coverage combine`ed
coverage==4.5.4

# locking packages defined as deps from azure-sdk-tools or azure-devtools
pytoml==0.1.21
readme-renderer[md]==25.0
pyOpenSSL==19.1.0
json-delta==2.0
ConfigArgParse==1.1
ConfigArgParse==1.2.3
six==1.14.0
vcrpy==3.0.0
pyyaml==5.3.1
packaging==20.4
wheel==0.34.2
Jinja2==2.11.1
Jinja2==2.11.2

# Locking pylint and required packages
pylint==2.3.1; python_version >= '3.4'
pylint==1.8.4; python_version < '3.4'
astroid==2.3.3; python_version >= '3.4'
pylint==2.5.2; python_version >= '3.4'

../../../tools/azure-devtools
../../../tools/azure-sdk-tools
4 changes: 2 additions & 2 deletions pylintrc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ ignore-patterns=test_*,conftest,setup
reports=no

# PYLINT DIRECTORY BLACKLIST.
ignore=_generated,samples,examples,test,tests,doc,.tox
ignore=_vendor,_generated,samples,examples,test,tests,doc,.tox

init-hook='import sys; sys.path.insert(0, os.path.abspath(os.getcwd().rsplit("azure-sdk-for-python", 1)[0] + "azure-sdk-for-python/scripts/pylint_custom_plugin"))'
load-plugins=pylint_guidelines_checker
Expand All @@ -14,7 +14,7 @@ load-plugins=pylint_guidelines_checker
# cyclic-import: because of https://github.com/PyCQA/pylint/issues/850
# too-many-arguments: Due to the nature of the CLI many commands have large arguments set which reflect in large arguments set in corresponding methods.
# Let's black deal with bad-continuation
disable=useless-object-inheritance,missing-docstring,locally-disabled,fixme,cyclic-import,too-many-arguments,invalid-name,duplicate-code,too-few-public-methods,bad-continuation,check-docstrings
disable=useless-object-inheritance,missing-docstring,locally-disabled,fixme,cyclic-import,too-many-arguments,invalid-name,duplicate-code,too-few-public-methods,bad-continuation,check-docstrings,import-outside-toplevel

[FORMAT]
max-line-length=120
Expand Down
2 changes: 1 addition & 1 deletion scripts/pylint_custom_plugin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
In order to lint for the guidelines, you must make sure you are using the pylintrc file.
It is recommended you run pylint at the library package level to be consistent with how the CI runs pylint.

Check that you are running pylint version >=2.31 and astroid version >=2.25.
Check that you are running pylint version >=2.5.2 and astroid version >=2.4.1.

**How to run pylint locally using the pylintrc:**

Expand Down
68 changes: 34 additions & 34 deletions scripts/pylint_custom_plugin/pylint_guidelines_checker.py

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion sdk/core/azure-core/azure/core/pipeline/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ def prepare_requests(req):

with concurrent.futures.ThreadPoolExecutor() as executor:
# List comprehension to raise exceptions if happened
[ # pylint: disable=expression-not-assigned
[ # pylint: disable=expression-not-assigned, unnecessary-comprehension
_ for _ in executor.map(prepare_requests, requests)
]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ def __init__(self, credential, *scopes, **kwargs):
super().__init__(credential, *scopes, **kwargs)
self._lock = asyncio.Lock()

async def on_request(self, request: PipelineRequest):
async def on_request(self, request: PipelineRequest): # pylint:disable=invalid-overridden-method
"""Adds a bearer token Authorization header to request and sends request to next policy.
:param request: The pipeline request object to be modified.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ class AsyncRedirectPolicy(RedirectPolicy, AsyncHTTPPolicy):
:caption: Configuring an async redirect policy.
"""

async def send(self, request):
async def send(self, request): # pylint:disable=invalid-overridden-method
"""Sends the PipelineRequest object to the next policy.
Uses redirect settings to send the request to redirect endpoint if necessary.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ class AsyncRetryPolicy(RetryPolicy, AsyncHTTPPolicy):
:caption: Configuring an async retry policy.
"""

async def _sleep_for_retry(self, response, transport):
async def _sleep_for_retry(self, response, transport): # pylint:disable=invalid-overridden-method
"""Sleep based on the Retry-After response header value.
:param response: The PipelineResponse object.
Expand All @@ -93,7 +93,7 @@ async def _sleep_for_retry(self, response, transport):
return True
return False

async def _sleep_backoff(self, settings, transport):
async def _sleep_backoff(self, settings, transport): # pylint:disable=invalid-overridden-method
"""Sleep using exponential backoff. Immediately returns if backoff is 0.
:param dict settings: The retry settings.
Expand All @@ -104,7 +104,7 @@ async def _sleep_backoff(self, settings, transport):
return
await transport.sleep(backoff)

async def sleep(self, settings, transport, response=None):
async def sleep(self, settings, transport, response=None): # pylint:disable=invalid-overridden-method
"""Sleep between retry attempts.
This method will respect a server's ``Retry-After`` response header
Expand All @@ -123,7 +123,7 @@ async def sleep(self, settings, transport, response=None):
return
await self._sleep_backoff(settings, transport)

async def send(self, request):
async def send(self, request): # pylint:disable=invalid-overridden-method
"""Uses the configured retry policy to send the request to the next policy in the pipeline.
:param request: The PipelineRequest object
Expand Down
2 changes: 1 addition & 1 deletion sdk/core/azure-core/azure/core/pipeline/transport/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -626,7 +626,7 @@ def parse_responses(response):

with concurrent.futures.ThreadPoolExecutor() as executor:
# List comprehension to raise exceptions if happened
[ # pylint: disable=expression-not-assigned
[ # pylint: disable=expression-not-assigned, unnecessary-comprehension
_ for _ in executor.map(parse_responses, responses)
]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,10 +79,10 @@ async def __aenter__(self):
async def __aexit__(self, *exc_details): # pylint: disable=arguments-differ
return super(AsyncioRequestsTransport, self).__exit__()

async def sleep(self, duration):
async def sleep(self, duration): # pylint:disable=invalid-overridden-method
await asyncio.sleep(duration)

async def send(self, request: HttpRequest, **kwargs: Any) -> AsyncHttpResponse: # type: ignore
async def send(self, request: HttpRequest, **kwargs: Any) -> AsyncHttpResponse: # type: ignore # pylint:disable=invalid-overridden-method
"""Send the request using this HTTP sender.
:param request: The HttpRequest
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -150,10 +150,10 @@ async def __aenter__(self):
async def __aexit__(self, *exc_details): # pylint: disable=arguments-differ
return super(TrioRequestsTransport, self).__exit__()

async def sleep(self, duration):
async def sleep(self, duration): # pylint:disable=invalid-overridden-method
await trio.sleep(duration)

async def send(self, request: HttpRequest, **kwargs: Any) -> AsyncHttpResponse: # type: ignore
async def send(self, request: HttpRequest, **kwargs: Any) -> AsyncHttpResponse: # type: ignore # pylint:disable=invalid-overridden-method
"""Send the request using this HTTP sender.
:param request: The HttpRequest
Expand Down
2 changes: 1 addition & 1 deletion sdk/core/azure-core/azure/core/polling/_async_poller.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ def from_continuation_token(cls, continuation_token: str, **kwargs) -> Tuple[Any
class AsyncNoPolling(_NoPolling):
"""An empty async poller that returns the deserialized initial response.
"""
async def run(self):
async def run(self): # pylint:disable=invalid-overridden-method
"""Empty run, no polling.
Just override initial run to add "async"
"""
Expand Down
12 changes: 6 additions & 6 deletions sdk/core/azure-core/azure/core/polling/async_base_polling.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ class AsyncLROBasePolling(LROBasePolling):
"""A subclass or LROBasePolling that redefine "run" as async.
"""

async def run(self):
async def run(self): # pylint:disable=invalid-overridden-method
try:
await self._poll()
except BadStatus as err:
Expand All @@ -62,7 +62,7 @@ async def run(self):
response=self._pipeline_response.http_response, error=err
)

async def _poll(self):
async def _poll(self): # pylint:disable=invalid-overridden-method
"""Poll status of operation so long as operation is incomplete and
we have an endpoint to query.
Expand All @@ -85,17 +85,17 @@ async def _poll(self):
self._pipeline_response = await self.request_status(final_get_url)
_raise_if_bad_http_status_and_method(self._pipeline_response.http_response)

async def _sleep(self, delay):
async def _sleep(self, delay): # pylint:disable=invalid-overridden-method
await self._transport.sleep(delay)

async def _delay(self):
async def _delay(self): # pylint:disable=invalid-overridden-method
"""Check for a 'retry-after' header to set timeout,
otherwise use configured timeout.
"""
delay = self._extract_delay()
await self._sleep(delay)

async def update_status(self):
async def update_status(self): # pylint:disable=invalid-overridden-method
"""Update the current status of the LRO.
"""
self._pipeline_response = await self.request_status(
Expand All @@ -104,7 +104,7 @@ async def update_status(self):
_raise_if_bad_http_status_and_method(self._pipeline_response.http_response)
self._status = self._operation.get_status(self._pipeline_response)

async def request_status(self, status_link):
async def request_status(self, status_link): # pylint:disable=invalid-overridden-method
"""Do a simple GET to this status link.
This method re-inject 'x-ms-client-request-id'.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ class AsyncARMAutoResourceProviderRegistrationPolicy(
"""Auto register an ARM resource provider if not done yet.
"""

async def send(self, request: PipelineRequest):
async def send(self, request: PipelineRequest): # pylint: disable=invalid-overridden-method
http_request = request.http_request
response = await self.next.send(request)
if response.http_response.status_code == 409:
Expand Down
2 changes: 1 addition & 1 deletion sdk/eventhub/azure-eventhub/azure/eventhub/_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ def trace_link_message(events, parent_span=None):
Will extract DiagnosticId if available.
"""
trace_events = events if isinstance(events, Iterable) else (events,)
trace_events = events if isinstance(events, Iterable) else (events,) # pylint:disable=isinstance-second-argument-not-valid-type
try: # pylint:disable=too-many-nested-blocks
span_impl_type = settings.tracing_implementation() # type: Type[AbstractSpan]
if span_impl_type is not None:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ async def __aexit__(self, *args):
async def close(self) -> None:
await self._pipeline.__aexit__()

async def request_token(
async def request_token( # pylint:disable=invalid-overridden-method
self,
scopes: "Iterable[str]",
method: "Optional[str]" = "POST",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ async def close(self):

await self._client.__aexit__()

async def get_token(self, *scopes: str, **kwargs: "Any") -> "AccessToken": # pylint:disable=unused-argument
async def get_token(self, *scopes: str, **kwargs: "Any") -> "AccessToken":
"""Asynchronously request an access token for `scopes`.
.. note:: This method is called by Azure SDK clients. It isn't intended for use in application code.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
Policy = Union[AsyncHTTPPolicy, SansIOHTTPPolicy]


# pylint:disable=invalid-overridden-method
class AadClient(AadClientBase):
async def __aenter__(self):
await self._pipeline.__aenter__()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ def finalize(self, data):
padded = self._ctx.finalize()
return self._padder.update(padded) + self._padder.finalize()

@property
def block_size(self):
# return self._cipher.block_size
raise NotImplementedError()
Expand All @@ -59,7 +58,6 @@ def finalize(self, data):
padded = self._padder.finalize()
return self._ctx.update(padded) + self._ctx.finalize()

@property
def block_size(self):
# return self._cipher.block_size
raise NotImplementedError()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@

class _AesCbcHmacCryptoTransform(AuthenticatedCryptoTransform):
def __init__(self, key, iv, auth_data, auth_tag):
super(_AesCbcHmacCryptoTransform, self).__init__()

self._aes_key = key[: len(key) // 2]
self._hmac_key = key[len(key) // 2 :]
hash_algo = {256: hashes.SHA256(), 384: hashes.SHA384(), 512: hashes.SHA512()}[len(key) * 8]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@

class _EcdsaSignatureTransform(SignatureTransform):
def __init__(self, key, hash_algorithm):
super(_EcdsaSignatureTransform, self).__init__()

self._key = key
self._hash_algorithm = hash_algorithm

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

class RsaSignatureTransform(SignatureTransform):
def __init__(self, key, padding_function, hash_algorithm):
super(RsaSignatureTransform, self).__init__()
self._key = key
self._padding_function = padding_function
self._hash_algorithm = hash_algorithm
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

class _Sha2DigestTransform(DigestTransform):
def __init__(self, algorithm):
super(_Sha2DigestTransform, self).__init__()
self._digest = hashes.Hash(algorithm=algorithm, backend=default_backend())

def update(self, data):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -434,7 +434,7 @@ def sequence_number(self):
return None


class ReceivedMessage(PeekMessage):
class ReceivedMessageBase(PeekMessage):
"""
A Service Bus Message received from service side.
Expand All @@ -452,7 +452,7 @@ class ReceivedMessage(PeekMessage):
"""

def __init__(self, message, mode=ReceiveSettleMode.PeekLock, **kwargs):
super(ReceivedMessage, self).__init__(message=message)
super(ReceivedMessageBase, self).__init__(message=message)
self._settled = (mode == ReceiveSettleMode.ReceiveAndDelete)
self._is_deferred_message = kwargs.get("is_deferred_message", False)
self.auto_renew_error = None
Expand Down Expand Up @@ -594,6 +594,8 @@ def _settle_via_receiver_link(self, settle_operation, dead_letter_reason=None, d
return functools.partial(self.message.modify, True, True)
raise ValueError("Unsupported settle operation type: {}".format(settle_operation))


class ReceivedMessage(ReceivedMessageBase):
def _settle_message(
self,
settle_operation,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
_LOGGER = logging.getLogger(__name__)


class ReceivedMessage(sync_message.ReceivedMessage):
class ReceivedMessage(sync_message.ReceivedMessageBase):
"""A Service Bus Message received from service side.
"""
Expand Down
Loading

0 comments on commit 0f9c3be

Please sign in to comment.