Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HttpLoggingPolicy supports integrating logs into one record #19925

Closed
freeloop914 opened this issue Jul 23, 2021 · 5 comments
Closed

HttpLoggingPolicy supports integrating logs into one record #19925

freeloop914 opened this issue Jul 23, 2021 · 5 comments
Assignees
@xiangyan99
Labels
Azure.Core Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. feature-request This issue requires a new behavior in the product in order be resolved. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team Service Attention Workflow: This issue is responsible by Azure service team.
Milestone
Backlog

Comments

@freeloop914
Copy link

freeloop914 commented Jul 23, 2021 •
edited
Loading

I'm using azure function app and using SDK to operate Azure resources like blob list/download.
I need these HTTP request/response logs for troubleshooting debug.
But currently they are separated to multiple records.
It's very difficult to query.

For example, this is what current log looks like:

[7/29/2020 6:55:16 PM] Request URL: 'https://xxx.documents.azure.com:443/'
[7/29/2020 6:55:16 PM] Request method: 'GET'
[7/29/2020 6:55:16 PM] Request headers:
[7/29/2020 6:55:16 PM]     'Cache-Control': 'no-cache'
[7/29/2020 6:55:16 PM]     'x-ms-version': 'REDACTED'
[7/29/2020 6:55:16 PM]     'x-ms-documentdb-query-iscontinuationexpected': 'REDACTED'
[7/29/2020 6:55:16 PM]     'x-ms-consistency-level': 'REDACTED'
[7/29/2020 6:55:16 PM]     'x-ms-session-token': 'REDACTED'
[7/29/2020 6:55:16 PM]     'x-ms-date': 'REDACTED'
[7/29/2020 6:55:16 PM]     'authorization': 'REDACTED'
[7/29/2020 6:55:16 PM]     'Accept': 'application/json'
[7/29/2020 6:55:16 PM]     'Content-Length': '0'
[7/29/2020 6:55:16 PM]     'User-Agent': 'azsdk-python-cosmos/4.0.0 Python/3.8.5 (Windows-10-10.0.19041-SP0)'

What I expect is :

[7/29/2020 6:55:16 PM] Request URL: 'https://xxx.documents.azure.com:443/'
 Request method: 'GET'
 Request headers:
     'Cache-Control': 'no-cache'
     'x-ms-version': 'REDACTED'
     'x-ms-documentdb-query-iscontinuationexpected': 'REDACTED'
     'x-ms-consistency-level': 'REDACTED'
     'x-ms-session-token': 'REDACTED'
     'x-ms-date': 'REDACTED'
     'authorization': 'REDACTED'
     'Accept': 'application/json'
     'Content-Length': '0'
     'User-Agent': 'azsdk-python-cosmos/4.0.0 Python/3.8.5 (Windows-10-10.0.19041-SP0)'

It looks more clean and tidy ( At least from Azure function app Monitoring Log interface. )
and when I query the log by

trace | where message contains 'Request URL'

I can get useful and related information instead of one line meaning nothing.
@ghost ghost added needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jul 23, 2021
@mccoyp
Copy link
Member

mccoyp commented Jul 23, 2021

Hi @freeloop914, thank you for opening an issue! It looks like you're using Cosmos here -- please correct me if I'm wrong. I'll tag the appropriate folks so we can take a look at this as soon as possible.

@mccoyp mccoyp added Client This issue points to a problem in the data-plane of the library. Cosmos feature-request This issue requires a new behavior in the product in order be resolved. Service Attention Workflow: This issue is responsible by Azure service team. and removed needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jul 23, 2021
@ghost ghost added the needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team label Jul 23, 2021
@ghost
Copy link

ghost commented Jul 23, 2021

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @zfoster, @southpolesteve.

Issue Details

I'm using azure function app and using SDK to operate Azure resources like blob list/download.
I need these HTTP request/response logs for troubleshooting debug.
But currently they are separated to multiple records.
It's very difficult to query.

For example, this is what current log looks like:

[7/29/2020 6:55:16 PM] Request URL: 'https://xxx.documents.azure.com:443/'
[7/29/2020 6:55:16 PM] Request method: 'GET'
[7/29/2020 6:55:16 PM] Request headers:
[7/29/2020 6:55:16 PM]     'Cache-Control': 'no-cache'
[7/29/2020 6:55:16 PM]     'x-ms-version': 'REDACTED'
[7/29/2020 6:55:16 PM]     'x-ms-documentdb-query-iscontinuationexpected': 'REDACTED'
[7/29/2020 6:55:16 PM]     'x-ms-consistency-level': 'REDACTED'
[7/29/2020 6:55:16 PM]     'x-ms-session-token': 'REDACTED'
[7/29/2020 6:55:16 PM]     'x-ms-date': 'REDACTED'
[7/29/2020 6:55:16 PM]     'authorization': 'REDACTED'
[7/29/2020 6:55:16 PM]     'Accept': 'application/json'
[7/29/2020 6:55:16 PM]     'Content-Length': '0'
[7/29/2020 6:55:16 PM]     'User-Agent': 'azsdk-python-cosmos/4.0.0 Python/3.8.5 (Windows-10-10.0.19041-SP0)'

What I expect is :

[7/29/2020 6:55:16 PM] Request URL: 'https://xxx.documents.azure.com:443/'
 Request method: 'GET'
 Request headers:
     'Cache-Control': 'no-cache'
     'x-ms-version': 'REDACTED'
     'x-ms-documentdb-query-iscontinuationexpected': 'REDACTED'
     'x-ms-consistency-level': 'REDACTED'
     'x-ms-session-token': 'REDACTED'
     'x-ms-date': 'REDACTED'
     'authorization': 'REDACTED'
     'Accept': 'application/json'
     'Content-Length': '0'
     'User-Agent': 'azsdk-python-cosmos/4.0.0 Python/3.8.5 (Windows-10-10.0.19041-SP0)'

It looks more clean and tidy ( At least from Azure function app Monitoring Log interface. )
and when I query the log by

trace | where message contains 'Request URL'

I can get useful and related information instead of one line meaning nothing.

Author: freeloop914
Assignees: Rodrigossz
Labels:

Client, Cosmos, Service Attention, customer-reported, feature-request
Milestone: -

@freeloop914
Copy link
Author

Hi @freeloop914, thank you for opening an issue! It looks like you're using Cosmos here -- please correct me if I'm wrong. I'll tag the appropriate folks so we can take a look at this as soon as possible.

Thanks @mccoyp ,
My function app will access Azure Blob/SQL/Cosmos,
As I know, they are all using logger named azure.core.pipeline.policies.http_logging_policy
and it propagates to Root logger.

Another example from blob downloading:

[2021/7/24 02:41:10 AM] Request URL: 'https://XXXXXXXXXXX.blob.core.windows.net/XXXXX/XXXXXXXXXXXXXX.zip'
[2021/7/24 02:41:10 AM] Request method: 'GET'
[2021/7/24 02:41:10 AM] Request headers:
[2021/7/24 02:41:10 AM]     'x-ms-range': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-encryption-algorithm': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-version': 'REDACTED'
[2021/7/24 02:41:10 AM]     'Accept': 'application/xml'
[2021/7/24 02:41:10 AM]     'User-Agent': 'azsdk-python-storage-blob/12.7.1 Python/3.8.5 (Windows-10-10.0.19041-SP0)'
[2021/7/24 02:41:10 AM]     'x-ms-date': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-client-request-id': '9a8baccf-ec28-11eb-8923-c8348e334d1c'
[2021/7/24 02:41:10 AM]     'Authorization': 'REDACTED'
[2021/7/24 02:41:10 AM] No body was attached to the request
[2021/7/24 02:41:10 AM] Response status: 206
[2021/7/24 02:41:10 AM] Response headers:
[2021/7/24 02:41:10 AM]     'Content-Length': '4612104'
[2021/7/24 02:41:10 AM]     'Content-Type': 'application/x-zip-compressed'
[2021/7/24 02:41:10 AM]     'Content-Range': 'REDACTED'
[2021/7/24 02:41:10 AM]     'Last-Modified': 'Fri, 23 Jul 2021 10:10:43 GMT'
[2021/7/24 02:41:10 AM]     'Accept-Ranges': 'REDACTED'
[2021/7/24 02:41:10 AM]     'ETag': '"0x8D94DC221CA60AA"'
[2021/7/24 02:41:10 AM]     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
[2021/7/24 02:41:10 AM]     'x-ms-request-id': '124ec128-f01e-005d-4a35-80a71f000000'
[2021/7/24 02:41:10 AM]     'x-ms-client-request-id': '9a8baccf-ec28-11eb-8923-c8348e334d1c'
[2021/7/24 02:41:10 AM]     'x-ms-version': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-meta-processed_ts': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-meta-processed_id': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-meta-trigger': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-meta-refilled_metadatadb': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-meta-refilled_resultdb': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-meta-test_result': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-creation-time': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-blob-content-md5': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-lease-status': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-lease-state': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-blob-type': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-copy-id': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-copy-source': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-copy-status': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-copy-progress': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-copy-completion-time': 'REDACTED'
[2021/7/24 02:41:10 AM]     'x-ms-server-encrypted': 'REDACTED'
[2021/7/24 02:41:10 AM]     'Date': 'Sat, 24 Jul 2021 02:41:10 AM GMT'

@xiangyan99 xiangyan99 added this to the Backlog milestone Jul 27, 2021
@xiangyan99 xiangyan99 removed the Cosmos label Jul 27, 2021
@Rodrigossz Rodrigossz removed their assignment Aug 10, 2021
@xiangyan99 xiangyan99 mentioned this issue Aug 16, 2021
Merged
@xiangyan99
Copy link
Member

The issue is fixed. It will be available in next release.

@freeloop914
Copy link
Author

Thanks @xiangyan99 . Good to know

@github-actions github-actions bot locked and limited conversation to collaborators Apr 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@xiangyan99 xiangyan99

Labels
Azure.Core Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. feature-request This issue requires a new behavior in the product in order be resolved. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team Service Attention Workflow: This issue is responsible by Azure service team.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@Rodrigossz @xiangyan99 @mccoyp @freeloop914