Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logging: parse entries created by bigquery_resource #5313

Closed
yan-hic opened this issue May 8, 2018 · 6 comments
Closed

Logging: parse entries created by bigquery_resource #5313

yan-hic opened this issue May 8, 2018 · 6 comments
Assignees
Labels
api: logging Issues related to the Cloud Logging API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@yan-hic
Copy link

yan-hic commented May 8, 2018

client_logging.list_entries() on BigQuery log entries renders a list of ProtobufEntry objects.

Unfortunately there is no documentation on how to read ("unpack" ?) the payload_pb property, for instance to retrieve the jobId.

Request to extent the documentation beyond retrieving entries that were written to a logger, and include snippets on how-to parse protoPayload messages like those created by GCP services.

@tseaver tseaver added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. documentation api: logging Issues related to the Cloud Logging API. priority: p2 Moderately-important priority. Fix may not be included in next release. labels May 9, 2018
@tseaver
Copy link
Contributor

tseaver commented May 10, 2018

@yiga2 We don't have any visibility here as to what kind of message has been passed by the originator as the proto_payload (gRPC) or protoPayload (HTTP): handling them requires knowledge of the protobuf message schema. Can you point to docs for the entries you are trying to parse?

@yan-hic
Copy link
Author

yan-hic commented May 10, 2018

EDITED - replaced and in filter with AND as it's case-sensitive. Note that a line-feed or space between filter entries works too.

Understood. I have a hard time getting the list_entries as expected for the below - no issue using the API (explorer) but here how to reproduce.
To get the log entries from biquery (used https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/bigquery/cloud-client/simple_app.py):

from google.cloud import bigquery
bq = bigquery.Client()
query_job = bq.query("""
        SELECT
          CONCAT(
            'https://stackoverflow.com/questions/',
            CAST(id as STRING)) as url,
          view_count
        FROM `bigquery-public-data.stackoverflow.posts_questions`
        WHERE tags like '%google-bigquery%'
        ORDER BY view_count DESC
        LIMIT 10""")

then here to retrieve the log entries (I am project owner)

from google.cloud import logging
log= logging.Client()
filter = """
resource.type="bigquery_resource" 
AND protoPayload.serviceData.jobCompletedEvent.eventName="query_job_completed"
AND timestamp >= "%s"
""" % query_job.started.isoformat() 
res = list(log.list_entries(filter_=filter))   

res should will return the job_completed entry, with the payload_pb. Somehow I don't get anything even with playing with the filter.

@tseaver
Copy link
Contributor

tseaver commented May 16, 2018

@yiga2 I've found the docs for the AuditLog message type.

@tseaver tseaver added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels May 31, 2018
@tseaver
Copy link
Contributor

tseaver commented Jun 21, 2018

@yiga2 If you know / expect that the log entry contains an AuditLog instance, you can unmarshall it using ProtobufEntry.parse_message, e.g.:

from google..... import AuditLog  # I don't know the path of the proto-generated module

def unmarshall_audit_log(entry):
    if entry.payload_pb.Is(AuditLog.DESCRIPTOR):
        message = AuditLog()
        protobuf_entry.parse_message(message)
        return message

@tseaver tseaver closed this as completed Jun 21, 2018
@yan-hic
Copy link
Author

yan-hic commented Jun 21, 2018

Not sure why it gets closed as what seems to be missing is the module compiled from proto, as per https://stackoverflow.com/questions/50301632/using-python-to-query-gcp-stackdriver-logs

Instead of each of us compiling a python module for it, it would make sense to have a new google.cloud.audit package containing, at a minimum, the AuditLog module. After that, I agree @tseaver's code makes it straightforward to parse.

@snarfed
Copy link

snarfed commented Dec 19, 2019

as a fully fleshed out example, here's how i generated my own request_log_pb2.py file based on request_log.prototo parse google.appengine.logging.v1.RequestLog serialized protobufs: https://stackoverflow.com/a/59414384/186123. more background in #2572.

@yiga2 is right, though, Google should ideally maintain official versions of these Python protobuf bindings instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: logging Issues related to the Cloud Logging API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

3 participants