[Filebeat] Error while parsing lastError field in Azure platform logs

[jmmcorreia]

• Version: 7.11.1 ( But also had this issue on 7.11.0)
• Operating System: Ubuntu (AKS Cluster)
• Discuss Forum URL: https://discuss.elastic.co/t/filebeat-error-while-parsing-lasterror-field-in-azure-platform-logs/265940

Hi everyone,

so I was testing the Azure module for filebeat to pull the logs into ES and run into the following issue. Basically, if there is any error message present in the APIM(API Management service) logs, those will not reach the ES backend. In other words, when the request is not successful on the APIM side, the service can add 6 extra fields to its log entry which are : LastErrorElaspsed, LastErrorSource, LastErrorSection, LasErrorReason, LastErrorMessage, LastErrorScope.

These are the fields whose data is being lost. However, they are being sent by azure as shown by the printed entry in the Filebeat logs:

{
    Fields: {
        "agent": {
            "type": "filebeat",
            "version": "7.11.1"
        },
        "ecs": {
            "version": "1.6.0"
        },
        "event": {
            "dataset": "azure.platformlogs",
            "module": "azure"
        },
        "fileset": {
            "name": "platformlogs"
        },
        "input": {
            "type": "azure-eventhub"
        },
        "message": "{\"Level\":4,\"callerIpAddress\":\"ipHere\",\"category\":\"GatewayLogs\",\"correlationId\":\"idHere\",\"durationMs\":0,\"isRequestSuccess\":false,\"location\":\"North Central US\",\"operationName\":\"Microsoft.ApiManagement/GatewayLogs\",\"properties\":{\"apiId\":\"api-id\",\"apiRevision\":\"1\",\"cache\":\"none\",\"clientProtocol\":\"HTTP/1.1\",\"clientTlsVersion\":\"1.2\",\"lastError\":{\"message\":\"Claim value mismatch: email=sampleemail@something.com.\",\"path\":\"choose\\\\when[1]\",\"reason\":\"TokenClaimValueMismatch\",\"scope\":\"api\",\"section\":\"inbound\",\"source\":\"validate-jwt\"},\"method\":\"GET\",\"operationId\":\"GET\",\"responseCode\":401,\"responseSize\":316,\"url\":\"https://some.url.here\"},\"resourceId\":\"/resourceIdHere\",\"time\":\"2021-03-02T10:15:00.2443029Z\"}",
        "service": {
            "type": "azure"
        }
    }
}

The following warning message accompanied the log entry shown above

2021-03-02T10:16:45.987Z	WARN	[elasticsearch]	elasticsearch/client.go:408	Cannot index event publisher.Event {azure log content here with lastError field included},  Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [azure.platformlogs.properties.lastError] of type [keyword] in document with id 'qLxw8ncBPAyIKXCBqjeG'. Preview of field's value: '{path=choose\\when[1], reason=TokenClaimValueMismatch, scope=api, section=inbound, source=validate-jwt, message=Claim value mismatch: email=sampleemail@something.com.}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1127"}}

These are the steps to reproduce the issue:
1: Use the Filebeat Azure Module to pull platform logs from Azure using the following config:

filebeat:
  modules:
  - module: azure
    platformlogs:
      enabled: true
      var:
        eventhub: "eventhub"
        consumer_group: "consumer"
        connection_string: "connstring"
        storage_account: "storageaccount"
        storage_account_key: "key"

2. Send APIM logs to the eventhub filebeat is reading from
3. Configure the APIM such a way that a request will fail. (ex: token validation failure)
4. Check logs in kibana. The usual log fields should be prensent, but not the LastError fields.

Just a few extra details about my setup. I'm using the kubernetes operator to deploy ES, Kibana and Filebeat. They are all running in an AKS cluster.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[narph]

@jmmcorreia , the problem is described right at the end of the error log

failed to parse field [azure.platformlogs.properties.lastError] of type [keyword] in document with ....","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1127"}}

We map by default only the common fields we see appear in the platform logs as it would be a lot of work to map each property from each type of platform logs from each resource type.

It looks like the lastError field had initially a string/keyword value so it dynamically mapped the lastError property type as a keyword. In the new event, lastError is sent as an object so mapping is not validated since it was expecting a keyword.
You can confirm this by retrieving the mapping for the platformlogs index and checking lastError property type.
One workaround in your case would be to modify the platformlogs pipeline and check if lastError is an object then either rename it or convert it into a string.

[jmmcorreia]

Thanks for helping me @narph

I tried to check the lastError field mapping, but couldn't find any info. First I tried this request:

GET /my-azure-logs-index/_mapping/field/lastError
That gave me the following answer:

{
  "my-azure-logs-index" : {
    "mappings" : { }
  }
}

Then I tried GET /my-azure-logs-index/_mapping and could not find any mention to lastError in the response. I also had a look at the index pattern fields and cannot find any mention to lastError.

I am still trying the workaround to see if I am able to make it work. Will provide an update when I have any more news.

[narph]

closed by #26148, please reopen if this will not fix your issue