PANW incorrectly parse the timezone to UTC when Timezone in the event

[ph]

I was involved in a user case concerning the PANW module and I have found out that the date doesn't seem to be correctly parsed by the ingest pipeline when the timezone is found in the event. I've changed the ingest pipeline for the following and it appears to have solved the user problem.

Is there something that I've missed or the pipeline is indeed incorrect?

diff --git a/x-pack/filebeat/module/panw/panos/ingest/pipeline.yml b/x-pack/filebeat/module/panw/panos/ingest/pipeline.yml
index 17ae7788d..887002a7f 100644
--- a/x-pack/filebeat/module/panw/panos/ingest/pipeline.yml
+++ b/x-pack/filebeat/module/panw/panos/ingest/pipeline.yml
@@ -6,48 +6,50 @@ processors:
      field: message
      target_field: log.original
 
-# Set @timestamp to the time when the entry was generated at the data plane.
  - date:
+     if: "ctx.event.timezone == null"
      field: "_temp_.generated_time"
      ignore_failure: true
      formats:
        - "yyyy/MM/dd HH:mm:ss"
  - date:
-    if: "ctx.event.timezone != null"
-    field: "@timestamp"
-    formats: ["ISO8601"]
-    timezone: "{{ event.timezone }}"
-    on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]
+     if: "ctx.event.timezone != null"
+     field: "_temp_.generated_time"
+     formats: ["yyyy/MM/dd HH:mm:ss"]
+     timezone: "{{ event.timezone }}"
+     on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]
 
 # event.created is the time the event was received at the management plane.
  - date:

+     if: "ctx.event_timezone == null"
      field: "event.created"
      target_field: "event.created"
      ignore_failure: true
      formats:
        - "yyyy/MM/dd HH:mm:ss"
  - date:
-    if: "ctx.event.timezone != null && ctx.event.created != null "
-    field: "event.created"
-    target_field: "event.created"
-    formats: ["ISO8601"]
-    timezone: "{{ event.timezone }}"
-    on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]
+     if: "ctx.event.timezone != null && ctx.event.created != null "
+     field: "event.created"
+     target_field: "event.created"
+     formats: ["yyyy/MM/dd HH:mm:ss"]
+     timezone: "{{ event.timezone }}"
+     on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]
 
 # event.start (traffic only) is the time the session started.
  - date:
+     if: "ctx.event.timezone == null"
      field: "event.start"
      target_field: "event.start"
      ignore_failure: true
      formats:
        - "yyyy/MM/dd HH:mm:ss"
  - date:
-    if: "ctx.event.timezone != null && ctx.event.start != null"
-    field: "event.start"
-    target_field: "event.start"
-    formats: ["ISO8601"]
-    timezone: "{{ event.timezone }}"
-    on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]
+     if: "ctx.event.timezone != null && ctx.event.start != null"
+     field: "event.start"
+     target_field: "event.start"
+     formats: ["yyyy/MM/dd HH:mm:ss"]
+     timezone: "{{ event.timezone }}"
+     on_failure: [{"append": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]
 
 # convert integer fields as the output of the CSV processor is always a string.
  - convert: { type: long, ignore_missing: true, field: client.bytes }

[ph]

Maybe @elastic/observability could comment on that?

[jsoriano]

I am also investigating these issues, I have opened a PR to avoid using UTC in pipelines testing, so we better see if changes in pipelines incorrectly change the resulting events (#13874).

This way I have found that there are some other incorrect modules.

The fix would be the one you posted here yes, @ph would you want to open a PR with this fix for this module? I will take care of others I have found.

[jsoriano]

I have created #13877 to keep track of this issue in affected modules.

[ph]

@jsoriano can you take it over, so only one review is necessary ?:)

[jsoriano]

@ph ok, I will take it as part of #13877 :)