Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spool to disk not working with time.Time fields #10099

Closed
andrewkroh opened this issue Jan 16, 2019 · 1 comment · Fixed by #15534
Closed

Spool to disk not working with time.Time fields #10099

andrewkroh opened this issue Jan 16, 2019 · 1 comment · Fixed by #15534
Assignees
Labels
bug libbeat Team:Integrations Label for the Integrations team

Comments

@andrewkroh
Copy link
Member

I noticed this problem with beats that produce event.start and event.end which are time.Time values. The same thing happens with Filebeat's netflow input which also produces these fields.

See how event.start and event.end are {}.

$ ./packetbeat -c packetbeat.queue.yml -I tests/system/pcaps/dns_google_com.pcap 
{
  "@timestamp": "2019-01-16T04:53:31.637Z",
  "@metadata": {
    "beat": "packetbeat",
    "type": "_doc",
    "version": "7.0.0"
  },
  "event": {
    "start": {},
    "end": {},
    "dataset": "dns",
    "duration": 66221937
  },
  "type": "dns"
}
packetbeat.interfaces.device: any
packetbeat.shutdown_timeout: 2s
packetbeat.protocols:
- type: dns
  ports: [53]

queue.spool.size: 10 MiB

output.console.pretty: true

processors:
  - include_fields:
      fields: event

If spooling is disabled the problem goes away.

$ ./packetbeat -c packetbeat.queue.yml -I tests/system/pcaps/dns_google_com.pcap -E queue.spool.enabled=false
{
  "@timestamp": "2019-01-16T04:56:33.861Z",
  "@metadata": {
    "beat": "packetbeat",
    "type": "_doc",
    "version": "7.0.0"
  },
  "event": {
    "dataset": "dns",
    "duration": 66212229,
    "start": "2019-01-16T04:56:33.861Z",
    "end": "2019-01-16T04:56:33.927Z"
  },
  "type": "dns"
}
andrewkroh added a commit to andrewkroh/beats that referenced this issue Feb 5, 2019
Register encoders for time.Time and common.Time so that events containing timestamps
can be serialized.

Fixes elastic#10099
andrewkroh added a commit that referenced this issue Feb 5, 2019
Register encoders for time.Time and common.Time so that events containing timestamps
can be serialized.

Fixes #10099
@andrewkroh
Copy link
Member Author

Reopening because this needs a unit test to prevent a regression.

@andrewkroh andrewkroh reopened this Feb 5, 2019
andrewkroh added a commit to andrewkroh/beats that referenced this issue Mar 4, 2019
Register encoders for time.Time and common.Time so that events containing timestamps
can be serialized.

Fixes elastic#10099

(cherry picked from commit 2b63496)
andrewkroh added a commit that referenced this issue Mar 6, 2019
Register encoders for time.Time and common.Time so that events containing timestamps
can be serialized.

Fixes #10099

(cherry picked from commit 2b63496)
@urso urso added the good first issue Indicates a good issue for first-time contributors label Apr 17, 2019
@urso urso mentioned this issue Dec 11, 2019
39 tasks
@urso urso removed the good first issue Indicates a good issue for first-time contributors label Jan 13, 2020
@blakerouse blakerouse self-assigned this Jan 14, 2020
@andresrc andresrc added the Team:Integrations Label for the Integrations team label Mar 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug libbeat Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants