Skip to content

Releases: opensnowcat/opensnowcat-enrich

1.2.2

10 Jan 19:42
@joaolcorreia joaolcorreia
1.2.2
bf1424c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
1.2.2 Latest
Latest

What's Changed

Full Changelog: 1.2.1...1.2.2

Contributors

oermolaev
Assets 7
Loading

1.2.1

21 Nov 18:18
@joaolcorreia joaolcorreia
1.2.1
4b75bad
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
1.2.1

What's Changed

New Contributors

Full Changelog: 1.2.0...1.2.1

Contributors

oermolaev and AlexITC
Assets 7
Loading

1.2.0

21 Aug 21:35
@AlexITC AlexITC
1.2.0
4b2f90d
Compare
Choose a tag to compare
1.2.0

What's New

What's Changed

Full Changelog: 1.1.0...1.2.0

Contributors

Fattouh92, AlexITC, and wiringbits-bot
Assets 7
Loading

1.1.0

01 Mar 23:33
@joaolcorreia joaolcorreia
1.1.0
44f5528
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
1.1.0

What's New

JSON Output for all modules

In this version, we expanded support for flattened JSON output to all modules.
JSON enables easier integration with data infrastructure without creating dependencies on vendor-specific loaders.

JSON format improvements

A new entry is available in the experimental settings which allows defining the output format:

  "experimental": {
    # Whether to use an alternative output format  (either "FlattenedJson" or "EventbridgeJson")
    # - FlattenedJson: encodes the output as JSON, where unstruct_event, contexts and derived_contexts are flattened
    # - EventbridgeJson: encodes the output as JSON with extra details, intended exclusively to use with eventbridge module.
    "customOutputFormat": {
      "type": "FlattenedJson"
    }
  }

This is an example produced when enabling the FlattenedJson format (null values omitted for simplicity):

{
  "collector_tstamp" : "2011-12-03T10:15:30Z",
  "contexts_org_ietf_http_header_1" : [
    {
      "name" : "Host",
      "value" : "test"
    }
  ],
  "event_id" : "236392af-ffec-4def-a0de-86929e9615be",
}

Previously, this JSON format was available only for Kinesis but it can now be enabled in all modules.

NOTE: If you use the PII output. the JSON format is discouraged because we have not done enough tests on this.

EventbridgeJson format

This is a superset of the FlattenedJson format, it has the option to include a few extra fields in the output, it can be turned on with:

  "experimental": {
    # Whether to use an alternative output format  (either "FlattenedJson" or "EventbridgeJson")
    # - FlattenedJson: encodes the output as JSON, where unstruct_event, contexts and derived_contexts are flattened
    # - EventbridgeJson: encodes the output as JSON with extra details, intended exclusively to use with eventbridge module.
    "customOutputFormat": {
      "type": "EventbridgeJson"

      # Add a "payload" parameter with the original base64 encoded TSV. Defaults to false (Useful for EventbridgeJson only).
      "payload": true

      # Add a "collector" parameter from contexts_org_ietf_http_header_1 Host Value. Defaults to false (Useful for EventbridgeJson only).
      "collector": true
    }
  }

This is an example produced when enabling the FlattenedJson format (null values omitted for simplicity):

{
  "collector_tstamp" : "2011-12-03T10:15:30Z",
  "contexts_org_ietf_http_header_1" : [
    {
      "name" : "Host",
      "value" : "test"
    }
  ],
  "event_id" : "236392af-ffec-4def-a0de-86929e9615be",
  # produced because "collector" = true
  "collector" : "test",
  # produced because "payload" = true
  "payload" : "<<BASE64ENCODEDTDV>>,
}

NOTE: This format is intended for the eventbridge module only.

Breaking changes

If you are using the JSON output from Kinesis it means that you have these settings:

{
  ...
  output {
    good: {
      ...
      jsonOutput = true
    }

    pii: {
      ...
      jsonOutput = true
    }
    bad: {
      ...
      jsonOutput = true
    }
  }
}

These need to be removed, then, set the experimental settings like this:

  ...
  "experimental": {
    "customOutputFormat": {
      "type": "FlattenedJson"
    }
  }

If you are use eventbridge, you may have these settings:

{
  ...
  output {
    good: {
      ...
      collector = true
      payload = true
    }
  }
}

Which need to be replaced by:

{
  ...
  "experimental": {
    "customOutputFormat": {
      "type": "EventbridgeJson"
      "payload": true
      "collector": true
    }
  }
}

Changelog

Read more

Contributors

joaolcorreia, AlexITC, and 2 other contributors
Assets 7
Loading

1.0.0

11 Jan 22:13
@AlexITC AlexITC
1.0.0
0f2b2e7
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
1.0.0

What's Changed

New Contributors

Full Changelog: https://github.com/opensnowcat/opensnowcat-enrich/commits/1.0.0

NOTE: This is a fork from https://github.com/snowplow/enrich/tree/cc06fa29beefdcdbb4facd2559a53d7ae027d88b

Contributors

joaolcorreia and AlexITC
Assets 7
Loading