Skip to content

Latest commit

 

History

History
219 lines (171 loc) · 3.54 KB

restructure.md

File metadata and controls

219 lines (171 loc) · 3.54 KB

restructure operator

The restructure operator facilitates changing the structure of a record by adding, removing, moving, and flattening fields.

The operator is configured with a list of ops, which are small operations that are applied to a record in the order they are defined.

Configuration Fields

Field Default Description
id restructure A unique identifier for the operator
output Next in pipeline The connected operator(s) that will receive all outbound entries
ops required A list of ops. The available op types are defined below
on_error send The behavior of the operator if it encounters an error. See on_error

Op types

Add

The add op adds a field to a record. It must have a field key and exactly one of value or value_expr.

field is a field that will be set to value or the result of value_expr

value is a static string that will be added to each entry at the field defined by field

value_expr is an expression with access to the record object

Example usage:

- type: restructure
  ops:
    - add:
        field: "key1"
        value: "val1"
    - add:
        field: "key2"
        value_expr: 'record["key1"] + "-suffix"'
Input record Output record 
{}
 
{
  "key1": "val1",
  "key2": "val1-suffix"
}

Remove

The remove op removes a field from a record.

Example usage:

- type: restructure
  ops:
    - remove: "key1"
Input record Output record 
{
  "key1": "val1",
  "key2": "val2"
}
 
{
  "key2": "val2"
}

Retain

The retain op keeps the specified list of fields, and removes the rest.

Example usage:

- type: restructure
  ops:
    - retain:
      - "key1"
      - "key2"
Input record Output record 
{
  "key1": "val1",
  "key2": "val2",
  "key3": "val3",
  "key4": "val4"
}
 
{
  "key1": "val1",
  "key2": "val2"
}

Move

The move op moves (or renames) a field from one location to another. Both the from and to fields are required.

Example usage:

- type: restructure
  ops:
    - move:
        from: "key1"
        to: "key3"
Input record Output record 
{
  "key1": "val1",
  "key2": "val2"
}
 
{
  "key3": "val1",
  "key2": "val2"
}

Flatten

The flatten op flattens a field by moving its children up to the same level as the field.

Example usage:

- type: restructure
  ops:
    - flatten: "key1"
Input record Output record 
{
  "key1": {
    "nested1": "nestedval1",
    "nested2": "nestedval2"
  },
  "key2": "val2"
}
 
{
  "nested1": "nestedval1",
  "nested2": "nestedval2",
  "key2": "val2"
}