[Fleet] Add support for routing rules in integrations

[kpollich]

In order to support document-based routing in Fleet, integrations need to expose their routing rules as part of their data stream manifest files.

These rules would be translated to reroute processors appropriately by Fleet during integration installation.

A routing rule is composed of a few pieces of data:

• "Source" dataset: in which dataset should Fleet place this rule?
• "Destination" dataset: where should documents be routed by this rule?
• Condition: what logic determines when a document is routed?
• Namespace: under what namespace should this document be written once routed?

We'll need to support two types of routing rules defined by an integration:

• "Local" routing rules that route from a dataset on a given integration to other datasets on that same integration
• "Injected" routing rules that route from a dataset on a external integration back to the given integration

As far as integrations are concerned, though, there is no meaningful difference between writing a "local" routing rule and an "injected" routing rule in a data stream manifest. Fleet will be responsible for generating the appropriate processors in the appropriate ingest pipelines based on these rules. So, the implementation on the package-spec side will be a generic routing_rules object at the data stream manifest level.

Typically, routing rules will be defined on a "catch-all" or "data sink" style dataset like kubernetes.router that is essentially a passthrough to more specific data streams.

For example, we might have an nginx catch-all dataset that routes Nginx logs to more specific data sets like nginx.error and nginx.access based on the logfile path reported in each document.

Here's a proposed example of the above in action. Please see the annotative comments for more details:

# nginx/data_stream/nginx/manifest.yml
title: Nginx logs
type: logs

# This is a catch-all "sink" data stream that routes documents to 
# other datasets based on conditions or variables
dataset: nginx

# Ensures agents have permissions to write data to `logs-nginx.*-*`
elasticsearch.dynamic_dataset: true
elasticsearch.dynamic_namespace: true

routing_rules:
  # "Local" routing rules are included under this current dataset, not a special case
  nginx:
    # Route error logs to `nginx.error` when they're sourced from an error logfile
    - dataset: nginx.error
      if: "ctx?.file?.path?.contains('/var/log/nginx/error')"
      namespace:
        - {{labels.data_stream.namespace}}
        - default

    # Route access logs to `nginx.access` when they're sourced from an access logfile
    - dataset: nginx.access
      if: "ctx?.file?.path?.contains('/var/log/nginx/access')"
      namespace:
        - {{labels.data_stream.namespace}}
        - default
  
  # Route K8's container logs to this catch-all dataset for further routing
  k8s.router: 
    - dataset: nginx
      if: "ctx?.container?.image?.name == 'nginx'"
      namespace:
        - {{labels.data_stream.namespace}}
        - default
   
  # Route syslog entries tagged with nginx to this catch-all dataset
  syslog:
    - dataset: nginx
      if: "ctx?.tags?.contains('nginx')"
      namespace:
        - {{labels.data_stream.namespace}}
        - default

Fleet support will be implemented as follow:

• Phase 1: [Fleet] Support local routing rules during integration installation kibana#155910
• Phase 2: [Fleet] Support injected routing rules during integration installation kibana#157422

[mrodm]

By reviewing this definition, I have some questions about it:

• Are these two settings mandatory for every package that uses routing rules ?

  # Ensures agents have permissions to write data to `logs-nginx.*-*`
  elasticsearch.dynamic_dataset: true
  elasticsearch.dynamic_namespace: true

  To avoid defining this in every package (datastream) using routing rules, could it be done automatically by Fleet when installing the package if any routing rule is defined ?

• In this example, it is used dataset that it is already defined in package-spec:

  # This is a catch-all "sink" data stream that routes documents to 
  # other datasets based on conditions or variables
  dataset: nginx

  Should it be updated the description ? Should we add another key for this ?

• Checking how to define this new routing_rules key in package-spec:

  • As each data-stream manifest could be very long, I was thinking to move this definition to its own file inside the data-stream folder, is that ok?
  • I was wondering if this could be rewritten as an array of objects:
    • instead of name, would it better dataset or another naming?

    routing_rules:
      - name: nginx
        rules:
          - dataset: ...
      - name: k8s.router
        rules:
          - dataset: ... 

cc @juliaElastic @jsoriano

[juliaElastic]

• Are these two settings mandatory for every package that uses routing rules ?

  # Ensures agents have permissions to write data to `logs-nginx.*-*`
  elasticsearch.dynamic_dataset: true
  elasticsearch.dynamic_namespace: true

  To avoid defining this in every package (datastream) using routing rules, could it be done automatically by Fleet when installing the package if any routing rule is defined ?

I think it makes sense to add this during Fleet installation to avoid repeating in every package, it seems mandatory if routing_rules are present. We can add it as a task in elastic/kibana#155910

• In this example, it is used dataset that it is already defined in package-spec:

  # This is a catch-all "sink" data stream that routes documents to 
  # other datasets based on conditions or variables
  dataset: nginx

  Should it be updated the description ? Should we add another key for this ?

I think it's good to use the existing data stream.

• Checking how to define this new routing_rules key in package-spec:

  • As each data-stream manifest could be very long, I was thinking to move this definition to its own file inside the data-stream folder, is that ok?

I think it makes sense to add to its own file.

• I was wondering if this could be rewritten as an array of objects:

  • instead of name, would it better dataset or another naming?

  routing_rules:
    - name: nginx
      rules:
        - dataset: ...
    - name: k8s.router
      rules:
        - dataset: ... 

I think array of objects sounds good.
The name refers to the current package in the first phase, so I wouldn't call it dataset. Maybe keeping it name is generic enough.

[mrodm]

• In this example, it is used dataset that it is already defined in package-spec:

  # This is a catch-all "sink" data stream that routes documents to 
  # other datasets based on conditions or variables
  dataset: nginx

  Should it be updated the description ? Should we add another key for this ?

I think it's good to use the existing data stream.

Just one more question about this. Should dataset field be mandatory in case routing_rules key is defined ? @juliaElastic
If so, probably routing_rules should be defined at the same manifest file so it can be expressed that requirement.

[juliaElastic]

I found that Fleet sets the default dataset name as packageInfo.name + '.' + policyTemplate.name if not set.

I think it's a good idea to make dataset mandatory if routing_rules are defined, as the routing rule name has to match the dataset, so it's better to be explicit about it.

[mrodm]

Once #535 has been completed.
These routing rules are going to be defined in its own file in each data stream folder. For instance:

• data_stream/rules/routing_rules.yml

And this is one example of the contents/syntax of this file:

  # "Local" routing rules are included under this current dataset, not a special case
- source_dataset:  nginx
  # Route error logs to `nginx.error` when they're sourced from an error logfile
  - target_dataset: nginx.error
    if: "ctx?.file?.path?.contains('/var/log/nginx/error')"
    namespace:
      - {{labels.data_stream.namespace}}
      - default

  # Route access logs to `nginx.access` when they're sourced from an access logfile
  - target_dataset: nginx.access
    if: "ctx?.file?.path?.contains('/var/log/nginx/access')"
    namespace:
      - {{labels.data_stream.namespace}}
      - default
  
# Route K8's container logs to this catch-all dataset for further routing
- source_dataset: k8s.router
  - target_dataset: nginx
    if: "ctx?.container?.image?.name == 'nginx'"
    namespace:
      - {{labels.data_stream.namespace}}
      - default
   
  # Route syslog entries tagged with nginx to this catch-all dataset
- source_dataset: syslog
  - target_dataset: nginx
    if: "ctx?.tags?.contains('nginx')"
    namespace:
      - {{labels.data_stream.namespace}}
      - default

[zmoog]

These routing rules are going to be defined in its own file in each data stream folder.

@mrodm the data_stream/rules/routing_rules.yml file replaces the routing_rules attribute in the manifest.yml file, right?

[mrodm]

These routing rules are going to be defined in its own file in each data stream folder.

@mrodm the data_stream/rules/routing_rules.yml file replaces the routing_rules attribute in the manifest.yml file, right?

@zmoog Exactly, routing rules are going to be defined in its own file in each datastream (not in the manifest of the datastream):
https://github.com/elastic/package-spec/blob/991ae115d7ac50175902072d2e3a4bb685f36c02/spec/integration/data_stream/routing_rules.spec.yml