[Fleet] Provide docs for experimental indexing features

[kpollich]

Ref elastic/kibana#132818

Fleet currently supports experimental indexing features behind a feature flag and via some (contentious) UI toggles. We'd like to remove the toggles entirely and simply allow advanced users (or those trialing prerelease integrations) to enable/disable these features via the component template API.

We'd like to add some docs for how to do this, though it may make sense to hide them as they aren't designed for general consumption. The approach here will be for the observability team to work with certain users to trial new versions of integrations that make heavy use of these features, and they need some docs they can link to to get some of the setup out of the way.

I've drafted some docs below:

---

Enabling and disabling experimental indexing features for Fleet-managed data streams

Fleet provides support for several experimental features around its data streams including

• Synthetic _source - Link
• Time-series data streams (TSDS, or TSDB) - Link

These features can be enabled and disabled for Fleet-managed data streams via the index template API and a few key settings.

Note: If you are already making use of @custom component templates for ingest/retention customization (e.g. Tutorial: Customize data retention policies), you should exercise care and ensure you don't overwrite your customizations when making these requests.

It is recommended to execute the following requests via Kibana dev tools. Replace <NAME> with the name of a given integration data stream, e.g. metrics-nginx.stubstatus would result in making a PUT request to _component_template/metrics-nginx.stubstatus@custom. Use the index management interface to explore what integration data streams are available to you.

Once you've executed a given request below, you'll also need to execute a data stream rollover to ensure any incoming data is ingested with your new settings immediately, e.g.

POST metrics-nginx.stubstatus/_rollover

Enabling synthetic _source

PUT _component_template/<NAME>@custom

{
  "template": {
    "mappings": {
      "_source": {
        "mode": "synthetic"
      }
    }
  }
}

Disabling synthetic _source

PUT _component_template/<NAME>@custom

{
  "template": {
    "mappings": {
      "_source": {}
    }
  }
}

Enabling TSDS

Note: TSDS utilizes synthetic _source under the hood, so if you want to trial both features you only need to enable TSDS

TSDS must be enabled at the index template level, due to restrictions in the Elasticsearch API. So, you'll need to make some sequential requests to enable or disable TSDB.

# Use the resulting JSON payload from this request to populate the following PUT request
GET _index_template/<NAME>

PUT _index_template/<NAME>
{
  # You can copy/paste this directly from the GET request above
  "index_patterns": [
    "<index pattern from GET request>"
  ],

  # Make sure this is added
  "template": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    }
  },

  # You can copy/paste this directly from the GET request above
  "composed_of": [ 
    "<NAME>@package",
    "<NAME>@custom",
    ".fleet_globals-1",
    ".fleet_agent_id_verification-1"
  ],

  # You can copy/paste this directly from the GET request above
  "priority": 200,

  # Make sure this is added
  "data_stream": {
    "allow_custom_routing": false
  }
}

Disabling TSDS

Disabling TSDS follows the same procedure as above, but specifying null for index.mode instead of time_series, e.g.

# Use the resulting JSON payload from this request to populate the following PUT request
GET _index_template/<NAME>

PUT _index_template/<NAME>
{
  # You can copy/paste this directly from the GET request above
  "index_patterns": [
    "<index pattern from GET request>"
  ],

  # Make sure this is added
  "template": {
    "settings": {
      "index": {
        "mode": null
      }
    }
  },

  # You can copy/paste this directly from the GET request above
  "composed_of": [ 
    "<NAME>@package",
    "<NAME>@custom",
    ".fleet_globals-1",
    ".fleet_agent_id_verification-1"
  ],

  # You can copy/paste this directly from the GET request above
  "priority": 200,

  # Make sure this is added
  "data_stream": {
    "allow_custom_routing": false
  }
}

cc @andresrc @mlunadia @lalit-satapathy @ruflin

[lalit-satapathy]

@kpollich.

Testing these recommendations as-is on 8.8.0-snapshot, finding errors. Can we cross-test and update the recommendation, so that they can be blindly copied to work as-is for any end user?

• Enabling TSDS:
  Errors "reason": "[index.mode=time_series] requires a non-empty [index.routing_path]"

       "index": { 
         "mode": "time_series"
      }

• Disabling TSDS:
  Errors for "null": illegal_argument_exception

      "index": { 
        "mode": "null"
      }

But works for: null

      "index": { 
         "mode": null
      }

[kpollich]

Errors "reason": "[index.mode=time_series] requires a non-empty [index.routing_path]"

Which integration was TSDB attempted to be enabled for? I'd like to try and reproduce this. This sounds like it may be intended behavior based on the manifest for the integration.

Errors for "null": illegal_argument_exception

Fixed in the description above.

[lalit-satapathy]

Which integration was TSDB attempted to be enabled for? I'd like to try and reproduce this. This sounds like it may be intended behavior based on the manifest for the integration.

Trying on nginx latest package; this is the specific command used, Let me know, if I should be following different steps? Can you check the same once.

PUT /_component_template/metrics-nginx.stubstatus@custom
{
  "template": {
    "settings": { 
      "index": { 
         "mode": "time_series"
      }
    }
  }
}

Fixed in the description above.

Thanks!

[kpollich]

I see the same behavior on the nginx package. Fleet doesn't actually generate the index.routing_path value here - Elasticsearch is supposed to generate it automatically based on the dimension fields present in the mappings for a given data stream.

@juliaElastic @nchaulet - you both worked in this area. Does anything stand out about the Nginx package in particular that might ring a bell here? I took a look and I think this may be a bug we need to look into on the Fleet side but I'm not 100% sure.

[nchaulet]

@kpollich actually Fleet do not put the mode: 'time_series' when using the experimental feature in the component template but in the index template because of that, it maybe something to fix at the elasticsearch level

[kpollich]

Seems like I run into the same error trying to update the index template, e.g.

PUT /_index_template/metrics-nginx.stubstatus
{
  "index_patterns": [
    "metrics-nginx.stubstatus-*"
  ],
  "template": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    }
  }
}


...


{
  "error": {
    "root_cause": [
      {
        "type": "invalid_index_template_exception",
        "reason": "index_template [metrics-nginx.stubstatus] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]"
      }
    ],
    "type": "invalid_index_template_exception",
    "reason": "index_template [metrics-nginx.stubstatus] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]"
  },
  "status": 400
}

[nchaulet]

@kpollich I think you may two issue here the PUT need to have the whole index template with the composed_of and you need to have at least one time_series_dimension: true property in your mapping.

PUT /_component_template/metrics-nginx.stubstatus@custom
{
  "template": {
    "mappings": {
      "properties": {
        "test_dimension": {
                    "time_series_dimension": true,
                    "type": "keyword"
                }
      }
    }
  },
  "version": 123
}

PUT /_index_template/metrics-nginx.stubstatus
{
    
        "index_patterns": [
          "metrics-nginx.stubstatus-*"
        ],
        "template": {
          "settings": {
            "index": {
        "mode": "time_series"
      }
          },
          "mappings": {
            "_meta": {
              "package": {
                "name": "nginx"
              },
              "managed_by": "fleet",
              "managed": true
            }
          }
        },
        "composed_of": [
          "metrics-nginx.stubstatus@package",
          "metrics-nginx.stubstatus@custom",
          ".fleet_globals-1",
          ".fleet_agent_id_verification-1"
        ],
        "priority": 200,
        "_meta": {
          "package": {
            "name": "nginx"
          },
          "managed_by": "fleet",
          "managed": true
        },
        "data_stream": {
          "hidden": false,
          "allow_custom_routing": false
        }
    
    }

[kpollich]

Got it - thanks. I'll update the description here to make this clear.

[kpollich]

you need to have at least one time_series_dimension: true property in your mapping

This seems to be the root cause of the issue on the nginx integration. There's no time_series_dimension: true fields coming through in the version of Nginx I'm seeing in EPR/integrations UI which is 1.11.0. Seems like there are several dimension: true fields defined by Nginx as of elastic/integrations#5315. However the mappings generated in the metrics-nginx.stubstatus@package component template don't contain any time_series_dimension: true denotations. According to the package spec this is supposed to be the case.

Looking at the code, Fleet will only append time_series_dimension: true if index_mode: "time_series" is set for the given datastream or if the experimental feature toggle for TSDB is enabled. So, on-the-fly enabling of TSDB is challenging because these settings aren't actually present on the mappings.

@nchaulet - Do you think we could change Fleet such that we always generate time_series_dimension: true if dimension: true is set in a fields.yml file, regardless of whether index_mode: "time_series" is set? I'm not sure if this causes Elasticsearch breakages. I'm going to try it out regardless just curious for your thoughts here. It'd be nice if we could leave the dimension/metrics settings in place and if they were ignored by Elasticsearch until index_mode: "time_series" was set.

[nchaulet]

@nchaulet - Do you think we could change Fleet such that we always generate time_series_dimension: true if dimension: true is set in a fields.yml file, regardless of whether index_mode: "time_series" is set? I'm not sure if this causes Elasticsearch breakages. I'm going to try it out regardless just curious for your thoughts here. It'd be nice if we could leave the dimension/metrics settings in place and if they were ignored by Elasticsearch until index_mode: "time_series" was set.

We could try it we had some issues with adding time_series_dimension: true to non time_series index before, so it may be interesting to try to install all the package to see if there any other issues here elastic/kibana#127327

[kpollich]

I removed the conditions to prevent dimension/metric settings from being generated for mappings unless index_mode: "time_series" was set and it's seemed to have no ill effects on data ingestion for the system integration. I was able to install the Kubernetes integration as well with no issues. I'll put up a PR to see how CI responds.

[nchaulet]

It may be worth running the node scripts/install_all_packages in fleet against your PR

[lalit-satapathy]

This seems to be the root cause of the issue on the nginx integration

The issue seems to occur in other integrations with dimensions (ex: oracle). You can check completed packages here.

[ruflin]

I removed the conditions to prevent dimension/metric settings from being generated for mappings unless index_mode: "time_series" was set and it's seemed to have no ill effects on data ingestion for the system integration. I was able to install the Kubernetes integration as well with no issues. I'll put up a PR to see how CI responds.

I remember it had some side effects in Elasticsearch on older versions but as Kibana and Elasticsearch should be on the same version, this should work as expected.

@kpollich Being able to roll back is critical for enabling TSDB. As the toggles are not "visible" anymore, the custom path must work.

[ruflin]

Follow up thought: To make sure we are aligned, the expectation is that before and after enabling / disabling TSDB, the template is 100% the same, only the setting changes. This applies to dimensions but also the metric types and potential other things we add.

[lalit-satapathy]

Hi @kpollich,

Can you help summarise where we stand today, on the following?

• Manual process to enable/disable TSDB via dev tools?
• Any issues, on the field mapping; When we upgrade a package from tsdb disabled to tsdb enabled or upgrade a package from tsdb enabled to tsdb disabled?

[kpollich]

Hi @lalit-satapathy, I'll summarize below.

I've filed elastic/integrations#6128 which fixes invalid mappings on the GCP and Elastic Package Registry integrations. I'm waiting on codeowner review from obs-cloud-monitoring and the SEI teams before I can merge this.

The PR above blocks elastic/kibana#157047 which is the root cause fix for the enable/disable TSDB issue we see here. I've pinged both teams above for codeowner review but will ping in Slack if I don't hear anything soon.

Any issues, on the field mapping; When we upgrade a package from tsdb disabled to tsdb enabled or upgrade a package from tsdb enabled to tsdb disabled?

I am not sure what you're asking here. We haven't discovered any new issues as far as I'm aware.

[kpollich]

We haven't discovered any new issues as far as I'm aware.

I should mention elastic/kibana#157345 actually, which is still in the early stages but I believe the team has tracked down the root issue.

[lalit-satapathy]

The PR above blocks elastic/kibana#157047 which is the root cause fix for the enable/disable TSDB issue we see here. I've pinged both teams above for codeowner review but will ping in Slack if I don't hear anything soon.

Can we use the "Disable TSDS" manual option now, since this is fixed?

[kpollich]

Posting a long explanation around still getting the routing_path error on the latest after our PR's

GET _component_template/metrics-nginx.stubstatus@custom

{
  "component_templates": [
    {
      "name": "metrics-nginx.stubstatus@custom",
      "component_template": {
        "template": {
          "settings": {}
        },
        "_meta": {
          "package": {
            "name": "nginx"
          },
          "managed_by": "fleet",
          "managed": true
        }
      }
    }
  ]
}

PUT _component_template/metrics-nginx.stubstatus@custom
{
  "template": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    }
  }
}

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "[index.mode=time_series] requires a non-empty [index.routing_path]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "[index.mode=time_series] requires a non-empty [index.routing_path]"
  },
  "status": 400
}

Seems like Elasticsearch isn't inferring index.routing_path as we expect in this case, even though the nginx integration defines time series dimensions and metrics now. e.g.

GET _component_template/metrics-nginx.stubstatus@package

{
  "component_templates": [
    {
      "name": "metrics-nginx.stubstatus@package",
      "component_template": {
        "template": {
          "settings": {
            "index": {
              "lifecycle": {
                "name": "metrics"
              },
              "codec": "best_compression",
              "default_pipeline": "metrics-nginx.stubstatus-1.11.1",
              "mapping": {
                "total_fields": {
                  "limit": "10000"
                }
              },
              "query": {
                "default_field": [
                  "cloud.account.id",
                  "cloud.availability_zone",
                  "cloud.instance.id",
                  "cloud.instance.name",
                  "cloud.machine.type",
                  "cloud.provider",
                  "cloud.region",
                  "cloud.project.id",
                  "cloud.image.id",
                  "container.id",
                  "container.image.name",
                  "container.name",
                  "host.architecture",
                  "host.hostname",
                  "host.id",
                  "host.mac",
                  "host.name",
                  "host.os.family",
                  "host.os.kernel",
                  "host.os.name",
                  "host.os.platform",
                  "host.os.version",
                  "host.os.build",
                  "host.os.codename",
                  "host.type",
                  "agent.id",
                  "ecs.version",
                  "service.address",
                  "service.type",
                  "nginx.stubstatus.hostname"
                ]
              }
            }
          },
          "mappings": {
            "dynamic_templates": [
              {
                "container.labels": {
                  "path_match": "container.labels.*",
                  "mapping": {
                    "type": "keyword"
                  },
                  "match_mapping_type": "string"
                }
              }
            ],
            "properties": {
              "cloud": {
                "properties": {
                  "availability_zone": {
                    "time_series_dimension": true,
                    "type": "keyword"
                  },
                  "image": {
                    "properties": {
                      "id": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      }
                    }
                  },
                  "instance": {
                    "properties": {
                      "name": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "id": {
                        "time_series_dimension": true,
                        "type": "keyword"
                      }
                    }
                  },
                  "provider": {
                    "time_series_dimension": true,
                    "type": "keyword"
                  },
                  "machine": {
                    "properties": {
                      "type": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      }
                    }
                  },
                  "project": {
                    "properties": {
                      "id": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      }
                    }
                  },
                  "region": {
                    "time_series_dimension": true,
                    "type": "keyword"
                  },
                  "account": {
                    "properties": {
                      "id": {
                        "time_series_dimension": true,
                        "type": "keyword"
                      }
                    }
                  }
                }
              },
              "container": {
                "properties": {
                  "image": {
                    "properties": {
                      "name": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      }
                    }
                  },
                  "name": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "id": {
                    "time_series_dimension": true,
                    "type": "keyword"
                  }
                }
              },
              "agent": {
                "properties": {
                  "id": {
                    "time_series_dimension": true,
                    "type": "keyword"
                  }
                }
              },
              "@timestamp": {
                "type": "date"
              },
              "ecs": {
                "properties": {
                  "version": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "nginx": {
                "properties": {
                  "stubstatus": {
                    "properties": {
                      "hostname": {
                        "time_series_dimension": true,
                        "type": "keyword"
                      },
                      "current": {
                        "time_series_metric": "gauge",
                        "meta": {},
                        "type": "long"
                      },
                      "waiting": {
                        "time_series_metric": "gauge",
                        "meta": {},
                        "type": "long"
                      },
                      "accepts": {
                        "time_series_metric": "counter",
                        "meta": {},
                        "type": "long"
                      },
                      "handled": {
                        "time_series_metric": "counter",
                        "meta": {},
                        "type": "long"
                      },
                      "writing": {
                        "time_series_metric": "gauge",
                        "meta": {},
                        "type": "long"
                      },
                      "dropped": {
                        "time_series_metric": "counter",
                        "meta": {},
                        "type": "long"
                      },
                      "active": {
                        "time_series_metric": "gauge",
                        "meta": {},
                        "type": "long"
                      },
                      "reading": {
                        "time_series_metric": "gauge",
                        "meta": {},
                        "type": "long"
                      },
                      "requests": {
                        "time_series_metric": "counter",
                        "meta": {},
                        "type": "long"
                      }
                    }
                  }
                }
              },
              "data_stream": {
                "properties": {
                  "namespace": {
                    "type": "constant_keyword"
                  },
                  "type": {
                    "type": "constant_keyword"
                  },
                  "dataset": {
                    "type": "constant_keyword"
                  }
                }
              },
              "service": {
                "properties": {
                  "address": {
                    "time_series_dimension": true,
                    "type": "keyword"
                  },
                  "type": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "host": {
                "properties": {
                  "hostname": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "os": {
                    "properties": {
                      "build": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "kernel": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "codename": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "name": {
                        "ignore_above": 1024,
                        "type": "keyword",
                        "fields": {
                          "text": {
                            "type": "text"
                          }
                        }
                      },
                      "family": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "version": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "platform": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      }
                    }
                  },
                  "domain": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "ip": {
                    "type": "ip"
                  },
                  "containerized": {
                    "type": "boolean"
                  },
                  "name": {
                    "time_series_dimension": true,
                    "type": "keyword"
                  },
                  "id": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "type": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "mac": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "architecture": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "event": {
                "properties": {
                  "module": {
                    "type": "constant_keyword",
                    "value": "nginx"
                  },
                  "dataset": {
                    "type": "constant_keyword",
                    "value": "nginx.stubstatus"
                  }
                }
              }
            }
          }
        },
        "_meta": {
          "package": {
            "name": "nginx"
          },
          "managed_by": "fleet",
          "managed": true
        }
      }
    }
  ]
}

I tried setting this on the index template level as well with the same error

PUT _index_template/metrics-nginx.stubstatus
{
  "index_patterns": [
    "metrics-nginx.stubstatus-*"
  ],
  "template": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    }
  },
  "composed_of": [
    "metrics-nginx.stubstatus@package",
    "metrics-nginx.stubstatus@custom",
    ".fleet_globals-1",
    ".fleet_agent_id_verification-1"
  ]
}

{
  "error": {
    "root_cause": [
      {
        "type": "invalid_index_template_exception",
        "reason": "index_template [metrics-nginx.stubstatus] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]"
      }
    ],
    "type": "invalid_index_template_exception",
    "reason": "index_template [metrics-nginx.stubstatus] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]"
  },
  "status": 400
}

The same thing happens in the UI when I try to enable TSDB for this data stream via the toggles

{
  "index_patterns": [
    "metrics-nginx.stubstatus-*"
  ],
  "template": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    }
  },
  "composed_of": [
    "metrics-nginx.stubstatus@package",
    "metrics-nginx.stubstatus@custom",
    ".fleet_globals-1",
    ".fleet_agent_id_verification-1"
  ],
  "_meta": {
    "has_experimental_data_stream_indexing_features": false
  }
}
[2023-05-12T11:27:28.782-04:00][ERROR][plugins.fleet] ResponseError: invalid_index_template_exception
	Root causes:
		invalid_index_template_exception: index_template [metrics-nginx.stubstatus] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]
    at KibanaTransport.request (/Users/kylepollich/Workspace/elastic/kibana/node_modules/@elastic/transport/src/Transport.ts:535:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

So, either something has regressed in Elasticsearch around the inference of the index.routing_path or this has never actually worked (which I find hard to believe as we extensively tested this in prior releases).

Interestingly, I was able to toggle TSDB on for the metrics-system.cpu datastream via the UI.

GET _index_template/metrics-system.cpu

{
  "index_templates": [
    {
      "name": "metrics-system.cpu",
      "index_template": {
        "index_patterns": [
          "metrics-system.cpu-*"
        ],
        "template": {
          "settings": {
            "index": {
              "mode": "time_series"
            }
          },
          "mappings": {
            "_meta": {
              "package": {
                "name": "system"
              },
              "managed_by": "fleet",
              "managed": true
            }
          }
        },
        "composed_of": [
          "metrics-system.cpu@package",
          "metrics-system.cpu@custom",
          ".fleet_globals-1",
          ".fleet_agent_id_verification-1"
        ],
        "priority": 200,
        "_meta": {
          "has_experimental_data_stream_indexing_features": true
        },
        "data_stream": {
          "hidden": false,
          "allow_custom_routing": false
        }
      }
    }
  ]
}

However executing the exact same request via dev tools results in a failure

PUT _index_template/metrics-system.cpu
{
  "index_patterns": [
    "metrics-system.cpu-*"
  ],
  "template": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    },
    "mappings": {
      "_meta": {
        "package": {
          "name": "system"
        },
        "managed_by": "fleet",
        "managed": true
      }
    }
  },
  "composed_of": [
    "metrics-system.cpu@package",
    "metrics-system.cpu@custom",
    ".fleet_globals-1",
    ".fleet_agent_id_verification-1"
  ],
  "priority": 200
}

{
  "error": {
    "root_cause": [
      {
        "type": "invalid_index_template_exception",
        "reason": "index_template [metrics-system.cpu] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]"
      }
    ],
    "type": "invalid_index_template_exception",
    "reason": "index_template [metrics-system.cpu] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]"
  },
  "status": 400
}

And once I get this error from dev tools, I can no longer toggle the TSDB setting via the UI

Looking at the two payloads (one previously successful generated by the Fleet API and one from dev tools), there's one major difference: the one generated by the TSDB toggle/Fleet API has

"data_stream": {
    "hidden": false,
    "allow_custom_routing": false
  }

If you include this, everything works as expected. So, Elasticsearch won't generate index.routing_path unless allow_custom_routing is specified explicitly, it seems.

[kpollich]

tl;dr of the above: we need to update the API requests to

• Operate directly on the index template
• Include datastream.allow_custom_routing: false in the request body

I'll take care of this now and verify everything is working end-to-end

[kpollich]

Updated description + tested synthetic source + TSDS instructions. Everything worked as expected. cc @lalit-satapathy

[lalit-satapathy]

Updated description + tested synthetic source + TSDS instructions. Everything worked as expected. cc @lalit-satapathy

Testing manual TSDB disabling for nginx. TSDB disablement and rollover seems to work fine.
@kpollich Let's know, if the steps below looks fine.

• Install package with TSDB enabled.
• In "Dev Tools"
  • GET _index_template/metrics-nginx.stubstatus
  • Copy the only relevant sections from the GET command as-below and make index.mode as NULL.
  • Disable TSDB using PUT _index_template/metrics-nginx.stubstatus

{
  "index_patterns": [
      "metrics-nginx.stubstatus-*"
  ],

  "template": {
    "settings": {
      "index": {
        "mode": null
      }
    }
  },

  "composed_of": [ 
    "metrics-nginx.stubstatus@package",
    "metrics-nginx.stubstatus@custom",
    ".fleet_globals-1",
    ".fleet_agent_id_verification-1"
  ],

  "priority": 200,

  "data_stream": {
    "allow_custom_routing": false
  }
}

• Do index rollover using POST metrics-nginx.stubstatus-default/_rollover

CC @tetianakravchenko to check on k8 package.

[kpollich]

@lalit-satapathy - Looks great to me. Thanks for testing and for your patience as we fix up bugs here 🙏

[jlind23]

@karenzone could you please add this to your radar for your next sprint content?

[jlind23]

@kilfoyle could you please start working on this issue as soon as you can? We are close from the TSDB official launch and it would be great to see this documented.

[kilfoyle]

@jlind23 Will do. 👍

[mlunadia]

Thanks @kilfoyle can you keep @lalit-satapathy in the loop or ping him if you need help, our priority is having a place we can send customers to show how to disable TSDB should a problem occur.

[andresrc]

@kilfoyle do we have any ETA for this? We have the first packages ready to ship.

[kilfoyle]

@kpollich, @lalit-satapathy, the docs page is ready for review via PR #227 There's a preview link in the PR description.

@andresrc I'll have the docs PR merged as soon as I can (ideally today or Monday) and will update this issue.

[kilfoyle]

The docs page is live:
Enabling and disabling advanced indexing features for Fleet-managed data streams

I assume we can close this issue now. I'm happy to help with whatever revisions we need.