Add Filestream integration

[belimawr]

Proposed commit message

This commit adds the Filestream integration

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• Use fingerprint as default file identity

How to test this PR locally

1. Build the integration
2. Start a stack with Elastic-Package or upload the integration to an existing stack
3. Go to the "Add integrations page"
4. Make sure "Display beta integrations" is selected
5. Add the "Custom Filestream integration", configure paths to read from any text file
6. Deploy the Elastic-Agent enrolled in this policy
7. Ensure the data from the file is ingested.

The Custom Filestream integration uses the fingerprint file identity by default, so make sure the file used for testing has a size of at least 1024 bytes or change the fingerprint length in the integration settings.

Related issues

• Closes [Elastic Agent] Add a new Custom Filestream integration elastic-agent#5370

Screenshots

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[leehinman]

Looking good.

Couple of questions in-line. The biggest is on field types. We probably need to double check other integrations to make sure they are defined the same across all of them. We don't want any mismatches.

[belimawr]

Looking good.

Couple of questions in-line. The biggest is on field types. We probably need to double check other integrations to make sure they are defined the same across all of them. We don't want any mismatches.

I checked them and they're all keyword in other integrations, thanks for spotting that!

[strawgate]

Can we document that users migrating from a log integration can temporarily use ignore_inactive: since_first_start to set the offset in the registry to the end of discovered files?

[belimawr]

Can we document that users migrating from a log integration can temporarily use ignore_inactive: since_first_start to set the offset in the registry to the end of discovered files?

That's a good idea @strawgate ! Where do you think we should add it? To the overview or directly to the configuration option? Or should we have a migration guide (for an ideal migration we'd have to implement some extra fetures)?

[strawgate]

I think there's some prior art in having migration information in the overview/readme so that makes sense to me.

[belimawr]

I think there's some prior art in having migration information in the overview/readme so that makes sense to me.

I like the idea, I'm thinking if we should involve the docs team and questioning what is the best way to communicate it.

I tried to write something, however we're not at a point where we can provide provide a migration guide without data loss, while those options can help with preventing, or reducing, data duplication, there is still the possibility from data loss between the last line ingested by the Log input and when Filestream actually starts. So I fear it could cause confiscation among our users.

We could merge the PR as is and work on a better documentation for migration that will be released when it goes GA or close to it. What do you think?

[strawgate]

When I made a version of this in the past I landed on including language like:

As Filestream configures a new input, configuring it to collect from a file that was previously collected by Custom Logs (Legacy) will result in duplicate data. You may wish to configure ignore_older or temporarily set ignore_inactive: since_first_start to limit the amount of duplicate data collected.

I think we could:

1. Start with something like that and refine later with docs links, or
2. At least include a warning about duplicate data

Do we already have an issue created for improving the migration information?

[belimawr]

I love the wording @strawgate ! I'll add it to the overview page and create an issue to follow up on improving it.

[belimawr]

@strawgate, I've added this bit at the end of the README (89429ce):

As Filestream configures a new input, configuring it to collect data
from a file that was previously collected by Custom Logs integration
will result in duplicate data. You may wish to configure
ignore_older or temporarily set ignore_inactive: since_first_start
to limit the amount of duplicate data collected.

If the Custom Logs integration is removed and the Custom Filestream
Logs is added in the same policy change, there risk of data being
missed between the last entry ingested by the Custom Logs and the
first one ingested by the Custom Filestream Logs.

What do you think?

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: aa58639

History

• 💚 Build #18285 succeeded 29cb21c
• 💚 Build #18246 succeeded aa02680
• 💔 Build #18073 failed 7ee3b84
• 💔 Build #18020 failed 7ee3b84
• 💔 Build #17988 failed cff0f45
• 💔 Build #17644 failed a5e9ef55022e008cf073429ce5369425793f5305

cc @belimawr

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[strawgate]

That seems good to me. Though I think we should rename Custom Logs to Custom Logs (Legacy) or something similar -- @kpollich ? If we are going to rename it we should reference the updated name in the readme here too.

[belimawr]

My understanding is that we'll rename the Custom Logs integration once the Custom Filestream is GA, this PR releases it in technical preview.

[elastic-vault-github-plugin-prod]

Package filestream - 0.0.1 containing this change is available at https://epr.elastic.co/package/filestream/0.0.1/

[kpollich]

+1 on renaming to Custom Logs (Legacy) once the filestream integration is GA