[Elastic Agent] Create a new Custom log integration's that relies on filestream

[nimarezainia]

log input is being deprecated and superseded by filestream. Custom log integration is still using log input and we would need a new integration relying on filestream.

Acceptance Criteria:

• The already existing Custom log integration should be renamed to something like - Custom log (legacy)
• A new Custom log integration should be created based on a filestream

[pierrehilbert]

Hey Nima,
Could you please fill a bit more the issue with the Definition of Done?

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[nimarezainia]

@cmacknz are you happy with the estimate on this?

[cmacknz]

I think so, but it may change once someone takes a closer look at the changes that are needed.

The complexity here comes from the fact that if we change the input type every existing custom logs input will re-ingest previously ingested data. I think the filestream take over mode can address this but I don't think we've tried it with an integration yet. We'll need someone to investigate the best approach here.

[mbudge]

We need these filestream settings to be accessible in Fleet UI

scan_frequency
ignore_older
close_inactive
harvester_limit
prospector.scanner.include_files

parsers:
- multiline:
type: pattern
max_lines: 1000
pattern: '^======'
negate: true
match: after
flush_pattern: '======

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[nimarezainia]

To minimize disruption to existing deployments we should have a legacy log input as the default and an option for the user to enable a filestream input as well.

[strawgate]

@nimarezainia will review with @jlind23 tomorrow and report back if we can proceed with this plan!

[mbudge]

Any update?

[andrewkroh]

Relates: https://github.com/elastic/ingest-dev/issues/4932

[strawgate]

We're looking into making a Filestream integration available and marking the existing custom log integration as legacy (but still available). We will keep this issue open to track interest in a more seamless migration capability.

[jlind23]

@strawgate @nimarezainia @pierrehilbert updated this issue description and title.

[nimarezainia]

I created #5370 a few days ago. We believe that adding a new filestream custom integration is the better approach.

cc: @flexitrev

[jlind23]

Thanks @nimarezainia i'll close this one as duplicate.