Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add filesource provider #6587

Merged
merged 8 commits into from
Jan 28, 2025
Merged

Conversation

blakerouse
Copy link
Contributor

@blakerouse blakerouse commented Jan 23, 2025

What does this PR do?

Adds the filesource provide to composable.

The provider watches for changes of the files and updates the values of the variables when the content of the file changes.

Why is it important?

This allows information from the filesystem to be used as variables in the Elastic Agent configuration. This information is only allowed by explicitly configuring the provider to read this information from the disk. The policy cannot just read any file, it has to be explicitly configured to allow it.

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • [ ] I have added an integration test or an E2E test (covered by unit tests well)

Disruptive User Impact

None

How to test this PR locally

Create a configuration:

elastic-agent.yml

provides:
  filesource:
    sources:
      file1:
        path: ./file1

inputs:
 - id: filestream
   type: filestream
   paths:
     - ${filesource.file1}

echo "/var/log/syslog" > ./file1

Observe that syslog is set to paths.

Related issues

@blakerouse blakerouse added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team backport-skip labels Jan 23, 2025
@blakerouse blakerouse self-assigned this Jan 23, 2025
@blakerouse blakerouse marked this pull request as ready for review January 23, 2025 21:12
@blakerouse blakerouse requested a review from a team as a code owner January 23, 2025 21:12
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

Copy link
Contributor

@kaanyalti kaanyalti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changes look good to me

Copy link
Contributor

@pkoutsovasilis pkoutsovasilis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with a note: I’d prefer a design with more isolation between event consumption and the file content reading. For instance, a goroutine could handle fsnotify events and mark map entries as “dirty,” while another periodically processes these at its own pace. This would mitigate risks like event overflow or frequent re-rendering caused by rapidly changing files (which even a malicious entity could try to exploit).

That said, let’s remain profile-driven - if issues arise, we can iterate and refine the solution as needed.

@blakerouse blakerouse merged commit 69e7612 into elastic:main Jan 28, 2025
14 checks passed
@blakerouse blakerouse deleted the filesource-provider branch January 28, 2025 14:35
@cmacknz
Copy link
Member

cmacknz commented Jan 29, 2025

You can file an issue in https://github.com/elastic/ingest-docs describing how this works to get it included in the provider documentation, assuming we want this available outside of internal use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-skip Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add filesource provider to composable inputs
6 participants