diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md deleted file mode 100644 index 5dec55d6c..000000000 --- a/CODE_OF_CONDUCT.md +++ /dev/null @@ -1,5 +0,0 @@ -# Stanza Community Code of Conduct - -Stanza project follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). - -Please report violations of the Code of Conduct to a Stanza [maintainer](MAINTAINERS.md). \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 38f403b07..c1f8da4b2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,29 +7,26 @@ You can view and edit the source code by cloning this repository: ```bash git clone https://github.com/observiq/stanza.git ``` + ## Pull Requests -### How to Send Pull Requests +### How to Submit Pull Requests -Everyone is welcome to contribute code to `stanza` via -GitHub pull requests (PRs). +Everyone is welcome to contribute code to `stanza` via GitHub pull requests (PRs). -To create a new PR, fork the project in GitHub and clone the upstream -repo: +To create a new PR, fork the project in GitHub and clone the upstream repo: ```sh $ go get -d github.com/observiq/stanza ``` -This will put the project in `${GOPATH}/src/github.com/observiq/stanza`. You -can alternatively use `git` directly with: +This will put the project in `${GOPATH}/src/github.com/observiq/stanza`. You can alternatively use `git` directly with: ```sh $ git clone https://github.com/observiq/stanza ``` -This would put the project in the `stanza` directory in -current working directory. +This would put the project in the `stanza` directory in current working directory. Enter the newly created directory and add your fork as a new remote: @@ -37,8 +34,7 @@ Enter the newly created directory and add your fork as a new remote: $ git remote add git@github.com:/opentelemetry-go ``` -Check out a new branch, make modifications, run linters and tests, and -push the branch to your fork: +Check out a new branch, make modifications, run linters and tests, and push the branch to your fork: ```sh $ git checkout -b @@ -51,13 +47,14 @@ $ git push Open a pull request against the main `stanza` repo. + ### How to Receive Comments -* If the PR is not ready for review, please put `[WIP]` in the title, - tag it as `work-in-progress`, or mark it as - [`draft`](https://github.blog/2019-02-14-introducing-draft-pull-requests/). +* If the PR is not ready for review, please put `[WIP]` in the title, tag it as `work-in-progress`, or mark it as [`draft`](https://github.blog/2019-02-14-introducing-draft-pull-requests/). +* If you're stuck, tag a maintainer and ask a question. We're here to help each other. * Make sure CI passes. + ### How to Get PRs Merged A PR is considered to be **ready to merge** when: @@ -66,14 +63,15 @@ A PR is considered to be **ready to merge** when: * CI passes. * Major feedback is resolved. + ## Design Choices -Best practices for developing a builtin operator are documented below, but for changes to -the core agent, we are happy to discuss proposals in the issue tracker. +Best practices for developing a builtin operator are documented below, but for changes to the core agent, we are happy to discuss proposals in the issue tracker. + -### Builtin Operator Development +### Operator Development -In order to write a builtin operator, follow these three steps: +In order to write an operator, follow these three steps: 1. Build a unique struct that satisfies the [`Operator`](operator/operator.go) interface. This struct will define what your operator does when executed in the pipeline. ```go @@ -108,10 +106,11 @@ func init() { } ``` -## Any tips for building operators? + +#### Any tips for building operators? We highly recommend that developers take advantage of [helpers](operator/helper) when building their operators. Helpers are structs that help satisfy common behavior shared across many operators. By embedding these structs, you can skip having to satisfy certain aspects of the `operator` and `config` interfaces. -For example, almost all operators should embed the [BasicOperator](operator/helper/basic_operator.go) helper, as it provides simple functionality for returning an operator id and operator type. +For example, almost all operators should embed the [BasicOperator](operator/helper/operator.go) helper, as it provides simple functionality for returning an operator id and operator type. ```go // ExampleOperator is a basic operator, with a basic lifecycle, that consumes diff --git a/GOVERNANCE.md b/GOVERNANCE.md new file mode 100644 index 000000000..835dd6d7b --- /dev/null +++ b/GOVERNANCE.md @@ -0,0 +1,44 @@ +# Governance + +The governance model adopted in Stanza is influenced by many CNCF projects. + +## Principles + +- **Open**: Stanza is an open source community. +- **Welcoming and respectful**: Stanza follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). +- **Transparent and accessible**: Work and collaboration should be done in public. +- **Merit**: Ideas and contributions are accepted according to their technical merit and alignment with project objectives, scope, and design principles. + +## Project Maintainers + +Maintainers are the first and foremost contributors that are committed to the success of Stanza project. Each maintainer should fulfill several of the following responsibilities: + +* Strive to make stanza widely while taking care not to unjustifiably inflate its scope. +* Carefully consider input from the community and seek consensus with amongst maintainers. +* Create and classify GitHub issues and perform pull request reviews for other maintainers and the community. +* Use ones best judgement to apply all applicable [labels](https://github.com/observiq/stanza/labels) to each new GitHub issue. +* Make sure that ongoing PRs are moving forward at the right pace or closing them if they are not moving in a productive direction. +* Create releases of the library as necessary. + +## Process of becoming a maintainer + +* Understand that maintainers are expected to have a deep understanding of the project and that they should be able to independently fulfill maintainer responsibilities. +* Talk to some of the existing [maintainers](MAINTAINERS.md). Explain that you are interested in becoming a maintainer and why you think you are a good fit for the role. +* Contribute increasingly complicated PRs under the guidance of the existing maintainers. +* When a maintainer believes you are ready, he or she may nominate you by creating a PR to add your to the list in [MAINTAINERS.md](MAINTAINERS.md). +* Once the existing maintainers have reached consensus that the nominatee is ready for the role and approved the PR, it will be merged and the new maintainer becomes active. + +## Process of removing maintainer status + +* If a maintainer is no longer interested or cannot perform the maintainer duties listed above, they should volunteer to be moved to emeritus status. +* In extreme cases a PR may be created to remove a maintainer. This PR must be approved by a 2/3 majority of maintainers to be approved. + +## Decision making process + +Decisions are made based on consensus between maintainers. Proposals and ideas should be submitted as a github issue or PR. + +In general, we prefer that technical issues and maintainer membership are amicably worked out between the persons involved. If a dispute cannot be decided independently, then a third-party maintainer (e.g. a mutual contact with some background on the issue, but not involved in the conflict) to arbitrate and the final decision will be made. Decision making process should be transparent to adhere to the principles of Stanza project. + +## Credits + +Some contents in this documents have been borrowed from the [OpenYurt](https://github.com/alibaba/openyurt) project. diff --git a/MAINTAINERS.md b/MAINTAINERS.md index 27232c0d5..8e50a74ff 100644 --- a/MAINTAINERS.md +++ b/MAINTAINERS.md @@ -7,4 +7,5 @@ This file lists the maintainers of the Stanza project. The responsibilities of m | ---- | --------- | ----------- | | [Mike Kelly](mailto:mike.kelly@bluemedora.com) | [mkelly](https://github.com/mkelly) | observIQ | | [Dan Jaglowski](mailto:dan.jaglowski@bluemedora.com) | [djaglowski](https://github.com/djaglowski) | observIQ | -| [Camden Cheek](mailto:camden.cheek@bluemedora.com) | [camdencheek](https://github.com/camdencheek) | observIQ | \ No newline at end of file +| [Camden Cheek](mailto:camden.cheek@bluemedora.com) | [camdencheek](https://github.com/camdencheek) | observIQ | +| [Joseph Howell](mailto:joseph.howell@bluemedora.com) | [jhowellbm](https://github.com/jhowellbm) | observIQ | \ No newline at end of file diff --git a/README.md b/README.md index bc32a9f04..e03bf62d2 100644 --- a/README.md +++ b/README.md @@ -1,134 +1,34 @@ -# stanza -Stanza is a fast and lightweight log transport and processing agent. It's designed as a modern replacement for Fluentd, Fluent Bit, and Logstash. +

[![](https://circleci.com/gh/observIQ/stanza.svg?style=shield&circle-token=980a514f9dc5a48ac2b8e61a4cdb7555ea5646ca)](https://app.circleci.com/pipelines/github/observIQ/stanza) [![codecov](https://codecov.io/gh/observIQ/stanza/branch/master/graph/badge.svg)](https://codecov.io/gh/observIQ/stanza) [![Go Report Card](https://goreportcard.com/badge/github.com/observIQ/stanza)](https://goreportcard.com/report/github.com/observIQ/stanza) -## How do I install the agent? - -We recommend using our single-line installer provided with each release: - -### Linux -```shell -sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download/unix-install.sh)" unix-install.sh -``` -### macOS -```shell -sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download/unix-install.sh)" unix-install.sh -``` -### Windows -```pwsh -[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-Expression ((New-Object net.webclient).DownloadString('https://github.com/observiq/stanza/releases/latest/download/windows-install.ps1')); Log-Agent-Install -``` - -## How do I run the agent? - -If you installed the agent using the single-line installer above, it's already running as a service! If you'd like to start or stop the agent, here's how: - -### Linux -```shell -# systemd -systemctl start stanza -systemctl stop stanza - -# sysv -service stanza start -service stanza stop -``` -### macOS -```shell -launchctl start com.observiq.stanza -launchctl stop com.observiq.stanza -``` -### Windows -```pwsh -Start-Service -Name "stanza" -Stop-Service -Name "stanza" -``` - -### Manual - -If you'd like to run the agent manually rather than as a service, you can do that, too! - -```shell -# Example Command -stanza - -# Supported flags: ---config The location of the agent config file (default: ./config.yaml) ---plugin_dir The location of the plugins directory (default: ./plugins) ---database The location of the offsets database file. If this is not specified, offsets will not be maintained across agent restarts ---log_file The location of the agent log file. If not specified, stanza will log to `stderr` ---debug Enables debug logging -``` - -## How do I configure the agent? -A simple configuration file (config.yaml) is included in the installation. By default it doesn't do much, but is an easy way to get started. By default, it generates a single log entry and sends it to STDOUT every time the agent is restarted. - -```yaml -pipeline: - # An example input that generates a single log entry when Stanza starts up. - - type: generate_input - count: 1 - entry: - record: This is a sample log generated by Stanza -... - # An example output that sends captured logs to stdout. - - type: stdout -... -``` - -The first step in configuring stanza is to setup your output. The sample configuration provides examples for sending data to the Elastic stack or Google Cloud Monitoring. Uncomment the destination of your choice and add in your own credentials and host. Restart the agent to generate another log line. - -```yaml -... -pipeline: - # An example input that generates a single log entry when Stanza starts up. - - type: generate_input - count: 1 - entry: - record: This is a sample log generated by Stanza - - # An example output that sends captured logs to elasticsearch. - # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/elastic_output.md - - type: elastic_output - addresses: - - http://my_node_address:9200 - api_key: my_api_key -... -``` - -Once you've confirmed you're able to send a log entry, you'll want to connect stanza to a log file you're interested in monitoring. We've included a sample `file_input` configuration in the config file to use (stanza also provide options for UDP, TCP, syslog, and other input streams. They're available [here](./docs/README.md)). -```yaml -... -pipeline: -... - # An example input that monitors the contents of a file. - # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/file_input.md - - type: file_input - include: - - /sample/file/path - ... - - # An example output that sends captured logs to elasticsearch. - # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/elastic_output.md - - type: elastic_output - addresses: - - http://my_node_address:9200 - api_key: my_api_key -... -``` - -That's it! You should have logs streaming to Elasticsearch. From here you can explore all the options available within stanza! You can use existing plugins from our plugin repository or build your own custom pipelines. - -To learn more about configuration, visit our [docs](./docs/README.md). - -## How do I contribute? - -First, check out our section on [`getting started with development`](./docs/development.md) - -Next, take a look at our contribution guidelines in [`CONTRIBUTING.md`](./CONTRIBUTING.md) + +# About Stanza + +Stanza is a fast and lightweight log transport and processing agent. It's designed as a modern replacement for Fluentd, Fluent Bit, and Logstash. Stanza can run as a standalone agent on all major operating systems, or it can be embedded directly into any Golang application. + + +# Quick Start + +Want to get started right away? Check out our [Quick Start Guide](./docs/README.md)! + + +## Community + +Stanza is an open source project. If you'd like to contribute, check out our First, take a look at our [contribution guidelines](./CONTRIBUTING.md) and [developer guide](./docs/development.md). We look forward to building with you. + + +## Roadmap + +Stanza uses GitHub [projects](https://github.com/observIQ/stanza/projects) to organize future work. There's a lot to do, and more use cases are identified almost daily. We'll keep it organized and moving forward the best we can, but we're always looking for great ideas and contributions of all sizes. + + +## Code of Conduct + +Stanza follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). Please report violations of the Code of Conduct to any or all [maintainers](MAINTAINERS.md). + ## Other questions? diff --git a/docs/README.md b/docs/README.md index a8b4ae34b..bfc503a60 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,57 +1,130 @@ -# Stanza Log Agent Documentation -This repo contains documentation for the Stanza Log Agent. +# Quick Start Guide -## How do I configure the agent? -The agent is configured using a YAML config file that is passed in using the `--config` flag. This file defines a collection of operators beneath a top-level `pipeline` key. Each operator possesses a `type` and `id` field. +## Installation + +We recommend using our single-line installer provided with each release: + +### Linux +```shell +sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download/unix-install.sh)" unix-install.sh +``` +### macOS +```shell +sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download/unix-install.sh)" unix-install.sh +``` +### Windows +```pwsh +[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-Expression ((New-Object net.webclient).DownloadString('https://github.com/observiq/stanza/releases/latest/download/windows-install.ps1')); Log-Agent-Install +``` + +#### Alternately, feel free to download the [latest release](https://github.com/observIQ/stanza/releases) directly. + + +## Running Stanza + +If you installed the agent using the single-line installer above, it's already running as a service! If you'd like to start or stop the agent, here's how: + +### Linux +```shell +# systemd +systemctl start stanza +systemctl stop stanza + +# sysv +service stanza start +service stanza stop +``` +### macOS +```shell +launchctl start com.observiq.stanza +launchctl stop com.observiq.stanza +``` +### Windows +```pwsh +Start-Service -Name "stanza" +Stop-Service -Name "stanza" +``` + +### Manual + +If you'd like to run the agent manually rather than as a service, you can do that, too! + +```shell +# Example Command +stanza + +# Supported flags: +--config The location of the agent config file (default: ./config.yaml) +--plugin_dir The location of the plugins directory (default: ./plugins) +--database The location of the offsets database file. If this is not specified, offsets will not be maintained across agent restarts +--log_file The location of the agent log file. If not specified, stanza will log to `stderr` +--debug Enables debug logging +``` + + +## Configuration +A simple configuration file (config.yaml) is included in the installation. By default it doesn't do much, but is an easy way to get started. By default, it generates a single log entry and sends it to STDOUT every time the agent is restarted. ```yaml pipeline: - - type: udp_input - listen_address: :5141 + # An example input that generates a single log entry when Stanza starts up. + - type: generate_input + count: 1 + entry: + record: This is a sample log generated by Stanza +... + # An example output that sends captured logs to stdout. + - type: stdout +... +``` - - type: syslog_parser - parse_from: message - protocol: rfc5424 +The first step in configuring stanza is to setup your output. The sample configuration provides examples for sending data to the Elastic stack or Google Cloud Monitoring. Uncomment the destination of your choice and add in your own credentials and host. Restart the agent to generate another log line. +```yaml +... +pipeline: + # An example input that generates a single log entry when Stanza starts up. + - type: generate_input + count: 1 + entry: + record: This is a sample log generated by Stanza + + # An example output that sends captured logs to elasticsearch. + # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/elastic_output.md - type: elastic_output + addresses: + - http://my_node_address:9200 + api_key: my_api_key +... ``` -## What is an operator? -An operator is the most basic unit of log processing. Each operator fulfills only a single responsibility, such as reading lines from a file, or parsing JSON from a field. These operators are then chained together in a pipeline to achieve a desired result. - -For instance, a user may read lines from a file using the `file_input` operator. From there, the results of this operation may be sent to a `regex_parser` operator that creates fields based on a regex pattern. And then finally, these results may be sent to a `elastic_output` operator that writes each line to Elasticsearch. - -## What operators are available? - -Inputs: -- [File](/docs/operators/file_input.md) -- [Windows Event Log](/docs/operators/windows_eventlog_input.md) -- [TCP](/docs/operators/tcp_input.md) -- [UDP](/docs/operators/udp_input.md) -- [Journald](/docs/operators/journald_input.md) -- [Generate](/docs/operators/generate_input.md) - -Parsers: -- [JSON](/docs/operators/json_parser.md) -- [Regex](/docs/operators/regex_parser.md) -- [Syslog](/docs/operators/syslog_parser.md) -- [Severity](/docs/operators/severity_parser.md) -- [Time](/docs/operators/time_parser.md) - -Outputs: -- [Google Cloud Logging](/docs/operators/google_cloud_output.md) -- [Elasticsearch](/docs/operators/elastic_output.md) -- [Stdout](/docs/operators/stdout.md) -- [File](docs/operators/file_output.md) - -General purpose: -- [Rate Limit](/docs/operators/rate_limit.md) -- [Filter](/docs/operators/filter.md) -- [Router](/docs/operators/router.md) -- [Metadata](/docs/operators/metadata.md) -- [Restructure](/docs/operators/restructure.md) -- [Host Metadata](/docs/operators/host_metadata.md) -- [Kubernetes Metadata Decorator](/docs/operators/k8s_metadata_decorator.md) - -Or create your own [plugins](/docs/plugins.md) for a technology-specific use case. +Once you've confirmed you're able to send a log entry, you'll want to connect stanza to a log file you're interested in monitoring. We've included a sample `file_input` configuration in the config file to use (stanza also provide options for UDP, TCP, syslog, and other input streams. They're available [here](./README.md)). +```yaml +... +pipeline: +... + # An example input that monitors the contents of a file. + # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/file_input.md + - type: file_input + include: + - /sample/file/path + ... + + # An example output that sends captured logs to elasticsearch. + # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/elastic_output.md + - type: elastic_output + addresses: + - http://my_node_address:9200 + api_key: my_api_key +... +``` + +That's it! You should have logs streaming to Elasticsearch. From here you can explore all the options available within stanza! You can use existing plugins from our plugin repository or build your own custom pipelines. + + +## Next Steps + +- Check out stanza's list of [operators](./operators/README.md). +- Check out the [FAQ](./faq.md). +- Let us know what you think! [Email us](mailto:stanza@observiqlabs.com), or open a GitHub issue. diff --git a/docs/faq.md b/docs/faq.md index cbc0ed58f..da8a7b573 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -1,7 +1,27 @@ # Frequently asked questions -## Does Stanza support HTTP proxies? +## How do I configure the agent? +The agent is configured using a YAML file. Use the `--config` flag to tell stanza where to find this file. + +This file defines a collection of operators that make up a `pipeline`. Each operator possesses a `type` field, and can optionally be given an `id` as well. + +```yaml +pipeline: + - type: udp_input + listen_address: :5141 + + - type: syslog_parser + parse_from: message + protocol: rfc5424 -Yes. See [here](/docs/proxy.md) for details. + - type: elastic_output +``` +## What is a plugin? + +A plugin is a templated set of operators. Read more about plugins [here](/docs/plugins.md). + + +## Does Stanza support HTTP proxies? +Yes. Read about the details [here](/docs/proxy.md). diff --git a/docs/operators/README.md b/docs/operators/README.md new file mode 100644 index 000000000..bfda83b5d --- /dev/null +++ b/docs/operators/README.md @@ -0,0 +1,39 @@ +## What is an operator? +An operator is the most basic unit of log processing. Each operator fulfills a single responsibility, such as reading lines from a file, or parsing JSON from a field. Operators are then chained together in a pipeline to achieve a desired result. + +For instance, a user may read lines from a file using the `file_input` operator. From there, the results of this operation may be sent to a `regex_parser` operator that creates fields based on a regex pattern. And then finally, these results may be sent to a `elastic_output` operator that writes each line to Elasticsearch. + + +## What operators are available? + +Inputs: +- [File](/docs/operators/file_input.md) +- [Windows Event Log](/docs/operators/windows_eventlog_input.md) +- [TCP](/docs/operators/tcp_input.md) +- [UDP](/docs/operators/udp_input.md) +- [Journald](/docs/operators/journald_input.md) +- [Generate](/docs/operators/generate_input.md) + +Parsers: +- [JSON](/docs/operators/json_parser.md) +- [Regex](/docs/operators/regex_parser.md) +- [Syslog](/docs/operators/syslog_parser.md) +- [Severity](/docs/operators/severity_parser.md) +- [Time](/docs/operators/time_parser.md) + +Outputs: +- [Google Cloud Logging](/docs/operators/google_cloud_output.md) +- [Elasticsearch](/docs/operators/elastic_output.md) +- [Stdout](/docs/operators/stdout.md) +- [File](docs/operators/file_output.md) + +General purpose: +- [Rate Limit](/docs/operators/rate_limit.md) +- [Filter](/docs/operators/filter.md) +- [Router](/docs/operators/router.md) +- [Metadata](/docs/operators/metadata.md) +- [Restructure](/docs/operators/restructure.md) +- [Host Metadata](/docs/operators/host_metadata.md) +- [Kubernetes Metadata Decorator](/docs/operators/k8s_metadata_decorator.md) + +Or create your own [plugins](/docs/plugins.md) for a technology-specific use case. diff --git a/docs/plugins.md b/docs/plugins.md index 3e0ae80ff..a6dcd0266 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -1,6 +1,6 @@ # Plugins -Plugins can be defined by using a file that contains a templated set of operators. +A plugin is a templated set of operators. Plugins are defined by using a file with a name that corresponds to the plugin's `type`. For example, a very simple plugin for monitoring Apache Tomcat access logs could look like this: `tomcat.yaml`: @@ -16,7 +16,7 @@ pipeline: regex: '(?P[^\s]+) - (?P[^\s]+) \[(?P[^\]]+)\] "(?P[A-Z]+) (?P[^\s]+)[^"]+" (?P\d+) (?P[^\s]+)' ``` -Once a plugin config has been defined, it can be used in the stanza config file with a `type` matching the filename of the plugin. +Plugins can be used in the stanza config file with a `type` matching the filename of the plugin. `config.yaml`: ```yaml @@ -28,7 +28,8 @@ pipeline: - type: stdout ``` -The `tomcat_access` plugin is replaced with the operators from the rendered config in `tomcat.yaml`. +The `tomcat` plugin is replaced with the operators from the rendered config in `tomcat.yaml`. + ## Building a plugin