From ae1d6e2303f3c03033e1ecc9991b761404d1fb66 Mon Sep 17 00:00:00 2001 From: Austin Senkowski <28813380+asenkowski@users.noreply.github.com> Date: Mon, 24 Jan 2022 17:09:14 -0500 Subject: [PATCH] Quickstart updates (#527) quickstart and install guide updates --- README.md | 73 ++++++++++++++++++++++--- docs/README.md | 146 ++++++++++++++++++++++++++++++++++++++----------- 2 files changed, 179 insertions(+), 40 deletions(-) diff --git a/README.md b/README.md index 03f9fa40d..77c74e211 100644 --- a/README.md +++ b/README.md @@ -30,34 +30,89 @@ Stanza has been contributed to the [OpenTelemetry](https://opentelemetry.io/) pr - Written in pure Go - Everything is self contained into a single binary, there are no external dependencies. - High Performance - - Stanza is proven to be significantly faster than fluentd and more stable at high throughput than fluentbit. + - Stanza is lightweight, blazing-fast, and designed to scale. ## Supported [Plugins](https://github.com/observIQ/stanza-plugins) -Utilize Plugins to get up and running quickly. Here are some of our top Plugins: +Utilize Plugins to get up and running quickly. Here's a quick list of Stanza's most popular plugins:

These are many of the Plugins supported by Stanza, with more being developed all the time. View a full list of Plugins [here](https://github.com/observIQ/stanza-plugins/tree/master/plugins). -# Documentation +# Quick Start -## [Quick Start](./docs/README.md) - -### Installation +## Installation To install Stanza, we recommend using our single-line installer provided with each release. Stanza will automatically be running as a service upon completion. -#### Linux/macOS +### Linux/macOS ```shell sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download/unix-install.sh)" unix-install.sh ``` -#### Windows +### Windows ```pwsh [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-Expression ((New-Object net.webclient).DownloadString('https://github.com/observiq/stanza/releases/latest/download/windows-install.ps1')); Log-Agent-Install ``` -For more details on installation and configuration, check out our full [Quick Start Guide](./docs/README.md)! +### Kubernetes +For Kubernetes, there are several guides to install and configure Stanza found [here](./examples/k8s). + +## Configuration + +To get started navigate to the `config.yaml` file in the Stanza install directory, located in the following locations by default: + +- Linux: `/opt/observiq/stanza` +- MacOS: `/Users//observiq/stanza` +- Windows: `C:\observiq\stanza` + +You can utilize [operators](./docs/operators/README.md) and [plugins](./docs/plugins.md) in a pipeline to easily configure Stanza to ship logs to your target destination. + +Stanza also offers several outputs to be configured for sending data, including: + +- [Stdout](./docs/operators/stdout.md) +- [File](./docs/operators/file_output.md) +- [Google Cloud Logging](./docs/operators/google_cloud_output.md) + +In the below examples, Stanza is configured to ship logs to Google Cloud logging using the file_input operator, and the MySQL plugin. You will need to have a `credentials.json` for your GCP environment which can be generated by following Google's documentation [here](https://cloud.google.com/iam/docs/creating-managing-service-account-keys). + +### Operators +This `config.yaml` collects logs from a file and sends them to Google Cloud. A full list of available operators can be found [here](./docs/operators/README.md). + +```yaml +pipeline: + # An example input that monitors the contents of a file. + # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/file_input.md + - type: file_input + include: + - /sample/file/path.log + + # An example output that sends captured logs to Google Cloud. + # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/google_cloud_output.md + - type: google_cloud_output + credentials_file: /tmp/credentials.json +``` + +### Plugins +This `config.yaml` collects logs from MySQL via a plugin and sends them to Google Cloud. By default, MySQL plugin collects general, slow query, and error logs. More details of the MySQL plugin can be viewed [here](https://github.com/observIQ/stanza-plugins/blob/master/plugins/mysql.yaml). A full list of available plugins can be found [here](https://github.com/observIQ/stanza-plugins/blob/master/plugins/). + +```yaml +pipeline: + # An example input that configures a MySQL plugin. + # For more info: https://github.com/observIQ/stanza/blob/master/docs/plugins.md + - type: mysql + enable_general_log: true + general_log_path: "/var/log/mysql/general.log" + + # An example output that sends captured logs to Google Cloud. + # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/google_cloud_output.md + - type: google_cloud_output + credentials_file: /tmp/credentials.json +``` + +That's it! Logs should be streaming to Google Cloud. + +For more details on installation and configuration, check out our full [Install Guide](./docs/README.md)! # Community diff --git a/docs/README.md b/docs/README.md index bcbcb621a..8597a6a1f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,5 +1,5 @@ -# Quick Start Guide +# Install Guide ### Contents @@ -8,7 +8,7 @@ 3. [Configuration](#configuration) 4. [Next Steps](#next-steps) -## Installation +# Installation We recommend using our single-line installer provided with each release: @@ -21,26 +21,101 @@ sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-Expression ((New-Object net.webclient).DownloadString('https://github.com/observiq/stanza/releases/latest/download/windows-install.ps1')); Log-Agent-Install ``` -#### Alternately, feel free to download the [latest release](https://github.com/observIQ/stanza/releases) directly. +### Kubernetes +For Kubernetes, there are several guides to install and configure Stanza found [here](./examples/k8s). -## Local File System Stanza Mirrior +## Manual Installation + +Alternately, feel free to download the [latest release](https://github.com/observIQ/stanza/releases) directly. + +### Linux + +1. Downlaod Stanza from the releases page +```shell +VERSION=v1.5.0 +curl -L -o stanza "https://github.com/observIQ/stanza/releases/download/${VERSION}/stanza_linux_amd64" +``` +2. Configure file permissions +```shell +chmod +x stanza +sudo chown root:root stanza +``` +3. Configure install directory +```shell +sudo mkdir -p /opt/observiq/stanza +sudo mv stanza /opt/observiq/stanza/stanza +``` +4. Create Systemd service file with the following content +```shell +sudo vim /etc/systemd/system/stanza.service +``` +```shell +[Unit] +Description=Stanza Log Agent +After=network.target +StartLimitIntervalSec=120 +StartLimitBurst=5 +[Service] +Type=simple +PIDFile=/tmp/log-agent.pid +User=root +Group=root +Environment=PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin +WorkingDirectory=/opt/observiq/stanza +ExecStart=/opt/observiq/stanza/stanza --log_file /opt/observiq/stanza/stanza.log --database /opt/observiq/stanza/stanza.db +SuccessExitStatus=143 +TimeoutSec=120 +StandardOutput=null +Restart=on-failure +RestartSec=5s +[Install] +WantedBy=multi-user.target +``` +5. Bootstrap config file +```shell +sudo vim /opt/observiq/stanza/config.yaml +``` +```shell +pipeline: +- type: file_input + include: + - /var/log/messages +- type: stdout +``` +6. Reload Systemd, Enable Stanza, Start Stanza +```shell +sudo systemctl daemon-reload +sudo systemctl enable stanza +sudo systemctl start stanza +``` +7. Install plugins (optional) +```shell +curl -L -o stanza-plugins.tar.gz https://github.com/observIQ/stanza/releases/download/v1.5.0/stanza-plugins.tar.gz +sudo tar -xf stanza-plugins.tar.gz -C /opt/observiq/stanza +``` + +#### Questions: +1. **Why does Stanza run as root?** Running as root gives Stanza the ability to listen on privileged network ports and read any log file on the system. Using a non root user is supported by updating the systemd service. + +## Local File System Stanza Mirror For Linux and macOS it is possible to run the script from a local mirror, passing in the URL. See the [Local Mirror](MIRRORS.md) documentation. +# Running Stanza -## Running Stanza - -If you installed the agent using the single-line installer above, it's already running as a service! If you'd like to start or stop the agent, here's how: +If you installed the agent using the single-line installer above, it's already running as a service! If you'd like to start or stop the agent or check its status, here's how: ### Linux ```shell # systemd systemctl start stanza systemctl stop stanza +systemctl status stanza # sysv service stanza start service stanza stop +service stanza status ``` ### macOS ```shell @@ -51,6 +126,7 @@ launchctl stop com.observiq.stanza ```pwsh Start-Service -Name "stanza" Stop-Service -Name "stanza" +Get-Service -Name "stanza" ``` ### Manual @@ -73,7 +149,7 @@ stanza ``` -## Configuration +# Configuration A simple configuration file (config.yaml) is included in the installation. By default it doesn't do much, but is an easy way to get started. By default, it generates a single log entry and sends it to STDOUT every time the agent is restarted. ```yaml @@ -83,16 +159,13 @@ pipeline: count: 1 entry: record: This is a sample log generated by Stanza -... # An example output that sends captured logs to stdout. - type: stdout -... ``` -The first step in configuring stanza is to setup your output. The sample configuration provides examples for sending data to the Elastic stack or Google Cloud Monitoring. Uncomment the destination of your choice and add in your own credentials and host. Restart the agent to generate another log line. +The first step in configuring stanza is to setup your output. The sample configuration provides examples for sending data to Google Cloud Monitoring. Add in your own credentials and restart the agent to generate another log line. The `project_id` field is optional as the credentials file will include the project ID by default. ```yaml -... pipeline: # An example input that generates a single log entry when Stanza starts up. - type: generate_input @@ -100,40 +173,51 @@ pipeline: entry: record: This is a sample log generated by Stanza - # An example output that sends captured logs to elasticsearch. - # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/elastic_output.md - - type: elastic_output - addresses: - - http://my_node_address:9200 - api_key: my_api_key -... + # An example output that sends captured logs to Google Cloud. + # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/google_cloud_output.md + - type: google_cloud_output + project_id: sample_project + credentials_file: /tmp/credentials.json ``` Once you've confirmed you're able to send a log entry, you'll want to connect stanza to a log file you're interested in monitoring. We've included a sample `file_input` configuration in the config file to use (stanza also provide options for UDP, TCP, syslog, and other input streams. They're available [here](/docs/operators/README.md)). + ```yaml -... pipeline: -... # An example input that monitors the contents of a file. # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/file_input.md - type: file_input include: - /sample/file/path - ... - # An example output that sends captured logs to elasticsearch. - # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/elastic_output.md - - type: elastic_output - addresses: - - http://my_node_address:9200 - api_key: my_api_key -... + # An example output that sends captured logs to Google Cloud. + # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/google_cloud_output.md + - type: google_cloud_output + project_id: sample_project + credentials_file: /tmp/credentials.json +``` + +ALternatively, you can use a plugin for log monitoring. This `config.yaml` collects logs from MySQL via a plugin and sends them to Google Cloud. By default, MySQL plugin collects general, slow query, and error logs. More details of the MySQL plugin can be viewed [here](https://github.com/observIQ/stanza-plugins/blob/master/plugins/mysql.yaml). + +```yaml +pipeline: + # An example input that configures a MySQL plugin. + # For more info: https://github.com/observIQ/stanza/blob/master/docs/plugins.md + - type: mysql + enable_general_log: true + general_log_path: "/var/log/mysql/general.log" + + # An example output that sends captured logs to Google Cloud. + # For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/google_cloud_output.md + - type: google_cloud_output + project_id: sample_project + credentials_file: /tmp/credentials.json ``` -That's it! You should have logs streaming to Elasticsearch. From here you can explore all the options available within stanza! You can use existing plugins from our plugin repository or build your own custom pipelines. +That's it! You should have logs streaming to Google Cloud. From here you can explore all the options available within stanza! You can use existing plugins from our plugin repository or build your own custom pipelines. -## Next Steps +# Next Steps - Read up on how to write a stanza [pipeline](/docs/pipeline.md). - Check out stanza's list of [operators](/docs/operators/README.md).