Skip to content

Commit

Permalink
Quickstart updates (#527)
Browse files Browse the repository at this point in the history 
quickstart and install guide updates
  • Loading branch information
@asenkowski
asenkowski authored Jan 24, 2022
1 parent 94b7d22 commit ae1d6e2
Show file tree
Hide file tree
Showing 2 changed files with 179 additions and 40 deletions.
73 changes: 64 additions & 9 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,34 +30,89 @@ Stanza has been contributed to the [OpenTelemetry](https://opentelemetry.io/) pr
- Written in pure Go
- Everything is self contained into a single binary, there are no external dependencies.
- High Performance
- Stanza is proven to be significantly faster than fluentd and more stable at high throughput than fluentbit.
- Stanza is lightweight, blazing-fast, and designed to scale.

## Supported [Plugins](https://github.com/observIQ/stanza-plugins)

Utilize Plugins to get up and running quickly. Here are some of our top Plugins:
Utilize Plugins to get up and running quickly. Here's a quick list of Stanza's most popular plugins:

<p align="center"><img src="docs/images/stanza_plugins.png?raw=true"></p>

These are many of the Plugins supported by Stanza, with more being developed all the time. View a full list of Plugins [here](https://github.com/observIQ/stanza-plugins/tree/master/plugins).

# Documentation
# Quick Start

## [Quick Start](./docs/README.md)

### Installation
## Installation

To install Stanza, we recommend using our single-line installer provided with each release. Stanza will automatically be running as a service upon completion.

#### Linux/macOS
### Linux/macOS
```shell
sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download/unix-install.sh)" unix-install.sh
```
#### Windows
### Windows
```pwsh
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-Expression ((New-Object net.webclient).DownloadString('https://github.com/observiq/stanza/releases/latest/download/windows-install.ps1')); Log-Agent-Install
```

For more details on installation and configuration, check out our full [Quick Start Guide](./docs/README.md)!
### Kubernetes
For Kubernetes, there are several guides to install and configure Stanza found [here](./examples/k8s).

## Configuration

To get started navigate to the `config.yaml` file in the Stanza install directory, located in the following locations by default:

- Linux: `/opt/observiq/stanza`
- MacOS: `/Users/<user>/observiq/stanza`
- Windows: `C:\observiq\stanza`

You can utilize [operators](./docs/operators/README.md) and [plugins](./docs/plugins.md) in a pipeline to easily configure Stanza to ship logs to your target destination.

Stanza also offers several outputs to be configured for sending data, including:

- [Stdout](./docs/operators/stdout.md)
- [File](./docs/operators/file_output.md)
- [Google Cloud Logging](./docs/operators/google_cloud_output.md)

In the below examples, Stanza is configured to ship logs to Google Cloud logging using the file_input operator, and the MySQL plugin. You will need to have a `credentials.json` for your GCP environment which can be generated by following Google's documentation [here](https://cloud.google.com/iam/docs/creating-managing-service-account-keys).

### Operators
This `config.yaml` collects logs from a file and sends them to Google Cloud. A full list of available operators can be found [here](./docs/operators/README.md).

```yaml
pipeline:
# An example input that monitors the contents of a file.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/file_input.md
- type: file_input
include:
- /sample/file/path.log

# An example output that sends captured logs to Google Cloud.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/google_cloud_output.md
- type: google_cloud_output
credentials_file: /tmp/credentials.json
```
### Plugins
This `config.yaml` collects logs from MySQL via a plugin and sends them to Google Cloud. By default, MySQL plugin collects general, slow query, and error logs. More details of the MySQL plugin can be viewed [here](https://github.com/observIQ/stanza-plugins/blob/master/plugins/mysql.yaml). A full list of available plugins can be found [here](https://github.com/observIQ/stanza-plugins/blob/master/plugins/).

```yaml
pipeline:
# An example input that configures a MySQL plugin.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/plugins.md
- type: mysql
enable_general_log: true
general_log_path: "/var/log/mysql/general.log"
# An example output that sends captured logs to Google Cloud.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/google_cloud_output.md
- type: google_cloud_output
credentials_file: /tmp/credentials.json
```

That's it! Logs should be streaming to Google Cloud.

For more details on installation and configuration, check out our full [Install Guide](./docs/README.md)!

# Community

Expand Down
146 changes: 115 additions & 31 deletions docs/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

# Quick Start Guide
# Install Guide

### Contents

Expand All @@ -8,7 +8,7 @@
3. [Configuration](#configuration)
4. [Next Steps](#next-steps)

## Installation
# Installation

We recommend using our single-line installer provided with each release:

Expand All @@ -21,26 +21,101 @@ sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-Expression ((New-Object net.webclient).DownloadString('https://github.com/observiq/stanza/releases/latest/download/windows-install.ps1')); Log-Agent-Install
```

#### Alternately, feel free to download the [latest release](https://github.com/observIQ/stanza/releases) directly.
### Kubernetes
For Kubernetes, there are several guides to install and configure Stanza found [here](./examples/k8s).

## Local File System Stanza Mirrior
## Manual Installation

Alternately, feel free to download the [latest release](https://github.com/observIQ/stanza/releases) directly.

### Linux

1. Downlaod Stanza from the releases page
```shell
VERSION=v1.5.0
curl -L -o stanza "https://github.com/observIQ/stanza/releases/download/${VERSION}/stanza_linux_amd64"
```
2. Configure file permissions
```shell
chmod +x stanza
sudo chown root:root stanza
```
3. Configure install directory
```shell
sudo mkdir -p /opt/observiq/stanza
sudo mv stanza /opt/observiq/stanza/stanza
```
4. Create Systemd service file with the following content
```shell
sudo vim /etc/systemd/system/stanza.service
```
```shell
[Unit]
Description=Stanza Log Agent
After=network.target
StartLimitIntervalSec=120
StartLimitBurst=5
[Service]
Type=simple
PIDFile=/tmp/log-agent.pid
User=root
Group=root
Environment=PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
WorkingDirectory=/opt/observiq/stanza
ExecStart=/opt/observiq/stanza/stanza --log_file /opt/observiq/stanza/stanza.log --database /opt/observiq/stanza/stanza.db
SuccessExitStatus=143
TimeoutSec=120
StandardOutput=null
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
```
5. Bootstrap config file
```shell
sudo vim /opt/observiq/stanza/config.yaml
```
```shell
pipeline:
- type: file_input
include:
- /var/log/messages
- type: stdout
```
6. Reload Systemd, Enable Stanza, Start Stanza
```shell
sudo systemctl daemon-reload
sudo systemctl enable stanza
sudo systemctl start stanza
```
7. Install plugins (optional)
```shell
curl -L -o stanza-plugins.tar.gz https://github.com/observIQ/stanza/releases/download/v1.5.0/stanza-plugins.tar.gz
sudo tar -xf stanza-plugins.tar.gz -C /opt/observiq/stanza
```

#### Questions:
1. **Why does Stanza run as root?** Running as root gives Stanza the ability to listen on privileged network ports and read any log file on the system. Using a non root user is supported by updating the systemd service.

## Local File System Stanza Mirror
For Linux and macOS it is possible to run the script from a local mirror,
passing in the URL. See the [Local Mirror](MIRRORS.md) documentation.

# Running Stanza

## Running Stanza

If you installed the agent using the single-line installer above, it's already running as a service! If you'd like to start or stop the agent, here's how:
If you installed the agent using the single-line installer above, it's already running as a service! If you'd like to start or stop the agent or check its status, here's how:

### Linux
```shell
# systemd
systemctl start stanza
systemctl stop stanza
systemctl status stanza

# sysv
service stanza start
service stanza stop
service stanza status
```
### macOS
```shell
Expand All @@ -51,6 +126,7 @@ launchctl stop com.observiq.stanza
```pwsh
Start-Service -Name "stanza"
Stop-Service -Name "stanza"
Get-Service -Name "stanza"
```

### Manual
Expand All @@ -73,7 +149,7 @@ stanza
```


## Configuration
# Configuration
A simple configuration file (config.yaml) is included in the installation. By default it doesn't do much, but is an easy way to get started. By default, it generates a single log entry and sends it to STDOUT every time the agent is restarted.

```yaml
Expand All @@ -83,57 +159,65 @@ pipeline:
count: 1
entry:
record: This is a sample log generated by Stanza
...
# An example output that sends captured logs to stdout.
- type: stdout
...
```
The first step in configuring stanza is to setup your output. The sample configuration provides examples for sending data to the Elastic stack or Google Cloud Monitoring. Uncomment the destination of your choice and add in your own credentials and host. Restart the agent to generate another log line.
The first step in configuring stanza is to setup your output. The sample configuration provides examples for sending data to Google Cloud Monitoring. Add in your own credentials and restart the agent to generate another log line. The `project_id` field is optional as the credentials file will include the project ID by default.

```yaml
...
pipeline:
# An example input that generates a single log entry when Stanza starts up.
- type: generate_input
count: 1
entry:
record: This is a sample log generated by Stanza
# An example output that sends captured logs to elasticsearch.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/elastic_output.md
- type: elastic_output
addresses:
- http://my_node_address:9200
api_key: my_api_key
...
# An example output that sends captured logs to Google Cloud.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/google_cloud_output.md
- type: google_cloud_output
project_id: sample_project
credentials_file: /tmp/credentials.json
```

Once you've confirmed you're able to send a log entry, you'll want to connect stanza to a log file you're interested in monitoring. We've included a sample `file_input` configuration in the config file to use (stanza also provide options for UDP, TCP, syslog, and other input streams. They're available [here](/docs/operators/README.md)).

```yaml
...
pipeline:
...
# An example input that monitors the contents of a file.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/file_input.md
- type: file_input
include:
- /sample/file/path
...
# An example output that sends captured logs to elasticsearch.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/elastic_output.md
- type: elastic_output
addresses:
- http://my_node_address:9200
api_key: my_api_key
...
# An example output that sends captured logs to Google Cloud.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/google_cloud_output.md
- type: google_cloud_output
project_id: sample_project
credentials_file: /tmp/credentials.json
```

ALternatively, you can use a plugin for log monitoring. This `config.yaml` collects logs from MySQL via a plugin and sends them to Google Cloud. By default, MySQL plugin collects general, slow query, and error logs. More details of the MySQL plugin can be viewed [here](https://github.com/observIQ/stanza-plugins/blob/master/plugins/mysql.yaml).

```yaml
pipeline:
# An example input that configures a MySQL plugin.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/plugins.md
- type: mysql
enable_general_log: true
general_log_path: "/var/log/mysql/general.log"
# An example output that sends captured logs to Google Cloud.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/google_cloud_output.md
- type: google_cloud_output
project_id: sample_project
credentials_file: /tmp/credentials.json
```

That's it! You should have logs streaming to Elasticsearch. From here you can explore all the options available within stanza! You can use existing plugins from our plugin repository or build your own custom pipelines.
That's it! You should have logs streaming to Google Cloud. From here you can explore all the options available within stanza! You can use existing plugins from our plugin repository or build your own custom pipelines.


## Next Steps
# Next Steps

- Read up on how to write a stanza [pipeline](/docs/pipeline.md).
- Check out stanza's list of [operators](/docs/operators/README.md).
Expand Down

0 comments on commit ae1d6e2

Please sign in to comment.