Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: introduce a fundamentals section #4403

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/sources/fundamentals/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
title: Fundamentals
weight: 150
---
# Grafana Loki Fundamentals

Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
title: Architecture
weight: 1000
weight: 200
aliases:
- /docs/loki/latest/architecture/
---
# Loki's Architecture

Expand Down Expand Up @@ -173,7 +175,7 @@ Logs from each unique set of labels are built up into "chunks" in memory and
then flushed to the backing storage backend.

If an ingester process crashes or exits abruptly, all the data that has not yet
been flushed could be lost. Loki is usually configured with a [Write Ahead Log](../operations/storage/wal) which can be _replayed_ on restart as well as with a `replication_factor` (usually 3) of each log to mitigate this risk.
been flushed could be lost. Loki is usually configured with a [Write Ahead Log](../../operations/storage/wal) which can be _replayed_ on restart as well as with a `replication_factor` (usually 3) of each log to mitigate this risk.

When not configured to accept out-of-order writes,
all lines pushed to Loki for a given stream (unique combination of
Expand All @@ -189,7 +191,7 @@ nanosecond timestamps:
different content, the log line is accepted. This means it is possible to
have two different log lines for the same timestamp.

#### Handoff - Deprecated in favor of the [WAL](../operations/storage/wal)
#### Handoff - Deprecated in favor of the [WAL](../../operations/storage/wal)

By default, when an ingester is shutting down and tries to leave the hash ring,
it will wait to see if a new ingester tries to enter before flushing and will
Expand Down Expand Up @@ -243,7 +245,7 @@ Caching log (filter, regexp) queries are under active development.

### Querier

The **querier** service handles queries using the [LogQL](../logql/) query
The **querier** service handles queries using the [LogQL](../../logql/) query
language, fetching logs both from the ingesters and from long-term storage.

Queriers query all ingesters for in-memory data before falling back to
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@ weight: 1000
---
# Distributor Component

This document builds upon the information in the [Loki Architecture](./) page.

Distributors are stateless and communicate with ingesters via [gRPC](https://grpc.io). The quantity of distributors can be increased or decreased as needed.

## Where does it live?
Expand All @@ -26,7 +24,7 @@ Currently the only way the distributor mutates incoming data is by normalizing l

The distributor can also rate limit incoming logs based on the maximum per-tenant bitrate. It does this by checking a per tenant limit and dividing it by the current number of distributors. This allows the rate limit to be specified per tenant at the cluster level and enables us to scale the distributors up or down and have the per-distributor limit adjust accordingly. For instance, say we have 10 distributors and tenant A has a 10MB rate limit. Each distributor will allow up to 1MB/second before limiting. Now, say another large tenant joins the cluster and we need to spin up 10 more distributors. The now 20 distributors will adjust their rate limits for tenant A to `(10MB / 20 distributors) = 500KB/s`! This is how global limits allow much simpler and safer operation of the Loki cluster.

**Note: The distributor uses the `ring` component under the hood to register itself amongst it's peers and get the total number of active distributors. This is a different "key" than the ingesters use in the ring and comes from the distributor's own [ring config](../../configuration#distributor_config).**
**Note: The distributor uses the `ring` component under the hood to register itself amongst it's peers and get the total number of active distributors. This is a different "key" than the ingesters use in the ring and comes from the distributor's own [ring configuration](../../../configuration#distributor_config).**

### Forwarding

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
---
title: Labels
weight: 300
aliases:
- /docs/loki/latest/getting-started/labels/
---
# Labels

Expand Down Expand Up @@ -189,7 +192,7 @@ Now let's talk about Loki, where the index is typically an order of magnitude sm

Loki will effectively keep your static costs as low as possible (index size and memory requirements as well as static log storage) and make the query performance something you can control at runtime with horizontal scaling.

To see how this works, let's look back at our example of querying your access log data for a specific IP address. We don't want to use a label to store the IP. Instead we use a [filter expression](../../logql#filter-expression) to query for it:
To see how this works, let's look back at our example of querying your access log data for a specific IP address. We don't want to use a label to store the IP address. Instead we use a [filter expression](../../logql/log_queries#line-filter-expression) to query for it:

```
{job="apache"} |= "11.11.11.11"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
title: Overview
weight: 150
weight: 100
aliases:
- /docs/loki/latest/overview/
---
# Overview

Expand All @@ -19,7 +21,7 @@ An agent (also called a client) acquires logs,
turns the logs into streams,
and pushes the streams to Loki through an HTTP API.
The Promtail agent is designed for Loki installations,
but many other [Agents](../clients/) seamlessly integrate with Loki.
but many other [Agents](../../clients/) seamlessly integrate with Loki.

![Loki agent interaction](loki-overview-2.png)

Expand All @@ -28,7 +30,7 @@ Each stream identifies a set of logs associated with a unique set of labels.
A quality set of labels is key to the creation of an index that is both compact
and allows for efficient query execution.

[LogQL](../logql) is the query language for Loki.
[LogQL](../../logql) is the query language for Loki.

## Loki features

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ horizontally-scalable mode data is stored in a cloud storage system such as S3,
GCS, or Cassandra. Logs are stored in plaintext form tagged with a set of label
names and values, where only the label pairs are indexed. This tradeoff makes it
cheaper to operate than a full index and allows developers to aggressively log
from their applications. Logs in Loki are queried using [LogQL](../../logql).
from their applications. Logs in Loki are queried using [LogQL](../../../logql).
However, because of this design tradeoff, LogQL queries that filter based on
content (i.e., text within the log lines) require loading all chunks within the
search window that match the labels defined in the query.
Expand Down