diff --git a/docs/sources/fundamentals/_index.md b/docs/sources/fundamentals/_index.md new file mode 100644 index 0000000000000..14ec169518832 --- /dev/null +++ b/docs/sources/fundamentals/_index.md @@ -0,0 +1,6 @@ +--- +title: Fundamentals +weight: 150 +--- +# Grafana Loki Fundamentals + diff --git a/docs/sources/architecture/_index.md b/docs/sources/fundamentals/architecture/_index.md similarity index 98% rename from docs/sources/architecture/_index.md rename to docs/sources/fundamentals/architecture/_index.md index 27a55844ba5ec..76b40e9071e21 100644 --- a/docs/sources/architecture/_index.md +++ b/docs/sources/fundamentals/architecture/_index.md @@ -1,6 +1,8 @@ --- title: Architecture -weight: 1000 +weight: 200 +aliases: + - /docs/loki/latest/architecture/ --- # Loki's Architecture @@ -173,7 +175,7 @@ Logs from each unique set of labels are built up into "chunks" in memory and then flushed to the backing storage backend. If an ingester process crashes or exits abruptly, all the data that has not yet -been flushed could be lost. Loki is usually configured with a [Write Ahead Log](../operations/storage/wal) which can be _replayed_ on restart as well as with a `replication_factor` (usually 3) of each log to mitigate this risk. +been flushed could be lost. Loki is usually configured with a [Write Ahead Log](../../operations/storage/wal) which can be _replayed_ on restart as well as with a `replication_factor` (usually 3) of each log to mitigate this risk. When not configured to accept out-of-order writes, all lines pushed to Loki for a given stream (unique combination of @@ -189,7 +191,7 @@ nanosecond timestamps: different content, the log line is accepted. This means it is possible to have two different log lines for the same timestamp. -#### Handoff - Deprecated in favor of the [WAL](../operations/storage/wal) +#### Handoff - Deprecated in favor of the [WAL](../../operations/storage/wal) By default, when an ingester is shutting down and tries to leave the hash ring, it will wait to see if a new ingester tries to enter before flushing and will @@ -243,7 +245,7 @@ Caching log (filter, regexp) queries are under active development. ### Querier -The **querier** service handles queries using the [LogQL](../logql/) query +The **querier** service handles queries using the [LogQL](../../logql/) query language, fetching logs both from the ingesters and from long-term storage. Queriers query all ingesters for in-memory data before falling back to diff --git a/docs/sources/architecture/chunks_diagram.png b/docs/sources/fundamentals/architecture/chunks_diagram.png similarity index 100% rename from docs/sources/architecture/chunks_diagram.png rename to docs/sources/fundamentals/architecture/chunks_diagram.png diff --git a/docs/sources/architecture/distributor.md b/docs/sources/fundamentals/architecture/distributor.md similarity index 97% rename from docs/sources/architecture/distributor.md rename to docs/sources/fundamentals/architecture/distributor.md index f0ed888c30cea..2bcbda2bb8b65 100644 --- a/docs/sources/architecture/distributor.md +++ b/docs/sources/fundamentals/architecture/distributor.md @@ -4,8 +4,6 @@ weight: 1000 --- # Distributor Component -This document builds upon the information in the [Loki Architecture](./) page. - Distributors are stateless and communicate with ingesters via [gRPC](https://grpc.io). The quantity of distributors can be increased or decreased as needed. ## Where does it live? @@ -26,7 +24,7 @@ Currently the only way the distributor mutates incoming data is by normalizing l The distributor can also rate limit incoming logs based on the maximum per-tenant bitrate. It does this by checking a per tenant limit and dividing it by the current number of distributors. This allows the rate limit to be specified per tenant at the cluster level and enables us to scale the distributors up or down and have the per-distributor limit adjust accordingly. For instance, say we have 10 distributors and tenant A has a 10MB rate limit. Each distributor will allow up to 1MB/second before limiting. Now, say another large tenant joins the cluster and we need to spin up 10 more distributors. The now 20 distributors will adjust their rate limits for tenant A to `(10MB / 20 distributors) = 500KB/s`! This is how global limits allow much simpler and safer operation of the Loki cluster. -**Note: The distributor uses the `ring` component under the hood to register itself amongst it's peers and get the total number of active distributors. This is a different "key" than the ingesters use in the ring and comes from the distributor's own [ring config](../../configuration#distributor_config).** +**Note: The distributor uses the `ring` component under the hood to register itself amongst it's peers and get the total number of active distributors. This is a different "key" than the ingesters use in the ring and comes from the distributor's own [ring configuration](../../../configuration#distributor_config).** ### Forwarding diff --git a/docs/sources/architecture/loki_architecture_components.svg b/docs/sources/fundamentals/architecture/loki_architecture_components.svg similarity index 100% rename from docs/sources/architecture/loki_architecture_components.svg rename to docs/sources/fundamentals/architecture/loki_architecture_components.svg diff --git a/docs/sources/architecture/modes_of_operation.png b/docs/sources/fundamentals/architecture/modes_of_operation.png similarity index 100% rename from docs/sources/architecture/modes_of_operation.png rename to docs/sources/fundamentals/architecture/modes_of_operation.png diff --git a/docs/sources/getting-started/labels.md b/docs/sources/fundamentals/labels.md similarity index 98% rename from docs/sources/getting-started/labels.md rename to docs/sources/fundamentals/labels.md index 87e8cc3a5c35e..e11bb2b7c2c5e 100644 --- a/docs/sources/getting-started/labels.md +++ b/docs/sources/fundamentals/labels.md @@ -1,5 +1,8 @@ --- title: Labels +weight: 300 +aliases: + - /docs/loki/latest/getting-started/labels/ --- # Labels @@ -189,7 +192,7 @@ Now let's talk about Loki, where the index is typically an order of magnitude sm Loki will effectively keep your static costs as low as possible (index size and memory requirements as well as static log storage) and make the query performance something you can control at runtime with horizontal scaling. -To see how this works, let's look back at our example of querying your access log data for a specific IP address. We don't want to use a label to store the IP. Instead we use a [filter expression](../../logql#filter-expression) to query for it: +To see how this works, let's look back at our example of querying your access log data for a specific IP address. We don't want to use a label to store the IP address. Instead we use a [filter expression](../../logql/log_queries#line-filter-expression) to query for it: ``` {job="apache"} |= "11.11.11.11" diff --git a/docs/sources/overview/_index.md b/docs/sources/fundamentals/overview/_index.md similarity index 92% rename from docs/sources/overview/_index.md rename to docs/sources/fundamentals/overview/_index.md index a0ed54d47d866..06f9f5dd6fab4 100644 --- a/docs/sources/overview/_index.md +++ b/docs/sources/fundamentals/overview/_index.md @@ -1,6 +1,8 @@ --- title: Overview -weight: 150 +weight: 100 +aliases: + - /docs/loki/latest/overview/ --- # Overview @@ -19,7 +21,7 @@ An agent (also called a client) acquires logs, turns the logs into streams, and pushes the streams to Loki through an HTTP API. The Promtail agent is designed for Loki installations, -but many other [Agents](../clients/) seamlessly integrate with Loki. +but many other [Agents](../../clients/) seamlessly integrate with Loki. ![Loki agent interaction](loki-overview-2.png) @@ -28,7 +30,7 @@ Each stream identifies a set of logs associated with a unique set of labels. A quality set of labels is key to the creation of an index that is both compact and allows for efficient query execution. -[LogQL](../logql) is the query language for Loki. +[LogQL](../../logql) is the query language for Loki. ## Loki features diff --git a/docs/sources/overview/comparisons.md b/docs/sources/fundamentals/overview/comparisons.md similarity index 99% rename from docs/sources/overview/comparisons.md rename to docs/sources/fundamentals/overview/comparisons.md index 54338b92fbdc2..11aab75c45a4f 100644 --- a/docs/sources/overview/comparisons.md +++ b/docs/sources/fundamentals/overview/comparisons.md @@ -18,7 +18,7 @@ horizontally-scalable mode data is stored in a cloud storage system such as S3, GCS, or Cassandra. Logs are stored in plaintext form tagged with a set of label names and values, where only the label pairs are indexed. This tradeoff makes it cheaper to operate than a full index and allows developers to aggressively log -from their applications. Logs in Loki are queried using [LogQL](../../logql). +from their applications. Logs in Loki are queried using [LogQL](../../../logql). However, because of this design tradeoff, LogQL queries that filter based on content (i.e., text within the log lines) require loading all chunks within the search window that match the labels defined in the query. diff --git a/docs/sources/overview/loki-overview-1.png b/docs/sources/fundamentals/overview/loki-overview-1.png similarity index 100% rename from docs/sources/overview/loki-overview-1.png rename to docs/sources/fundamentals/overview/loki-overview-1.png diff --git a/docs/sources/overview/loki-overview-2.png b/docs/sources/fundamentals/overview/loki-overview-2.png similarity index 100% rename from docs/sources/overview/loki-overview-2.png rename to docs/sources/fundamentals/overview/loki-overview-2.png