Skip to content

Commit

Permalink
(chore): update concepts documentation (#2107)
Browse files Browse the repository at this point in the history
  • Loading branch information
@aj-emerich
aj-emerich authored Jan 21, 2025
1 parent 44f212c commit d86ccff
Show file tree
Hide file tree
Showing 11 changed files with 141 additions and 141 deletions.
70 changes: 35 additions & 35 deletions content/docs/05.concepts/02.namespace-files.md
View file

Large diffs are not rendered by default.

22 changes: 11 additions & 11 deletions content/docs/05.concepts/04.secret.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Secret is a mechanism that allows you to securely store sensitive information, s

---

To retrieve secrets in a flow, use the `secret()` function, e.g. `"{{ secret('API_TOKEN'') }}"`. You can leverage your existing secrets manager as a secrets backend.
To retrieve secrets in a flow, use the `secret()` function, e.g., `"{{ secret('API_TOKEN'') }}"`. You can leverage your existing secrets manager as a secrets backend.

Your flows often need to interact with external systems. To do that, they need to programmatically authenticate using passwords or API keys. Secrets help you securely store such variables and avoid hard-coding sensitive information within your workflow code.

Expand All @@ -24,7 +24,7 @@ You can leverage the `secret()` function to retrieve sensitive variables within

### Adding a new Secret from the UI

If you are using a managed Kestra version, you can add **new Secrets** directly from the UI. In the left navigation menu, go to **Namespaces**, select the namespace to which you want to add a new secret. Then, add a new secret within the Secrets tab.
If you are using a managed Kestra version, you can add **new Secrets** directly from the UI. In the left navigation menu, go to **Namespaces** and select the namespace to which you want to add a new secret. Next, add a new secret within the Secrets tab.

![Secrets EE](/docs/developer-guide/secrets/secrets-ee-1.png)

Expand Down Expand Up @@ -56,21 +56,21 @@ Imagine that so far, you were setting the following environment variable:
export MYPASSWORD=myPrivateCode
```

Here is how you can encode the sensitive value of that environment variable:
Below is how you can encode the sensitive value of that environment variable:

```bash
echo -n "myPrivateCode" | base64
```

This should output the value: `bXlQcml2YXRlQ29kZQ==`
This outputs the value: `bXlQcml2YXRlQ29kZQ==`

To use that value as a Secret in your Kestra instance, you would need to add a prefix `SECRET_` to the variable key (here: `SECRET_MYPASSWORD`) and set that key to the encoded value:

```bash
export SECRET_MYPASSWORD=bXlQcml2YXRlQ29kZQ==
```

If you would add the environment variable to the `kestra` container section in a [Docker Compose file](https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml#L22), it would look as follows:
If you want to add the environment variable to the `kestra` container section in a [Docker Compose file](https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml#L22), it would look as follows:

```yaml
kestra:
Expand All @@ -79,17 +79,17 @@ If you would add the environment variable to the `kestra` container section in a
SECRET_MYPASSWORD: bXlQcml2YXRlQ29kZQ==
```
This secret can then be used in a flow using the `{{ secret('MYPASSWORD') }}` syntax, and it will base64-decoded during flow execution. Make sure to not include the prefix `SECRET_` when calling the `secret('MYPASSWORD')` function, as this prefix is only there in the environment variable definition to prevent Kestra from treating other system variables as secrets (for better performance and increased security).
This secret can be used in a flow using the `{{ secret('MYPASSWORD') }}` syntax, and it will be base64-decoded during flow execution. Make sure to not include the prefix `SECRET_` when calling the `secret('MYPASSWORD')` function, as this prefix is only there in the environment variable definition to prevent Kestra from treating other system variables as secrets (for better performance and increased security).

Lastly, shall you wish to reference any non_encoded environment variables in your flows definition, you can always use the syntax `{{envs.lowercase_environment_variable_key}}`.
Lastly, if you want to reference any non_encoded environment variables in your flows definition, you can always use the syntax `{{envs.lowercase_environment_variable_key}}`.

::alert{type="warning"}
Note that Kestra has built-in protection to prevent its logs from revealing any encoded secret you would have defined.
Note that Kestra has built-in protection to prevent its logs from revealing any encoded secret you have defined.
::

### Convert all variables in an `.env` file

The previous section showed the process for one Secret. But what if you have tens or hundreds of them? This is where `.env` file can come in handy.
The previous section showed the process for one Secret, but if you have tens or hundreds of them, then the `.env` is better suited.

Let's assume that you have an `.env` file with the following content:

Expand All @@ -101,7 +101,7 @@ AWS_SECRET_ACCESS_KEY=myawssecretaccesskey
```

Make sure to keep the last line empty, otherwise the bash script below won't encode the last secret AWS_SECRET_ACCESS_KEY correctly.
Make sure to keep the last line empty, otherwise the bash script below won't encode the last secret `AWS_SECRET_ACCESS_KEY` correctly.

Using the bash script shown below, you can:
1. Encode all values using base64-encoding
Expand Down Expand Up @@ -144,7 +144,7 @@ with the encoded version of the file:

### Use a macro within your `.env` file

As an alternative to replacing values in your environment variables by encoded counterparts, you may also leverage the `base64encode` macro and keep the values intact.
As an alternative to replacing values in your environment variables with encoded counterparts, you can also leverage the `base64encode` macro and keep the values intact.

The original `.env` file:

Expand Down
50 changes: 25 additions & 25 deletions content/docs/05.concepts/05.kv-store.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ Build stateful workflows with the KV Store.

## Overview

Kestra's workflows are stateless by design. All workflow executions and task runs are isolated from each other by default to avoid any unintended side effects. When you pass data between tasks, you do so explicitly by passing outputs from one task to another and that data is stored transparently in Kestra's internal storage. This stateless execution model ensures that workflows are idempotent and can be executed anywhere in parallel at scale.
Kestra's workflows are stateless by design. All workflow executions and task runs are isolated from each other by default to avoid any unintended side effects. When you pass data between tasks, you do so explicitly by passing outputs from one task to another, and that data is stored transparently in Kestra's internal storage. This stateless execution model ensures that workflows are idempotent and can be executed anywhere in parallel at scale.

However, in certain scenarios, your workflow might need to share data beyond passing outputs from one task to another. For example, you might want to persist data across executions or even across different workflows. This is where the Key Value (KV) store comes into play.

KV Store allows you to store any data in a convenient key-value format. You can create them directly from the UI, via dedicated tasks, Terraform or through the API.
KV Store allows you to store any data in a convenient key-value format. You can create them directly from the UI, via dedicated tasks, Terraform, or through the API.

The KV store is a powerful tool that allows you to build stateful workflows and share data across executions and workflows.

Expand All @@ -35,10 +35,10 @@ In short, the KV Store gives you full control and privacy over your data, and Ke

`Keys` are arbitrary strings. Keys can contain:

* characters in uppercase and or lowercase
* standard ASCII characters
- characters in uppercase and or lowercase
- standard ASCII characters

`Values` are stored as ION files in Kestra's internal storage. Values are strongly typed, and can be of one of the following types:
`Values` are stored as ION files in Kestra's internal storage. Values are strongly typed and can be of one of the following types:

- string
- number
Expand All @@ -52,13 +52,13 @@ For each KV pair, you can set a `Time to Live` (TTL) to avoid cluttering your st

## Namespace binding

Key value pairs are defined at a namespace level and you can access them from the namespace page in the UI in the KV Store tab.
Key value pairs are defined at a namespace level, and you can access them from the namespace page in the UI in the KV Store tab.

You can create and read KV pairs across namespaces as long as those namespaces are [allowed](../06.enterprise/allowed-namespaces.md).

## UI: How to Create, Read, Update and Delete KV pairs from the UI

Kestra follows a philosophy of Everything as Code and from the UI. Therefore, you can create, read, update, and delete KV pairs both from the UI and Code.
Kestra follows a philosophy of Everything as Code and also from the UI. Therefore, you can create, read, update, and delete KV pairs both from the UI and Code.

Here is a list of the different ways to manage KV pairs:
1. **Kestra UI**: select a Namespace and go to the KV Store tab — from here, you can create, edit, and delete KV pairs.
Expand Down Expand Up @@ -86,15 +86,15 @@ You can create, read, update, and delete KV pairs from the UI in the following w

### Update and Delete KV pairs from the UI

You can edit or delete any KV pair by clicking on the `Edit` button on the right side of each KV pair.
You can edit or delete any KV pair by clicking on the **Edit** button on the right side of each KV pair.

![edit_delete_kv_pair](/docs/concepts/kv-store/edit_delete_kv_pair.png)

## CODE: How to Create, Read, Update and Delete KV pairs in your flow code

### Create a new KV pair with the `Set` task in a flow

To create a KV pair from a flow, you can use the `io.kestra.plugin.core.kv.Set` task. Here's an example of how to create a KV pair in a flow:
To create a KV pair from a flow, you can use the `io.kestra.plugin.core.kv.Set` task. Below is an example of how to create a KV pair in a flow:

```yaml
id: add_kv_pair
Expand Down Expand Up @@ -141,7 +141,7 @@ You can use the `io.kestra.plugin.core.kv.Set` task to create or modify any KV p

The easiest way to retrieve a value by key is to use the `{{ kv('YOUR_KEY'') }}` Pebble function.

Here is the full syntax of that function:
Below is the full syntax of that function:

```
{{ kv(key='your_key_name', namespace='your_namespace_name', errorOnMissing=false) }}
Expand Down Expand Up @@ -180,7 +180,7 @@ tasks:
format: "{{ kv('non_existing_key', errorOnMissing=true) }}"
```

The function arguments such as the `errorOnMissing` keyword can be skipped for brevity as long as you fill in all positional arguments i.e. `{{ kv(key='your_key_name', namespace='your_namespace_name', errorOnMissing=false) }}` — the version below will have the same effect:
The function arguments such as the `errorOnMissing` keyword can be skipped for brevity as long as you fill in all positional arguments i.e., `{{ kv(key='your_key_name', namespace='your_namespace_name', errorOnMissing=false) }}` — the version below has the same effect:
{{ kv(key='my_key', namespace='company.team') }}
```yaml
id: read_non_existing_kv_pair
Expand All @@ -193,9 +193,9 @@ tasks:

### Read KV pairs with the `Get` task

You can also retrieve the value of any KV pair using the `Get` task. The `Get` task will produce the `value` output, which you can use in subsequent tasks. This option is a little more verbose but it has two benefits:
1. More declarative syntax.
2. Useful when you need to pass the current state of that value to multiple downstream tasks.
You can also retrieve the value of any KV pair using the `Get` task. The `Get` task produces the `value` output, which you can use in subsequent tasks. This option is a little more verbose, but it has two benefits:
1. More declarative syntax
2. Useful when you need to pass the current state of that value to multiple downstream tasks

```yaml
id: get_kv_pair
Expand All @@ -215,7 +215,7 @@ tasks:

### Read and parse JSON-type values from KV pairs

To parse JSON values in Kestra's templated expressions, make sure to wrap the `kv()` call in the `json()` function, e.g. `"{{ json(kv('your_json_key')).json_property }}"`.
To parse JSON values in Kestra's templated expressions, make sure to wrap the `kv()` call in the `json()` function like the following: `"{{ json(kv('your_json_key')).json_property }}"`.

The following example demonstrates how to parse values from JSON-type KV pairs in a flow:
```yaml
Expand Down Expand Up @@ -273,11 +273,11 @@ tasks:
message: "{{ outputs.get.keys }}"
```

The output will be a list of keys - if no keys were found, an empty list will be returned.
The output is a list of keys - if no keys were found, an empty list will be returned.

### Delete a KV pair with the `Delete` task

The `io.kestra.plugin.core.kv.Delete` task will produce the boolean output `deleted` to confirm whether a given KV pair was deleted or not.
The `io.kestra.plugin.core.kv.Delete` task produces the boolean output `deleted` to confirm whether a given KV pair was deleted or not.

```yaml
id: delete_kv_pair
Expand Down Expand Up @@ -315,7 +315,7 @@ For example:
curl -X PUT -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv/my_key -d '"Hello World"'
```

The above `curl` command will create the KV pair with key `my_key` and the `Hello World` string value in the `company.team` namespace. The API does not return any response.
The above `curl` command creates the KV pair with key `my_key` and the `Hello World` string value in the `company.team` namespace. The API does not return any response.

### Read the value by key

Expand All @@ -331,7 +331,7 @@ For example:
curl -X GET -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv/my_key
```

This will retrieve a KV pair with the key `my_key` in the `company.team` namespace. The output of the API will contain the data type of the value and the retrieved value of the KV pair:
This `curl` command retrieves a KV pair with the key `my_key` in the `company.team` namespace. The output of the API contains the data type of the value and the retrieved value of the KV pair:

```json
{"type": "STRING", "value": "Hello World"}
Expand All @@ -345,13 +345,13 @@ You can list all keys in the namespace as follows:
curl -X GET -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/{namespace}/kv
```

The `curl` command below will return all keys in the `company.team` namespace:
The `curl` command below returns all keys in the `company.team` namespace:

```bash
curl -X GET -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv
```

The output will be returned as a JSON array of all keys in the namespace:
The output is returned as a JSON array of all keys in the namespace:
```json
[
{"key":"my_key","creationDate":"2024-07-27T06:10:33.422Z","updateDate":"2024-07-27T06:11:08.911Z"},
Expand All @@ -369,13 +369,13 @@ curl -X DELETE -H "Content-Type: application/json" http://localhost:8080/api/v1/

This call returns a boolean indicating whether the key was deleted.

For example, the following `curl` command will return `false` because the key `non_existing_key` does not exist:
For example, the following `curl` command returns `false` because the key `non_existing_key` does not exist:

```bash
curl -X DELETE -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv/non_existing_key
```

However, when we try to delete a key `my_key` which exists in the `company.team` namespace, the same API call will return `true`:
However, when we try to delete a key `my_key` which exists in the `company.team` namespace, the same API call returns `true`:

```bash
curl -X DELETE -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv/my_key
Expand All @@ -389,7 +389,7 @@ curl -X DELETE -H "Content-Type: application/json" http://localhost:8080/api/v1/

You can create a KV pair via Terraform by using the `kestra_kv` resource.

Here is an example of how to create a KV pair:
Below is an example of how to create a KV pair:

```hcl
resource "kestra_kv" "my_key" {
Expand All @@ -404,7 +404,7 @@ resource "kestra_kv" "my_key" {

You can read a KV pair via Terraform by using the `kestra_kv` data source.

Here is an example of how to read a KV pair:
Below is an example of how to read a KV pair:

```hcl
data "kestra_kv" "new" {
Expand Down
Loading

0 comments on commit d86ccff

Please sign in to comment.