Skip to content

Commit

Permalink
Updated for docs
Browse files Browse the repository at this point in the history
  • Loading branch information
dat-a-man committed Jan 29, 2025
1 parent 8654625 commit e9773b6
Showing 1 changed file with 23 additions and 0 deletions.
23 changes: 23 additions & 0 deletions docs/website/docs/reference/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,3 +319,26 @@ volumes {
}
}
```
## Handling storage limits

If your storage reaches its limit, you are likely running dlt in a cloud environment with restricted disk space. To prevent issues, mount an external cloud storage location and set the `DLT_DATA_DIR` environment variable to point to it. This ensures that dlt uses the mounted storage as its data directory instead of local disk space.


### Setting `DLT_DATA_DIR`

You can configure `DLT_DATA_DIR` in your environment setup as follows:

```py
import os
from dlt.common.known_env import DLT_DATA_DIR


# Define the path to your mounted external storage
data_dir = "/path/to/mounted/bucket/dlt_pipeline_data"

# Set the DLT_DATA_DIR environment variable
os.environ[DLT_DATA_DIR] = data_dir

# Rest of your pipeline code
```
This directs dlt to use the specified external storage for all data operations, preventing local storage constraints.

0 comments on commit e9773b6

Please sign in to comment.