Skip to content

Commit

Permalink
Documentation cleanup (nebari-dev#889)
Browse files Browse the repository at this point in the history
* DOCS: Fix warnings in docs/README.md

* DOCS: Fix warnings in docs/index.md

* DOCS: Fix warnings in docs/source/user_guide/faq.md

* DOCS: Fix warnings in docs/source/admin_guide/faq.md

* DOCS: Fix warnings in docs/source/admin_guide/index.md

* DOCS: Fix warnings in docs/source/admin_guide/clearml.md

* DOCS: Fix warnings in docs/source/admin_guide/monitoring.md

* DOCS: Fix warnings in docs/source/admin_guide/preemptible-spot-instances.md

* DOCS: Fix warnings in docs/source/admin_guide/backup.md

* DOCS: Fix warnings in docs/source/admin_guide/cost.md

* FIX: Fix Value issues on the CI

* FIX: Add more ignore patterns on the broken link checker
  • Loading branch information
HarshCasper authored Nov 2, 2021
1 parent 155b761 commit 0491c9a
Show file tree
Hide file tree
Showing 19 changed files with 170 additions and 265 deletions.
6 changes: 6 additions & 0 deletions .github/workflows/markdown.links.config.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@
},
{
"pattern": "linuxize.com"
},
{
"pattern": "github-actions.qhub.dev"
},
{
"pattern": "127.0.0.1:8000"
}
]
}
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ build/
*.egg
*.egg-info/
dist/
env/

# IntelliJ
.idea/
Expand All @@ -28,10 +29,13 @@ quansight-terraform-modules
config.yaml
data*

# modules - removed this because .terraform folder is in .gitignore of deployments,
# modules - removed this because .terraform folder is in .gitignore of deployments,
# this should cover imported modules while not gitignoring our new direct 'modules' folder.

# ignore qhub files
qhub-config.yaml

.vscode/

# Ignore Vale
vale
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# qhub docs
# QHub docs

To build the docs:

Expand Down
12 changes: 2 additions & 10 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,11 @@
[![Conda version](https://img.shields.io/badge/conda--forge-v0.3.0-%234f28a8)](https://anaconda.org/conda-forge/qhub)
[![GitHub issues](https://img.shields.io/github/issues/quansight/qhub?style=plastic)](https://github.com/Quansight/qhub/issues/new/choose)


Open source tooling for data science research, development, and deployment.

# What is Qhub?

Integrated data science environment designed and developed by
scientists at [**Quansight**](https://www.quansight.com/). It enables
teams to build and maintain a cost effective and scalable compute/data
science platform in the Cloud, providing an [**Infrastructure as
Code**](https://en.wikipedia.org/wiki/Infrastructure_as_code) platform
that streamlines the deployment of data science infrastructure.
# What is QHub?

QHUb is an integrated data science environment designed and developed by scientists at [**Quansight**](https://www.quansight.com/). It enables teams to build and maintain a cost effective and scalable compute/data science platform in the Cloud, providing an [**Infrastructure as Code**](https://en.wikipedia.org/wiki/Infrastructure_as_code) platform that streamlines the deployment of data science infrastructure.

```{toctree}
:maxdepth: 1
Expand All @@ -26,4 +19,3 @@ source/user_guide/index.md
source/admin_guide/index.md
source/dev_guide/index.md
```

86 changes: 33 additions & 53 deletions docs/source/admin_guide/backup.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,21 @@
# Manual Backups
This method for backing up and restoring requires amount to:
# Manual backups

- tarballing the /home directory
- saving to block storage [s3, google cloud storage, etc]
- downloading and untaring to new cluster
Manual backups requires amount to:

This specific guide will show how to do this on a cluster on AWS
- Tarballing the /home directory
- Saving to block storage [s3, google cloud storage, etc]
- Downloading and untaring to new cluster

This specific guide shows how to do this on a cluster on AWS.

## Pre-requisites

- [Install kubectl](<https://kubernetes.io/docs/tasks/tools/>)
- [Install AWS CLI](<https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html>)
- [Install kubectl](<https://kubernetes.io/docs/tasks/tools/>)
- [Install AWS command-line tool](<https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html>)

## Kubectl configuration

To setup kubectl, first we must obtain the name of the cluster. So
long as you know the region your current cluster is deployed in, this
is straightforward:
To setup kubectl, obtain the name of the cluster. If the user knows the deployment region of the current cluster, this is straightforward:

```shell
aws eks list-clusters --region=us-west-2
Expand All @@ -30,9 +29,7 @@ aws eks update-kubeconfig --region us-west-2 --name <relevant-name>

## Pod deployment

With kubectl configured, we now need to deploy the pod that will allow
us to access the cluster files. First save the follow pod spec to a
file named `pod.yaml`
With kubectl configured, the user now needs to deploy the pod that allows the user to access the cluster files. First save the follow pod specification to a file named `pod.yaml`

```yaml
kind: Pod
Expand All @@ -54,20 +51,15 @@ spec:
name: volume-to-debug-ubuntu
```
To determine what should replace `<mount-share-drive>` run `kubectl
get pvc -n dev`. This will be the volume that doesn't have conda in
the name, and will have the same storage space as specified by the
`shared<sub>filesystem</sub>` line in `qhub-config.yaml`. In my example the name
is `nfs-mount-dev-share`
To determine what should replace `<mount-share-drive>` run `kubectl get pvc -n dev`. This is the volume that doesn't has conda in the name, and has the same storage space as specified by the `shared<sub>filesystem</sub>` line in `qhub-config.yaml`. In the example the name is `nfs-mount-dev-share`

With the name of the nfs volume saved in the file, run:

```shell
kubectl apply -f pod.yaml -n dev
```

If you have a namespace other than the default dev, replace `dev` with
your namespace. To get a shell to this running pod, run:
If you have a namespace other than the default dev, replace `dev` with your namespace. To get a shell to this running pod, run:

```shell
kubectl exec -n dev --stdin --tty volume-debugger-ubuntu -- /bin/bash
Expand All @@ -77,16 +69,15 @@ Again replacing the `dev` namespace as needed.

## Installations

The pod we spun up is a basic pod, so several apt packages must be
installed. The following commands will install them:
The user must install several `apt` packages, as the pod spun up is a basic pod. The following commands installs them:

```shell
apt update
apt install curl -y
apt install unzip -y
```

Because we are on aws, we will then install the AWS CLI:
Because the user is on AWS, the AWS command-line tool is also installed.:

```shell
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
Expand All @@ -95,47 +86,40 @@ unzip awscliv2.zip
aws configure
```

The last command from above will prompt for your aws public/private
key, and default region. Past each of these and press enter. The
output can be ignored and skipped by pressing enter.
The last command from preceding prompts for your AWS public/private key, and default region. Past each of these and press enter. To ignore and skip the output, press enter.

## Backups

The file system can now be backed up with the following:
To backup the file system, run:

```shell
cd /data
tar -cvf <custom_name>.tar /home
```

My preferred naming scheme includes a year-month-day,
e.g. `2021-04-23<sub>home</sub><sub>backup.tar</sub>`. This helps when
multiple backups are used. This step will take several minutes
depending on the size of the home directories.
The preferred naming scheme includes a year-month-day, example `2021-04-23<sub>home</sub><sub>backup.tar</sub>`. The user can utilize multi-backups through this step. This step takes several minutes depending on the size of the home directories.

## Upload to block storage
Once this is complete, the AWS CLI can be used to upload the tar file to s3.

Once this is complete, the user uploads the tar file to S3 using the AWS command-line tool:

```shell
aws s3 cp 2021-04-23.tar s3://<your_bucket_name>/backups/2021-04-23.tar
```

Replacing your <your<sub>bucket</sub><sub>name</sub>> with a bucket
you have created. If you don't have an existing bucket, instructions
are here:
Replacing your <your<sub>bucket</sub><sub>name</sub>> with a bucket you have created. If you don't have an existing bucket, instructions are here:
<https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html>

\## Download from block storage and decompress
## Download from block storage and decompress

Now that we have the data backed up,
perform the same steps above for the new cluster. This includes:
Now that the data backed up, perform the same steps preceding for the new cluster. This includes:

- configuring kubectl for the new cluster
- creating a pod on the new cluster and getting shell access into it
- installing the apt packages
- configuring AWS
- Configuring kubectl for the new cluster.
- Creating a pod on the new cluster and getting shell access into it.
- Installing the apt packages.
- Configuring AWS.

Once AWS is configured on the new pod, we can then download the backup with:
Once AWS gets configured on the new pod, the user can then download the backup with:

```shell
cd /data
Expand All @@ -149,14 +133,11 @@ cd /data
tar -xvf 2021-04-23.tar
```

By default tar will keep all of the same file permissions as the
original files that were tarballed. That's it! Enjoy.
The file permissions for the default tar is same as the original files.

## Google Cloud Provider
## Google cloud provider

The same instructions as above, except install
(gsutil)[https://cloud.google.com/storage/docs/gsutil_install) and use
these commands for copy/download of the backup
To use the Google Cloud provider, install [gsutil](https://cloud.google.com/storage/docs/gsutil_install). The instructions are same as the preceding steps. Additionally, use these commands for copy/download of the backup:

```shell
cd /data
Expand All @@ -165,9 +146,8 @@ gsutil cp 2021-04-23.tar gs://<your_bucket_name>/backups/2021-04-23.tar
cd /data
gsutil cp gs://<your_bucket_name>/backups/2021-04-23.tar .
```

## Digital Ocean

Similar instructions, but use digital ocean spaces. This guide will
explain installation of the cli:
Similar instructions, but use Digital Ocean spaces. This guide explains installation of the command-line tool:
https://www.digitalocean.com/community/tutorials/how-to-migrate-from-amazon-s3-to-digitalocean-spaces-with-rclone
21 changes: 7 additions & 14 deletions docs/source/admin_guide/clearml.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,20 @@
# ClearML

ClearML integration comes built in with QHub, here is how you would
enable this integration. Currently ClearML integration is only
supported on Google Cloud Platform.
ClearML integration comes built in with QHub, here is how you would enable this integration. Currently ClearML integration is only supported on Google Cloud Platform.

## Setting subdomain DNS Record for ClearML

ClearML components requires subdomains, you would need to set a CNMAE
or A record for the following subdomains on your QHub.
ClearML components requires subdomains, you would need to set a `CNAME` or A record for the following subdomains on your QHub.

- app.clearml.your-qhub-domain.com
- files.clearml.your-qhub-domain.com
- api.clearml.your-qhub-domain.com

Note: These domains are automatically setup for you, if you're using Cloudflare and the
args `--dns-provider cloudflare --dns-auto-provision` passed to `qhub deploy`.

These domains are automatically setup for you, if you're using Cloudflare and the args `--dns-provider cloudflare --dns-auto-provision` passed to `qhub deploy`.

## Create a node group

1. To enable ClearML integration on GCP QHub deployments, simply enable the feature flag within your `qhub-config.yaml` file. For example:
1. To enable ClearML integration on Google Cloud QHub deployments, simply enable the feature flag within your `qhub-config.yaml` file. For example:

```yaml
clearml:
Expand All @@ -41,15 +36,13 @@ google_cloud_platform:
app: clearml
```

## Accessing the ClearML server
## Accessing the server

The ClearML server can be accessed at: `app.clearml.your-qhub-domain.com`
Users can access the ClearML server at: `app.clearml.your-qhub-domain.com`

## Authentication

The ClearML dashboard is secured by default with JupyterHub OAuth
via Traefik ForwardAuth. You can disable it via a flag in the QHub
config yaml:
QHub secures ClearML dashboard by default with JupyterHub OAuth via Traefik ForwardAuth. You can turn it off via a flag in the QHub config YAML:

```yaml
clearml:
Expand Down
Loading

0 comments on commit 0491c9a

Please sign in to comment.