Skip to content

Commit

Permalink
Jobs API docs, pt 2 (dapr#4240)
Browse files Browse the repository at this point in the history
* update overview, add to how-to

Signed-off-by: Hannah Hunter <hannahhunter@microsoft.com>

* mark review

Signed-off-by: Hannah Hunter <hannahhunter@microsoft.com>

* Update daprdocs/content/en/developing-applications/building-blocks/jobs/howto-schedule-jobs.md

Signed-off-by: Mark Fussell <markfussell@gmail.com>

* Update daprdocs/content/en/developing-applications/building-blocks/jobs/howto-schedule-jobs.md

Signed-off-by: Mark Fussell <markfussell@gmail.com>

---------

Signed-off-by: Hannah Hunter <hannahhunter@microsoft.com>
Signed-off-by: Mark Fussell <markfussell@gmail.com>
Co-authored-by: Mark Fussell <markfussell@gmail.com>
  • Loading branch information
hhunter-ms and msfussell authored Jul 3, 2024
1 parent d3f74bd commit 3231bfc
Show file tree
Hide file tree
Showing 4 changed files with 45 additions and 31 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,25 @@ weight: 2000
description: "Learn how to use the jobs API to schedule jobs"
---

Now that you've learned what the [jobs building block]({{< ref jobs-overview.md >}}) provides, let's look at an example of how to use the API. The code example below describes an application that schedules and orchestrates ***TBD***.
Now that you've learned what the [jobs building block]({{< ref jobs-overview.md >}}) provides, let's look at an example of how to use the API. The code example below describes an application that schedules jobs for a **TBD** application.

<!--
Include a diagram or image, if possible.
-->

<!--
Make sure the how-to includes examples for multiple programming languages, OS, or deployment targets, if applicable.
-->

## Configure jobs

## Set up the Scheduler service

<!--
Each H2 step should start with a verb/action word.
-->
When you run `dapr init` in either self-hosted mode or on Kubernetes, the Dapr scheduler service is started.

## Run the Dapr sidecar

Run the Dapr sidecar alongside your application.

```bash
dapr run --app-id=jobs --app-port 50070 --app-protocol grpc --log-level debug -- go run main.go
```

## Next steps

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,29 +6,16 @@ weight: 1000
description: "Overview of the jobs API building block"
---

Many applications require job scheduling, the need to take an action in the future. The jobs API is an orchestrator for scheduling these jobs in the future, either at a specific time or a specific interval.
Some typically example scenarios include;
- **Automated Database Backups**: Ensure a database is backed up daily to prevent data loss. Schedule a backup script to run every night at 2 AM, which will create a backup of the database and store it in a secure location.
- **Regular Data Processing and ETL (Extract, Transform, Load)**: Process and transform raw data from various sources and load it into a data warehouse. Schedule ETL jobs to run at specific times (for example: hourly, daily) to fetch new data, process it, and update the data warehouse with the latest information.
**Email Notifications and Reports**: Receive daily sales reports and weekly performance summaries via email. Schedule a job that generates the required reports and sends them via email at 6 AM every day for daily reports and 8 AM every Monday for weekly summaries.
**Maintenance Tasks and System Updates**: Perform regular maintenance tasks such as clearing temporary files, updating software, and checking system health. Schedule various maintenance scripts to run at off-peak hours, such as weekends or late nights, to minimize disruption to users.
**Batch Processing for Financial Transactions**: Processes a large number of transactions that need to be batched and settled at the end of each business day. Schedule batch processing jobs to run at 5 PM every business day, aggregating the day’s transactions and performing necessary settlements and reconciliations.
Using the jobs API in these scenarios ensures that tasks are performed consistently and reliably without manual intervention, improving efficiency and reducing the risk of errors. The jobs API helps you with scheduling jobs, and internally it is also used by Dapr to schedule actor reminders.
Many applications require job scheduling, or the need to take an action in the future. The jobs API is an orchestrator for scheduling these future jobs, either at a specific time or for a specific interval.

### Delayed pub/sub
Not only does the jobs API help you with scheduling jobs, but internally, Dapr uses the scheduler service to schedule actor reminders.

Use jobs to delay your pub/sub messaging. You can publish a message in a future specific time -- for example, a week from today, or a specific UTC date/time.

### Scheduled service invocation

Use jobs with [service invocation]({{< ref service-invocation-overview.md >}}) to schedules method calls between applications.



Jobs consist of:
Jobs in Dapr consist of:
- The jobs API building block
- [The Scheduler control plane service]({{< ref "concepts/dapr-services/scheduler.md" >}})

[See example scenarios.]({{< ref "#scenarios" >}})

<img src="/images/scheduler/scheduler-architecture.png" alt="Diagram showing the Scheduler control plane service and the jobs API">

## How it works
Expand All @@ -38,6 +25,31 @@ The jobs API is a job scheduler, not the executor which runs the job. The design
- **Not guaranteed:** A ceiling time on when the job is invoked *after* the due time is reached.

All job details and user-associated data for scheduled jobs are stored in an embedded Etcd database in the Scheduler service.
You can use jobs to:

- **Delay your [pub/sub messaging]({<< ref pubsub-overview.md >>}).** You can publish a message in a future specific time (for example: a week from today, or a specific UTC date/time).
- **Schedule [service invocation]({{< ref service-invocation-overview.md >}}) method calls between applications.**

## Scenarios

Job scheduling can prove helpful in the following scenarios:

- **Automated Database Backups**:
Ensure a database is backed up daily to prevent data loss. Schedule a backup script to run every night at 2 AM, which will create a backup of the database and store it in a secure location.

- **Regular Data Processing and ETL (Extract, Transform, Load)**:
Process and transform raw data from various sources and load it into a data warehouse. Schedule ETL jobs to run at specific times (for example: hourly, daily) to fetch new data, process it, and update the data warehouse with the latest information.

- **Email Notifications and Reports**:
Receive daily sales reports and weekly performance summaries via email. Schedule a job that generates the required reports and sends them via email at 6 a.m. every day for daily reports and 8 a.m. every Monday for weekly summaries.

- **Maintenance Tasks and System Updates**:
Perform regular maintenance tasks such as clearing temporary files, updating software, and checking system health. Schedule various maintenance scripts to run at off-peak hours, such as weekends or late nights, to minimize disruption to users.

- **Batch Processing for Financial Transactions**:
Processes a large number of transactions that need to be batched and settled at the end of each business day. Schedule batch processing jobs to run at 5 PM every business day, aggregating the day’s transactions and performing necessary settlements and reconciliations.

Dapr's jobs API ensures the tasks represented in these scenarios are performed consistently and reliably without manual intervention, improving efficiency and reducing the risk of errors.

## Features

Expand Down
6 changes: 3 additions & 3 deletions daprdocs/content/en/reference/cli/dapr-init.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ dapr init [flags]
| `--runtime-version` | | `latest` | The version of the Dapr runtime to install, for example: `1.0.0` |
| `--image-variant` | | | The image variant to use for the Dapr runtime, for example: `mariner` |
| `--set` | | | Configure options on the command line to be passed to the Dapr Helm chart and the Kubernetes cluster upon install. Can specify multiple values in a comma-separated list, for example: `key1=val1,key2=val2` |
| `--slim`, `-s` | | `false` | Exclude placement service, Redis and Zipkin containers from self-hosted installation |
| `--slim`, `-s` | | `false` | Exclude placement service, scheduler service, and the Redis and Zipkin containers from self-hosted installation |
| `--timeout` | | `300` | The wait timeout for the Kubernetes installation |
| `--wait` | | `false` | Wait for Kubernetes initialization to complete |
| N/A |DAPR_DEFAULT_IMAGE_REGISTRY| | It is used to specify the default container registry to pull images from. When its value is set to `GHCR` or `ghcr` it pulls the required images from Github container registry. To default to Docker hub, unset the environment variable or leave it blank|
Expand All @@ -55,7 +55,7 @@ dapr init [flags]

**Install**

Install Dapr by pulling container images for Placement, Redis, and Zipkin. By default, these images are pulled from Docker Hub.
Install Dapr by pulling container images for Placement, Scheduler, Redis, and Zipkin. By default, these images are pulled from Docker Hub.

```bash
dapr init
Expand Down Expand Up @@ -187,7 +187,7 @@ Use the `--set` flag to configure a set of [Helm Chart values](https://github.co
dapr init -k --set global.tag=1.0.0 --set dapr_operator.logLevel=error
```

You can also specify a private registry to pull container images from. As of now `dapr init -k` does not use specific images for sentry, operator, placement and sidecar. It relies on only Dapr runtime container image `dapr` for all these images.
You can also specify a private registry to pull container images from. As of now `dapr init -k` does not use specific images for sentry, operator, placement, scheduler, and sidecar. It relies on only Dapr runtime container image `dapr` for all these images.

Scenario 1 : dapr image hosted directly under root folder in private registry -
```bash
Expand Down
4 changes: 2 additions & 2 deletions daprdocs/content/en/reference/cli/dapr-uninstall.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ dapr uninstall [flags]

| Name | Environment Variable | Default | Description |
| -------------------- | -------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--all` | | `false` | Remove Redis, Zipkin containers in addition to actor placement container. Remove default dapr dir located at `$HOME/.dapr or %USERPROFILE%\.dapr\`. |
| `--all` | | `false` | Remove Redis, Zipkin containers in addition to the scheduler service and the actor placement container. Remove default dapr dir located at `$HOME/.dapr or %USERPROFILE%\.dapr\`. |
| `--help`, `-h` | | | Print this help message |
| `--kubernetes`, `-k` | | `false` | Uninstall Dapr from a Kubernetes cluster |
| `--namespace`, `-n` | | `dapr-system` | The Kubernetes namespace to uninstall Dapr from |
Expand All @@ -38,7 +38,7 @@ dapr uninstall [flags]
dapr uninstall
```

You can also use option `--all` to remove .dapr directory, Redis, Placement and Zipkin containers
You can also use option `--all` to remove .dapr directory, Redis, Placement, Scheduler, and Zipkin containers

```bash
dapr uninstall --all
Expand Down

0 comments on commit 3231bfc

Please sign in to comment.