Skip to content

Commit

Permalink
Merge pull request #83 from orkes-io/staging
Browse files Browse the repository at this point in the history 
added Best practices for HTTP
  • Loading branch information
@dougsillars
dougsillars authored Apr 19, 2022
2 parents aeae444 + 1424995 commit f326b1b
Show file tree
Hide file tree
Showing 2 changed files with 62 additions and 97 deletions.
141 changes: 45 additions & 96 deletions docs/how-tos/sdks/conductor-go/main/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,106 +6,42 @@ To find out more about Conductor visit: [https://github.com/Netflix/conductor](h

## Quick Start

1. [Setup conductor-go package](#Setup-conductor-go-package)
1. [Write worker as a function](#Write-worker-as-a-function)
2. [Run workers](#Run-workers)
3. [Configuration](#Configuration)
1. [Run workers](#Run-workers)
1. [Configuration](#Configuration)

### Setup-conductor-go-package

Create a folder to build your package:
```shell
$ mkdir conductor-go/
$ cd conductor-go/
```

### Write worker as a function
Create a `go.mod` file inside this folder, with this content:
```
module conductor_test
```go
package example
go 1.18
import (
"github.com/netflix/conductor/client/go/conductor_client/model"
"github.com/netflix/conductor/client/go/conductor_client/model/enum/task_result_status"
log "github.com/sirupsen/logrus"
require (
github.com/conductor-sdk/conductor-go v1.0.8
)

func TaskExecuteFunctionExample(t *model.Task) (taskResult *model.TaskResult, err error) {
log.Debug("Executing Task_Execution_Function_Example for", t.TaskType)
taskResult = model.NewTaskResult(t)
taskResult.OutputData = map[string]interface{}{
"task": "task_1",
"key2": "value2",
"key3": 3,
"key4": false,
}
taskResult.Logs = append(taskResult.Logs, model.LogMessage{Log: "Hello World"})
taskResult.Status = task_result_status.COMPLETED
err = nil
return taskResult, err
}
```
### Run workers
Create main method that does the following:
1. Adds configurations such as metrics, authentication, thread count, Conductor server URL
2. Add your workers
3. Start the workers to poll for work

You can copy this code and put into `main.go` file

```go
package main
Now you may be able to create your workers and main function.

import (
"os"

"github.com/netflix/conductor/client/go/example/task_execute_function"
"github.com/netflix/conductor/client/go/metrics"
"github.com/netflix/conductor/client/go/orkestrator"
"github.com/netflix/conductor/client/go/settings"
log "github.com/sirupsen/logrus"
)
### Write worker as a function
You can download [this code](examples/task_execute_function/task_execute_function.go) into the repository folder with:
```shell
$ wget "https://github.com/conductor-sdk/conductor-go/blob/main/examples/task_execute_function/task_execute_function.go"
```

// Example init function that shows how to configure logging
// Using json formatter and changing level to Debug
func init() {
// Log as JSON instead of the default ASCII formatter.
log.SetFormatter(&log.JSONFormatter{})
//Stdout, change to a file for production use case
log.SetOutput(os.Stdout)
// Set to debug for demonstration. Change to Info for production use cases.
log.SetLevel(log.DebugLevel)
}

// Example main function to start workers
func main() {
// MetricsSettings is optional,
// could be nil instead and use default settings
go metrics.ProvideMetrics(
settings.NewMetricsSettings(
"/metrics", // api endpoint
2112, // port
),
)
// AuthenticationSettings and HttpSettings are optional,
// could be nil instead and use default settings
workerOrkestrator := orkestrator.NewWorkerOrkestrator(
settings.NewAuthenticationSettings(
"keyId", // key id from your application
"keySecret", // key secret from your application
),
settings.NewHttpSettings(
"https://play.orkes.io/api", // conductor http server url
),
)
workerOrkestrator.StartWorker(
"go_task_example", // task definition name
task_execute_function.Example1, // task execution function
1, // parallel go routines amount
5000, // 5000ms
)
workerOrkestrator.StartWorker(
"go_task_example", // task definition name
task_execute_function.Example2, // task execution function
1, // parallel go routines amount
100, // 100ms
)
// Wait for all workers to finish, otherwise would terminate them
workerOrkestrator.WaitWorkers()
}
### Run workers
You can download [this code](examples/main/main.go) into the repository folder with:
```shell
$ wget "https://github.com/conductor-sdk/conductor-go/blob/main/examples/main/main.go"
```

### Running Conductor server locally in 2-minute
Expand Down Expand Up @@ -188,7 +124,7 @@ curl -X 'POST' \

## Configuration

### Authentication settings
### Authentication settings (optional)
Use if your conductor server requires authentication
* keyId: Key
* keySecret: Secret for the Key
Expand All @@ -200,13 +136,26 @@ authenticationSettings := settings.NewAuthenticationSettings(
),
```

### HTTP Settings
### External Storage Settings (optional)
Use if you would like to upload large payload at an external storage
You may define max payload size and threshold for uploading, also with a function capable of returning the path where it is stored.

```go
externalStorageSettings := settings.NewExternalStorageSettings(
4, // taskOutputPayloadThresholdKB
10, // taskOutputMaxPayloadThresholdKB
external_storage_handler.UploadAndGetPath, // External Storage Handler function
),
```

### HTTP Settings (optional)

* baseUrl: Conductor server address. e.g. http://localhost:8000 if running locally

```go
httpSettings := settings.NewHttpSettings(
"https://play.orkes.io/api",
externalStorageSettings,
)
```

Expand Down Expand Up @@ -237,9 +186,9 @@ You can create a new worker by calling `workerOrkestrator.StartWorker` with:

```go
workerOrkestrator.StartWorker(
"go_task_example",
example.TaskExecuteFunctionExample1,
1,
100,
"go_task_example", // task definition name
task_execute_function.Example1, // task execution function
1, // parallel go routines amount
5000, // 5000ms
)
```
```
18 changes: 17 additions & 1 deletion docs/reference-docs/system-tasks/http-task.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,4 +142,20 @@ Following is the example of HTTP task with `DELETE` method.
1. Why are my HTTP tasks not getting picked up?
1. We might have too many HTTP tasks in the queue. There is a concept called Isolation Groups that you can rely on
for prioritizing certain HTTP tasks over others. Read more here: [Isolation Groups](https://netflix.github.io/conductor/configuration/isolationgroups/)

2. Why is my HTTP Task timing out with ```Failed to invoke HTTP task due to: java.lang.Exception: I/O error on GET request for "<url>": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out```?
1. The default timeout for an HTTP request is 150ms. If your API takes longer than this, you will need to increase the timeout parameters. In your ```inputParameters``` under ```http_request``` add the two following parameters (the timeouts are in milliseconds):
```json
"connectionTimeOut": 1000,
"readTimeOut": 1000
```

3. Can I retry my HTTP Tasks?
1. Yes. See [Extending System Tasks](/content/docs/how-tos/Tasks/extending-system-tasks) to see how to add retries, and retry parameters to your HTTP Task.

4. I'm getting rate limited. Can I slow down my HTTP Task?
1. Yes! By [Extending System Tasks](/content/docs/how-tos/Tasks/extending-system-tasks), and adding the following parameters:
```json
"rateLimitPerFrequency": 100,
"rateLimitFrequencyInSeconds": 60,
```
2. This will allow only 100 calls to the API endpoint in 60 seconds. Modify the values as required.

0 comments on commit f326b1b

Please sign in to comment.