Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs][Connector-V2][MyHours]Reconstruct the MyHours connector document #5129

Merged
merged 3 commits into from
Aug 11, 2023
+101 −114
Merged

[Docs][Connector-V2][MyHours]Reconstruct the MyHours connector document #5129

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0e88f44
[Docs][Connector-V2][MyHours]Reconstruct the MyHours connector document
TaoZex Jul 20, 2023
cd74a80
fix format
TaoZex Jul 25, 2023
2c12af0
fix format
TaoZex Jul 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
215 changes: 101 additions & 114 deletions docs/en/connector-v2/source/MyHours.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,13 @@

> My Hours source connector
## Description
## Support Those Engines

Used to read data from My Hours.
> Spark<br/>
> Flink<br/>
> SeaTunnel Zeta<br/>
## Key features
## Key Features

- [x] [batch](../../concept/connector-v2-features.md)
- [ ] [stream](../../concept/connector-v2-features.md)
Expand All @@ -15,71 +17,103 @@ Used to read data from My Hours.
- [ ] [parallelism](../../concept/connector-v2-features.md)
- [ ] [support user-defined split](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|-----------------------------|---------|----------|---------------|
| url | String | Yes | - |
| email | String | Yes | - |
| password | String | Yes | - |
| method | String | No | get |
| schema | Config | No | - |
| schema.fields | Config | No | - |
| format | String | No | json |
| params | Map | No | - |
| body | String | No | - |
| json_field | Config | No | - |
| content_json | String | No | - |
| poll_interval_ms | int | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
| retry_backoff_max_ms | int | No | 10000 |
| enable_multi_lines | boolean | No | false |
| common-options | config | No | - |

### url [String]

http request url

### email [String]

email for login

### password [String]

password for login

### method [String]

http request method, only supports GET, POST method

### params [Map]

http params

### body [String]

http body

### poll_interval_ms [int]
## Description

request http api interval(millis) in stream mode
Used to read data from My Hours.

### retry [int]
## Key features

The max retry times if request http return to `IOException`
- [x] [batch](../../concept/connector-v2-features.md)
- [ ] [stream](../../concept/connector-v2-features.md)
- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [ ] [column projection](../../concept/connector-v2-features.md)
- [ ] [parallelism](../../concept/connector-v2-features.md)
- [ ] [support user-defined split](../../concept/connector-v2-features.md)

### retry_backoff_multiplier_ms [int]
## Supported DataSource Info

In order to use the My Hours connector, the following dependencies are required.
They can be downloaded via install-plugin.sh or from the Maven central repository.

| Datasource | Supported Versions | Dependency |
|------------|--------------------|------------------------------------------------------------------------------------------------|
| My Hours | universal | [Download](https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-connectors-v2) |

## Source Options

| Name | Type | Required | Default | Description |
|-----------------------------|---------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------|
| url | String | Yes | - | Http request url. |
| email | String | Yes | - | My hours login email address. |
| password | String | Yes | - | My hours login password. |
| schema | Config | No | - | Http and seatunnel data structure mapping |
| schema.fields | Config | No | - | The schema fields of upstream data |
| json_field | Config | No | - | This parameter helps you configure the schema,so this parameter must be used with schema. |
| content_json | String | No | - | This parameter can get some json data.If you only need the data in the 'book' section, configure `content_field = "$.store.book.*"`. |
| format | String | No | json | The format of upstream data, now only support `json` `text`, default `json`. |
| method | String | No | get | Http request method, only supports GET, POST method. |
| headers | Map | No | - | Http headers. |
| params | Map | No | - | Http params. |
| body | String | No | - | Http body. |
| poll_interval_ms | Int | No | - | Request http api interval(millis) in stream mode. |
| retry | Int | No | - | The max retry times if request http return to `IOException`. |
| retry_backoff_multiplier_ms | Int | No | 100 | The retry-backoff times(millis) multiplier if request http failed. |
| retry_backoff_max_ms | Int | No | 10000 | The maximum retry-backoff times(millis) if request http failed |
| enable_multi_lines | Boolean | No | false | |
| common-options | | No | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details |

## How to Create a My Hours Data Synchronization Jobs

The retry-backoff times(millis) multiplier if request http failed
```hocon
env {
execution.parallelism = 1
job.mode = "BATCH"
}
### retry_backoff_max_ms [int]
MyHours{
url = "https://api2.myhours.com/api/Projects/getAll"
email = "seatunnel@test.com"
password = "seatunnel"
schema {
fields {
name = string
archived = boolean
dateArchived = string
dateCreated = string
clientName = string
budgetAlertPercent = string
budgetType = int
totalTimeLogged = double
budgetValue = double
totalAmount = double
totalExpense = double
laborCost = double
totalCost = double
billableTimeLogged = double
totalBillableAmount = double
billable = boolean
roundType = int
roundInterval = int
budgetSpentPercentage = double
budgetTarget = int
budgetPeriodType = string
budgetSpent = string
id = string
}
}
}
The maximum retry-backoff times(millis) if request http failed
# Console printing of the read data
sink {
Console {
parallelism = 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
parallelism = 1

}
}
```

### format [String]
## Parameter Interpretation

the format of upstream data, now only support `json` `text`, default `json`.
### format

when you assign format is `json`, you should also assign schema option, for example:

Expand All @@ -98,11 +132,11 @@ you should assign schema as the following:
```hocon
schema {
fields {
code = int
data = string
success = boolean
}
fields {
code = int
data = string
success = boolean
}
}
```
Expand Down Expand Up @@ -131,13 +165,7 @@ connector will generate data as the following:
|----------------------------------------------------------|
| {"code": 200, "data": "get success", "success": true} |

### schema [Config]

#### fields [Config]

the schema fields of upstream data

### content_json [String]
### content_json
ic4y marked this conversation as resolved.
Show resolved Hide resolved

This parameter can get some json data.If you only need the data in the 'book' section, configure `content_field = "$.store.book.*"`.

Expand Down Expand Up @@ -212,14 +240,14 @@ Here is an example:
- Test data can be found at this link [mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
- See this link for task configuration [http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).

### json_field [Config]
### json_field
ic4y marked this conversation as resolved.
Show resolved Hide resolved

This parameter helps you configure the schema,so this parameter must be used with schema.

If your data looks something like this:

```json
{
{
"store": {
"book": [
{
Expand Down Expand Up @@ -273,47 +301,6 @@ source {
- Test data can be found at this link [mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
- See this link for task configuration [http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).

### common options

Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details

## Example

```hocon
MyHours{
url = "https://api2.myhours.com/api/Projects/getAll"
email = "seatunnel@test.com"
password = "seatunnel"
schema {
fields {
name = string
archived = boolean
dateArchived = string
dateCreated = string
clientName = string
budgetAlertPercent = string
budgetType = int
totalTimeLogged = double
budgetValue = double
totalAmount = double
totalExpense = double
laborCost = double
totalCost = double
billableTimeLogged = double
totalBillableAmount = double
billable = boolean
roundType = int
roundInterval = int
budgetSpentPercentage = double
budgetTarget = int
budgetPeriodType = string
budgetSpent = string
id = string
}
}
}
```

## Changelog

### next version
Expand Down