Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue #417] gRPC design doc and protobuf models #593

Closed
wants to merge 30 commits into from
Closed
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
ba67c5c
Merge pull request #1 from apache/develop
jinrongluo May 10, 2021
d638ec4
[Issue #337] Fix HttpSubscriber startup issue
May 10, 2021
5ebfb54
[Issue #337] test commit
jinrongluo May 10, 2021
a3afff3
[Issue #337] revert test commit
jinrongluo May 10, 2021
50f959d
[Issue #337] Enhance Http Demo Subscriber by using ExecutorService, C…
jinrongluo May 11, 2021
7adc322
Merge remote-tracking branch 'origin/develop' into develop
jinrongluo May 11, 2021
d48ead5
[Issue #337] Enhance Http Demo Subscriber by using ExecutorService, C…
jinrongluo May 11, 2021
c9021fe
[Issue #337] Address code review comment for Subscriber Demo App
jinrongluo May 12, 2021
c6d732e
Merge branch 'apache:develop' into develop
jinrongluo May 14, 2021
c613be8
Merge branch 'apache:develop' into develop
jinrongluo May 18, 2021
66ac95e
Merge branch 'apache:develop' into develop
jinrongluo May 19, 2021
9e636c0
Merge branch 'apache:develop' into develop
jinrongluo May 26, 2021
a0f44b0
Merge branch 'apache:develop' into develop
jinrongluo Jun 4, 2021
37f5d7a
Merge branch 'apache:develop' into develop
jinrongluo Jun 12, 2021
93e711c
Merge branch 'apache:develop' into develop
jinrongluo Jun 15, 2021
e75d67a
Merge branch 'apache:develop' into develop
jinrongluo Jun 21, 2021
2fb485b
Merge branch 'apache:develop' into develop
jinrongluo Jul 5, 2021
80ab9af
Merge branch 'apache:develop' into develop
jinrongluo Sep 14, 2021
87d1747
Merge branch 'apache:develop' into develop
jinrongluo Sep 17, 2021
c9d3537
Merge branch 'apache:develop' into develop
jinrongluo Sep 26, 2021
c9e38f5
Merge branch 'apache:develop' into develop
jinrongluo Sep 27, 2021
63728fc
Merge branch 'apache:develop' into develop
jinrongluo Oct 12, 2021
ad18b29
Merge branch 'apache:develop' into develop
jinrongluo Oct 16, 2021
0c58188
Merge branch 'apache:develop' into develop
jinrongluo Oct 18, 2021
0141006
Merge branch 'apache:develop' into develop
jinrongluo Nov 12, 2021
98b32c6
add eventmesh gric protobuf
jinrongluo Nov 15, 2021
99fe0b9
update eventmesh client protos and generated codes
jinrongluo Nov 16, 2021
834a6f8
update grpc design docs
jinrongluo Nov 16, 2021
733af59
[Issue# 417] adding license header
jinrongluo Nov 16, 2021
87c5c2f
Merge branch 'apache:develop' into grpc-dev
jinrongluo Nov 16, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 120 additions & 0 deletions docs/cn/instructions/eventmesh-runtime-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,3 +258,123 @@ public class LiteMessage {
| 场景 | Server向Client发送消息请求码 | Client回复Server消息响应码 | 说明 |
| ------------------ | ---------------------------- | -------------------------- | ---------------------- |
| 客户端接收异步事件 | HTTP_PUSH_CLIENT_ASYNC(105) | retCode | retCode值为0时代表成功 |


## gRPC 协议文档

#### 1. protobuf

在 `eventmesh-protocol-gprc` 模块有 Eventmesh gRPC 客户端的 protobuf 文件. the protobuf 文件路径是 `/src/main/proto/eventmesh-client.proto`.

用gradle build 生成 gRPC 代码在 `/build/generated/source/proto/main`. 生成代码用于 `eventmesh-sdk-java` 模块.

#### 2. gRPC 数据模型

- 消息

以下消息数据模型用于 `publish()`, `requestReply()` 和 `broadcast()` APIs.

```
message RequestHeader {
string env = 1;
string region = 2;
string idc = 3;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

region, namespace, idc, rack and so on. these tenant or geographic info should be carefully designed in our v2 version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, any guys want to add comment for every field in protocol?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is V1 version of protobuf and we are experimenting this grpc transport feature.

Please review and comment on this set of header attributes and improve it in V2. @qqeasonchen @ruanwenjun

Thanks.

string ip = 4;
string pid = 5;
string sys = 6;
string username = 7;
string password = 8;
string version = 9;
string language = 10;
string seqNum = 11;
}

message Message {
RequestHeader header = 1;
string productionGroup = 2;
string topic = 3;
string content = 4;
string ttl = 5;
string uniqueId = 6;
}

message Response {
string respCode = 1;
string respMsg = 2;
string respTime = 3;
string seqNum = 4;
}
```

- 订阅

以下订阅数据模型用于 `subscribe()` 和 `unsubscribe()` APIs.

```
message Subscription {
RequestHeader header = 1;
string consumerGroup = 2;

message SubscriptionItem {
string topic = 1;
string mode = 2;
string type = 3;
string url = 4;
}

repeated SubscriptionItem subscriptionItems = 3;
}
```

- 心跳

以下心跳数据模型用于 `heartbeat()` API.

```
message Heartbeat {
RequestHeader header = 1;
string clientType = 2;
string producerGroup = 3;
string consumerGroup = 4;

message HeartbeatItem {
string topic = 1;
string url = 2;
}

repeated HeartbeatItem heartbeatItems = 5;
}
```

#### 3. gRPC 服务接口

- 事件生产端服务 APIs

```
# 异步事件生产
rpc publish(Message) returns (Response);

# 同步事件生产
rpc requestReply(Message) returns (Response);

# 事件广播
rpc broadcast(Message) returns (Response);
```

- 事件消费端服务 APIs

```
# 所消费事件通过 HTTP Webhook推送事件
rpc subscribe(Subscription) returns (Response);

# 所消费事件通过 TCP stream推送事件
rpc subscribeStream(Subscription) returns (stream Message);

rpc unsubscribe(Subscription) returns (Response);
```

- 客户端心跳服务 API

```
rpc heartbeat(Heartbeat) returns (Response);
```
123 changes: 123 additions & 0 deletions docs/en/instructions/eventmesh-runtime-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,3 +259,126 @@ same with RequestHeader of Heartbeat Msg
| Scene | Server Send | Client Reply | Remark |
| ------------------ | ---------------------------- | -------------------------- | ---------------------- |
| Push async msg to client | HTTP_PUSH_CLIENT_ASYNC(105) | retCode | retCode=0,send success |


## gRPC Protocol Document In Eventmesh-Runtime

#### 1. protobuf

The `eventmesh-protocol-gprc` module contains the protobuf file of the evenmesh client. the protobuf file
is located as `/src/main/proto/eventmesh-client.proto`.

Run the gradle build to generate the gRPC codes. The generated codes are located at `/build/generated/source/proto/main`.

These generated grpc codes will be used in `eventmesh-sdk-java` module.

#### 2. data models

- message

The following is the message data model, used by `publish()`, `requestReply()` and `broadcast()` APIs.

```
message RequestHeader {
string env = 1;
string region = 2;
string idc = 3;
string ip = 4;
string pid = 5;
string sys = 6;
string username = 7;
string password = 8;
string version = 9;
string language = 10;
string seqNum = 11;
}

message Message {
RequestHeader header = 1;
string productionGroup = 2;
string topic = 3;
string content = 4;
string ttl = 5;
string uniqueId = 6;
}

message Response {
string respCode = 1;
string respMsg = 2;
string respTime = 3;
string seqNum = 4;
}
```

- subscription

The following data model is used by `subscribe()` and `unsubscribe()` APIs.

```
message Subscription {
RequestHeader header = 1;
string consumerGroup = 2;

message SubscriptionItem {
string topic = 1;
string mode = 2;
string type = 3;
string url = 4;
}

repeated SubscriptionItem subscriptionItems = 3;
}
```

- heartbeat

The following data model is used by `heartbeat()` API.

```
message Heartbeat {
RequestHeader header = 1;
string clientType = 2;
string producerGroup = 3;
string consumerGroup = 4;

message HeartbeatItem {
string topic = 1;
string url = 2;
}

repeated HeartbeatItem heartbeatItems = 5;
}
```

#### 3. service operations

- event publisher service APIs

```
# Async event publish
rpc publish(Message) returns (Response);

# Sync event publish
rpc requestReply(Message) returns (Response);

# event broadcast
rpc broadcast(Message) returns (Response);
```

- event consumer service APIs

```
# The subscribed event will be delivered by invoking the webhook url in the Subscription
rpc subscribe(Subscription) returns (Response);

# The subscribed event will be delivered through stream of Message
rpc subscribeStream(Subscription) returns (stream Message);

rpc unsubscribe(Subscription) returns (Response);
```

- client heartbeat service API

```
rpc heartbeat(Heartbeat) returns (Response);
```
48 changes: 48 additions & 0 deletions eventmesh-connector-plugin/eventmesh-protocol-grpc/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
plugins {
id 'java'
id 'com.google.protobuf' version '0.8.17'
}

group 'org.apache.eventmesh'
version '1.3.0-SNAPSHOT'

repositories {
mavenCentral()
}

def grpcVersion = '1.15.0' // CURRENT_GRPC_VERSION
def protobufVersion = '3.5.1'
def reactiveGrpcVersion = '1.2.3'
def protocVersion = protobufVersion

dependencies {
implementation "io.grpc:grpc-protobuf:${grpcVersion}"
implementation "io.grpc:grpc-stub:${grpcVersion}"
implementation "com.google.protobuf:protobuf-java-util:${protobufVersion}"
implementation "io.reactivex.rxjava2:rxjava:2.2.21"
implementation "com.salesforce.servicelibs:rxgrpc-stub:1.2.3"

testImplementation 'org.junit.jupiter:junit-jupiter-api:5.6.0'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine'
}

protobuf {
protoc { artifact = "com.google.protobuf:protoc:${protocVersion}" }
plugins {
grpc { artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}" }
rxgrpc {
artifact = "com.salesforce.servicelibs:rxgrpc:${reactiveGrpcVersion}"
}
}
generateProtoTasks {
ofSourceSet("main")*.plugins {
grpc { }
rxgrpc {}
}
}
}


test {
useJUnitPlatform()
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
syntax = "proto3";

package tutorial;

option java_multiple_files = true;
option java_package = "com.example.tutorial.protos";
option java_outer_classname = "AddressBookProtos";

message Person {
string name = 1;
int32 id = 2;
string email = 3;

enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}

message PhoneNumber {
string number = 1;
PhoneType type = 2;
}

repeated PhoneNumber phones = 4;
}

message AddressBook {
repeated Person people = 1;
}

message GetAddressBookRequest {
string bookId = 1;
}

service AddressService {
rpc addPerson(Person) returns (AddressBook);

rpc getAddressBook(GetAddressBookRequest) returns (stream Person);
}
36 changes: 36 additions & 0 deletions eventmesh-protocol-plugin/eventmesh-protocol-grpc/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
plugins {
id 'java'
id 'com.google.protobuf' version '0.8.17'
}

repositories {
mavenCentral()
}

def grpcVersion = '1.15.0' // CURRENT_GRPC_VERSION
def protobufVersion = '3.5.1'
def protocVersion = protobufVersion

dependencies {
implementation "io.grpc:grpc-protobuf:${grpcVersion}"
implementation "io.grpc:grpc-stub:${grpcVersion}"
implementation "com.google.protobuf:protobuf-java-util:${protobufVersion}"
testImplementation 'org.junit.jupiter:junit-jupiter-api:5.6.0'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine'
}

protobuf {
protoc { artifact = "com.google.protobuf:protoc:${protocVersion}" }
plugins {
grpc { artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}" }
}
generateProtoTasks {
all()*.plugins {
grpc {}
}
}
}

test {
useJUnitPlatform()
}
Loading