Skip to content

Commit

Permalink
Add open inference protocol adoptions
Browse files Browse the repository at this point in the history
Signed-off-by: Dan Sun <dsun20@bloomberg.net>
  • Loading branch information
yuzisun committed Feb 19, 2023
1 parent fc7b5b1 commit 0f649b5
Show file tree
Hide file tree
Showing 3 changed files with 14 additions and 9 deletions.
17 changes: 11 additions & 6 deletions docs/blog/articles/2023-02-05-KServe-0.10-release.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Announcing: KServe v0.10.0

We are excited to announce KServe 0.10 release. In this release we have enabled more KServe networking options,
improved metrics instruments for supported serving runtimes and increased support coverage for [Open(aka v2) inference protocol](https://kserve.github.io/website/0.10/modelserving/data_plane/v2_protocol/) for both standard and ModelMesh InferenceService.
improved KServe telemetry for supported serving runtimes and increased support coverage for [Open(aka v2) inference protocol](https://kserve.github.io/website/0.10/modelserving/data_plane/v2_protocol/) for both standard and ModelMesh InferenceService.

## KServe Networking Options

Istio is now optional for both `Serverless` and `RawDeployment` mode. Please see the [alternative networking guide](https://kserve.github.io/website/0.10/admin/serverless/kourier_networking/) for how you can enable other ingress options supported by Knative with Serverless mode.
Istio is now optional for both [Serverless](https://kserve.github.io/website/0.10/admin/serverless/serverless/) and [RawDeployment](https://kserve.github.io/website/0.10/admin/kubernetes_deployment/) mode. Please see the [alternative networking guide](https://kserve.github.io/website/0.10/admin/serverless/kourier_networking/) for how you can enable other ingress options supported by Knative with Serverless mode.
For Istio users, if you want to turn on full service mesh mode to secure InferenceService with mutual TLS and enable the traffic policies, please read the [service mesh setup guideline](https://kserve.github.io/website/0.10/admin/serverless/servicemesh/).

## KServe Telemetry for Serving Runtimes
Expand All @@ -16,7 +16,11 @@ Please read the [prometheus metrics setup guideline](https://kserve.github.io/we

## Open(v2) Inference Protocol Support Coverage

In KServe 0.10, we have added support for Open(v2) inference protocol for KServe custom runtimes.
As there have been increasing adoptions for `KServe v2 Inference Protocol` from [AMD Inference ServingRuntime](https://kserve.github.io/website/0.10/modelserving/v1beta1/amd/) which
supports FPGAs and OpenVINO which now provides KServe [REST](https://docs.openvino.ai/latest/ovms_docs_rest_api_kfs.html) and [gRPC](https://docs.openvino.ai/latest/ovms_docs_grpc_api_kfs.html) compatible API,
in [the issue](https://github.com/kserve/kserve/issues/2663) we have proposed to rename to `KServe Open Inference Protocol`.

In KServe 0.10, we have added Open(v2) inference protocol support for KServe custom runtimes.
Now, you can enable v2 REST/gRPC for both custom transformer and predictor with images built by implementing KServe Python SDK API.
gRPC enables high performance inference data plane as it is built on top of HTTP/2 and binary data transportation which is more efficient to send over the wire compared to REST.
Please see the detailed example for [transformer](https://kserve.github.io/website/0.10/modelserving/v1beta1/transformer/torchserve_image_transformer/) and
Expand Down Expand Up @@ -64,9 +68,9 @@ You can use the same Python API type `InferRequest` and `InferResponse` for both
If you have existing custom transformer or predictor, the `headers` argument is now required to add to the `preprocess`, `predict` and `postprocess` handlers.


Please check the following matrix for supported ServingRuntimes and ModelFormats.
Please check the following matrix for supported ModelFormats and [ServingRuntimes](https://kserve.github.io/website/0.10/modelserving/v1beta1/serving_runtime/).

| Model Format | v1 | v2 REST/gRPC |
| Model Format | v1 | Open(v2) REST/gRPC |
| ------------------- |--------------| ----------------|
| Tensorflow | ✅ TFServing | ✅ Triton |
| PyTorch | ✅ TorchServe | ✅ TorchServe |
Expand All @@ -85,7 +89,7 @@ KServe control plane images [kserve-controller](https://hub.docker.com/r/kserve/
[kserve/agent](https://hub.docker.com/r/kserve/agent/tags), [kserve/router](https://hub.docker.com/r/kserve/router/tags) are now supported
for multiple architectures: `ppc64le`, `arm64`, `amd64`, `s390x`.

## KServe Storage Credentials
## KServe Storage Credentials Support

- Currently, AWS users need to create a secret with long term/static IAM credentials for downloading models stored in S3.
Security best practice is to use [IAM role for service account(IRSA)](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/)
Expand Down Expand Up @@ -114,6 +118,7 @@ For a complete change list please read the release notes from [KServe v0.10](htt


Thanks for all the contributors who have made the commits to 0.10 release!

- [Steve Larkin](https://github.com/sel)
- [Stephan Schielke](https://github.com/stephanschielke)
- [Curtis Maddalozzo](https://github.com/cmaddalozzo)
Expand Down
2 changes: 1 addition & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ nav:
- V1 Inference Protocol: modelserving/data_plane/v1_protocol.md
- Open Inference Protocol (V2 Inference Protocol): modelserving/data_plane/v2_protocol.md
- Serving Runtimes: modelserving/servingruntimes.md
- Single Model Serving:
- Model Serving Runtimes:
- Supported Model Frameworks/Formats:
- Overview: modelserving/v1beta1/serving_runtime.md
- Tensorflow: modelserving/v1beta1/tensorflow/README.md
Expand Down
4 changes: 2 additions & 2 deletions overrides/home.html
Original file line number Diff line number Diff line change
Expand Up @@ -84,12 +84,12 @@ <h2>KServe Components</h2>
<div class="org-card--body">
<div class="org-card--body-heading">
<h5 class="org-card--heading" id="org-card--heading">
<a href="./modelserving/v1beta1/serving_runtime">Single Model Serving</a>
<a href="./modelserving/v1beta1/serving_runtime">Model Serving</a>
</h5>
</div>
<div class="org-card--body-content">
<div class="org-card--body-content-wrapper">
Provides Serverless deployment of single model inference on CPU/GPU for common ML frameworks
Provides Serverless deployment for model inference on CPU/GPU with common ML frameworks
<a href="https://scikit-learn.org/">Scikit-Learn</a>, <a href="https://xgboost.readthedocs.io/">XGBoost</a>, <a href="https://www.tensorflow.org/">Tensorflow</a>, <a href="https://pytorch.org/">PyTorch</a> as well as pluggable custom model runtime.
</div>
</div>
Expand Down

0 comments on commit 0f649b5

Please sign in to comment.