From 42c986b6217630e66ad80b04f9a19d2ae5ab560f Mon Sep 17 00:00:00 2001 From: Dan Sun Date: Sat, 18 Feb 2023 15:17:09 -0500 Subject: [PATCH] Add open inference protocol adoptions Signed-off-by: Dan Sun --- .../blog/articles/2023-02-05-KServe-0.10-release.md | 13 +++++++++---- mkdocs.yml | 2 +- overrides/home.html | 4 ++-- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/blog/articles/2023-02-05-KServe-0.10-release.md b/docs/blog/articles/2023-02-05-KServe-0.10-release.md index 6a062b636..d75d0a9c6 100644 --- a/docs/blog/articles/2023-02-05-KServe-0.10-release.md +++ b/docs/blog/articles/2023-02-05-KServe-0.10-release.md @@ -1,11 +1,11 @@ # Announcing: KServe v0.10.0 We are excited to announce KServe 0.10 release. In this release we have enabled more KServe networking options, -improved metrics instruments for supported serving runtimes and increased support coverage for [Open(aka v2) inference protocol](https://kserve.github.io/website/0.10/modelserving/data_plane/v2_protocol/) for both standard and ModelMesh InferenceService. +improved KServe telemetry for supported serving runtimes and increased support coverage for [Open(aka v2) inference protocol](https://kserve.github.io/website/0.10/modelserving/data_plane/v2_protocol/) for both standard and ModelMesh InferenceService. ## KServe Networking Options -Istio is now optional for both `Serverless` and `RawDeployment` mode. Please see the [alternative networking guide](https://kserve.github.io/website/0.10/admin/serverless/kourier_networking/) for how you can enable other ingress options supported by Knative with Serverless mode. +Istio is now optional for both [Serverless](https://kserve.github.io/website/0.10/admin/serverless/serverless/) and [RawDeployment](https://kserve.github.io/website/0.10/admin/kubernetes_deployment/) mode. Please see the [alternative networking guide](https://kserve.github.io/website/0.10/admin/serverless/kourier_networking/) for how you can enable other ingress options supported by Knative with Serverless mode. For Istio users, if you want to turn on full service mesh mode to secure InferenceService with mutual TLS and enable the traffic policies, please read the [service mesh setup guideline](https://kserve.github.io/website/0.10/admin/serverless/servicemesh/). ## KServe Telemetry for Serving Runtimes @@ -16,6 +16,10 @@ Please read the [prometheus metrics setup guideline](https://kserve.github.io/we ## Open(v2) Inference Protocol Support Coverage +As there have been increasing adoptions for `KServe v2 Inference Protocol` from [AMD Inference ServingRuntime](https://kserve.github.io/website/0.10/modelserving/v1beta1/amd/) which +supports FPGAs and OpenVINO which now provides KServe [REST](https://docs.openvino.ai/latest/ovms_docs_rest_api_kfs.html) and [gRPC](https://docs.openvino.ai/latest/ovms_docs_grpc_api_kfs.html) compatible API, +in [the issue](https://github.com/kserve/kserve/issues/2663) we have proposed to rename to `KServe Open Inference Protocol`. + In KServe 0.10, we have added support for Open(v2) inference protocol for KServe custom runtimes. Now, you can enable v2 REST/gRPC for both custom transformer and predictor with images built by implementing KServe Python SDK API. gRPC enables high performance inference data plane as it is built on top of HTTP/2 and binary data transportation which is more efficient to send over the wire compared to REST. @@ -64,9 +68,9 @@ You can use the same Python API type `InferRequest` and `InferResponse` for both If you have existing custom transformer or predictor, the `headers` argument is now required to add to the `preprocess`, `predict` and `postprocess` handlers. -Please check the following matrix for supported ServingRuntimes and ModelFormats. +Please check the following matrix for supported ModelFormats and [ServingRuntimes](https://kserve.github.io/website/0.10/modelserving/v1beta1/serving_runtime/). -| Model Format | v1 | v2 REST/gRPC | +| Model Format | v1 | Open(v2) REST/gRPC | | ------------------- |--------------| ----------------| | Tensorflow | ✅ TFServing | ✅ Triton | | PyTorch | ✅ TorchServe | ✅ TorchServe | @@ -114,6 +118,7 @@ For a complete change list please read the release notes from [KServe v0.10](htt Thanks for all the contributors who have made the commits to 0.10 release! + - [Steve Larkin](https://github.com/sel) - [Stephan Schielke](https://github.com/stephanschielke) - [Curtis Maddalozzo](https://github.com/cmaddalozzo) diff --git a/mkdocs.yml b/mkdocs.yml index fec41f73d..e8f800294 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -22,7 +22,7 @@ nav: - V1 Inference Protocol: modelserving/data_plane/v1_protocol.md - Open Inference Protocol (V2 Inference Protocol): modelserving/data_plane/v2_protocol.md - Serving Runtimes: modelserving/servingruntimes.md - - Single Model Serving: + - Model Serving Runtimes: - Supported Model Frameworks/Formats: - Overview: modelserving/v1beta1/serving_runtime.md - Tensorflow: modelserving/v1beta1/tensorflow/README.md diff --git a/overrides/home.html b/overrides/home.html index 809b0dd7e..43b9bd76c 100644 --- a/overrides/home.html +++ b/overrides/home.html @@ -84,12 +84,12 @@

KServe Components

- Single Model Serving + Model Serving
- Provides Serverless deployment of single model inference on CPU/GPU for common ML frameworks + Provides Serverless deployment for model inference on CPU/GPU with common ML frameworks Scikit-Learn, XGBoost, Tensorflow, PyTorch as well as pluggable custom model runtime.