FastAPI: Separate model server and data plane (kubeflow#2444)

* Update kserve python dependency Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Convert to FastAPI Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Convert to FastAPI Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add workers Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add Dataplane api and model repo extension api Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Setup up single source versioning for python sdk Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix tests Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add v2 routes with FastAPIRoute Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Extract errors and add exception handler for ModelServer Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add metrics route and handler Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add kserve version in __init__.py Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Remove tornado handlers Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Move ModelNotReady to errors.py Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Remove unused arguments - max_buffer_size - max_asyncio_workers Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Clean up validate_enable_latency_logging by using strtobool Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Remove tests for validate_enable_latency_logging Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Improve error response messages Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Update handlers - fix imports for model_repository_extension.py - raise ModelNotFound exception in model_ready handler Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add model ready and list handler - replace model_metadata with model_list_handler for /v1/models - use ModelRepositoryExtension for load and unload Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix misspelling Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Use FastAPI TestClient for testing Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add V1Endpoints and update tests - switch to use v1_endpoints in FastAPI routes - make TestModel and TestTFHttpServer pass Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix TestRayServer Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix model not ready test Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix cloud event tests Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Use pytest asyncio and lowercase header key Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Set default header to json Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Update tests - fixed TestModel - fixed load/unload tests Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add V2Endpoints and data models - in v1, rename `infer` to `predict` Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Improve Swagger UI support - disable redoc and add flag for swagger ui - add example for InferenceRequest model Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Revert "Add kserve version in __init__.py" This reverts commit a0204f8bcd7cc849ae55127355d07e9d5bd4eb92. Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix the version number in setup.py by using the pathlib to get the __file__ path instead of the cwd Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix the version file in dockerfile Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix linting issue Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix other lint issues Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix lint issue Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Use version from VERSION file Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Use `Dict` for typing Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix `Dict` typing Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix `List` typing Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Support v2 load and unload Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix test_model_repository_extension.py Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix dataplane Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add orjson in requirements.txt Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Use httpx.AsyncClient to replace tornado Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Remove tornado Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add httpx in requirements.txt Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Update doc strings for some functions Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Use orjson for decoding Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Use async for liveness handler Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Added a few comments Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Unify infer and predict in dataplane Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Get rid of json Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Use ORJSONResponse by default in FastAPI application Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix orjson exception match Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add more comments Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add handler for NotImplementedError Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Use orjson for InferenceRequest deserialization Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add schema examples Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix v2 live and ready endpoints Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Use asyncio to fix start() issue Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix missing asyncio import Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add missing asyncio to start model server Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Set default host to 0.0.0.0 to fix raw deployment readiness issue Signed-off-by: Xin Fu <xfu83@bloomberg.net> * add back nest_asyncio for explainers Signed-off-by: Dan Sun <dsun20@bloomberg.net> * Use async for load and unload for model_repository_extension.py and update some comments Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Get rid of DeprecationWarning from cloudevents Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Fix test_model_repository_extension.py with async Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Add license Signed-off-by: Xin Fu <xfu83@bloomberg.net> * Update comments and docstrings Signed-off-by: Xin Fu <xfu83@bloomberg.net> Signed-off-by: Xin Fu <xfu83@bloomberg.net> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Xin Fu <xfu83@bloomberg.net>
magdalenakuhn17 · Oct 20, 2022 · e126d01 · e126d01
1 parent 3ba300b
commit e126d01
Show file tree

Hide file tree

Showing 72 changed files with 1,935 additions and 806 deletions.
diff --git a/docs/samples/kafka/setup.py b/docs/samples/kafka/setup.py
@@ -10,6 +10,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import os
 
 from setuptools import setup, find_packages
 
@@ -19,6 +20,9 @@
     'mypy'
 ]
 
+with open(os.path.join(os.getcwd(), '../../../python/VERSION')) as version_file:
+    version = version_file.read().strip()
+
 setup(
     name='transformer',
     version='0.1.0',
@@ -30,7 +34,7 @@
     python_requires='>=3.7',
     packages=find_packages("transformer"),
     install_requires=[
-        "kserve",
+        f"kserve>={version}",
         "pandas>=0.24.2",
         "opencv-python-headless==4.2.0.32",
     ],

diff --git a/docs/samples/v1beta1/transformer/torchserve_image_transformer/setup.py b/docs/samples/v1beta1/transformer/torchserve_image_transformer/setup.py
@@ -10,6 +10,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import os
 
 from setuptools import setup, find_packages
 
@@ -19,6 +20,10 @@
     'mypy'
 ]
 
+with open(os.path.join(os.getcwd(), '../../../../../python/VERSION')) as version_file:
+    version = version_file.read().strip()
+
+
 setup(
     name='image_transformer',
     version='0.1.0',
@@ -30,7 +35,7 @@
     python_requires='>=3.7',
     packages=find_packages("image_transformer"),
     install_requires=[
-        "kserve",
+        f"kserve>={version}",
         "joblib>=0.13.2",
         "torchvision>=0.4.0",
         "pillow==9.0.1"

diff --git a/docs/samples/v1beta1/triton/bert/bert_tokenizer_v2/setup.py b/docs/samples/v1beta1/triton/bert/bert_tokenizer_v2/setup.py
@@ -1,3 +1,5 @@
+import os
+
 from setuptools import setup, find_packages
 
 tests_require = [
@@ -6,6 +8,9 @@
     'mypy'
 ]
 
+with open(os.path.join(os.getcwd(), '../../../../../../python/VERSION')) as version_file:
+    version = version_file.read().strip()
+
 setup(
     name='bert_transformer_v2',
     version='0.1.0',
@@ -14,7 +19,7 @@
     python_requires='>=3.7',
     packages=find_packages("bert_transformer"),
     install_requires=[
-        "kserve",
+        f"kserve>={version}",
         "tensorflow==2.7.2",
     ],
     tests_require=tests_require,

diff --git a/docs/samples/v1beta1/triton/torchscript/setup.py b/docs/samples/v1beta1/triton/torchscript/setup.py
@@ -10,6 +10,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import os
 
 from setuptools import setup, find_packages
 
@@ -19,6 +20,9 @@
     'mypy'
 ]
 
+with open(os.path.join(os.getcwd(), '../../../../../python/VERSION')) as version_file:
+    version = version_file.read().strip()
+
 setup(
     name='image_transformer_v2',
     version='0.1.0',
@@ -30,7 +34,7 @@
     python_requires='>=3.7',
     packages=find_packages("image_transformer_v2"),
     install_requires=[
-        "kserve",
+        f"kserve>={version}",
         "torchvision>=0.4.0",
         "pillow==9.0.1"
     ],

diff --git a/hack/aix_patch_dev.sh b/hack/aix_patch_dev.sh
diff --git a/python/VERSION b/python/VERSION
@@ -0,0 +1 @@
+0.9.0
diff --git a/python/aiffairness.Dockerfile b/python/aiffairness.Dockerfile
@@ -3,6 +3,7 @@ FROM python:3.9-slim-bullseye
 COPY third_party third_party
 
 COPY kserve kserve
+COPY VERSION VERSION
 RUN pip install --no-cache-dir --upgrade pip && pip install --no-cache-dir -e ./kserve
 
 COPY aiffairness aiffairness

diff --git a/python/aiffairness/aifserver/__main__.py b/python/aiffairness/aifserver/__main__.py
@@ -13,14 +13,15 @@
 # limitations under the License.
 
 import argparse
+import asyncio
+
 import kserve
 import json
 
 from .model import AIFModel
 
 DEFAULT_MODEL_NAME = "aifserver"
 
-
 parser = argparse.ArgumentParser(parents=[kserve.model_server.parser])
 
 parser.add_argument('--model_name',
@@ -56,7 +57,6 @@
                     nargs='+',
                     required=True)
 
-
 args, _ = parser.parse_known_args()
 
 if __name__ == "__main__":
@@ -71,4 +71,4 @@
         unprivileged_groups=args.unprivileged_groups
     )
     model.load()
-    kserve.ModelServer().start([model], nest_asyncio=True)
+    asyncio.run(kserve.ModelServer().start([model]))
diff --git a/python/aiffairness/aifserver/model.py b/python/aiffairness/aifserver/model.py
@@ -20,6 +20,9 @@
 from aif360.metrics import BinaryLabelDatasetMetric
 from aif360.datasets import BinaryLabelDataset
 
+import nest_asyncio
+nest_asyncio.apply()
+
 
 class AIFModel(kserve.Model):
     def __init__(self, name: str, predictor_host: str, feature_names: list, label_names: list, favorable_label: float,

diff --git a/python/aiffairness/setup.py b/python/aiffairness/setup.py
@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import pathlib
 
 from setuptools import setup, find_packages
 
@@ -20,9 +21,12 @@
     'mypy'
 ]
 
+with open(pathlib.Path(__file__).parent.parent / 'VERSION') as version_file:
+    version = version_file.read().strip()
+
 setup(
     name='aifserver',
-    version='0.9.0',
+    version=version,
     author_email='Andrew.Butler@ibm.com',
     license='https://github.com/kserve/kserve/LICENSE',
     url='https://github.com/kserve/kserve/python/aifserver',
@@ -32,7 +36,7 @@
     python_requires='>3.7',
     packages=find_packages("aifserver"),
     install_requires=[
-        "kserve",
+        f"kserve>={version}",
         "aif360 >= 0.2.3",
         "nest_asyncio>=1.4.0",
         "requests[security]>=2.24.0"

diff --git a/python/aixexplainer.Dockerfile b/python/aixexplainer.Dockerfile
@@ -3,6 +3,8 @@ FROM python:3.7-slim
 COPY third_party third_party
 
 COPY kserve kserve
+COPY VERSION VERSION
+
 RUN pip install --no-cache-dir --upgrade pip && pip install --no-cache-dir -e ./kserve
 
 RUN apt update && apt install -y build-essential

diff --git a/python/aixexplainer/aixserver/__main__.py b/python/aixexplainer/aixserver/__main__.py
@@ -13,8 +13,9 @@
 # limitations under the License.
 
 import argparse
-import kserve
+import asyncio
 
+import kserve
 from .model import AIXModel
 
 DEFAULT_MODEL_NAME = "aixserver"
@@ -51,4 +52,4 @@
                      top_labels=args.top_labels, min_weight=args.min_weight,
                      positive_only=args.positive_only, explainer_type=args.explainer_type)
     model.load()
-    kserve.ModelServer().start([model], nest_asyncio=True)
+    asyncio.run(kserve.ModelServer().start([model]))
diff --git a/python/aixexplainer/aixserver/model.py b/python/aixexplainer/aixserver/model.py
@@ -20,6 +20,8 @@
 from aix360.algorithms.lime import LimeImageExplainer
 from lime.wrappers.scikit_image import SegmentationAlgorithm
 from aix360.algorithms.lime import LimeTextExplainer
+import nest_asyncio
+nest_asyncio.apply()
 
 
 class AIXModel(kserve.Model):  # pylint:disable=c-extension-no-member

diff --git a/python/aixexplainer/setup.py b/python/aixexplainer/setup.py
@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import pathlib
 
 from setuptools import setup, find_packages
 
@@ -19,9 +20,13 @@
     'pytest-tornasync',
     'mypy'
 ]
+
+with open(pathlib.Path(__file__).parent.parent / 'VERSION') as version_file:
+    version = version_file.read().strip()
+
 setup(
     name='aixserver',
-    version='0.9.0',
+    version=version,
     author_email='Andrew.Butler@ibm.com',
     license='https://github.com/kserve/kserve/LICENSE',
     url='https://github.com/kserve/kserve/python/aixserver',
@@ -31,7 +36,8 @@
     python_requires='>3.7',
     packages=find_packages("aixserver"),
     install_requires=[
-        "kserve",
+        f"kserve>={version}",
+        "argparse >= 1.4.0",
         "aix360 >= 0.2.0",
         "lime >= 0.1.1.37",
         "nest_asyncio>=1.4.0",

diff --git a/python/alibiexplainer.Dockerfile b/python/alibiexplainer.Dockerfile
@@ -3,6 +3,7 @@ FROM python:3.9-slim-bullseye
 COPY third_party third_party
 
 COPY kserve kserve
+COPY VERSION VERSION
 RUN pip install --no-cache-dir --upgrade pip && pip install --no-cache-dir -e ./kserve
 
 COPY alibiexplainer alibiexplainer

diff --git a/python/alibiexplainer/alibiexplainer/__main__.py b/python/alibiexplainer/alibiexplainer/__main__.py
@@ -11,16 +11,18 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-import dill
-import kserve
+import asyncio
 import logging
 import os
 import sys
+
+import dill
 from alibiexplainer import AlibiExplainer
 from alibiexplainer.explainer import ExplainerMethod  # pylint:disable=no-name-in-module
 from alibiexplainer.parser import parse_args
 
+import kserve
+
 logging.basicConfig(level=kserve.constants.KSERVE_LOGLEVEL)
 
 EXPLAINER_FILENAME = "explainer.dill"
@@ -47,7 +49,7 @@ def main():
         alibi_model,
     )
     explainer.load()
-    kserve.ModelServer().start(models=[explainer], nest_asyncio=True)
+    asyncio.run(kserve.ModelServer().start(models=[explainer]))
 
 
 if __name__ == "__main__":

diff --git a/python/alibiexplainer/alibiexplainer/explainer.py b/python/alibiexplainer/alibiexplainer/explainer.py
@@ -24,6 +24,9 @@
 from alibiexplainer.anchor_text import AnchorText
 from alibiexplainer.explainer_wrapper import ExplainerWrapper
 
+import nest_asyncio
+nest_asyncio.apply()
+
 logging.basicConfig(level=kserve.constants.KSERVE_LOGLEVEL)
 
 

diff --git a/python/alibiexplainer/setup.py b/python/alibiexplainer/setup.py
@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import pathlib
 
 from setuptools import setup, find_packages
 
@@ -20,9 +21,12 @@
     'mypy'
 ]
 
+with open(pathlib.Path(__file__).parent.parent / 'VERSION') as version_file:
+    version = version_file.read().strip()
+
 setup(
     name='alibiexplainer',
-    version='0.9.0',
+    version=version,
     author_email='cc@seldon.io',
     license='../../LICENSE.txt',
     url='https://github.com/kserve/kserve/python/alibiexplainer',
@@ -32,7 +36,7 @@
     python_requires='>=3.7',
     packages=find_packages("alibiexplainer"),
     install_requires=[
-        "kserve",
+        f"kserve>={version}",
         "nest_asyncio>=1.4.0",
         "alibi==0.6.4",
         "joblib>=0.13.2",

diff --git a/python/artexplainer.Dockerfile b/python/artexplainer.Dockerfile
@@ -3,6 +3,7 @@ FROM python:3.9-slim-bullseye
 COPY third_party third_party
 
 COPY kserve kserve
+COPY VERSION VERSION
 RUN pip install --no-cache-dir --upgrade pip && pip install --no-cache-dir -e ./kserve
 
 COPY artexplainer artexplainer

diff --git a/python/artexplainer/artserver/__main__.py b/python/artexplainer/artserver/__main__.py
@@ -13,10 +13,12 @@
 # limitations under the License.
 
 import argparse
-import kserve
+import asyncio
 
 from artserver import ARTModel
 
+import kserve
+
 DEFAULT_MODEL_NAME = "art-explainer"
 DEFAULT_ADVERSARY_TYPE = "SquareAttack"
 
@@ -40,4 +42,4 @@
     model = ARTModel(args.model_name, args.predictor_host, adversary_type=args.adversary_type,
                      nb_classes=args.nb_classes, max_iter=args.max_iter)
     model.load()
-    kserve.ModelServer().start([model], nest_asyncio=True)
+    asyncio.run(kserve.ModelServer().start([model]))
diff --git a/python/artexplainer/artserver/model.py b/python/artexplainer/artserver/model.py
@@ -21,6 +21,9 @@
 
 import kserve
 
+import nest_asyncio
+nest_asyncio.apply()
+
 
 class ARTModel(kserve.Model):  # pylint:disable=c-extension-no-member
     def __init__(self, name: str, predictor_host: str, adversary_type: str,

diff --git a/python/artexplainer/setup.py b/python/artexplainer/setup.py
@@ -11,17 +11,21 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import pathlib
 
 from setuptools import setup, find_packages
 
+with open(pathlib.Path(__file__).parent.parent / 'VERSION') as version_file:
+    version = version_file.read().strip()
+
 tests_require = [
     'pytest',
     'pytest-tornasync',
     'mypy'
 ]
 setup(
     name='artserver',
-    version='0.7.0',
+    version=version,
     author_email='Andrew.Butler@ibm.com',
     license='https://github.com/kserve/kserve/LICENSE',
     url='https://github.com/kserve/kserve/python/artserver',