Building TensorFlow Serving

Building TensorFlow Serving

The instructions provided here specify the steps to build TensorFlow Serving version 2.18.0 on Linux on IBM Z for the following distributions:

• Ubuntu (20.04, 22.04, 24.04)

General Notes:

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

1. Build using script

TensorFlow Serving can be built manually using STEP 2.

Use the following commands to build TensorFlow Serving using the build script. Please ensure wget is installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/TensorflowServing/2.18.0/build_tensorflow_serving.sh
bash build_tensorflow_serving.sh    [Provide -t option for executing build with tests, -p option for choosing the Python version from {3.9, 3.10, 3.11, 3.12}, if not specified, the script will use the default Python version as 3.11]

If the build completes successfully, go to STEP 4. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2. Install Dependencies

2.1. Install Basic Dependencies

export SOURCE_ROOT=/<source_root>/
export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/TensorflowServing/2.18.0/patch"

sudo apt-get update
sudo apt-get install -y wget automake libtool

2.2. Build and Install TensorFlow

cd $SOURCE_ROOT
wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Tensorflow/2.18.0/build_tensorflow.sh
bash build_tensorflow.sh -y

3. Build and Install

3.1. Download source code

cd $SOURCE_ROOT
git clone -b 2.18.0 https://github.com/tensorflow/serving
cd serving

3.2. Apply patches

curl -o tfs.patch $PATCH_URL/tfs.patch
curl -o rules.patch $PATCH_URL/rules.patch
sed -i "s?SOURCE_ROOT?$SOURCE_ROOT?" tfs.patch
git apply tfs.patch
cp rules.patch third_party/
touch third_party/BUILD

3.3. Build TensorFlow Serving

Tensorflow Serving can be built as follows:

cd $SOURCE_ROOT/serving
bazel build tensorflow_serving/...

Note: TensorFlow Serving build is resource intensive operation. If build continues to fail try increasing the swap space or reducing the number of concurrent jobs by specifying --jobs=n in the build command above, where n is the number of concurrent jobs.

3.4. Install TensorFlow Serving

Copy binary to access it from anywhere, make sure /usr/local/bin is in $PATH. Run command:

sudo cp bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server /usr/local/bin/
tensorflow_model_server --version

The following output should be seen in the console:

TensorFlow ModelServer: 2.18.0-rc0+dev.sha.5815bfdd
TensorFlow Library: 2.18.0

3.5. Install TensorFlow Serving Python API PIP package

pip3 install tensorflow-serving-api==2.18.0

4. Verify TensorFlow Serving (Optional)

Run TensorFlow Serving from command Line

tensorflow_model_server --rest_api_port=8501 --model_name=<model_name> --model_base_path=<model_path> &

For example:

export TESTDATA="$SOURCE_ROOT/serving/tensorflow_serving/servables/tensorflow/testdata"
# Start TensorFlow Serving model server and open the REST API port
tensorflow_model_server --rest_api_port=8501 --model_name=half_plus_two --model_base_path=$TESTDATA/saved_model_half_plus_two_cpu &
# Query the model using the predict API
curl -d '{"instances": [1.0, 2.0, 5.0]}' -X POST http://localhost:8501/v1/models/half_plus_two:predict

Output should look like:

{
    "predictions": [2.5, 3.0, 4.5
    ]
}

5. Execute Test Suite (Optional)

Run complete testsuite

cd $SOURCE_ROOT/serving
bazel test tensorflow_serving/...

All test cases should pass.

Note:

• Below test cases will fail if IPv6 is not enabled.

  //tensorflow_serving/util/net_http/server/internal:evhttp_request_test
  //tensorflow_serving/util/net_http/server/internal:evhttp_server_test
  

References:

• https://www.tensorflow.org/tfx/guide/serving
• https://github.com/tensorflow/serving