Building TensorFlow

Building TensorFlow

The instructions provided below specify the steps to build TensorFlow version 2.11.0 on Linux on IBM Z for the following distributions:

• Ubuntu (20.04, 22.04)

General Notes:

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Step 1: Build and Install TensorFlow v2.11.0

1.1) Build using script

If you want to build TensorFlow using manual steps, go to STEP 1.2.

Use the following commands to build TensorFlow using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Tensorflow/2.11.0/build_tensorflow.sh

# Build Tensorflow
bash build_tensorflow.sh [Provide -t option for executing build with tests, -p option for choosing the Python version from {3.7, 3.8, 3.9, 3.10}, if not specified, the script will use the distro provided Python version (i.e., Python 3.8 on Ubuntu 20.04 and Python 3.10 on Ubuntu 22.04).]

If the build completes successfully, go to STEP 2. In case of error, check logs for more details or go to STEP 1.2 to follow manual build steps.

1.2) Install the dependencies

export SOURCE_ROOT=/<source_root>/
export NUMPY_VERSION="1.22.4"

• Only when choosing Python version 3.7

  PYTHON_VERSION=3.7.4
  NUMPY_VERSION=1.21.6

• Only when choosing Python version 3.8

  PYTHON_VERSION=3.8.6

• Only when choosing Python version 3.9

  PYTHON_VERSION=3.9.7

• Only when choosing Python version 3.10

  PYTHON_VERSION=3.10.6

• Only on Ubuntu 20.04

  sudo apt-get update
  sudo apt-get install wget git unzip zip python3-dev python3-pip openjdk-11-jdk pkg-config libhdf5-dev libssl-dev libblas-dev liblapack-dev gfortran curl -y
  REPO_PYTHON=$(python3 -V | sed -n 's/Python \([0-9.]*\)/\1/p')
  if [[ "$PYTHON_VERSION" != "$REPO_PYTHON" ]]; then
    wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Python3/$PYTHON_VERSION/build_python3.sh
    sed -i 's/apt-get install/DEBIAN_FRONTEND=noninteractive apt-get install/g' build_python3.sh
    bash build_python3.sh -y
    sudo update-alternatives --install /usr/local/bin/python python /usr/local/bin/python3 40
  else
    sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 40
  fi
  sudo ldconfig
  sudo pip3 install --upgrade pip
  sudo pip3 install --no-cache-dir numpy==$NUMPY_VERSION wheel scipy==1.7.3 portpicker protobuf==3.13.0 opt_einsum packaging requests psutil setuptools==59.5.0

• Only on Ubuntu 22.04

  sudo apt-get update
  sudo apt-get install wget git unzip zip python3-dev python3-pip openjdk-11-jdk pkg-config libhdf5-dev libssl-dev gcc-9* g++-9* curl libblas-dev liblapack-dev gfortran -y
  REPO_PYTHON=$(python3 -V | sed -n 's/Python \([0-9.]*\)/\1/p')
  if [[ "$PYTHON_VERSION" != "$REPO_PYTHON" ]]; then
    wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Python3/$PYTHON_VERSION/build_python3.sh
    sed -i 's/apt-get install/DEBIAN_FRONTEND=noninteractive apt-get install/g' build_python3.sh
    sed -i 's/ubuntu-20.04/ubuntu-22.04/g' build_python3.sh
    bash build_python3.sh -y
    sudo update-alternatives --install /usr/local/bin/python python /usr/local/bin/python3 40
  else
    sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 40
  fi
  sudo ldconfig
  sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 60 --slave /usr/bin/g++ g++ /usr/bin/g++-9
  sudo update-alternatives --auto gcc
  sudo pip3 install --upgrade pip
  sudo pip3 install --no-cache-dir numpy==$NUMPY_VERSION wheel packaging requests opt_einsum portpicker protobuf==3.13.0 scipy==1.7.3 psutil setuptools==59.5.0

• Build ICU data in big-endian format

  cd $SOURCE_ROOT
  export ICU_MAJOR_VERSION="69"
  export ICU_RELEASE="release-${ICU_MAJOR_VERSION}-1"
  git clone --depth 1 --single-branch --branch "$ICU_RELEASE" https://github.com/unicode-org/icu.git
  cd icu/icu4c/source/
  # create ./filters.json
  cat << 'EOF' > filters.json
  {
    "localeFilter": {
      "filterType": "language",
      "includelist": [
        "en"
      ]
    }
  }
  EOF
  ICU_DATA_FILTER_FILE=filters.json ./runConfigureICU Linux
  make clean && make
  # Workaround makefile issue where not all of the resource files may have been processed
  find data/out/build/ -name '*pool.res' -print0 | xargs -0 touch
  make
  cd data/out/tmp
  LD_LIBRARY_PATH=../../../lib ../../../bin/genccode "icudt${ICU_MAJOR_VERSION}b.dat"
  echo "U_CAPI const void * U_EXPORT2 uprv_getICUData_conversion() { return icudt${ICU_MAJOR_VERSION}b_dat.bytes; }" >> "icudt${ICU_MAJOR_VERSION}b_dat.c"
  cp icudt${ICU_MAJOR_VERSION}b_dat.c icu_conversion_data_big_endian.c
  gzip icu_conversion_data_big_endian.c
  split -a 3 -b 100000 icu_conversion_data_big_endian.c.gz icu_conversion_data_big_endian.c.gz.

• Install grpcio

  cd $SOURCE_ROOT
  export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=True
  sudo -E pip3 install grpcio

• Build Bazel v5.3.0

  cd $SOURCE_ROOT
  wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/5.3.2/build_bazel.sh
  sed -i 's/5.3.2/5.3.0/g' build_bazel.sh
  sed -i 's#Bazel/${PACKAGE_VERSION}/patch#Bazel/5.3.2/patch#g' build_bazel.sh
  sed -i 's/apt-get install/DEBIAN_FRONTEND=noninteractive apt-get install/g' build_bazel.sh
  bash build_bazel.sh -y
  sudo cp $SOURCE_ROOT/bazel/output/bazel /usr/local/bin/bazel

1.3) Build TensorFlow

• Download source code

  cd $SOURCE_ROOT
  git clone https://github.com/tensorflow/tensorflow
  cd tensorflow
  git checkout v2.11.0

• Apply patches

  curl -o tf_v2.11.0.patch https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Tensorflow/2.11.0/patch/tf_v2.11.0.patch
  patch -p1 < tf_v2.11.0.patch
  rm -f tf_v2.11.0.patch

• Copy the big-endian version of ICU data

  cp ${SOURCE_ROOT}/icu/icu4c/source/data/out/tmp/icu_conversion_data_big_endian.c.gz.* third_party/icu/data/

• Configure

    yes "" | ./configure || true

• Build TensorFlow

    bazel build //tensorflow/tools/pip_package:build_pip_package

  Note:
  1. TensorFlow build is resource intensive operation. If build continues to fail try increasing the swap space and reduce the number of concurrent jobs by specifying --jobs=n in the build command above, where n is the number of concurrent jobs.

  2. Building TensorFlow from source can use a lot of RAM. If your system is memory-constrained, limit Bazel's RAM usage with: --local_ram_resources=2048.

  3. Since our patch has changed the implementation of ParseTensorOp and SerializeTensorOp to make sure the serialized binary string is in little-endian format, s390x users who had used Python function io.serialize_tensor() in previous releases of TensorFlow to serialize Tensor data in storage (which would be in big-endian format) should apply endianness conversion to the data (for example, by invoking byteswap() on the data) before feed it to io.parse_tensor() function. Serialized tensor data via io.serialize_tensor() from this release of TensorFlow will be in little-endian format on s390x.

1.4) Build tensorflow_io_gcs_filesystem wheel

sudo pip install --upgrade pip
cd $SOURCE_ROOT
git clone https://github.com/tensorflow/io.git
cd io/
git checkout v0.23.1
python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem          
cd dist
sudo pip3 install ./tensorflow_io_gcs_filesystem-0.23.1-cp*-cp*-linux_s390x.whl

1.5) Build and Install TensorFlow wheel

cd $SOURCE_ROOT/tensorflow
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_wheel
sudo pip3 install /tmp/tensorflow_wheel/tensorflow-2.11.0-cp*-linux_s390x.whl

Step 2: Verify TensorFlow (Optional)

• Run TensorFlow from command Line and check the installed version

   $ cd $SOURCE_ROOT
   $ python -c "import tensorflow as tf; print(tf.__version__)"
     2.11.0
  
   $ python
    >>> import tensorflow as tf
    >>> tf.add(1, 2).numpy()
    3
    >>> hello = tf.constant('Hello, TensorFlow!')
    >>> hello.numpy()
    b'Hello, TensorFlow!'
    >>>

Step 3: Execute Test Suite (Optional)

• Run complete testsuite

  cd $SOURCE_ROOT/tensorflow
  
  bazel test -- //tensorflow/... -//tensorflow/compiler/... -//tensorflow/java/... -//tensorflow/python/kernel_tests/math_ops:approx_topk_test_cpu -//tensorflow/python/tools:saved_model_cli_test -//tensorflow/lite/delegates/gpu/cl/kernels/... -//tensorflow/lite/experimental/acceleration/mini_benchmark/c:c_api_test -tensorflow/lite/tools/delegates/compatibility/gpu:gpu_delegate_compatibility_checker_test -//tensorflow/lite:tensorflow_profiler_logger_build_test -//tensorflow/lite/delegates/nnapi:nnapi_delegate_test -//tensorflow/lite/experimental/acceleration/mini_benchmark:runner_test -//tensorflow/lite/tools/delegates/compatibility/nnapi:nnapi_delegate_compatibility_checker_test -//tensorflow/lite/toco/tflite:import_test -//tensorflow/lite/kernels:reshape_test -//tensorflow/lite/kernels:squeeze_test -//tensorflow/lite/tools/signature:signature_def_util_test -//tensorflow/lite/delegates/flex:buffer_map_test -//tensorflow/lite/tools:flatbuffer_utils_test -//tensorflow/lite/tools:visualize_test -//tensorflow/lite/tools/optimize:quantize_model_test -//tensorflow/lite/tools/signature:signature_def_utils_test -//tensorflow/lite/experimental/acceleration/mini_benchmark:big_little_affinity_test
  

• Run individual test

  bazel test //tensorflow/<module_name>:<testcase_name>

  For example,

  bazel test //tensorflow/python/kernel_tests:topk_op_test

  Note:
  

  1. Few test cases from //tensorflow/java/... fail on s390x and Intel, refer to this for more details.

  2. Test case //tensorflow/python/kernel_tests/math_ops:approx_topk_test_cpu and a few tests from compiler module fail due to lack of certain FP16 intrinsics in SystemZ LLVM backend. Refer to this for more details. Part of them can pass by passing --copt=-fno-ipa-cp flag to bazel command.

  3. Test case //tensorflow/python/tools:saved_model_cli_test fails due to incorrect target_triple value for s390x. Refer to this for more details.

  4. A few tests (such as //tensorflow/lite/delegates/gpu/cl/kernels/...) from lite module fail on both s390x and Intel.

Step 4: Model Training Verification (Optional)

Follow the instructions provided here to verify TensorFlow v2.11.0.

References:

• https://www.tensorflow.org/
• https://github.com/tensorflow/tensorflow
• http://bazel.io/