c_bindings_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Purposes to have Python Interface to C\\C++"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "* Create Python API for existing C\\C++ projects\n",
    "* Speed up things (especially useful on platforms where optimized python packages\n",
    "might not be available)\n",
    "* Access to low-level HW and System capabilities from high-level language\n",
    "* Ease UI (Command Line or Graphical) creation\n",
    "* Test C\\C++ code from python"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# CFFI and Ctypes comparison"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Ctypes:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Pros**\n",
    "* Built-in module\n",
    "* Some optimized modules relies on ctypes to access low-level objects representation or row memory\n",
    "\n",
    "**Cons**\n",
    "* Works only in binary (ABI) mode through libffi\n",
    "* A lot of extra code required to describe functions, structures, types, etc."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## CFFI"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Pros**\n",
    "* Both ABI and API level available\n",
    "* Standalone ready for distribution Python library can be created\n",
    "* No need to manually describe function signatures and types\n",
    "* Better performance comparing to ctypes\n",
    "\n",
    "**Cons**\n",
    "* Requires compilation\n",
    "* **Limitations in C code support**\n",
    "    * Preprocessor directives are not supported\\supported partially\n",
    "    * *\"For now some C99 constructs are not supported, but all C89 should be, including macros\"*"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Install binary dependencies and required python modules:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "* CMake\n",
    "* CFFI\n",
    "* Cython\n",
    "* PyBind11\n",
    "* numpy\n",
    "* tabulate"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ INFO ] Checking for installed CMake...\n",
      "[ INFO ] CMake executable found at /usr/local/bin/cmake\n",
      "\n",
      "[ INFO ] Installing required python modules...\n",
      "[ INFO ] Installation succeed\n"
     ]
    }
   ],
   "source": [
    "from sys import platform\n",
    "import logging as log\n",
    "import subprocess\n",
    "\n",
    "log.basicConfig(format='[ %(levelname)s ] %(message)s', level=log.DEBUG)\n",
    "\n",
    "log.info(\"Checking for installed CMake...\")\n",
    "p = subprocess.run(['which', 'cmake'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n",
    "if p.returncode == 0:\n",
    "    log.info(f\"CMake executable found at {p.stdout.decode()}\")\n",
    "else:\n",
    "    log.warn(\"CMake executable not found! Installing ... \")\n",
    "    cmd = ['brew', 'install', 'cmake'] if platform == 'darwin' else ['apt', 'install', 'cmake']\n",
    "    log.info(f\"Running: {' '.join(cmd)}...\")\n",
    "    p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n",
    "    if p.returncode != 0:\n",
    "        log.error(f\"Failed to install cmake.\\n\"\n",
    "                  f\"Process STDOUT:\\n{p.stdout.decode()}\\n\"\n",
    "                  f\"Process STDERR:\\n{p.stderr.decode()}\")\n",
    "    else:\n",
    "        log.info(\"Installation succeed!\")\n",
    "\n",
    "log.info(\"Installing required python modules...\")\n",
    "p = subprocess.run(['pip', 'install', '-r', 'requirements.txt'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n",
    "if p.returncode != 0:\n",
    "    log.error(f\"Failed to install required python modules.\\n\"\n",
    "              f\"Process STDOUT:\\n{p.stdout.decode()}\\n\"\n",
    "              f\"Process STDERR:\\n{p.stderr.decode()}\")\n",
    "else:\n",
    "    log.info(\"Installation succeed\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# C Library listing"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bellow you can find C library functions declaration. For more convenience listings with each function implmentation\n",
    "will be added bellow right before using it from python.\n",
    "\n",
    "c_library/library.h\n",
    "```c\n",
    "/*\n",
    "CDefError: cannot parse \"#include <stdio.h>\"\n",
    "<cdef source string>:1:1: Directives not supported yet\n",
    "\n",
    "CFFI doesn't support directives in headers\n",
    "For compatibility between ctypes and CFFI all includes moved to .c files\n",
    "#include <stdio.h>\n",
    "#include <string.h>\n",
    "#include <unistd.h>\n",
    "*/\n",
    "\n",
    "int func_ret_int(int val);\n",
    "double func_ret_double(double val);\n",
    "char *func_ret_str(char *val);\n",
    "char func_many_args(int val1, double val2, char val3, short val4);\n",
    "\n",
    "void arr_minus_one(int *data, unsigned int arr_size);\n",
    "int *gen_arr(unsigned int size);\n",
    "void fill_arr(int *arr, unsigned int size);\n",
    "void qsort_wrap(int *arr, unsigned int size, unsigned int el_size);\n",
    "int comp (const int *a, const int *b);\n",
    "\n",
    "void upd_arr(int *arr, unsigned int size, int(*callback)(int));\n",
    "void free_arr(int *p);\n",
    "\n",
    "char *do_nothing_function(char *some_str);\n",
    "```"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Here is brief description of each method and its purpose:\n",
    "* Functions demonstrating working with basic types. Do nothing besides of receiving arguments of certain types,\n",
    "printing basic information, which will show us that real C function was invoked, and returning some value.\n",
    "```c\n",
    "int func_ret_int(int val);\n",
    "double func_ret_double(double val);\n",
    "char *func_ret_str(char *val);\n",
    "char func_many_args(int val1, double val2, char val3, short val4);\n",
    "```\n",
    "* Functions demonstrating how to manipulate with arrays:\n",
    "    *  `void arr_minus_one(int *data, unsigned int arr_size);` - demonstrates how to update existing array\n",
    "    * `int *gen_arr(unsigned int size);` - demonstrates dynamic memory allocation for creating new array\n",
    "    * `void gen_arr(int *arr, unsigned int size);` - fill pre-allocated integer array `int *arr` of given size\n",
    "    `unsigned int size` with integer values in range `[0, size - 1]`\n",
    "    * `void upd_arr(int *arr, unsigned int size, int(*callback)(int))` - function for callbacks performance profiling\n",
    "    *  `void free_arr(int *p)` - memory deallocator.\n",
    "    * `void qsort_wrap(int *arr, unsigned int size, unsigned int el_size);` - wraps standard quick sort algorithm from libc.\n",
    "    Will be used for performance comparison between C, Python and mixed quick sort implementation\n",
    "    * `int comp (const int *a, const int *b)` - C implementation of comparison function required by standard qsort\n",
    "    function from libc\n",
    "    * `char *do_nothing_function(char *some_str);` - dummy function receiving and returning same value to asses overhead\n",
    "    of calling C functions from Python\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Building C library from sources."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ INFO ] Running CMake...\n",
      "[ INFO ] CMake files generation succeed!\n",
      "STDOUT:\n",
      " -- The C compiler identification is AppleClang 12.0.0.12000031\n",
      "-- Detecting C compiler ABI info\n",
      "-- Detecting C compiler ABI info - done\n",
      "-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped\n",
      "-- Detecting C compile features\n",
      "-- Detecting C compile features - done\n",
      "-- Configuring done\n",
      "-- Generating done\n",
      "-- Build files have been written to: /Users/mikhailtreskin/repos/py_bindings_tutorial/c_library/build\n",
      "\n",
      "\n",
      "[ INFO ] Running library building with make...\n",
      "[ INFO ] Building suceed succeed!\n",
      "STDOUT:\n",
      " Scanning dependencies of target bindings_demo\n",
      "[ 25%] Building C object CMakeFiles/bindings_demo.dir/basic_funcs.c.o\n",
      "[ 50%] Building C object CMakeFiles/bindings_demo.dir/arrays.c.o\n",
      "[ 75%] Building C object CMakeFiles/bindings_demo.dir/vars.c.o\n",
      "[100%] Linking C shared library libbindings_demo.dylib\n",
      "[100%] Built target bindings_demo\n",
      "\n",
      "\n",
      "[ INFO ] Compiled library located at /Users/mikhailtreskin/repos/py_bindings_tutorial/c_library/build/libbindings_demo.dylib\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import shutil\n",
    "\n",
    "TUTOR_ROOT = os.getcwd()\n",
    "CMAKE_ROOT_FOLDER = os.path.join(TUTOR_ROOT, 'c_library',)\n",
    "BUILD_FOLDER = os.path.join(CMAKE_ROOT_FOLDER, 'build')\n",
    "if os.path.exists(BUILD_FOLDER):\n",
    "    shutil.rmtree(BUILD_FOLDER)\n",
    "os.mkdir(BUILD_FOLDER)\n",
    "\n",
    "os.chdir(BUILD_FOLDER)\n",
    "log.info(\"Running CMake...\")\n",
    "p = subprocess.run(['cmake', os.path.pardir], stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n",
    "if p.returncode != 0:\n",
    "    log.error(\"CMake run failed!\\n\"\n",
    "              f\"STDOUT:\\n {p.stdout.decode()}\\n\"\n",
    "              f\"STDERR:\\n {p.stderr.decode()}\\n\")\n",
    "else:\n",
    "    log.info(\"CMake files generation succeed!\\n\"\n",
    "             f\"STDOUT:\\n {p.stdout.decode()}\\n\")\n",
    "\n",
    "log.info(\"Running library building with make...\")\n",
    "p = subprocess.run(['make'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n",
    "if p.returncode != 0:\n",
    "    log.error(\"Building failed!\\n\"\n",
    "              f\"STDOUT:\\n {p.stdout.decode()}\\n\"\n",
    "              f\"STDERR:\\n {p.stderr.decode()}\\n\")\n",
    "else:\n",
    "    log.info(\"Building suceed succeed!\\n\"\n",
    "             f\"STDOUT:\\n {p.stdout.decode()}\\n\")\n",
    "\n",
    "os.chdir(TUTOR_ROOT)\n",
    "\n",
    "if platform in (\"linux\", \"linux2\"):\n",
    "    C_LIB_NAME = \"libbindings_demo.so\"\n",
    "elif platform == \"darwin\":\n",
    "    C_LIB_NAME = \"libbindings_demo.dylib\"\n",
    "else:\n",
    "    raise OSError(f\"Unsupported platform {platform}\")\n",
    "C_LIB_PATH = os.path.join(BUILD_FOLDER, C_LIB_NAME)\n",
    "if not os.path.exists(C_LIB_PATH):\n",
    "    raise FileNotFoundError(f\"Compiled library not found at {C_LIB_PATH}\")\n",
    "else:\n",
    "    log.info(f\"Compiled library located at {C_LIB_PATH}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Ctypes demo"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "[ctypes](https://docs.python.org/3/library/ctypes.html) is a foreign function library for Python. It provides C\n",
    "compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python.\n",
    "\n",
    "What's stands for **foreign function library**? It's a library which implements foreign function interface (FFI) convention\n",
    "between two certain languages - between **C as a host language** and **Python as a guest language** in case of ctypes.\n",
    "Briefly, FFI is an interface that allows code written in one language to call code written in another language.\n",
    "Ctypes rely on [libffi](https://github.com/libffi/libffi) library implementing FFI calling convention for C as a host language.\n",
    "\n",
    "Ctypes allows to make calls to C functions and refer to C objects from Python in a relatively easy way. Let's go\n",
    "through several examples to on how to interact with precompiled C library from Python using ctypes"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "First of all we have to load dynamic library in a following way (on Windows WINDLL call should be used instead CDLL)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "outputs": [],
   "source": [
    "import ctypes\n",
    "\n",
    "lib = ctypes.CDLL(C_LIB_PATH)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "And now we have an access to all of the functions mentioned above and declared in `c_library/library.h`.\n",
    "Let's start from calling a simple functions consuming one or several \"simple\" arguments."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Working with generic data types\n",
    "*Functions listed bellow are defined in `c_library/basic_funcs.c`*"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Working with numeric types"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Working with integers** on example of pretty simple C function consuming integer argument performing logging print from C\n",
    "code and then returns same integer value, which was provided as an input argument:\n",
    "```c\n",
    "int func_ret_int(int val) {\n",
    "    printf(\"From C Library: func_ret_int: %d\\n\", val);\n",
    "    return val;\n",
    "}\n",
    "\n",
    "double func_ret_double(double val) {\n",
    "    printf(\"From C Library: func_ret_double: %f\\n\", val);\n",
    "    return val;\n",
    "}\n",
    "\n",
    "```\n",
    "\n",
    "To call functions from C library we have to describe it's signature in Python by specifying input arguments types\n",
    "and return value type of desired C function. And then we can call it as an usual Python method."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From C Library: func_ret_int: 101\n",
      "From Python Script: func_ret_int:  101\n",
      "From C Library: func_ret_double: 12.123457\n",
      "From Python Script: func_ret_double:  12.123456789\n"
     ]
    }
   ],
   "source": [
    "# Jupyter notebook suppress default output from our C library\n",
    "# so custom manager to capture stdout from C was added\n",
    "from stdout_capture import capture_c_stdout\n",
    "\n",
    "# Specifying return type of the func_ret_int C function\n",
    "lib.func_ret_int.restype = ctypes.c_int\n",
    "# Specifying input arguments type of the func_ret_int C function\n",
    "lib.func_ret_int.argtypes = [ctypes.c_int, ]\n",
    "\n",
    "# Same for double data type\n",
    "lib.func_ret_double.restype = ctypes.c_double\n",
    "lib.func_ret_double.argtypes = [ctypes.c_double]\n",
    "with capture_c_stdout():\n",
    "    print('From Python Script: func_ret_int: ', lib.func_ret_int(101))\n",
    "    print('From Python Script: func_ret_double: ', lib.func_ret_double(12.123456789))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Working with strings (char arrays in C)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The same for **char functions** like:\n",
    "```c\n",
    "char * func_ret_str(char *val) {\n",
    "    printf(\"From C Library: func_ret_str: %s\\n\", val);\n",
    "    return val;\n",
    "}\n",
    "```\n",
    "\n",
    "On Python level the only important aspects is that **we have to encode input strings to bytearray** and then also decode\n",
    "return value"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From C Library: func_ret_str: Hello!\n",
      "From Python Script: func_ret_str:  Hello!\n"
     ]
    }
   ],
   "source": [
    "lib.func_ret_str.restype = ctypes.c_char_p\n",
    "lib.func_ret_str.argtypes = [ctypes.c_char_p, ]\n",
    "with capture_c_stdout():\n",
    "    print('From Python Script: func_ret_str: ', lib.func_ret_str('Hello!'.encode('utf-8')).decode(\"utf-8\"))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Function of several arguments"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Also nothing outstanding for functions with several input arguments with different data types, e.g.:\n",
    "\n",
    "```c\n",
    "char func_many_args(int val1, double val2, char val3, short val4) {\n",
    "    printf(\"From C Library: func_many_args: int - %d, double - %f, char - %c, short - %d\\n\", val1, val2, val3, val4);\n",
    "    return val3;\n",
    "}\n",
    "```\n",
    "To call the function all input argument types have to be defined in proper order:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From C Library: func_many_args: int - 15, double - 18.161700, char - X, short - 32000\n",
      "From Python Script: func_many_args:  X\n"
     ]
    }
   ],
   "source": [
    "lib.func_many_args.restype = ctypes.c_char\n",
    "lib.func_many_args.argtypes = [ctypes.c_int, ctypes.c_double, ctypes.c_char, ctypes.c_short]\n",
    "with capture_c_stdout():\n",
    "    print('From Python Script: func_many_args: ',\n",
    "      lib.func_many_args(15, 18.1617, 'X'.encode('utf-8'), 32000).decode(\"utf-8\"))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Accessing variables"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Ctypes also allows to **access and modify variables** from C library. In `c_library/vars.c` we have defined following variables\n",
    "```c\n",
    "int a = 5;\n",
    "double b = 5.12345;\n",
    "char c = 'X';\n",
    "```\n",
    "\n",
    "To access variables from loaded C library we need ro use `in_dll()` class method of corresponding data type of variable\n",
    "we going to access. `in_dll()` requires two input arguments - the 1st is an loaded library handler, and the 2nd is string\n",
    "with variable name. E.g.:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From Python Script: a:  22\n",
      "From Python Script: b:  5.12345\n",
      "From Python Script: c:  X\n"
     ]
    }
   ],
   "source": [
    "a = ctypes.c_int.in_dll(lib, \"a\")\n",
    "print('From Python Script: a: ', a.value)\n",
    "b = ctypes.c_double.in_dll(lib, \"b\")\n",
    "print('From Python Script: b: ', b.value)\n",
    "c = ctypes.c_char.in_dll(lib, \"c\")\n",
    "print('From Python Script: c: ', c.value.decode(\"utf-8\"))\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Variables values modification also allowed using usual assignment operator"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'a' initial value: 22\n",
      "'a' modified value: 22\n"
     ]
    }
   ],
   "source": [
    "print(f\"'a' initial value: {a.value}\")\n",
    "a.value = 22\n",
    "a = ctypes.c_int.in_dll(lib, \"a\")\n",
    "print(f\"'a' modified value: {a.value}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Working with arrays and interaction with numpy"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "OK, above were a simple examples which barely will be used in real life and were provided as introduction and way\n",
    "to understand how ctypes works. But what about real life examples?\n",
    "\n",
    "Bellow we will have a look on **how to have a deal with C arrays** basing on manipulation with numpy arrays."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Updating existing array"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "*c_library/arrays.c*\n",
    "```c\n",
    "void arr_minus_one(int *data, unsigned int arr_size) {\n",
    "    printf(\"From C Library:\\n\");\n",
    "    for (int i = 0; i < arr_size; ++i) {\n",
    "        data[i] = data[i] - 1;\n",
    "        printf(\"%d \", data[i]);\n",
    "    }\n",
    "}\n",
    "```"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Getting pointer from numpy array"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python numpy array:\n",
      " [[1 2 3]\n",
      " [4 5 6]\n",
      " [7 8 9]]\n",
      "Updated python numpy array:\n",
      " [[0 1 2]\n",
      " [3 4 5]\n",
      " [6 7 8]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "arr = np.asarray([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.int32)\n",
    "data_p = arr.ctypes.data_as(ctypes.POINTER(ctypes.c_int))\n",
    "arr_size = arr.size\n",
    "el_size = arr.itemsize\n",
    "dtype = str(arr.dtype).encode('utf-8')\n",
    "\n",
    "lib.arr_minus_one.restype = None\n",
    "lib.arr_minus_one.argtypes = [ctypes.POINTER(ctypes.c_int), ctypes.c_uint]\n",
    "print(f\"Python numpy array:\\n {arr}\")\n",
    "with capture_c_stdout():\n",
    "    lib.arr_minus_one(data_p, arr_size, el_size, dtype)\n",
    "print(f\"Updated python numpy array:\\n {arr}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Getting pointer from python list"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python list:\n",
      " [1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
      "Updated python list:\n",
      " [1, 2, 3, 4, 5, 6, 7, 8, 9]\n"
     ]
    }
   ],
   "source": [
    "arr = [1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
    "arr_size = len(arr)\n",
    "data_p = (ctypes.c_int * arr_size)(*arr)\n",
    "print(f\"Python list:\\n {arr}\")\n",
    "with capture_c_stdout():\n",
    "    lib.arr_minus_one(data_p, arr_size)\n",
    "print(f\"Updated python list:\\n {arr}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Allocating dynamic array to create numpy array from the pointer"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "```c\n",
    "int *gen_arr(unsigned int size) {\n",
    "    int *arr = malloc(sizeof(int) * size);\n",
    "    for (unsigned int i = 0; i < size; ++i) {\n",
    "        arr[i] = i;\n",
    "    }\n",
    "    return arr;\n",
    "}\n",
    "\n",
    "void free_arr(int *p) {\n",
    "    printf(\"C: memory deallocation invoked\");\n",
    "    free(p);\n",
    "}\n",
    "```"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python: object destructor invoked\n",
      "C: memory deallocation invoked\n",
      "Shaped numpy array:\n",
      " [     0      1      2 ... 999997 999998 999999]\n"
     ]
    }
   ],
   "source": [
    "lib.gen_arr.restype = ctypes.POINTER(ctypes.c_int)\n",
    "lib.gen_arr.argtypes = [ctypes.c_int, ]\n",
    "arr_size = 1000000\n",
    "data_p = lib.gen_arr(arr_size)\n",
    "\n",
    "class ArrayWrap:\n",
    "    def __init__(self, ptr, shape):\n",
    "        self.array = np.ctypeslib.as_array(ptr, shape=shape)\n",
    "        self.__lib_path = C_LIB_PATH\n",
    "        self.__ptr = ptr\n",
    "    def __del__(self):\n",
    "        with capture_c_stdout():\n",
    "            print(\"Python: object destructor invoked\")\n",
    "            lib = ctypes.CDLL(self.__lib_path)\n",
    "            lib.free_arr.argtypes = [ctypes.POINTER(ctypes.c_int),]\n",
    "            lib.free_arr(self.__ptr)\n",
    "\n",
    "array_wrap = ArrayWrap(data_p, (arr_size, ))\n",
    "print(f\"\\nShaped numpy array:\\n {array_wrap.array}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "On the image bellow we can see that allocation and deallocation memory size caused in our library is equal.\n",
    "\n",
    "![\"Memory profiling snapshot\"](img/mem_prof.jpg \"Memory profiling snapshot\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Fill pre-allocated memory"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Write to ctypes array"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "```c\n",
    "void fill_arr(int *arr, unsigned int size) {\n",
    "    for (int i = 0; i < size; ++i) {\n",
    "        arr[i] = i;\n",
    "    }\n",
    "}\n",
    "```"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Resulting numpy array:\n",
      " [[ 0  1  2  3  4]\n",
      " [ 5  6  7  8  9]\n",
      " [10 11 12 13 14]]\n"
     ]
    }
   ],
   "source": [
    "lib.gen_arr.argtypes = [ctypes.POINTER(ctypes.c_int), ctypes.c_uint]\n",
    "lib.gen_arr.restype = None\n",
    "\n",
    "# Allocating memory\n",
    "i_arr = (ctypes.c_int * 15)()\n",
    "i_p = ctypes.cast(i_arr, ctypes.POINTER(ctypes.c_int))\n",
    "lib.fill_arr(i_p, 15)\n",
    "# Build numpy array\n",
    "shaped_array = np.ctypeslib.as_array(i_p, shape=(3, 5))\n",
    "print(f\"Resulting numpy array:\\n {shaped_array}\")\n",
    "\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Write directly to numpy array's memory"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shaped numpy array:\n",
      " [[ 0  1  2  3  4]\n",
      " [ 5  6  7  8  9]\n",
      " [10 11 12 13 14]\n",
      " [15 16 17 18 19]\n",
      " [20 21 22 23 24]]\n"
     ]
    }
   ],
   "source": [
    "arr = np.ndarray(shape=(5, 5), dtype=np.int32)\n",
    "arr_p = arr.ctypes.data_as(ctypes.POINTER(ctypes.c_int))\n",
    "lib.fill_arr(arr_p, 25)\n",
    "print(f\"Shaped numpy array:\\n {arr}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Find library and callbacks"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Ctypes provides functionality allowing to locate a library in a way similar to what the compiler or runtime loader\n",
    "does (on platforms with several versions of a shared library the most recent should be loaded)\n",
    "**To search fo a library in a default paths** `find_library()` method should be called with library name without extension\n",
    "name or 'lib' prefix on Linux\\MacOS. (In example bellow 'c' library name stands for `libc.so` or `libc.dyli`\n",
    "depending on platform)\n",
    "\n",
    "We will use `qsort` function from standard C library performing quick sort of an array.\n",
    "The function requires following arguments: array to sort, array length, element size, and comparing callback function.\n",
    "\n",
    "\n",
    "In case bellow we will pass Python function as sorting callback to `qsort` function to demonstrate **Python function\n",
    "calls from C code mechanism** using ctypes library.\n",
    "The callback will then be called with two pointers to items, and it must return a negative integer if the first\n",
    "item is smaller than the second, a zero if they are equal, and a positive integer otherwise."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 1, 5, 5, 7, 12, 33, 43, 99]\n"
     ]
    }
   ],
   "source": [
    "from ctypes.util import find_library\n",
    "# libc and find_library\n",
    "libc = ctypes.CDLL(find_library('c'))\n",
    "\n",
    "a = (ctypes.c_int * 10)(5, 1, 7, 33, 99, 43, 12, 5, 1, 0)\n",
    "libc.qsort.restype = None\n",
    "\n",
    "# Creating callback function using CFUNCTYPE function factory decorator\n",
    "@ctypes.CFUNCTYPE(ctypes.c_int, ctypes.POINTER(ctypes.c_int), ctypes.POINTER(ctypes.c_int))\n",
    "def py_cmp_func(a, b):\n",
    "    return a[0] - b[0]\n",
    "\n",
    "# Calling C qsort passing python function as sorting callback\n",
    "libc.qsort(a, len(a), ctypes.sizeof(ctypes.c_int), py_cmp_func)\n",
    "print([v for v in a])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Some performance comparison"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bellow we will compare calling of quick sort in a three different modes:\n",
    "* Pure python quick sort implementation\n",
    "* Calling C qsort implementation with Python comparison callback\n",
    "* Pure C qsort with C comparison callback implementation"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python qsort time: 4.709294989999989\n",
      "libc qsort with Python callback time: 10.811738563000006\n",
      "Pure C qsort: 0.1369612619999998\n"
     ]
    }
   ],
   "source": [
    "from timeit import default_timer\n",
    "import random\n",
    "\n",
    "def py_qsort(nums):\n",
    "    if len(nums) <= 1:\n",
    "        return nums\n",
    "    else:\n",
    "        q = random.choice(nums)\n",
    "    l_nums = [n for n in nums if n < q]\n",
    "\n",
    "    e_nums = [q] * nums.count(q)\n",
    "    b_nums = [n for n in nums if n > q]\n",
    "    return py_qsort(l_nums) + e_nums + py_qsort(b_nums)\n",
    "\n",
    "rand_arr = np.random.randint(low=0, high=500000, size=1000000, dtype=np.int32)\n",
    "\n",
    "rand_list = list(rand_arr)\n",
    "t0 = default_timer()\n",
    "sorted_arr = py_qsort(rand_list)\n",
    "t1 = default_timer()\n",
    "print(f\"Python qsort time: {t1 - t0}\")\n",
    "\n",
    "rand_data_p = rand_arr.ctypes.data_as(ctypes.POINTER(ctypes.c_int))\n",
    "t0 = default_timer()\n",
    "libc.qsort(rand_data_p, rand_arr.size, ctypes.sizeof(ctypes.c_int), py_cmp_func)\n",
    "t1 = default_timer()\n",
    "print(f\"libc qsort with Python callback time: {t1 - t0}\")\n",
    "\n",
    "rand_arr = np.random.randint(low=0, high=500000, size=1000000, dtype=np.int32)\n",
    "rand_data_p = rand_arr.ctypes.data_as(ctypes.POINTER(ctypes.c_int))\n",
    "lib.qsort_wrap.restype = None\n",
    "t0 = default_timer()\n",
    "lib.qsort_wrap(rand_data_p, rand_arr.size, ctypes.sizeof(ctypes.c_int))\n",
    "t1 = default_timer()\n",
    "print(f\"Pure C qsort: {t1 - t0}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Why callbacks are so slow?"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "To trace hotspot of C function which calls Python functions as callbacks we will profile slightly different script since\n",
    "qucik sort is a recursive function which pretty hard to analyze.\n",
    "\n",
    "```c\n",
    "void upd_arr(int *arr, unsigned int size, int(*callback)(int)) {\n",
    "    for (int i = 0; i < size; ++i) {\n",
    "        arr[i] = callback(i);\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "```python\n",
    "arr = np.ones(shape=(50000000), dtype=np.int32)\n",
    "\n",
    "arr_p = arr.ctypes.data_as(ctypes.POINTER(ctypes.c_int))\n",
    "func_type = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_int)\n",
    "lib.upd_arr.argtypes = [ctypes.POINTER(ctypes.c_int), ctypes.c_uint, func_type]\n",
    "\n",
    "@func_type\n",
    "def py_func(a):\n",
    "    return a\n",
    "\n",
    "lib.upd_arr(arr_p, arr.size, py_func)\n",
    "```"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bellow a profiling trace from VTune which demonstrates bottom-up distribution of code execution time which demonstrates\n",
    "that **calling of callback function is pretty expensive operation**. In general make sense to avoid callbacks in cases\n",
    "when it's going to be called from relatively fast function which called many times\n",
    "\n",
    "![\"Callback profiling snapshot\"](img/callback_prof.jpg \"Callback profiling snapshot\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# CFFI demo"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "CFFI library provides similar with ctypes functionality and also extends it with several modes of library usage.\n",
    "\n",
    "**Main advantages of cffi library:**\n",
    "* The goal is to call C code from Python without learning a 3rd language: existing alternatives require users to\n",
    "learn domain specific language (Cython, SWIG) or API (ctypes). The CFFI design requires users to know only C and Python,\n",
    "**minimizing the extra bits of API that need to be learned**.\n",
    "* Keep all the Python-related logic in Python so that you don’t need to write much C code\n",
    "(unlike CPython native C extensions).\n",
    "* **Functions signatures and types definition performed automatically** basing on headers parsing.\n",
    "User don't have to describe function arguments and return value, structure fields etc.\n",
    "* API and ABI usage modes\n",
    "\n",
    "**Main disadvantages:**\n",
    "* **Preprocessors directives** are not supported which force to implement single-header library since `#include`\n",
    "directives are not allowed on header parsing stage.\n",
    "* Little bit complicated building process especially in API mode."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## In-Line ABI Mode"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The simplest mode from implementation perspective.\n",
    "This mode is pretty the same as `ctypes.CDLL()` which **opens precompiled library** and works on ABI (\n",
    "Application Binary Interface) level."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "outputs": [],
   "source": [
    "# Parsing header file to get types and signatures\n",
    "import cffi\n",
    "ffi = cffi.FFI()\n",
    "include = os.path.join(\"c_library\", \"library.h\")\n",
    "with open(include) as f:\n",
    "    ffi.cdef(f.read())"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Opening precompiled library and accessing functions. No need to describe function signature.\n",
    "We will measure library loading time to compare In-Line and Out-of-Line ABI modes bellow"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ INFO ] In-line ABI loading time 0.00016225799998892398\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From C Library: func_ret_str: Some str\n",
      "From Python Script: Some str\n"
     ]
    }
   ],
   "source": [
    "t0 = default_timer()\n",
    "lib = ffi.dlopen(C_LIB_PATH)\n",
    "t1 = default_timer()\n",
    "log.info(f\"In-line ABI loading time {t1 - t0}\")\n",
    "with capture_c_stdout():\n",
    "    res = lib.func_ret_str(\"Some str\".encode(\"utf-8\"))\n",
    "    print(f\"From Python Script: {ffi.string(res).decode('utf-8')}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Out-of-Line ABI Mode"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As in In-Line ABI mode, opens precompiled library with **further compilation to create python module**\n",
    "with binary info which speed-ups import time and improves performance.\n",
    "\n",
    "No real compilation happens (C compiler doesn't invoked) since we doesn't provide any sources in `ffi.set_source()` call"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ INFO ] Content of resulting folder\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./cffi_libs/out_of_line_abi/:\n",
      "    __pycache__\n",
      "    _bindings_demo.py\n"
     ]
    }
   ],
   "source": [
    "# Need to reload cffi module and ffi object to allow to redefine header file and sources\n",
    "# (Such trick required only for this jupyter notebook)\n",
    "if \"cffi\" in globals():\n",
    "    globals().pop(\"cffi\")\n",
    "if \"ffi\" in globals():\n",
    "    globals().pop(\"ffi\")\n",
    "import cffi\n",
    "ffi = cffi.FFI()\n",
    "with open(include) as f:\n",
    "    ffi.cdef(f.read())\n",
    "\n",
    "# Set destination package structure\n",
    "ffi.set_source(\"cffi_libs.out_of_line_abi._bindings_demo\", None)\n",
    "# Compile\n",
    "ffi.compile()\n",
    "target_module = \"./cffi_libs/out_of_line_abi/_bindings_demo.py\"\n",
    "if not os.path.exists(target_module):\n",
    "    raise FileNotFoundError(f\"{target_module} not found! Possibly CFFI module compilation failed!\")\n",
    "log.info(\"Content of resulting folder\")\n",
    "print(\"./cffi_libs/out_of_line_abi/:\")\n",
    "for entry in os.listdir(\"./cffi_libs/out_of_line_abi/\"):\n",
    "    print(f\"    {entry}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Unlike In-Line ABI mode we will create `FFI` object not from `cffi` module but **import it from our compiled above package**\n",
    "(stored in `ffi` variable).\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ INFO ] Out-of-line ABI import time 0.00015120699998760756\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From C Library: func_ret_str: Some str\n",
      "From Python Script: Some str\n"
     ]
    }
   ],
   "source": [
    "from cffi_libs.out_of_line_abi._bindings_demo import ffi as out_of_line_abi_ffi\n",
    "t0 = default_timer()\n",
    "lib = out_of_line_abi_ffi.dlopen(C_LIB_PATH)\n",
    "t1 = default_timer()\n",
    "log.info(f\"Out-of-line ABI import time {t1 - t0}\")\n",
    "with capture_c_stdout():\n",
    "    res = lib.func_ret_str(\"Some str\".encode(\"utf-8\"))\n",
    "    print(f\"From Python Script: {ffi.string(res).decode('utf-8')}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As we can see Out-of-line ABI mode speed-up library opening comparing with In-Line ABI and also reduce ffi calls\n",
    "overhead (will see bellow)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Out-of-Line API Mode\n",
    "**Preferable mode**"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "This mode requires passing library sources to `cffi.set_source()` function to perform **importable python library\n",
    "compilation**. On compilation stage CFFI generates source code which wraps methods from provided in\n",
    "C library (or from source code) with required CPython code and then invokes C compiler to produce importable\n",
    "python library. This mode results to the best performance but requires preliminary compilation."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generating ./cffi_libs/out_of_line_api/_bindings_demo.c\n",
      "(already up-to-date)\n",
      "setting the current directory to '/Users/mikhailtreskin/repos/py_bindings_tutorial/cffi_libs/out_of_line_api'\n",
      "running build_ext\n",
      "building '_bindings_demo' extension\n",
      "clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/mikhailtreskin/repos/py_bindings_tutorial/venv/include -I/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/include/python3.8 -c _bindings_demo.c -o ./_bindings_demo.o\n",
      "clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk ./_bindings_demo.o -L/Users/mikhailtreskin/repos/py_bindings_tutorial/c_library/build -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -lbindings_demo -o ./_bindings_demo.cpython-38-darwin.so\n",
      "./cffi_libs/out_of_line_api\n",
      "    _bindings_demo.cpython-38-darwin.so\n",
      "    _bindings_demo.c\n",
      "    _bindings_demo.o\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ INFO ] Content of resulting folder\n"
     ]
    }
   ],
   "source": [
    "if \"cffi\" in globals():\n",
    "    globals().pop(\"cffi\")\n",
    "if \"ffi\" in globals():\n",
    "    globals().pop(\"ffi\")\n",
    "import cffi\n",
    "ffi = cffi.FFI()\n",
    "with open(include) as f:\n",
    "    ffi.cdef(f.read())\n",
    "module_name = \"_bindings_demo\"\n",
    "ffi.set_source(module_name,  # name of compiled library\n",
    "               # Source code to be compiled along with CFFI-generated wrapper code\n",
    "               f'#include \"../../{include}\"',\n",
    "               # Libraries to link compiled CFFI module with (no extension and 'lib' prefix)\n",
    "               # Passed directly to compiler as '-l' option\n",
    "               libraries=[\"bindings_demo\"],\n",
    "               library_dirs=[os.path.dirname(C_LIB_PATH)],\n",
    "               )\n",
    "target_dir = './cffi_libs/out_of_line_api'\n",
    "ffi.compile(tmpdir=target_dir, verbose=True)\n",
    "\n",
    "log.info(\"Content of resulting folder\")\n",
    "print(target_dir)\n",
    "target_module_found = False\n",
    "library_name = None\n",
    "for entry in os.listdir(target_dir):\n",
    "    print(f\"    {entry}\")\n",
    "    if entry.startswith(module_name) and entry.endswith(\".so\"):  # both Linux and MacOS creates .so file\n",
    "        library_name = entry\n",
    "        target_module_found = True\n",
    "\n",
    "if not target_module_found:\n",
    "    raise FileNotFoundError(f\"{target_module} library not found! Possibly CFFI module compilation failed!\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Note that compiled python module dynamically linked against initial 'bindings_demo' library"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ INFO ] _bindings_demo.cpython-38-darwin.so library dependencies:\n",
      "./cffi_libs/out_of_line_api/_bindings_demo.cpython-38-darwin.so:\n",
      "\t/Users/mikhailtreskin/repos/py_bindings_tutorial/c_library/build/libbindings_demo.dylib (compatibility version 0.0.0, current version 0.0.0)\n",
      "\t/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if platform == \"darwin\":\n",
    "    command = ['otool', '-L', os.path.join(target_dir, library_name)]\n",
    "elif platform.startswith(\"linux\"):\n",
    "    command = ['ldd', os.path.join(target_dir, library_name)]\n",
    "p = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n",
    "\n",
    "if p.returncode != 0:\n",
    "    log.error(\"otool execution failed!\\n\"\n",
    "              f\"STDOUT:\\n {p.stdout.decode()}\\n\"\n",
    "              f\"STDERR:\\n {p.stderr.decode()}\\n\")\n",
    "else:\n",
    "    log.info(f\"{library_name} library dependencies:\\n\"\n",
    "             f\"{p.stdout.decode()}\\n\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Compiled module usage example"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From C Library: func_ret_str: Some str\n",
      "From Python Script: Some str\n"
     ]
    }
   ],
   "source": [
    "from cffi_libs.out_of_line_api._bindings_demo import ffi as out_of_line_api_ffi\n",
    "from cffi_libs.out_of_line_api._bindings_demo import lib as out_of_line_api_lib\n",
    "\n",
    "# dlopen not required\n",
    "with capture_c_stdout():\n",
    "    res = out_of_line_api_lib.func_ret_str(\"Some str\".encode(\"utf-8\"))\n",
    "    print(f\"From Python Script: {out_of_line_api_ffi.string(res).decode('utf-8')}\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Some extra usage examples"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Declaring C source right in set_source() method"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Another possible way of using Out-of-Line API mode which allows to avoid dynamic dependencies\n",
    "is to specify all C sources right in `ffi.set_source()` method and do not add dependency libraries.\n",
    "``` python\n",
    "ffi.set_source(\"_my_module\",  # name of compiled library\n",
    "               # Source code to be compiled along with CFFI-generated wrapper code\n",
    "               '''\n",
    "                #include \"library.h\"\n",
    "\n",
    "                #include <stdio.h>\n",
    "                #include <string.h>\n",
    "                #include <unistd.h>\n",
    "\n",
    "                int func_ret_int(int val) {\n",
    "                    printf(\"From C Library: func_ret_int: %d\\n\", val);\n",
    "                    return val;\n",
    "                }\n",
    "                ''',\n",
    "               # Libraries to link compiled CFFI module with (no extension and 'lib' prefix)\n",
    "               # Passed directly to compiler as '-l' option\n",
    "               libraries=[],\n",
    "               )\n",
    "```"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Setuptools and distutils integration"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finally, you can (but don’t have to) use CFFI’s Distutils or Setuptools integration when writing a setup.py.\n",
    "\n",
    "For Distutils (only in out-of-line API mode):\n",
    "``` python\n",
    "# setup.py (requires CFFI to be installed first)\n",
    "from distutils.core import setup\n",
    "\n",
    "import foo_build   # possibly with sys.path tricks to find it\n",
    "\n",
    "setup(\n",
    "    ...,\n",
    "    ext_modules=[foo_build.ffibuilder.distutils_extension()],\n",
    ")\n",
    "```\n",
    "\n",
    "For Setuptools (out-of-line, but works in ABI or API mode; recommended):\n",
    "``` python\n",
    "# setup.py (with automatic dependency tracking)\n",
    "from setuptools import setup\n",
    "\n",
    "setup(\n",
    "    ...,\n",
    "    setup_requires=[\"cffi>=1.0.0\"],\n",
    "    cffi_modules=[\"package/foo_build.py:ffibuilder\"],\n",
    "    install_requires=[\"cffi>=1.0.0\"],\n",
    ")\n",
    "```\n",
    "Note again that the foo_build.py example contains the following lines, which mean that the ffi builder is not\n",
    "actually compiled when package.foo_build is merely imported—it will be compiled independently by the Setuptools logic,\n",
    "using compilation parameters provided by Setuptools:\n",
    "\n",
    "``` python\n",
    "if __name__ == \"__main__\":    # not when running with setuptools\n",
    "    ffibuilder.compile(verbose=True)\n",
    "```"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Partial C code declaration\n",
    "\n",
    "Using `...` in `ffi.cdef()` allows to skip some part of declarations in case of explicit declarations specification\n",
    "(as a multiline string) and 'ask' C compiler to fill the gaps. Such way of declaration can be convenient in some\n",
    "cases e.g. for `struct{}` or `union{}` declaration.\n",
    "More details in the [documentation](https://cffi.readthedocs.io/en/latest/cdef.html#letting-the-c-compiler-fill-the-gaps)."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# FFI overhead comparison"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In order to compare overheads of calling C functions from python in ctypes and CFFI (different modes)\n",
    "we will run `char *do_nothing_function(char *some_str)` C function which literally doesn't do anything\n",
    "except receiving and returning some value.\n",
    "In such approach all of the calling time will be spent on accessing C function from Python code which allows to asses\n",
    "only FFI interface overhead."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "╒══════════════════════╤═════════════╤══════════╕\n",
      "│ Library              │        Mean │    Ratio │\n",
      "╞══════════════════════╪═════════════╪══════════╡\n",
      "│ ctypes               │ 5.49664e-07 │ 0.760538 │\n",
      "├──────────────────────┼─────────────┼──────────┤\n",
      "│ cffi in-line         │ 5.17167e-07 │ 0.808328 │\n",
      "├──────────────────────┼─────────────┼──────────┤\n",
      "│ cffi out-of-line abi │ 4.67511e-07 │ 0.894184 │\n",
      "├──────────────────────┼─────────────┼──────────┤\n",
      "│ cffi out-of-line api │ 4.18041e-07 │ 1        │\n",
      "╘══════════════════════╧═════════════╧══════════╛\n"
     ]
    }
   ],
   "source": [
    "from  tabulate import tabulate\n",
    "ITERATIONS_COUNT = 500000\n",
    "c_types_lib = ctypes.CDLL(C_LIB_PATH)\n",
    "\n",
    "ffi = cffi.FFI()\n",
    "with open(include) as f:\n",
    "    ffi.cdef(f.read())\n",
    "cffi_in_line_abi_lib = ffi.dlopen(C_LIB_PATH)\n",
    "\n",
    "from cffi_libs.out_of_line_abi._bindings_demo import ffi as out_of_line_abi_ffi\n",
    "cffi_out_of_line_abi_lib = out_of_line_abi_ffi.dlopen(C_LIB_PATH)\n",
    "\n",
    "from cffi_libs.out_of_line_api._bindings_demo import lib as out_of_line_api_lib\n",
    "\n",
    "c_types_times = []\n",
    "for i in range(ITERATIONS_COUNT):\n",
    "    t0 = default_timer()\n",
    "    res = c_types_lib.do_nothing_function(\"Some str\".encode(\"utf-8\"))\n",
    "    c_types_times.append(default_timer() - t0)\n",
    "\n",
    "cffi_in_line_abi_lib_times = []\n",
    "for i in range(ITERATIONS_COUNT):\n",
    "    t0 = default_timer()\n",
    "    res = cffi_in_line_abi_lib.do_nothing_function(\"Some str\".encode(\"utf-8\"))\n",
    "    cffi_in_line_abi_lib_times.append(default_timer() - t0)\n",
    "\n",
    "cffi_out_of_line_abi_times = []\n",
    "for i in range(ITERATIONS_COUNT):\n",
    "    t0 = default_timer()\n",
    "    res = cffi_out_of_line_abi_lib.do_nothing_function(\"Some str\".encode(\"utf-8\"))\n",
    "    cffi_out_of_line_abi_times.append(default_timer() - t0)\n",
    "\n",
    "cffi_out_of_line_api_times = []\n",
    "for i in range(ITERATIONS_COUNT):\n",
    "    t0 = default_timer()\n",
    "    res = out_of_line_api_lib.do_nothing_function(\"Some str\".encode(\"utf-8\"))\n",
    "    cffi_out_of_line_api_times.append(default_timer() - t0)\n",
    "\n",
    "ctypes_mean = np.mean(c_types_times)\n",
    "cffi_in_line_abi_mean = np.mean(cffi_in_line_abi_lib_times)\n",
    "cffi_out_of_line_abi_mean = np.mean(cffi_out_of_line_abi_times)\n",
    "cffi_out_of_line_api_mean = np.mean(cffi_out_of_line_api_times)\n",
    "\n",
    "fastest_mean = np.min([ctypes_mean, cffi_in_line_abi_mean, cffi_out_of_line_abi_mean, cffi_out_of_line_api_mean])\n",
    "headers = [\"Library\", \"Mean\", \"Ratio\"]\n",
    "results = [\n",
    "    [\"ctypes\", ctypes_mean, fastest_mean / ctypes_mean],\n",
    "    [\"cffi in-line\", cffi_in_line_abi_mean, fastest_mean / cffi_in_line_abi_mean],\n",
    "    [\"cffi out-of-line abi\", cffi_out_of_line_abi_mean, fastest_mean / cffi_out_of_line_abi_mean],\n",
    "    [\"cffi out-of-line api\", cffi_out_of_line_api_mean, fastest_mean / cffi_out_of_line_api_mean],\n",
    "]\n",
    "print(tabulate(results, headers=headers, tablefmt='fancy_grid'))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# SWiG overview"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "*Pros*\n",
    "* Multi-language bindings library (Python, Java, C#, Perl etc. support)\n",
    "* Automatic STL containers support\n",
    "* Elegant Python interface\n",
    "\n",
    "*Cons*\n",
    "* Explicit compilation required or not really obvious CMake integration\n",
    "([tutorial here](https://github.com/danielunderwood/swig-example))\n",
    "* Need to create separate interface file with SWIG-own syntax\n",
    "* Explicit template instantiations required for C++\n",
    "* Only API mode"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Example (from docs)\n",
    "\n",
    "*Header file  example*\n",
    "```cpp\n",
    "// pair.h.  A pair like the STL\n",
    " namespace std {\n",
    "    template<class T1, class T2> struct pair {\n",
    "        T1 first;\n",
    "        T2 second;\n",
    "        pair() : first(T1()), second(T2()) { };\n",
    "        pair(const T1 &f, const T2 &s) : first(f), second(s) { }\n",
    "    };\n",
    " }\n",
    "```\n",
    "*SWIG interface file*\n",
    "```\n",
    " // pair.i - SWIG interface\n",
    " %module pair\n",
    " %{\n",
    " #include \"pair.h\"\n",
    " %}\n",
    "\n",
    " // Ignore the default constructor\n",
    " %ignore std::pair::pair();\n",
    "\n",
    " // Parse the original header file\n",
    " %include \"pair.h\"\n",
    "\n",
    " // Instantiate some templates\n",
    "\n",
    " %template(pairii) std::pair<int,int>;\n",
    " %template(pairdi) std::pair<double,int>;\n",
    "```\n",
    "\n",
    "*Compilation example*\n",
    "\n",
    "```shell\n",
    "swig -python -c++ pair.i\n",
    "g++ -c -fpic pair_wrap.c -I/usr/include/python3.8\n",
    "g++ -shared pair_wrap.o -o _pair.so\n",
    "```\n",
    "\n",
    "*Python usage example*\n",
    "\n",
    "```python\n",
    "import pair\n",
    "a = pair.pairii(3,4)\n",
    "print(a.first)\n",
    "print(a.second)\n",
    "a.second = 16\n",
    "print(a.second)\n",
    "b = pair.pairdi(3.5,8)\n",
    "print(b.first)\n",
    "print(b.second)\n",
    "```"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Boost-Python"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Outdated library, up-to-date alternative is pybind11.\n",
    "\n",
    "**From pybind11 docs:**\n",
    "\n",
    "*\"The main issue with Boost.Python—and the reason for creating such a similar project—is Boost.\\t the\n",
    "oldest and buggiest of compiler specimens. Now that C++11-compatible compilers are widely available,\n",
    "this heavy machinery has become an excessively large and unnecessary dependency.\"*\n",
    "\n",
    "[Tutorial example](https://habr.com/ru/post/168083/)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}