folly cython bindings OSS build fixes #1
Conversation
…3.13 api, augment setup.py (first hack)
…ll install with PYTHONPATH set
… add boost and other dep libs
… libiberty PIC build
#support macos zsh and bash
ECHO=$([[ "$SHELL" == *"zsh"* ]] && echo "echo" || echo "echo -e")
# Define color variables
COLOR_BLUE="\033[0;36m"
COLOR_GREEN="\033[0;32m"
COLOR_RED="\033[0;31m"
COLOR_OFF="\033[0m"
$ECHO "${COLOR_GREEN}Setting up Python/Cython test environment... ${COLOR_OFF}"
PYTHON_VERSION="3.13" # Latest Python version to install via uv
CYTHON_VERSION="3.0.11" # Target Cython release (3.1.0a1 crashes - use 3.1.0 when released)
PLATFORM="$(uname -s)"
case "${PLATFORM}" in
Linux*) PLATFORM=Linux;;
Darwin*) PLATFORM=Mac;;
*) PLATFORM="UNKNOWN:${PLATFORM}"
esac
$ECHO "Detected platform: ${COLOR_GREEN}$PLATFORM${COLOR_OFF}"
# Ensure python3 and pip3 are installed
if [ "${PLATFORM}" = "Linux" ]; then
if [ "${EUID}" -ne 0 ]; then
SUDO=sudo
fi
${SUDO} apt-get install -y python3 python3-pip python3-venv clang
elif [ "${PLATFORM}" = "Mac" ]; then
brew install python3
else
$ECHO "${COLOR_RED}[ ERROR ] Unrecognised platform: ${PLATFORM}${COLOR_OFF}"
exit 1
fi
VENV_DIR=".venv"
UV_PIP_INSTALL="uv pip install --no-config --upgrade"
export UV_PYTHON_INSTALL_DIR="${VENV_DIR}"
if [ ! -d "${VENV_DIR}" ]; then
[ -f ".python-version" ] && rm -f ".python-version"
python3 -m venv --prompt "pip" "${VENV_DIR}"
source "${VENV_DIR}/bin/activate"
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade uv
uv venv --prompt "uv" \
--seed --relocatable \
--allow-existing --no-config \
--python-preference only-managed \
--python "${PYTHON_VERSION}" \
"${VENV_DIR}"
fi
source ./.venv/bin/activate
if ! command -v "uv" >/dev/null 2>&1 ; then
$ECHO "${COLOR_RED}[ ERROR ] Failed to install uv... ${COLOR_OFF}"
exit -1
fi
# Ensure uv is upgraded to latest
${UV_PIP_INSTALL} uv
# Install desired python verion using uv
uv python install --no-config "${PYTHON_VERSION}"
uv python pin --no-config "${PYTHON_VERSION}"
# Install required packages
${UV_PIP_INSTALL} build wheel setuptools
#${UV_PIP_INSTALL} "git+https://github.com/cython/cython.git"
${UV_PIP_INSTALL} "cython==${CYTHON_VERSION}"
${UV_PIP_INSTALL} pytest pytest-asyncio pytest-cov pytest-xdist
${UV_PIP_INSTALL} aioquic
version=$(python --version 2>&1 | awk '{print $NF}')
if [ $? -ne 0 ]; then
$ECHO "${COLOR_RED}Python failed to install${COLOR_OFF}"
exit 1
fi
$ECHO "Python: ${COLOR_GREEN}${version}${COLOR_OFF}"
version=$(cython --version 2>&1 | awk '{print $NF}')
if [ $? -ne 0 ]; then
$ECHO "${COLOR_RED}Cython failed to install${COLOR_OFF}"
exit 1
fi
$ECHO "Cython: ${COLOR_GREEN}${version}${COLOR_OFF}"
cd "${VENV_DIR}"
PYTHON_DIR=$(python3 -c 'import sysconfig; print(sysconfig.get_config_var("prefix").split("/")[-1])')
if [ -d "include" ] && [ ! -L "include" ]; then
rm -rf "include"
fi
ln -sf "${PYTHON_DIR}/include" "include"
cd ..
deactivate
$ECHO "Python/Cython environment set up: ${COLOR_GREEN}SUCCESS${COLOR_OFF}"
$ECHO "activate with: ${COLOR_BLUE}source ${VENV_DIR}/bin/activate${COLOR_OFF}" |
…libs, target/min python version
…or test code, get tests passing
|
got all the test building and passing. I have taken some liberties with package and file names and i suspect the interface between test code and units under test can be made more clean and isolated. this is still pretty big (uv) gmarzot@GMARZOT-LAPTOP:~/Projects/moq/folly/folly/python$ pytest -n auto -v test test/test_iobuf.py::IOBufTests::test_bytes [gw3] [ 43%] PASSED test/test_iobuf.py::WritableIOBufTests::test_buffer_creation_with_size_prepend test/test_iobuf.py::WritableIOBufTests::test_buffer_creation_with_size_trim_start_fail_multiple =========================================== 65 passed in 1.02s =========================================== |
…ate test dir from folly/python source
After running this in the environment created using the shell script on macOS, I am getting the following error: I fixed it, by removing Line 167 in 1882ddb After doing so, the error becomes: |
|
ahh .. some merge issues it seems. added your fix and then have to re-fix
folly/CMakeLists.txt to not use the colliding cython generated filenames
executor.cpp -> executor_intf.cpp iobuf.cpp -> iobuf_intf.cpp
but now another compilation issue ... the FB OSS gets broken a lot it
seems. they are using c++17 too which I saw in @jmswen'shbrough getting it
to co PR. surprised not c++20
working through getting to compile again will push to branch when done
…On Sun, Feb 9, 2025 at 1:38 PM аэт ***@***.***> wrote:
./build/fbcode_builder/getdeps.py build --extra-cmake-defines '{ "CMAKE_CXX_STANDARD": "20", "CMAKE_CXX_FLAGS": "-std=c++20 -fPIC", "PYTHON_EXTENSIONS": "ON", "BUILD_SHARED_LIBS": "ON", "BOOST_LINK_STATIC": "ON" }' --extra-b2-args "cxxflags=-fPIC" --extra-b2-args "cflags=-fPIC" --no-build-cache --no-tests --scratch-path /tmp/folly | tee "/tmp/folly_build-$(date +%Y%m%d_%H%M).log"
After running this in the environment created using the shell script on
macOS, I am getting the following error:
CMake Error at folly/CMakeLists.txt:169:
Parse error. Expected a command name, got unterminated string with text ")
endif ()
".
I fixed it, by removing
https://github.com/gmarzot/folly/blob/1882ddbe6e1133e7bf2f26671eb3541b5a037b78/folly/CMakeLists.txt#L167
After doing so, the error becomes:
CMake Error at folly/CMakeLists.txt:107 (add_library):
Cannot find source file:
python/executor.cpp
Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .h .hh .h++
.hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .ispc
CMake Error at folly/CMakeLists.txt:107 (add_library):
No SOURCES given to target: folly_python_cpp
CMake Generate step failed. Build files cannot be regenerated correctly.
—
Reply to this email directly, view it on GitHub
<#1 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB7T3ICW7ANO7PNMCE4CSIT2O6OCRAVCNFSM6AAAAABTO3SI32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNBWGQ3TGOJZHE>
.
You are receiving this because you were assigned.Message ID:
***@***.***>
|
So far here are my fixes: Lines 107 to 112 in c7c622f add_library(
folly_python_cpp
python/error.cpp
python/executor_intf.cpp
python/iobuf_intf.cpp
)Lines 42 to 49 in c7c622f file(GLOB BindingFiles
"${CMAKE_CURRENT_SOURCE_DIR}/python/error.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/python/executor.pyx"
"${CMAKE_CURRENT_SOURCE_DIR}/python/executor_intf.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/python/iobuf.pyx"
"${CMAKE_CURRENT_SOURCE_DIR}/python/iobuf_intf.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/python/iobuf_ext.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/python/fiber_manager.pyx"
"${CMAKE_CURRENT_SOURCE_DIR}/python/fibers.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/python/build_mode.pyx"
"${CMAKE_CURRENT_SOURCE_DIR}/python/ProactorExecutor.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/python/*.pxd"
"${CMAKE_CURRENT_SOURCE_DIR}/python/*.h"
)Lines 80 to 91 in c7c622f execute_process(
COMMAND python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")'
OUTPUT_VARIABLE PYTHON_VERSION_CODE
OUTPUT_STRIP_TRAILING_WHITESPACE
)
execute_process(
COMMAND python3-config --prefix
OUTPUT_VARIABLE PYTHON_PREFIX
OUTPUT_STRIP_TRAILING_WHITESPACE
)
add_custom_command(TARGET folly_python_bindings POST_BUILD
COMMAND
CFLAGS="${CMAKE_C_FLAGS}" CXXFLAGS="${CMAKE_CXX_FLAGS}"
CC="${CMAKE_C_COMPILER}" CXX="${CMAKE_CXX_COMPILER}"
python3 ${CMAKE_CURRENT_SOURCE_DIR}/python/setup.py
build_ext -f ${incs} ${libs} -I${PYTHON_PREFIX}/include/python${PYTHON_VERSION_CODE} -L${PYTHON_PREFIX}/lib
BYPRODUCTS
${_cybld}/folly/executor_api.h
${_cybld}/folly/iobuf_api.h
${_cybld}/folly/fiber_manager_api.h
WORKING_DIRECTORY ${_cybld}
)I probably over complexed it Now I am hitting against the following error. My env was created using the shell script on python 3.13.2. |
|
Any news so far ? @gmarzot |
see attached file for bootstrapping python/cython venv into build environment
moved interface source files to not collide with cython gen names:
iobuf.cpp/h => iobuf_intf.cpp/h
executor.cpp/h => executor_intf.cpp/h
include paths for all cython generated .h _api.h <folly/...> and i left all the source references as <folly/python/...>
updated CMake ignore lists
small tweak to support python 3.13 Py_IsFinalizing()
rewrite setup.py - build all packages (folly, folly.iobuf, folly.executor, folly.fiber_manager, folly.build_mode)
several manifest dependency issues:
xz url redirects to sourceforge(?) and fails download (how is this working for anyone?)
ninja rather old version not compatible with modern python 3.13
libiberty was not supporting -fPIC needed to link to python bindings (hard-coded this change - could be improved - please review)
various changes to eliminate warnings from cython 3.0.11-1
I suggest eliminating all python support for versions prior to 3.0 and likewise requiring >=3.0 for cython would be reasonable.
once the cython build above completes and installs, the test directory can be built from folly/python/test
the tests may then be run from folly/python
pytest -n auto -v testMacOS Specific Setup: