folly/python/executor_api.h No such file or directory #1703
Comments
|
I think that the whole directory |
|
the problem has evolved it appears - there is no longer a missing file, executor_api.h, but an output naming collision with -DPYTHON_EXTENSIONS=ON enabled. Does anyone own folly/cython bindings? actively working in this area? Presumably the internal buck build works and only oss build had issues. Are there any active clients of these python bindings? cmake with -DPYTHON_EXTNSIONS=ON error output Compiling folly/executor.pyx because it changed. Error compiling Cython file:... Copyright (c) Meta Platforms, Inc. and affiliates.^folly/executor.pyx:1:0: The output file already exists and does not look like it was generated by Cython: "executor.cpp" |
|
@gmarzot any progress on this ? |
see: gmarzot#1 I am building like this: sudo ./build/fbcode_builder/getdeps.py install-system-deps --recursive DESTDIR=/tmp/folly ./build/fbcode_builder/getdeps.py build --scratch-path /tmp/folly --allow-system-packages --extra-cmake-defines='{"CMAKE_CXX_FLAGS": "-fPIC", "PYTHON_EXTENSIONS": "ON"}' --no-tests next step is to get a setup.py for folly/folly/python/test directory and see if those tests can run (i hove one weird problem - one dev host is ubuntu 20.04 lts and I cant get modern python3-dev installed so build fails with missing "Python.h" - clues and solutions welcome) |
Thank you ! I tried (macOS), and it's failing for some other issues, I am investigating.
I had a similar issue recently, I fixed It by compiling indicating the path to my python. For me it was missing: |
|
I manage to build it completely using the branch of your fork ! I encountered the same missing
(I am specifying Then I encountered: I solved it by adding Now I am stuck on: |
|
I think that lib needs to be in the cython setup.py
libraries=["folly", "fmt", "glog", "double-conversion"],
likely there are others. what does your test script look like and what is
PYTHONPATH?
…On Sun, Dec 15, 2024 at 7:59 AM аэт ***@***.***> wrote:
@gmarzot <https://github.com/gmarzot>
I manage to build it completely ! I encountered the same missing
"Python.h" issue as you, here is how I solved it:
DESTDIR=/tmp/folly python3.11 ./build/fbcode_builder/getdeps.py build
folly --scratch-path /tmp/folly --allow-system-packages
--extra-cmake-defines='{"CMAKE_CXX_FLAGS": "-fPIC $$(python3.11-config
--includes) $$(python3.11-config --ldflags)", "PYTHON_EXTENSIONS": "ON"}'
--no-tests
(I am specifying python3.11 because my python3 points to python3.13)
Now I am stuck with:
% python -u "/Users/n/cpp/2/t.py"
Traceback (most recent call last):
File "/Users/n/cpp/2/t.py", line 1, in <module>
from folly.iobuf import IOBuf, WritableIOBuf
ImportError: dlopen(/opt/homebrew/lib/python3.11/site-packages/folly/iobuf.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '__ZN3fmt3v117vformatENS0_17basic_string_viewIcEENS0_17basic_format_argsINS0_7contextEEE'
Which says that the function fmt::v11::vformat(fmt::v11::basic_string_view<char>,
fmt::v11::basic_format_args<fmt::v11::context>) was not included during
the build. It seems like adding -lfmt to the "CMAKE_CXX_FLAGS" isn't
fixing it.
—
Reply to this email directly, view it on GitHub
<#1703 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB7T3IHCTU7LTKEXS6TDO5D2FV4LVAVCNFSM6AAAAABTH2XYRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBTHA3DIMZZGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Ha.. I'm behind by one message. Looking |
in setup.py |
just import folly.executoror import folly.iobuf |
|
After resolving the missing ones, I can import all. Here is the setup.py: #!/usr/bin/env python3
# Copyright (c) Meta Platforms, Inc. and affiliates.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Do not call directly, use cmake
#
# Cython requires source files in a specific structure, the structure is
# created as tree of links to the real source files.
import sys, platform
from Cython.Build import cythonize
from Cython.Compiler import Options
from setuptools import Extension, setup
Options.fast_fail = True
if sys.platform == "darwin" and platform.machine() == "arm64":
# Macos (arm64, homebrew path is different than intel)
library_dirs = ["/opt/homebrew/lib"]
else:
# Debian/Ubuntu
library_dirs = ["/usr/lib", "/usr/lib/x86_64-linux-gnu"]
exts = [
Extension(
"folly.executor",
sources=[
"folly/executor.pyx",
"folly/executor_intf.cpp",
"folly/ProactorExecutor.cpp",
"folly/error.cpp",
],
libraries=["folly", "glog", "double-conversion"],
extra_compile_args=["-std=c++20"], # C++20 for coroutines
include_dirs=[".", "../.."], # cython generated code
library_dirs=library_dirs,
),
Extension(
"folly.iobuf",
sources=[
"folly/iobuf.pyx",
"folly/iobuf_intf.cpp",
"folly/iobuf_ext.cpp",
"folly/error.cpp",
],
libraries=["folly", "glog", "double-conversion", "fmt"],
extra_compile_args=["-std=c++20"], # C++20 for coroutines
include_dirs=[".","../.."], # cython generated code
library_dirs=library_dirs,
),
Extension(
"folly.fiber_manager",
sources=[
"folly/fiber_manager.pyx",
"folly/fibers.cpp",
"folly/error.cpp",
],
libraries=["folly", "glog", "double-conversion", "fmt", "event"],
extra_compile_args=["-std=c++20"], # C++20 for coroutines
include_dirs=[".","../.."], # cython generated code
library_dirs=library_dirs,
)
]
setup(
name="folly",
version="0.0.1",
packages=["folly"],
package_data={"": ["*.pxd", "*.pyi", "*.h"]},
setup_requires=["cython"],
zip_safe=False,
ext_modules=cythonize(exts, compiler_directives={"language_level": 3}),
)
|
|
the warnings remain concerning and in particular the ones related to std:move (you can see in the next cython release there are changes in this space: https://cython.readthedocs.io/en/latest/src/changes.html) getting folly/folly/python/test built and running will be very telling.. needs whole setup.py and very similr process as we did in folly/folly/python (@novitae thank you for pushing this forward) uhg ... now i am getting the can't link -lfmt problem again and i had that fixed at one point. DESTDIR=/tmp/folly python3.11 ./build/fbcode_builder/getdeps.py build folly --allow-system-packages --scratch-path /tmp/folly --extra-cmake-defines='{"CMAKE_CXX_FLAGS": "-fPIC -I.venv/cpython-3.13.1-linux-x86_64-gnu/include -L.venv/cpython-3.13.1-linux-x86_64-gnu/lib", "PYTHON_EXTENSIONS": "ON"}' --no-tests clang++ -pthread -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -fPIC -pthread -shared -Wl,--exclude-libs,ALL -LModules/_hacl build/temp.linux-x86_64-cpython-313/folly/error.o build/temp.linux-x86_64-cpython-313/folly/iobuf.o build/temp.linux-x86_64-cpython-313/folly/iobuf_ext.o build/temp.linux-x86_64-cpython-313/folly/iobuf_intf.o -L/usr/lib -L/usr/lib/x86_64-linux-gnu -L/tmp/folly/build/folly -L/install/lib -lfolly -lglog -ldouble-conversion -lfmt -o build/lib.linux-x86_64-cpython-313/folly/iobuf.cpython-313-x86_64-linux-gnu.so my environment a little diff with python3.13 and cython 3.0,11 for reference I am installing python3.13 and cython3.0.11 as follows: ECHO=$([[ "$SHELL" == *"zsh"* ]] && echo "echo" || echo "echo -e")
$ECHO "${COLOR_GREEN}Setting up Python test environment ${COLOR_OFF}"
PYTHON_VERSION="3.13" # Latest Python version to install via uv
CYTHON_VERSION="3.0.11" # Target Cython release (3.1.0a1 crashes - use 3.1.0 when released)
PLATFORM="$(uname -s)"
case "${PLATFORM}" in
Linux*) PLATFORM=Linux;;
Darwin*) PLATFORM=Mac;;
*) PLATFORM="UNKNOWN:${PLATFORM}"
esac
$ECHO "${COLOR_GREEN}Detected platform: $PLATFORM ${COLOR_OFF}"
# Ensure python3 and pip3 are installed
if [ "${PLATFORM}" = "Linux" ]; then
if [ "${EUID}" -ne 0 ]; then
SUDO=sudo
fi
${SUDO} apt-get install -y python3 python3-pip python3-venv clang
elif [ "${PLATFORM}" = "Mac" ]; then
brew install python3
else
$ECHO "${COLOR_RED}[ ERROR ] Unrecognised platform: ${PLATFORM}${COLOR_OFF}"
exit 1
fi
VENV_DIR=".venv"
UV_PIP_INSTALL="uv pip install --no-config --upgrade"
export UV_PYTHON_INSTALL_DIR="${VENV_DIR}"
if [ ! -d "${VENV_DIR}" ]; then
[ -f ".python-version" ] && rm -f ".python-version"
python3 -m venv --prompt "pip" "${VENV_DIR}"
source "${VENV_DIR}/bin/activate"
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade uv
uv venv --prompt "uv" \
--seed --relocatable \
--allow-existing --no-config \
--python-preference only-managed \
--python "${PYTHON_VERSION}" \
"${VENV_DIR}"
fi
source ./.venv/bin/activate
if ! command -v "uv" >/dev/null 2>&1 ; then
$ECHO "${COLOR_RED}[ ERROR ] Failed to install uv... ${COLOR_OFF}"
exit -1
fi
# Ensure uv is upgraded to latest
${UV_PIP_INSTALL} uv
# Install desired python verion using uv
uv python install --no-config "${PYTHON_VERSION}"
uv python pin --no-config "${PYTHON_VERSION}"
# Install required packages
${UV_PIP_INSTALL} build wheel setuptools
#${UV_PIP_INSTALL} "git+https://github.com/cython/cython.git@${CYTHON_VERSION}"
${UV_PIP_INSTALL} "cython==${CYTHON_VERSION}"
${UV_PIP_INSTALL} pytest pytest-asyncio pytest-cov
${UV_PIP_INSTALL} aioquic
cd "${VENV_DIR}"
PYTHON_DIR=$(python3 -c 'import sysconfig; print(sysconfig.get_config_var("prefix").split("/")[-1])')
if [ -d "include" ] && [ ! -L "include" ]; then
rm -rf "include"
fi
ln -sf "${PYTHON_DIR}/include" "include"
cd ..
deactivate
$ECHO "${COLOR_GREEN}Python test environment is set up ${COLOR_OFF}" |
It happened few times to me also during this whole process for other dependencies ...
I think something interesting we could try if you manage to pull this off, is to make a pypi package for folly, with a way more straightforward to install, that would just use the already installed package, without having to rebuild endlessly. We are trying to do the same thing on the same week, so other people must also struggle. |
|
@novitae # Add library paths from CMAKE_PREFIX_PATH
if cmake_prefix_path := os.environ.get("CMAKE_PREFIX_PATH"):
library_dirs.extend(p + "/lib" for p in cmake_prefix_path.split(":"))
# Make sure DESTDIR is set to something reasonable for install
if 'DESTDIR' not in os.environ and 'GETDEPS_INSTALL_DIR' in os.environ:
os.environ['DESTDIR'] = '.'and yes making a releasable package would be cool .. need to get the test dir running and understand better what is implied by the issues/warning around move semantics (see warning in attached successfult build) |
Nice ! So did you also manage to import and use the lib from python ?
I have no idea what could it be. Tbh with you, I started touching to all those makefile and C++ on Saturday fro the first time. I am a quick learner, but I don't think this quick 😅 |
|
-- Up-to-date: /tmp/folly/installed/folly/include/folly/python/executor_api.h progress... still some issues. pushing current branch updates |
|
Also could you merge the pr I've done to your branch ? It contains the edited setup.py and also a small edit to the getdeps for an issue I had where the dir wasn't made when trying to write the getdeps file |
The dir creation in getdeps should not have to occur. that file, build_marker, is in inst_dir which has to have been created prior to getting there. can you recreate the error without that patch. we shouldn't have to mess with core folly getdeps for the python stuff. all your other stuff is already included in my pr.. but we are still a ways from being able to offer this back to facebook/folly. still discovering various dependencies and link and run issues. and folly/folly/python/test not built/run yet.. i will keep my pr branch up to date |
Here it is, a the fail it raises. The line I previously added fixed this issue completely, without any side effects. Anyway this is unrelated, I should do my own pr to folly about it. |
|
and that is repeatable after the follwing? (note: that last one is the only cmd that should need sudo) its my assertion that a clean environment should never produce that problem BTW i completely removed the need for DESTDIR - it looked archaic/vestigial to some ancient python env and actually did some strange things to install paths |
Ok so I did this,
(btw homebrew on macOS refuses sudo, so the getdeps using it fails with it, so I did without)
Here is my command, using a completely empty python3.13 version: And I get stuck here: When running in python 3.11, I remember seeing warning about |
|
above is my bootstrap_python.sh script . it should run fine on MaOS and leave you in a uv based venv |
It still fails using your |
|
~/Projects/moq/folly$ PYTHONPATH=/tmp/folly/installed/folly/lib/python3.13/site-packages python3 -c "from folly import fiber_manager; print('folly fiber_manager import successful')" ~/Projects/moq/folly$ nm /tmp/folly/installed/folly/lib/libfolly.a | grep use_executor ~/Projects/moq/folly$ PYTHONPATH=/tmp/folly/installed/folly/lib/python3.13/site-packages python3 -c "from folly import iobuf; print('folly iobuf import successful')" ~/Projects/moq/folly$ PYTHONPATH=/tmp/folly/installed/folly/lib/python3.13/site-packages python3 -c "import folly; print('folly import successful')" ~/Projects/moq/folly$ PYTHONPATH=/tmp/folly/installed/folly/lib/python3.13/site-packages python3 -c "from folly import executor; print('folly executor import successful')" still luck importing folly.fiber_manager - the rest imports fine. I actually cant find anything else that wants to importfolly. fiber_manager. Anway the error above is baffling as they all link to libfolly.a attached is the build log if anyone from facebook has an idea @yfeldblum . @novitae i have a mac that i am setting up so maybe some news on that in a while. will bet you 5$ no code change in getdeps should be needed ;) |
@gmarzot send me your PayPal, you were right I used your latest version of your fork, and it builds successfully on python3.13, without even a venv. Please just remove the Here is my command (if I loose it, somehow):
Also, it turns out the upper issue was happening when not using |
|
ha .. i have a sponsor button on github i think... but i'd prefer you
donate to a charity or other worthy cause. I am working on this today and
still battling the errors i have for fiber_manager import related to
multiple symbols defs/conflicts... not all the folly build flags do what
they say. I think the setup needs to be very precise about what objects to
link against. more to come but holidays going on here
…On Mon, Dec 23, 2024 at 11:18 AM аэт ***@***.***> wrote:
@novitae <https://github.com/novitae> i have a mac that i am setting up
so maybe some news on that in a while. will bet you 5$ no code change in
getdeps should be needed ;)
@gmarzot <https://github.com/gmarzot> send me your PayPal, you were right
I used your latest version of your fork, and it builds successfully on
python3.13. Please just remove the unwind lib for non-linux in the
setup.py, as it is only on linux (I didn't check windows, but it's not on
macOS)
—
Reply to this email directly, view it on GitHub
<#1703 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB7T3IH7VV7OIRLQKO3Z4YD2HAZV5AVCNFSM6AAAAABTH2XYRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJYGA4DKOBZHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@novitae some success on macos - a number of changes needed. see gmarzot#1 for updates and notes. (uv) gmarzot@MarzBook folly % export PYTHONPATH=/tmp/folly/installed/folly/lib/python3.13/site-packages python3 -c "import folly; print('folly import successful')" |
|
Hey @gmarzot, big progress so far, I hope the PR will be accepted. I've seen on your recent activity that you are working around HTTP3 (I've seen aioquic here), as I am. Moreover, there are chances we want to do the exact same thing. Would there be a way I could contact you directly to talk in private ? Twitter, discord, email, ... ? Thank you |
Hi,
I'm trying the following steps to build folly,
cmake build executed successfully.
But
make -j4got error at 97% execution raisingfolly/python/executor_api.hNo such file or directory.I checked the same location there is one
executor.hfile but couldn't locate anyexecutor_api.hfilePlease suggest if someone got similar issue.
Thanks.
The text was updated successfully, but these errors were encountered: