Read only file system ?

[asmodehn]

Using industrial_ci with one of my package I get :

[pyros_utils:make] cd /root/catkin_ws/src/pyros-utils && /root/catkin_ws/build/pyros_utils/catkin_generated/env_cached.sh flock /root/catkin_ws/build/pyros_utils/catkin_pip_env/catkin_pip.lock /root/catkin_ws/build/pyros_utils/catkin_pip_env/bin/pip install -e /root/catkin_ws/src/pyros-utils --prefix /root/catkin_ws/devel/.private/pyros_utils --no-dependencies --ignore-installed
[pyros_utils:make] Obtaining file:///root/catkin_ws/src/pyros-utils
[pyros_utils:make]     Complete output from command python setup.py egg_info:
[pyros_utils:make]     running egg_info
[pyros_utils:make]     creating pyros_utils.egg-info
[pyros_utils:make]     error: could not create 'pyros_utils.egg-info': Read-only file system
[pyros_utils:make]     
[pyros_utils:make]     ----------------------------------------
[pyros_utils:make] Command "python setup.py egg_info" failed with error code 1 in /root/catkin_ws/src/pyros-utils/

I am not familiar with catkin_tools but :

• it works on my local machine using catkin build
• i used to work on travis, in a docker, with catkin_make (see https://github.com/pyros-dev/pyros-utils and https://travis-ci.org/pyros-dev/pyros-utils/branches)

Any idea of what may be the cause here ? some specific docker image setup ?

[mathias-luedtke]

some specific docker image setup ?

Yes, the source directory is only mounted read-only, because generated files should end up in build space.
There is a fix in #123, but I haven't seen a use case that really needs it yet.

[asmodehn]

I attempted to run that version of industrial_ci, but I get a similar error...
See https://travis-ci.org/pyros-dev/pyros-utils/jobs/260479082

[asmodehn]

OK I got it working after adding WRITABLE_SOURCE=true to the travis build.

[mathias-luedtke]

Why does catkin_pip_requirements puts ("generates") files in CMAKE_SOURCE_DIR?
It should end up in CMAKE_CURRENT_BINARY_DIR, to be able to clean-up.

[asmodehn]

Pip does that for (AFAIK) :

• editable installs : https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs
• remote source dependencies : https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-src (the usecase for catkin-pip-requirements)

pip can actually do the whole workspace management by itself. It s a feature not used by default in catkin_pip, but possible, because some python packages can have unreleased dependencies.

"cleanup" in python is not the same as C++. There is no "build" in python for example (you just run the source), so no need to "cleanup" (unless you totally break your python environment, but then cleaning is quite complicated and probably not automatable).

[mathias-luedtke]

There is no "build" in python for example (you just run the source), so no need to "cleanup" (unless you totally break your python environment, but then cleaning is quite complicated and probably not automatable).

As soon as you execute something from within CMake and adding some files somewhere, this is a build.
I you put --src in the binary dir (like the venv you install to), then removing the build directory cleans up everything.

The idea behind the read-only source directory is not related to the CI build itself, but it should enforce the best practices that users (of your packages) expect (out-of-source build).

[asmodehn]

When someone checkout a repository A and also checkout a repository B in their workspace, they don't get a " you should not create file in the source tree" error. Yet this step is part of a "install" step for python. no build, no clean. just install.

So I would say that C++ create workspace + clone all repos + checkout all proper version + build + clean + build (again as much as you like) + install is actually the same as when one does pip install. Therefore there is also no cleaning needed here, and therefore not much point in keeping the source directory clean. Just have a proper .gitignore or similar file in your repo.
The best practices from one language/ecosystem to another can be very different, and all kinds of people are confused moving from one to the other (that includes all kind of bad advices you can find on the web).

That being said, I created an issue there pyros-dev/catkin_pip#142 for discussing possible improvements to catkin_pip_requirements installs, since your remarks might be a sign that the cmake pip integration might be doing much more than what people are used to and might confuse catkin users... and maybe it could be better to integrate with some other ROS tool (rosdep?) ?

I wonder how you would consider managing a project that uses https://cmake.org/cmake/help/v3.0/module/ExternalProject.html to gather its unreleased dependencies, instead of using wstool + git/svn/... manually ? This to me seems similar to the pip install -rrequirements.txt with source dependencies usecase. The user execute something within CMake and then expect files to be created in his src/...

[gavanderhoorn]

(disclaimer: I'm probably as much a C++ 'dinosaur' as @ipa-mdl is, so my views on Python dev are a bit skewed)

I just wanted to quickly comment on the fact that some comments seems to suggest that catkin somehow is doing something special, especially when it comes to avoiding in-source builds.

That is not the case. It's really a CMake convention that catkin is merely following.

I wonder how you would consider managing a project that uses https://cmake.org/cmake/help/v3.0/module/ExternalProject.html to gather its unreleased dependencies, instead of using wstool + git/svn/... manually ? This to me seems similar to the pip install -rrequirements.txt with source dependencies usecase. The user execute something within CMake and then expect files to be created in his src/...

This is actually very different. ExternalProject is not supposed to be used as a tool similar to wstool (ie: managing a source space), but it's a way to get your build dependencies into your build space during the build phase. Dependencies can either be binary distributions or they might have to be build from sources, which is what ExternalProject supports.

But all of that should go into the build space, not in the source space, as they are not part of the sources of your project, they just happen to be dependencies distributed in source format.

[machinekoder]

I have some trouble executing Python tests. I get errors importing nested modules. I'm not sure what's causing this problem, but it is very likely related to the read-only file systems. In Python, especially tests are executed in place.

[mathias-luedtke]

All source is now writable in the new master branch, because the support for the devel space was dropped.