Fix/add option: do not compile fim c-extension #494

Moritz-Alexander-Kern · 2022-06-10T15:26:30Z

This PR adds an extra to elephant, which allows installation of Elephant without compiling the c++ extension fim.cpp used in SPADE.

Do not compile any extension:

development pip install -e . --install-option='--no-compile'
install pip install elephant --install-option='--no-compile'

Do not compile spade extension:

development pip install -e . --install-option='--no-compile-spade'
install pip install elephant --install-option='--no-compile-spade'

This option allows to have a workaround in cases were there is no suitable c-compiler available on the target system.

Additionally extensions are now optional, according to setuptools documentation:

 class setuptools.Extension(name, sources, *args, **kw)

optional (bool) – specifies that a build failure in the extension should not abort the build process, but simply not install the failing extension.

See elephant documentation:

https://elephant--494.org.readthedocs.build/en/494/install.html

kohlerca · 2022-07-13T12:31:46Z

I can confirm that the behavior is improved.
When installing from this branch, in a configuration that is known for build failure, the optional feature works. Apparently, the build fails, but the installation proceeds.

Collecting git+https://github.com/INM-6/elephant.git@fix/do_not_compile_extension
  Cloning https://github.com/INM-6/elephant.git (to revision fix/do_not_compile_extension) to /tmp/pip-req-build-bxf4m2qf
  Running command git clone --filter=blob:none --quiet https://github.com/INM-6/elephant.git /tmp/pip-req-build-bxf4m2qf
  Running command git checkout -b fix/do_not_compile_extension --track origin/fix/do_not_compile_extension
  Switched to a new branch 'fix/do_not_compile_extension'
  Branch 'fix/do_not_compile_extension' set up to track remote branch 'fix/do_not_compile_extension' from 'origin'.
  Resolved https://github.com/INM-6/elephant.git to commit 47a62e97a71f1cbcd12a54c9bb67cde20e2a31a0
  Preparing metadata (setup.py) ... done
Requirement already satisfied: neo>=0.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.10.2)
Requirement already satisfied: numpy>=1.18.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.23.1)
Requirement already satisfied: quantities>=0.12.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.13.0)
Requirement already satisfied: scipy>=1.5.4 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.8.1)
Requirement already satisfied: six>=1.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.16.0)
Requirement already satisfied: tqdm in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (4.64.0)
Building wheels for collected packages: elephant
  Building wheel for elephant (setup.py) ... done
  Created wheel for elephant: filename=elephant-0.12.0b1-cp39-cp39-linux_x86_64.whl size=511992 sha256=c7991819fdd9a75b90bc8eb3fd15c44de881e3e2fd8d17e492f332a2e55f968c
  Stored in directory: /tmp/pip-ephem-wheel-cache-xf7hejos/wheels/1f/3d/36/fb5e82c1bd6d7bd77d68df2aaa6aa211c5f2f28cf8c23531ad
Successfully built elephant
Installing collected packages: elephant
Successfully installed elephant-0.12.0b1

kohlerca · 2022-07-13T12:32:32Z

However, when using the `--install-option='--nofim', build is not attempted.

WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Collecting git+https://github.com/INM-6/elephant.git@fix/do_not_compile_extension
  Cloning https://github.com/INM-6/elephant.git (to revision fix/do_not_compile_extension) to /tmp/pip-req-build-0tk4j5lt
  Running command git clone --filter=blob:none --quiet https://github.com/INM-6/elephant.git /tmp/pip-req-build-0tk4j5lt
  Running command git checkout -b fix/do_not_compile_extension --track origin/fix/do_not_compile_extension
  Switched to a new branch 'fix/do_not_compile_extension'
  Branch 'fix/do_not_compile_extension' set up to track remote branch 'fix/do_not_compile_extension' from 'origin'.
  Resolved https://github.com/INM-6/elephant.git to commit 47a62e97a71f1cbcd12a54c9bb67cde20e2a31a0
  Preparing metadata (setup.py) ... done
Requirement already satisfied: neo>=0.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.10.2)
Requirement already satisfied: numpy>=1.18.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.23.1)
Requirement already satisfied: quantities>=0.12.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.13.0)
Requirement already satisfied: scipy>=1.5.4 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.8.1)
Requirement already satisfied: six>=1.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.16.0)
Requirement already satisfied: tqdm in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (4.64.0)
Skipping wheel build for elephant, due to binaries being disabled for it.
Installing collected packages: elephant
  Running setup.py install for elephant ... done
Successfully installed elephant-0.12.0b1

kohlerca · 2022-07-13T12:42:17Z

It works for the develop option (pip install -e . --install-option='--nofim') from the repo cloned locally.

Trying to use the optional by just passing pip install -e . fails:

  Preparing metadata (setup.py) ... done
Requirement already satisfied: neo>=0.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.10.2)
Requirement already satisfied: numpy>=1.18.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.23.1)
Requirement already satisfied: quantities>=0.12.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.13.0)
Requirement already satisfied: scipy>=1.5.4 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.8.1)
Requirement already satisfied: six>=1.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.16.0)
Requirement already satisfied: tqdm in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (4.64.0)
Installing collected packages: elephant
  Running setup.py develop for elephant
    error: subprocess-exited-with-error
    
    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [33 lines of output]
        running develop
        /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        running egg_info
        writing elephant.egg-info/PKG-INFO
        writing dependency_links to elephant.egg-info/dependency_links.txt
        writing requirements to elephant.egg-info/requires.txt
        writing top-level names to elephant.egg-info/top_level.txt
        reading manifest file 'elephant.egg-info/SOURCES.txt'
        reading manifest template 'MANIFEST.in'
        no previously-included directories found matching 'doc/_build'
        no previously-included directories found matching 'doc/tutorials/.ipynb_checkpoints'
        no previously-included directories found matching 'doc/reference/toctree'
        warning: no files found matching 'doc/reference/toctree/kernels/*'
        warning: no previously-included files matching '*.h5' found under directory '*'
        warning: no previously-included files matching '*~' found under directory '*'
        adding license file 'LICENSE.txt'
        writing manifest file 'elephant.egg-info/SOURCES.txt'
        running build_ext
        building 'elephant.spade_src.fim' extension
        gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -fPIC -Ielephant/spade_src/include -I/p/home/jusers/koehler4/jureca/envs/elephant-test/include -I/p/software/jurecadc/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/include/python3.9 -c elephant/spade_src/src/fim.cpp -o build/temp.linux-x86_64-cpython-39/elephant/spade_src/src/fim.o -DMODULE_NAME=fim -DUSE_OPENMP -DWITH_SIG_TERM -Dfim_EXPORTS -O3 -pedantic -Wextra -Weffc++ -Wunused-result -Werror -fopenmp -std=gnu++17
        In file included from elephant/spade_src/src/fim.cpp:41:
        elephant/spade_src/include/FPGrowth.h: In constructor ‘FPGrowth::FPGrowth(Transactions&, Support, uint32_t, uint32_t, ItemC, uint32_t, uint32_t, int32_t)’:
        elephant/spade_src/include/FPGrowth.h:198:37: error: loop variable ‘p’ of type ‘const RefPair&’ {aka ‘const std::pair<unsigned int, std::shared_ptr<FrequencyRef> >&’} binds to a temporary constructed from type ‘std::pair<const unsigned int, std::shared_ptr<FrequencyRef> >’ [-Werror=range-loop-construct]
          198 |                 for (const RefPair& p : F)
              |                                     ^
        elephant/spade_src/include/FPGrowth.h:198:37: note: use non-reference type ‘const RefPair’ {aka ‘const std::pair<unsigned int, std::shared_ptr<FrequencyRef> >’} to make the copy explicit or ‘const std::pair<const unsigned int, std::shared_ptr<FrequencyRef> >&’ to prevent copying
        cc1plus: all warnings being treated as errors
        warning: build_ext: building extension "elephant.spade_src.fim" failed: command '/p/software/jurecadc/stages/2022/software/GCCcore/11.2.0/bin/gcc' failed with exit code 1
        
        error: can't copy 'build/lib.linux-x86_64-cpython-39/elephant/spade_src/fim.cpython-39-x86_64-linux-gnu.so': doesn't exist or not a regular file
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [33 lines of output]
    running develop
    /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running egg_info
    writing elephant.egg-info/PKG-INFO
    writing dependency_links to elephant.egg-info/dependency_links.txt
    writing requirements to elephant.egg-info/requires.txt
    writing top-level names to elephant.egg-info/top_level.txt
    reading manifest file 'elephant.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    no previously-included directories found matching 'doc/_build'
    no previously-included directories found matching 'doc/tutorials/.ipynb_checkpoints'
    no previously-included directories found matching 'doc/reference/toctree'
    warning: no files found matching 'doc/reference/toctree/kernels/*'
    warning: no previously-included files matching '*.h5' found under directory '*'
    warning: no previously-included files matching '*~' found under directory '*'
    adding license file 'LICENSE.txt'
    writing manifest file 'elephant.egg-info/SOURCES.txt'
    running build_ext
    building 'elephant.spade_src.fim' extension
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -fPIC -Ielephant/spade_src/include -I/p/home/jusers/koehler4/jureca/envs/elephant-test/include -I/p/software/jurecadc/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/include/python3.9 -c elephant/spade_src/src/fim.cpp -o build/temp.linux-x86_64-cpython-39/elephant/spade_src/src/fim.o -DMODULE_NAME=fim -DUSE_OPENMP -DWITH_SIG_TERM -Dfim_EXPORTS -O3 -pedantic -Wextra -Weffc++ -Wunused-result -Werror -fopenmp -std=gnu++17
    In file included from elephant/spade_src/src/fim.cpp:41:
    elephant/spade_src/include/FPGrowth.h: In constructor ‘FPGrowth::FPGrowth(Transactions&, Support, uint32_t, uint32_t, ItemC, uint32_t, uint32_t, int32_t)’:
    elephant/spade_src/include/FPGrowth.h:198:37: error: loop variable ‘p’ of type ‘const RefPair&’ {aka ‘const std::pair<unsigned int, std::shared_ptr<FrequencyRef> >&’} binds to a temporary constructed from type ‘std::pair<const unsigned int, std::shared_ptr<FrequencyRef> >’ [-Werror=range-loop-construct]
      198 |                 for (const RefPair& p : F)
          |                                     ^
    elephant/spade_src/include/FPGrowth.h:198:37: note: use non-reference type ‘const RefPair’ {aka ‘const std::pair<unsigned int, std::shared_ptr<FrequencyRef> >’} to make the copy explicit or ‘const std::pair<const unsigned int, std::shared_ptr<FrequencyRef> >&’ to prevent copying
    cc1plus: all warnings being treated as errors
    warning: build_ext: building extension "elephant.spade_src.fim" failed: command '/p/software/jurecadc/stages/2022/software/GCCcore/11.2.0/bin/gcc' failed with exit code 1
    
    error: can't copy 'build/lib.linux-x86_64-cpython-39/elephant/spade_src/fim.cpython-39-x86_64-linux-gnu.so': doesn't exist or not a regular file
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

kohlerca · 2022-07-13T12:44:08Z

So, unless the develop install failure when not passing the --nofim flag is expected, this is the only current problem. Apart from that, as raised in the comments below, the naming of the flag could be revised before merge.

doc/install.rst

setup.py

…an be individually turned off

pep8speaks · 2022-07-22T12:30:05Z

Hello @Moritz-Alexander-Kern! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-08-09 12:51:10 UTC

…mentation page notes.

coveralls · 2022-08-09T12:55:59Z

Coverage increased (+0.009%) to 87.21% when pulling 1ab415b on INM-6:fix/do_not_compile_extension into 646cbd4 on NeuralEnsemble:master.

mdenker

I can confirm these new install options work for me, and also I find the new naming scheme for the options very good and future-proof.

mdenker · 2022-10-13T14:04:46Z

The problem highlighted by @kohlerca still persists: Development mode install fails without any options specified if the c++ file does not compile well. This happened to me with various versions of setuptools, but with different errors. This tells me it's a problem on the setuptools side. For regular installs, the module is skipped if there are errors -- this is probably the most important point. Thus, I think merging this is fine as is.

Moritz-Alexander-Kern added 2 commits June 10, 2022 15:03

add extra no_cpp to avoid compiling of fim.cpp

f2776ae

added description for no_cpp to installation documentation

581f655

Moritz-Alexander-Kern changed the title ~~Fix/do not compile extension~~ Fix/do not compile c-extension Jun 10, 2022

Moritz-Alexander-Kern changed the title ~~Fix/do not compile c-extension~~ Fix/add option: do not compile fim c-extension Jun 24, 2022

Moritz-Alexander-Kern closed this Jun 24, 2022

add new command "nofim" to setup.py

47a62e9

Moritz-Alexander-Kern reopened this Jul 6, 2022

Moritz-Alexander-Kern requested review from mdenker, kohlerca and rjurkus July 11, 2022 10:35

kohlerca reviewed Jul 13, 2022

View reviewed changes

doc/install.rst Outdated Show resolved Hide resolved

setup.py Outdated Show resolved Hide resolved

setup.py Show resolved Hide resolved

Moritz-Alexander-Kern added this to the v0.11.2 milestone Jul 18, 2022

add general --no-compile option, prepared for more extensions which c…

a1f0c57

…an be individually turned off

Moritz-Alexander-Kern added 2 commits July 22, 2022 14:35

fix pep8

517d545

add link for the install option to not compile also in the SPADE docu…

52af139

…mentation page notes.

Moritz-Alexander-Kern requested a review from kohlerca July 25, 2022 10:08

Moritz-Alexander-Kern added 3 commits July 25, 2022 16:19

add documentation for for 'no-compile' option

e829f3e

add documentation for for 'no-compile-spade' option

2633292

update reference to spade from CAD

55e5f8b

Moritz-Alexander-Kern mentioned this pull request Jul 26, 2022

[Bug] Pip install fails on linux due to fim issue #506

Closed

Moritz-Alexander-Kern linked an issue Jul 26, 2022 that may be closed by this pull request

[Bug] Pip install fails on linux due to fim issue #506

Closed

Moritz-Alexander-Kern added 2 commits August 9, 2022 14:49

Merge branch 'master' into fix/do_not_compile_extension

ab703e6

merge master, fix CI

1ab415b

mdenker approved these changes Oct 13, 2022

View reviewed changes

mdenker merged commit b384a4f into NeuralEnsemble:master Oct 13, 2022

Moritz-Alexander-Kern deleted the fix/do_not_compile_extension branch October 28, 2022 07:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix/add option: do not compile fim c-extension #494

Fix/add option: do not compile fim c-extension #494

Moritz-Alexander-Kern commented Jun 10, 2022 •

edited

Loading

kohlerca commented Jul 13, 2022

kohlerca commented Jul 13, 2022

kohlerca commented Jul 13, 2022

kohlerca commented Jul 13, 2022 •

edited

Loading

pep8speaks commented Jul 22, 2022 •

edited

Loading

coveralls commented Aug 9, 2022 •

edited

Loading

mdenker left a comment

mdenker commented Oct 13, 2022

Fix/add option: do not compile fim c-extension #494

Fix/add option: do not compile fim c-extension #494

Conversation

Moritz-Alexander-Kern commented Jun 10, 2022 • edited Loading

kohlerca commented Jul 13, 2022

kohlerca commented Jul 13, 2022

kohlerca commented Jul 13, 2022

kohlerca commented Jul 13, 2022 • edited Loading

pep8speaks commented Jul 22, 2022 • edited Loading

Comment last updated at 2022-08-09 12:51:10 UTC

coveralls commented Aug 9, 2022 • edited Loading

mdenker left a comment

Choose a reason for hiding this comment

mdenker commented Oct 13, 2022

Moritz-Alexander-Kern commented Jun 10, 2022 •

edited

Loading

kohlerca commented Jul 13, 2022 •

edited

Loading

pep8speaks commented Jul 22, 2022 •

edited

Loading

coveralls commented Aug 9, 2022 •

edited

Loading