Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MacOS Mojave, Python 3.8: "AttributeError: Can't pickle local object 'fork.<locals>.child_process'" #14102

Closed
AndrewGaspar opened this issue Dec 10, 2019 · 25 comments · Fixed by #18205 or #18124
Closed

MacOS Mojave, Python 3.8: "AttributeError: Can't pickle local object 'fork.<locals>.child_process'" #14102

AndrewGaspar opened this issue Dec 10, 2019 · 25 comments · Fixed by #18205 or #18124
Assignees
@scheibelp
Labels
bug Something isn't working impact-high macOS triage The issue needs to be prioritized

Comments

@AndrewGaspar
Copy link
Contributor

AndrewGaspar commented Dec 10, 2019 •
edited
Loading

I tried to install a package on MacOS Mojave using Python 3.8 installed from MacPorts. I got an error related to "pickle"ing an object.

The issue does not reproduce with Python 3.7. It reproduces on the latest commit of develop.

Steps to reproduce the issue

$ sudo port select --set python3 python38
$ spack install <package>

Error Message

$ spack -d install googletest%clang
==> [2019-12-10-16:40:53.367136] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/modules.yaml
==> [2019-12-10-16:40:53.375856] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/darwin/modules.yaml
==> [2019-12-10-16:40:53.378937] Reading config file /Users/agaspar/.spack/darwin/modules.yaml
==> [2019-12-10-16:40:53.382033] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/config.yaml
==> [2019-12-10-16:40:53.402445] Imported install from built-in commands
==> [2019-12-10-16:40:53.403393] Imported install from built-in commands
==> [2019-12-10-16:40:53.423052] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/repos.yaml
==> [2019-12-10-16:40:53.426926] Reading config file /Users/agaspar/.spack/darwin/repos.yaml
==> [2019-12-10-16:40:53.590667] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/packages.yaml
==> [2019-12-10-16:40:53.602863] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/darwin/packages.yaml
==> [2019-12-10-16:40:53.609408] Reading config file /Users/agaspar/.spack/packages.yaml
==> [2019-12-10-16:40:53.632981] WRITE LOCK: /Users/agaspar/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:53.633665] WRITE LOCK: /Users/agaspar/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.187669] WRITE LOCK: /Users/agaspar/.spack/cache/providers/.builtin-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.191428] WRITE LOCK: /Users/agaspar/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.192032] WRITE LOCK: /Users/agaspar/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.320890] WRITE LOCK: /Users/agaspar/.spack/cache/tags/.builtin-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.325895] WRITE LOCK: /Users/agaspar/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.326356] WRITE LOCK: /Users/agaspar/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.913941] WRITE LOCK: /Users/agaspar/.spack/cache/patches/.builtin-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.920255] READ LOCK: /Users/agaspar/.spack/cache/providers/.eap-cpp-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.920748] READ LOCK: /Users/agaspar/.spack/cache/providers/.eap-cpp-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.921247] READ LOCK: /Users/agaspar/.spack/cache/providers/.eap-cpp-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.921705] READ LOCK: /Users/agaspar/.spack/cache/tags/.eap-cpp-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.921978] READ LOCK: /Users/agaspar/.spack/cache/tags/.eap-cpp-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.922417] READ LOCK: /Users/agaspar/.spack/cache/tags/.eap-cpp-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.922960] READ LOCK: /Users/agaspar/.spack/cache/patches/.eap-cpp-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.923237] READ LOCK: /Users/agaspar/.spack/cache/patches/.eap-cpp-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.931963] READ LOCK: /Users/agaspar/.spack/cache/patches/.eap-cpp-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.933605] READ LOCK: /Users/agaspar/.spack/cache/providers/.agaspar-spack-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.934507] READ LOCK: /Users/agaspar/.spack/cache/providers/.agaspar-spack-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.935006] READ LOCK: /Users/agaspar/.spack/cache/providers/.agaspar-spack-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.935556] READ LOCK: /Users/agaspar/.spack/cache/tags/.agaspar-spack-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.935896] READ LOCK: /Users/agaspar/.spack/cache/tags/.agaspar-spack-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.936364] READ LOCK: /Users/agaspar/.spack/cache/tags/.agaspar-spack-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.936862] READ LOCK: /Users/agaspar/.spack/cache/patches/.agaspar-spack-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.937520] READ LOCK: /Users/agaspar/.spack/cache/patches/.agaspar-spack-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.937955] READ LOCK: /Users/agaspar/.spack/cache/patches/.agaspar-spack-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.938516] READ LOCK: /Users/agaspar/.spack/cache/providers/.sds-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.938796] READ LOCK: /Users/agaspar/.spack/cache/providers/.sds-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.939228] READ LOCK: /Users/agaspar/.spack/cache/providers/.sds-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.939772] READ LOCK: /Users/agaspar/.spack/cache/tags/.sds-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.940071] READ LOCK: /Users/agaspar/.spack/cache/tags/.sds-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.940524] READ LOCK: /Users/agaspar/.spack/cache/tags/.sds-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.941433] READ LOCK: /Users/agaspar/.spack/cache/patches/.sds-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.941899] READ LOCK: /Users/agaspar/.spack/cache/patches/.sds-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.942576] READ LOCK: /Users/agaspar/.spack/cache/patches/.sds-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.947954] Reading config file /Users/agaspar/.spack/darwin/compilers.yaml
==> [2019-12-10-16:41:00.015384] Warning: clang@11.0.0-apple cannot build optimized binaries for "skylake". Using best target possible: "x86_64"
==> [2019-12-10-16:41:00.066119] DATABASE LOCK TIMEOUT: 120s
==> [2019-12-10-16:41:00.066246] PACKAGE LOCK TIMEOUT: No timeout
==> [2019-12-10-16:41:00.069298] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.070192] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.116237] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.118529] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.119082] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.165906] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.167462] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.167934] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.218965] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.219271] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.219655] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.266983] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.267310] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.267728] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.477182] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.484131] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquiring]
==> [2019-12-10-16:41:00.485073] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquired]
==> [2019-12-10-16:41:00.485185] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.485539] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.538750] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.539044] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.539420] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.585088] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.588744] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Released]
==> [2019-12-10-16:41:00.598045] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquiring]
==> [2019-12-10-16:41:00.598586] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquired]
==> [2019-12-10-16:41:00.598719] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.599106] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.645810] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.646311] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Released]
==> [2019-12-10-16:41:00.646450] Installing googletest dependencies
==> [2019-12-10-16:41:00.646715] cmake@3.15.1 : externally installed in /opt/local
==> [2019-12-10-16:41:00.646790] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.647079] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.693385] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.693685] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.694108] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.740314] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.740884] cmake@3.15.1 : generating module file
==> [2019-12-10-16:41:00.740948] SKIP: shebang filtering [external package]
==> [2019-12-10-16:41:00.741077] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.741426] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.787969] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.788322] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.788725] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.834796] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.842742]        WRITE: cmake@3.15.1%clang@11.0.0-apple~doc+ncurses+openssl+ownlibs~qt arch=darwin-mojave-x86_64/6u2g2jh [/Users/agaspar/Code/spack/share/spack/modules/darwin-mojave-x86_64/cmake-3.15.1-clang-11.0.0-apple-6u2g2jh]
==> [2019-12-10-16:41:00.864711] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.865189] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.933812] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:01.114793] cmake@3.15.1 : registering into DB
==> [2019-12-10-16:41:01.114944] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:01.115530] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:01.164461] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:01.214115] Bootstrapping clang@11.0.0-apple compiler for googletest
==> [2019-12-10-16:41:01.214736] Installing googletest
==> [2019-12-10-16:41:01.214778] Searching for binary cache of googletest
==> [2019-12-10-16:41:01.214992] No Spack mirrors are currently configured
==> [2019-12-10-16:41:01.215405] No binary for googletest found: installing from source
==> [2019-12-10-16:41:01.215535] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:01.215997] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:01.262544] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:01.280533] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:01.282336] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:01.533037] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
Traceback (most recent call last):
  File "/Users/agaspar/Code/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/Users/agaspar/Code/spack/lib/spack/spack/main.py", line 715, in main
    return _invoke_command(command, parser, args, unknown)
  File "/Users/agaspar/Code/spack/lib/spack/spack/main.py", line 457, in _invoke_command
    return_val = command(parser, args)
  File "/Users/agaspar/Code/spack/lib/spack/spack/cmd/install.py", line 331, in install
    install_spec(args, kwargs, abstract, concrete)
  File "/Users/agaspar/Code/spack/lib/spack/spack/cmd/install.py", line 209, in install_spec
    spec.package.do_install(**kwargs)
  File "/Users/agaspar/Code/spack/lib/spack/spack/package.py", line 1761, in do_install
    PackageBase._verbose = spack.build_environment.fork(
  File "/Users/agaspar/Code/spack/lib/spack/spack/build_environment.py", line 860, in fork
    p.start()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'fork.<locals>.child_process'

Information on your system

This includes:

  1. MacOS Mojave
  2. Python 3.8 from MacPorts
@AndrewGaspar AndrewGaspar added the bug Something isn't working label Dec 10, 2019
@adamjstewart
Copy link
Member

@scheibelp can you take a look at this? It looks like our unit tests are missing a Python 3.8 bug.

@scheibelp scheibelp self-assigned this Dec 11, 2019
@scheibelp

This comment has been minimized.

Sign in to view
@scheibelp scheibelp mentioned this issue Dec 16, 2019
Closed
1 task
@scheibelp
Copy link
Member

I can replicate this using a Spack-installed Python 3.8.0 on Mac OS. I think the problem is that the implementation of Spack install wasn't following the requirements for pickling: in particular, there are locally-defined that are passed to multiprocessing.Process.

I can fix the initial error with the changes at #14187 but that does not fully resolve this issue so I will need to work on it more over the next day or two.

See also:

@scheibelp
Copy link
Member

I will need to work on it more over the next day or two.

Update: I've resolved some additional issues but more remain. I will be continuing with this in January.

@germasch
Copy link
Contributor

germasch commented Mar 9, 2020

I figured I'll just add a (not very helpful) "I'm having this problem, too" comment here, with current master and also python 3.8.

@DiegoMagdaleno
Copy link
Contributor

This bug is sadly still a thing :/

@adamjstewart
Copy link
Member

@scheibelp we should bump this up in priority. Spack, MacPorts, and Anaconda all offer Python 3.8 installations. From what I can tell, the only reason Anaconda doesn't default to Python 3.8 is that a few packages still don't support it (TensorFlow being the big one), so as soon as that changes, I think the default version of Python any Anaconda user will have will be Python 3.8. I was also planning on switching Spack's default Python version to 3.8 once Anaconda makes the switch. Homebrew will likely follow.

@DiegoMagdaleno
Copy link
Contributor

@adamjstewart I am taking a look at it myself, I am going to try to fix it, btw homebrew already switched.

@adamjstewart
Copy link
Member

btw homebrew already switched.

Did they? This formula still uses Python 3.7.

@DiegoMagdaleno
Copy link
Contributor

Oh perhaps I got a package that made it build from "head" also: #16314

I updated some of the old patches by @scheibelp, however other issues occur after this.

@DiegoMagdaleno
Copy link
Contributor

This is about to get bigger, since macOS Big Sur by Apple now ships with Python 3.8 by default, and most brew formulae has updated to Python 3.8 Space is still broken for me :/

@adamjstewart
Copy link
Member

Oddly enough, I've been using Spack on Ubuntu 20.04 (via WSL) which comes with Python 3.8.2 but I haven't encountered a single problem yet. I already have 160+ packages successfully installed.

@scheibelp
Copy link
Member

Oddly enough, I've been using Spack on Ubuntu 20.04 (via WSL) which comes with Python 3.8.2 but I haven't encountered a single problem yet. I already have 160+ packages successfully installed.

Spack should work fine with Python 3.8 in Linux, but not on Mac OS, because Python 3.8 multiprocessing behavior diverged for Mac OS and Linux.

@jrood-nrel
Copy link
Member

I'm starting to experience this issue as well.

@sethrj
Copy link
Contributor

sethrj commented Jul 25, 2020

Just hit this with a fresh Catalina install using homebrew. Worked around by removing /usr/local/bin from my PATH:

export PATH=/usr/bin:/bin:/usr/sbin:/sbin:$PATH

@DiegoMagdaleno
Copy link
Contributor

@sethrj This solution doesn't seem ideal, if I remove /usr/local/bin from my path, then I can't access my Homebrew tools

@sethrj
Copy link
Contributor

sethrj commented Jul 25, 2020

@sethrj This solution doesn't seem ideal, if I remove /usr/local/bin from my path, then I can't access my Homebrew tools

Sure, but you only need to do it temporarily in the shell in which you’re calling spack install. I’m not suggesting doing that permanently.

@adamqc
Copy link
Contributor

adamqc commented Jul 28, 2020

@DiegoMagdaleno You can also use brew unlink python to remove python from /usr/local/bin.

@ax3l
Copy link
Member

ax3l commented Jul 29, 2020 •
edited
Loading

Interesting, did brew roll out Python 3.8 in the last days or so (likely)? The problem occurs for me now as well in nightly macOS integration tests on Azure Pipelines (issue started July 22th).

alalazo added a commit to alalazo/spack that referenced this issue Jul 31, 2020
@alalazo
MacOS nightly builds: use Python 3.7 in CI
911c755 
Nightly builds with MacOS started failing again
due to an upgrade of the default virtual environment
that now uses Python 3.8

This makes us hit spack#14102 and every build fails. This
commit should be reverted along with the fix to spack#14102.
@alalazo alalazo mentioned this issue Jul 31, 2020
Closed
@tgamblin tgamblin mentioned this issue Aug 1, 2020
Merged
tgamblin pushed a commit that referenced this issue Aug 1, 2020
@alalazo @tgamblin
MacOS nightly builds: use Python 3.7 in CI
1f000a6 
Nightly builds with MacOS started failing again
due to an upgrade of the default virtual environment
that now uses Python 3.8

This makes us hit #14102 and every build fails. This
commit should be reverted along with the fix to #14102.
tgamblin pushed a commit that referenced this issue Aug 1, 2020
@alalazo @tgamblin
MacOS nightly builds: use Python 3.7 in CI
00aa67b 
Nightly builds with MacOS started failing again
due to an upgrade of the default virtual environment
that now uses Python 3.8

This makes us hit #14102 and every build fails. This
commit should be reverted along with the fix to #14102.
tgamblin pushed a commit that referenced this issue Aug 2, 2020
@alalazo @tgamblin
MacOS nightly builds: use Python 3.7 in CI
ba3f91b 
Nightly builds with MacOS started failing again
due to an upgrade of the default virtual environment
that now uses Python 3.8

This makes us hit #14102 and every build fails. This
commit should be reverted along with the fix to #14102.
scheibelp pushed a commit that referenced this issue Aug 4, 2020
@alalazo @scheibelp
MacOS nightly builds: use Python 3.7 in CI
b1133fa 
Nightly builds with MacOS started failing again
due to an upgrade of the default virtual environment
that now uses Python 3.8

This makes us hit #14102 and every build fails. This
commit should be reverted along with the fix to #14102.
@zhiyuanzhai
Copy link
Contributor

zhiyuanzhai commented Aug 11, 2020 •
edited
Loading

Actually, the problem disappears with conda environment activated.
The python version in my environment is still 3.7.6 now. I haven't upgraded my environment yet.
I will have my conda environment activated when using spack before the problem is finally fixed.

One more thing:
Notice that the command python still refers to python 2.7.16 in the latest version of Catalina, while python3 is the one that refers to python 3.8.

@r-xue r-xue mentioned this issue Aug 17, 2020
Merged
@r-xue
Copy link
Contributor

r-xue commented Aug 17, 2020

@scheibelp add a PR #18124

I am an inexperienced user, switching from MacPorts. Hope the change doesn't introduce unintended consequences.

@psakievich
Copy link
Contributor

FWIW Using pyenv is a good way to circumvent this issue until a long term fix is in place. Personally, I prefer that to unlinking the brew installation.

@sethrj
Copy link
Contributor

sethrj commented Aug 27, 2020

The sooner this gets resolved, the sooner I can start recommending spack to mac people again...

@ax3l ax3l added the triage The issue needs to be prioritized label Aug 27, 2020
@scheibelp scheibelp mentioned this issue Aug 27, 2020
Merged
@joshessman-llnl joshessman-llnl mentioned this issue Sep 8, 2020
Closed
tgamblin pushed a commit that referenced this issue Nov 12, 2020
@scheibelp
macos: update build process to use spawn instead of fork (#18205)
bb42470 
Spack creates a separate process to do package installation. Different
operating systems and Python versions use different methods to create
it but up until Python 3.8 both Linux and Mac OS used "fork" (which
duplicates process memory, file descriptor table, etc.).

Python >= 3.8 on Mac OS prefers creating an entirely new process
(referred to as the "spawn" start method) because "fork" was found to
cause issues (in other words "spawn" is the default start method used
by multiprocessing.Process). Spack was dependent on the particular
behavior of fork to replicate process memory and transmit file
descriptors.

This PR refactors the Spack internals to support starting a child
process with the "spawn" method. To achieve this, it makes the
following changes:

- ensure that the package repository and other global state are
  transmitted to the child process
- ensure that file descriptors are transmitted to the child process in
  a way that works with multiprocessing and spawn
- make all the state needed for the build process and tests picklable
  (package, stage, etc.)
- move a number of locally-defined functions into global scope so that
  they can be pickled
- rework tests where needed to avoid using local functions

This PR also reworks sbang tests to work on macOS, where temporary
directories are deeper than the Linux sbang limit. We make the limit
platform-dependent (macOS supports 512-character shebangs)

See: #14102
@scheibelp scheibelp mentioned this issue Jan 11, 2021
Closed
@GoneAwry GoneAwry mentioned this issue May 25, 2021
Closed
@RafalSkolasinski RafalSkolasinski mentioned this issue Nov 1, 2021
Closed
@mpkouznetsov mpkouznetsov mentioned this issue Jan 6, 2022
Merged
3 tasks
Kami pushed a commit to scalyr/scalyr-agent-2 that referenced this issue Jan 20, 2022
@tomaz-s1
Fix copy manager functionality so it works under Python >= 3.8 on OS X.
5f7b060 
In Python 3.8, local objects can't be pickled anymore which means we
need to move that class definition to a top module level scope so it can
be pickled by the multiprocess functionality.

See https://gist.github.com/Kami/8386c79d2db7c95329e6c182ec639f49 and
spack/spack#14102 for more details.
@Kami Kami mentioned this issue Jan 20, 2022
Merged
Kami added a commit to scalyr/scalyr-agent-2 that referenced this issue Jan 20, 2022
@Kami
Fix copy manager functionality so it works under Python >= 3.8 on OS X (
1d9aa23 
#818)

* Fix copy manager functionality so it works under Python >= 3.8 on OS X.

In Python 3.8, local objects can't be pickled anymore which means we
need to move that class definition to a top module level scope so it can
be pickled by the multiprocess functionality.

See https://gist.github.com/Kami/8386c79d2db7c95329e6c182ec639f49 and
spack/spack#14102 for more details.

* Use Python 3.8 for osx unit tests on CI.

* Don't run multiprocess copying manager tests on OSX since it doesn't
work there under Python 3.8.

* Add unit and smoke tests for python 3.8

Co-authored-by: arthur <arthurk@sentinelone.com>
@Kami Kami mentioned this issue Mar 13, 2022
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@scheibelp scheibelp

Labels
bug Something isn't working impact-high macOS triage The issue needs to be prioritized
Projects
None yet
Milestone
No milestone