crash on OS X 10.8.3 when using DYNAMIC_ARCH=1 USE_THREAD=1 #221

ViralBShah · 2013-05-21T02:09:42Z

OpenBLAS is crashing when built for the julia distribution on the mac, when used in multi-threaded mode. The relevant flags are:

DYNAMIC_ARCH=1 NO_AFFINITY=1 INTERFACE64=1 BINARY=64 USE_THREAD=1

I am using OS X 10.8.3, clang 4.2, and gfortran 4.8.0 from brew. Single threaded works fine, but multi-threaded crashes. I believe this crash is related to DYNAMIC_ARCH=1, since that flag is only used when building openblas for the julia binary distribution on OS X, and is otherwise not used.

The text was updated successfully, but these errors were encountered:

xianyi · 2013-05-21T14:56:05Z

Hi @ViralBShah ,

Is it failed at building stage or running stage?

Xianyi

ViralBShah · 2013-05-22T17:33:03Z

I get a segfault when trying to multiply two matrices - at runtime.

xianyi · 2013-05-30T07:29:39Z

Hi @ViralBShah ,

I just tried gcc & gfortran 4.7 with DYNAMIC_ARCH=1 NO_AFFINITY=1 INTERFACE64=1 BINARY=64 USE_THREAD=1 on OS X 10.8.3. The dgemm works fine.

Could you try it from http://xianyi.github.io/OpenBLAS/download/0.2.6/openblas_0.2.6_osx.tar.gz ?

I will try clang.

Xianyi

ViralBShah · 2013-06-06T05:25:26Z

Hi @xianyi

Is this the same as the released 0.2.6 openblas, or something you have patched?

ViralBShah · 2013-06-06T05:29:39Z

May be similar issue as #225

ViralBShah · 2013-06-06T05:43:26Z

Also, I do not observe the error running multi-threaded dgemm if I am using INTERFACE64=0 BINARY=64.

Sorry for these reports that are not very specific.

ViralBShah · 2013-06-06T08:37:27Z

I deleted everything and started from scratch, and this no longer crashes.

ViralBShah · 2013-06-06T08:58:48Z

Oops - this still crashes. Turns out I had added OPENBLAS_NUM_THREADS=1 in my .bashrc.

OpenMathLib/OpenBLAS#221

xianyi · 2013-06-06T14:09:00Z

Hi @ViralBShah ,

Please try the tarball. It is built on latest develop branch.

Xianyi

ViralBShah · 2013-06-07T03:14:57Z

Will do @xianyi. I also see the crash on 32-bit, now that I unset my environment variable for num threads.

ViralBShah · 2013-06-07T17:49:44Z

I built the develop branch for now, and the crash persists. I will try the tarball tomorrow.

ViralBShah · 2013-06-12T19:25:01Z

I am unable to try your provided openblas due to lack of LAPACK. Could you build it with LAPACK as well? There is a problem with libgfortran as well, since I am on 4.8.1, but I figure that I can just use the 4.8.1 files here.

ViralBShah · 2013-06-12T19:29:12Z

If you build with gcc -static-libgcc -Wl,-Bstatic -lstdc++ -lgfortran -lquadmath -Wl,-Bdynamic -lm, then all the gfortran dependencies can also be statically included in libopenblas.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46539

ViralBShah · 2013-06-12T19:58:28Z

This is the openblas build output (on develop branch). Perhaps on mac, the max number of threads can be kept at something like 16.

 OpenBLAS build complete.

  OS               ... Darwin             
  Architecture     ... x86_64               
  BINARY           ... 64bit                 
  Use 64 bits int    (equivalent to "-i8" in Fortran)      
  C compiler       ... CLANG  (command line : clang -mmacosx-version-min=10.6)
  Fortran compiler ... GFORTRAN  (command line : gfortran)
-n   Library Name     ... libopenblasp-r0.2.6.a
 (Multi threaded; Max num-threads is 256)

xianyi · 2013-06-13T14:49:03Z

Hi @ViralBShah ,

Could you try static link OpenBLAS library, e.g. libopenblas.a ?

Xianyi

ViralBShah · 2013-06-14T07:03:44Z

Good idea. I compiled it into a dynamic library with
gfortran -fPIC -shared -Wl,-all_load libopenblasp-r0.2.6.a -o libopenblasp-r0.2.6.dylib

This works fine and there are no crashes.

xianyi · 2013-06-14T07:18:30Z

Interesting. Could you double check the result? Is it multi-threaded and DYNAMIC_ARCH=1?

ViralBShah · 2013-06-14T07:58:39Z

I am using your provided library, which is built with DYNAMIC_ARCH and multi-threading. More so, I checked that I am using multiple cores. Julia has a peakflops() command, which does a matrix multiply and computes the flop rate, and I do get the expected results with 2 cores.

The only difference between your build and mine seems to be that you are using gfortran 4.7 and I am using 4.8.

xianyi · 2013-06-14T08:24:32Z

Could you try to convert your compiled static library as following, too?

gfortran -fPIC -shared -Wl,-all_load libopenblasp-r0.2.6.a -o libopenblasp-r0.2.6.dylib

I want to make sure whether this is a bug about generating dylib.

Xianyi

ViralBShah · 2013-06-14T09:36:06Z

Trying that on the develop branch, it fails as before. It seems that the dylib generation is fine.

ViralBShah · 2013-06-17T10:46:34Z

This is the stacktrace

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00007fff5f3f1800
0x00000001049511e4 in gemm_driver ()
(gdb) bt
#0  0x00000001049511e4 in gemm_driver ()
#1  0x00000001049511ca in dgemm_thread_nn ()
#2  0x000000010481ef73 in dgemm_ ()

Is openmp more stable / higher performance than the pthreads version?

xianyi · 2013-06-17T14:17:45Z

@ViralBShah , Could you try to enable DEBUG=1?

ViralBShah · 2013-07-01T06:12:35Z

This is what I get inside julia:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00007fff5f3f1038
0x0000000105278160 in gemm_driver () at common_x86_64.h:145
145 common_x86_64.h: No such file or directory.
    in common_x86_64.h
(gdb) bt
#0  0x0000000105278160 in gemm_driver () at common_x86_64.h:145
#1  0x0000000105278096 in dgemm_thread_nn (args=0x7fff5fbfe278, range_m=0x0, range_n=0x0, sa=0x10fe0a000, sb=0x10ff0a000, mypos=0) at level3_thread.c:705
#2  0x0000000105031bce in dgemm_ (TRANSA=0x7fff5fbfe460 "N?_?", TRANSB=0x7fff5fbfe450 "N/?", M=0x7fff5fbfe440, N=0x7fff5fbfe430, K=0x7fff5fbfe420, alpha=0x7fff5fbfe410, a=0x10c100000, ldA=0x7fff5fbfe400, b=0x10c100000, ldB=0x7fff5fbfe3f0, beta=0x7fff5fbfe3e0, c=0x10df85000, ldC=0x7fff5fbfe3d0) at gemm.c:430

ViralBShah · 2013-07-01T06:16:11Z

These are the compilers I am using:

viral-laptop 11:45:47 {master} ~/julia$ clang -v
Apple LLVM version 4.2 (clang-425.0.28) (based on LLVM 3.2svn)
Target: x86_64-apple-darwin12.4.0
Thread model: posix

viral-laptop 11:45:49 {master} ~/julia$ gfortran -v
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/local/Cellar/gfortran/4.8.1/gfortran/libexec/gcc/x86_64-apple-darwin12.3.0/4.8.1/lto-wrapper
Target: x86_64-apple-darwin12.3.0
Configured with: ../configure --prefix=/usr/local/Cellar/gfortran/4.8.1/gfortran --datarootdir=/usr/local/Cellar/gfortran/4.8.1/share --bindir=/usr/local/Cellar/gfortran/4.8.1/bin --enable-languages=fortran --with-system-zlib --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-cloog=/usr/local/opt/cloog --with-isl=/usr/local/opt/isl --enable-checking=release --disable-stage1-checking --disable-build-poststage1-with-cxx --disable-libstdcxx-pc --disable-nls
Thread model: posix
gcc version 4.8.1 (GCC)

ViralBShah · 2013-07-01T06:35:53Z

I built again this time by removing the -mmacosx-version-min=10.6 option to clang, and also setting the max number of threads to 128 and I do not get the crash.

ViralBShah · 2013-07-01T06:51:19Z

I finally narrowed it down. Compiling with 128 threads max instead of 256 fixed this for me on mac. Perhaps there is some kind of memory corruption when using a large number of threads - which for some reason only shows up when using DYNMAIC_ARCH.

xianyi · 2013-07-01T07:19:45Z

Thank you for the investigation.

I will debug this error.

2013/7/1 Viral B. Shah notifications@github.com

I finally narrowed it down. Compiling with 128 threads max instead of 256
fixed this for me on mac. Perhaps there is some kind of memory corruption
when using a large number of threads?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/221#issuecomment-20265773
.

…S X with DYNAMIC_ARCH=1 & NUM_THREADS=256.

xianyi · 2013-07-02T06:19:17Z

Hi @ViralBShah ,

I think I fixed this bug on develop branch. Could you try it?

Xianyi

When NUM_THREADS(MAX_CPU_NUNBERS) is very large ,e.g. 256. typedef struct { volatile BLASLONG working[MAX_CPU_NUMBER][CACHE_LINE_SIZE * DIVIDE_RATE]; } job_t; job_t job[MAX_CPU_NUMBER]; The job array is equal 8MB. Thus, We use malloc instead of stack allocation.

xianyi · 2013-07-07T18:14:30Z

@ViralBShah , I fixed this bug in last commit, which didn't change the stack limit on Mac OS X.

I used a smaller threshold since the stack size is 1MB on windows.

ViralBShah closed this as completed Jun 6, 2013

ViralBShah reopened this Jun 6, 2013

ViralBShah added a commit to JuliaLang/julia that referenced this issue Jun 6, 2013

Still need OPENBLAS_NUM_THREADS=1. This issue tracks this bug:

0e411ad

OpenMathLib/OpenBLAS#221

ViralBShah mentioned this issue Jun 12, 2013

OpenBLAS crash when using OPENBLAS_DYNAMIC_ARCH=1 JuliaLang/julia#3369

Closed

xianyi added a commit that referenced this issue Jul 2, 2013

Refs #221. Set stack limit to 16MB to prevent a SEGFAULT bug on Mac O…

32dbeb6

…S X with DYNAMIC_ARCH=1 & NUM_THREADS=256.

xianyi added a commit that referenced this issue Jul 10, 2013

Refs #214, #221, #246. Fixed the getrf overflow bug on Windows.

32d2ca3

I used a smaller threshold since the stack size is 1MB on windows.

xianyi closed this as completed Jul 11, 2013

ViralBShah mentioned this issue Jul 24, 2013

Multithreaded version of OpenBLAS causes intermittent segfaults when starting Julia #229

Closed

martin-frbg mentioned this issue Sep 19, 2020

Segfault with large NUM_THREADS #2839

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crash on OS X 10.8.3 when using DYNAMIC_ARCH=1 USE_THREAD=1 #221

crash on OS X 10.8.3 when using DYNAMIC_ARCH=1 USE_THREAD=1 #221

ViralBShah commented May 21, 2013

xianyi commented May 21, 2013

ViralBShah commented May 22, 2013

xianyi commented May 30, 2013

ViralBShah commented Jun 6, 2013

ViralBShah commented Jun 6, 2013

ViralBShah commented Jun 6, 2013

ViralBShah commented Jun 6, 2013

ViralBShah commented Jun 6, 2013

xianyi commented Jun 6, 2013

ViralBShah commented Jun 7, 2013

ViralBShah commented Jun 7, 2013

ViralBShah commented Jun 12, 2013

ViralBShah commented Jun 12, 2013

ViralBShah commented Jun 12, 2013

xianyi commented Jun 13, 2013

ViralBShah commented Jun 14, 2013

xianyi commented Jun 14, 2013

ViralBShah commented Jun 14, 2013

xianyi commented Jun 14, 2013

ViralBShah commented Jun 14, 2013

ViralBShah commented Jun 17, 2013

xianyi commented Jun 17, 2013

ViralBShah commented Jul 1, 2013

ViralBShah commented Jul 1, 2013

ViralBShah commented Jul 1, 2013

ViralBShah commented Jul 1, 2013

xianyi commented Jul 1, 2013

xianyi commented Jul 2, 2013

xianyi commented Jul 7, 2013

crash on OS X 10.8.3 when using DYNAMIC_ARCH=1 USE_THREAD=1 #221

crash on OS X 10.8.3 when using DYNAMIC_ARCH=1 USE_THREAD=1 #221

Comments

ViralBShah commented May 21, 2013

xianyi commented May 21, 2013

ViralBShah commented May 22, 2013

xianyi commented May 30, 2013

ViralBShah commented Jun 6, 2013

ViralBShah commented Jun 6, 2013

ViralBShah commented Jun 6, 2013

ViralBShah commented Jun 6, 2013

ViralBShah commented Jun 6, 2013

xianyi commented Jun 6, 2013

ViralBShah commented Jun 7, 2013

ViralBShah commented Jun 7, 2013

ViralBShah commented Jun 12, 2013

ViralBShah commented Jun 12, 2013

ViralBShah commented Jun 12, 2013

xianyi commented Jun 13, 2013

ViralBShah commented Jun 14, 2013

xianyi commented Jun 14, 2013

ViralBShah commented Jun 14, 2013

xianyi commented Jun 14, 2013

ViralBShah commented Jun 14, 2013

ViralBShah commented Jun 17, 2013

xianyi commented Jun 17, 2013

ViralBShah commented Jul 1, 2013

ViralBShah commented Jul 1, 2013

ViralBShah commented Jul 1, 2013

ViralBShah commented Jul 1, 2013

xianyi commented Jul 1, 2013

xianyi commented Jul 2, 2013

xianyi commented Jul 7, 2013