-
Notifications
You must be signed in to change notification settings - Fork 24
Sequoia BG Q
Those MPI enabled versions of CloverLeaf enable the MPI via a "use mpi". Doing this from XL Fortran on Sequoia and building with the default compiler wrappers, will encounter this error:
XL Fortran90 error: Module symbol file for module mpi is in a format not recognized by this compiler===
The cause of the error is that the mpi.mod referenced in the "default" software stack (/bgsys/drivers/ppcfloor/comm/default) is built with GNU Fortran.
MPI built using the XL compilers instead of the GNU compilers, thus containing a Fortran mpi module (mpi.mod) generated by XLF is avaiable from a non-default location. This software stack was introduced in V1R4M1. You may build with it using the wrappers in comm/xl:
/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90
OpenMP Compilation Needs Thread Safe Versions of the XL Compilers
So to build CloverLeaf on Sequoia issue the following command:
make COMPILER=XL MPI_COMPILER=/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90_r C_MPI_COMPILER=mpixlc_r
Current compiler versions: /bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90_r -qversion
IBM XL Fortran for Blue Gene, V14.1 Version: 14.01.0000.0001
mpixlc_r -qversion
IBM XL C/C++ for Blue Gene, V12.1 Version: 12.01.0000.0001
SLURM on here - different scripts needed to uDawn
Results
| Processor | Clock | Notes | Version | Build | Test | Nodes | MPI Tasks/ Node | OpenMP Threads per MPI Task | Time | | :-------------- |:--------: | :---: | :---: | :----------: | :----------: | :------------: | :---: | :---: | :-----: | :-----: | ----: | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm_short | 1 | 1 | 64 | 6.43 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm_short | 1 | 4 | 16 | 6.44 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm_short | 1 | 64 | 1 | 6.72 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm_short | 1 | 16 | 4 | 6.46 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm_short | 1 | 16 | 1 | 10.27 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm | 8 | 64 | 1 | 37.76 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm | 8 | 16 | 4 | 36.24 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm | 8 | 1 | 64 | 47.69 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm | 16 | 16 | 2 | 28.85 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm | 32 | 16 | 1 | 25.79 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 8 | 64 | 1 | 37.74 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 8 | 16 | 4 | 34.38 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 8 | 1 | 64 | 35.60 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 16 | 16 | 2 | 28.34 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 32 | 16 | 1 | 25.83 |
Using experimental compilers, but without the LOMP runtime enabled:
i.e. make COMPILER=XL MPI_COMPILER=/usr/local/tools/compilers/ibm/mpixlf90_r-lompbeta1 C_MPI_COMPILER=/usr/local/tools/compilers/ibm/mpixlc_r-lompbeta1
However, as soon as I try and invoke the LOMP runtime with the -qdebug=lompinterface I end up wit a host of internal compiler errors in both the C and Fortran that relate to the IPA.
Example from the C (Fortran is identical except references /bgsys/xlcompilers/lompbeta1/xlf/bg/14.1/bin/.orig/bgxlf90_r ):
Calling signal handler...
Trap encountered in IPA:
SIGSEGV - Segmentation violation
Traceback:
Offset 0x0000006c StatementNode::AddReportMessage(ReportMessage*)
Offset 0x00001090 LOMPOutlineParDoRegion(unsigned int, ControlFlowGraph&, AliasInstance*, LiveVariables*, Procedure*, unsigned int, unsigned int&)
Offset 0x000003ec TransformLoopDirectives(LoopOptimizer&, Procedure*)
Offset 0x000015c8 DoOutlineOMPDirectives::doTransformationLOMP()
Offset 0x00000038 DoOutlineOMPDirectives::doTransformation()
Offset 0x00001fc8 IntraTransformation::transform()
Offset 0x00002258 BaseBlockOutliner::OutlineCodeSections(AliasSetOf&, ListOf&, ListOf&, ListOf&, BitVector*, HashTable<unsigned int, unsigned int>, HashTable<unsigned int, unsigned int>)
Offset 0x00002c74 BaseBlockOutliner::OutlineBlocks(BitVector&, unsigned char, unsigned int, BitVector*, HashTable<unsigned int, unsigned int>, HashTable<unsigned int, unsigned int>)
Offset 0x00000f30 DoOutlineOMPDirectives::doTransformationLOMP()
Offset 0x00000038 DoOutlineOMPDirectives::doTransformation()
Offset 0x00001fc8 IntraTransformation::transform()
Offset 0x00000a2c CompilationInstance::EarlyTransformations()
Offset 0x00000614 CompilationInstance::Optimize()
Offset 0x000008d4 InvokePass1(int, char**)
Offset 0x000004c4 main
Location 0x8011bdbcd4
Location 0x8011bdbecc
--- End of call chain ---
1586-494 (U) INTERNAL COMPILER ERROR: Signal 11.
/bgsys/xlcompilers/lompbeta1/vac/bg/12.1/bin/.orig/bgxlc_r: 1501-230 (S) Internal compiler error; please contact your Service Representative. For more information visit:
http://www.ibm.com/support/docview.wss?uid=swg21110810
After V1R2M0 driver update. OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES are now enabled on system by default.
First 5 lines are copy of results before update. Next five are re-runs following re-compile to check numbers.
Slight update in compilers too.
/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90_r -qversion IBM XL Fortran for Blue Gene, V14.1 Version: 14.01.0000.0002
mpixlc_r -qversion IBM XL C/C++ for Blue Gene, V12.1 Version: 12.01.0000.0002
| Processor | Clock | Notes | Version | Build | Test | Nodes | MPI Tasks/ Node | OpenMP Threads per MPI Task | Time | | :-------------- |:--------: | :---: | :---: | :----------: | :----------: | :------------: | :---: | :---: | :-----: | :-----: | ----: | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 8 | 64 | 1 | 37.74 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 8 | 16 | 4 | 34.38 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 8 | 1 | 64 | 35.60 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 16 | 16 | 2 | 28.34 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 32 | 16 | 1 | 25.83 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.2 XLC 12.1.0.2 V1R2M0 driver update | 0.9 | Ref/F | bm | 8 | 64 | 1 | ? | | Sequoia BG/Q | 1.6 | XLF 14.1.0.2 XLC 12.1.0.2 V1R2M0 driver update | 0.9 | Ref/F | bm | 8 | 16 | 4 | 33.62 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.2 XLC 12.1.0.2 V1R2M0 driver update | 0.9 | Ref/F | bm | 8 | 1 | 64 | ? | | Sequoia BG/Q | 1.6 | XLF 14.1.0.2 XLC 12.1.0.2 V1R2M0 driver update | 0.9 | Ref/F | bm | 16 | 16 | 2 | ? | | Sequoia BG/Q | 1.6 | XLF 14.1.0.2 XLC 12.1.0.2 V1R2M0 driver update | 0.9 | Ref/F | bm | 32 | 16 | 1 | ?|