-
Notifications
You must be signed in to change notification settings - Fork 24
Sequoia BG Q
Those MPI enabled versions of CloverLeaf enable the MPI via a "use mpi". Doing this from XL Fortran on Sequoia and building with the default compiler wrappers, will encounter this error:
XL Fortran90 error: Module symbol file for module mpi is in a format not recognized by this compiler===
The cause of the error is that the mpi.mod referenced in the "default" software stack (/bgsys/drivers/ppcfloor/comm/default) is built with GNU Fortran.
MPI built using the XL compilers instead of the GNU compilers, thus containing a Fortran mpi module (mpi.mod) generated by XLF is avaiable from a non-default location. This software stack was introduced in V1R4M1. You may build with it using the wrappers in comm/xl:
/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90
OpenMP Compilation Needs Thread Safe Versions of the XL Compilers
So to build CloverLeaf on Sequoia issue the following command:
make COMPILER=XL MPI_COMPILER=/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90_r C_MPI_COMPILER=mpixlc_r
Current compiler versions: /bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90_r -qversion
IBM XL Fortran for Blue Gene, V14.1 Version: 14.01.0000.0001
mpixlc_r -qversion
IBM XL C/C++ for Blue Gene, V12.1 Version: 12.01.0000.0001
SLURM on here - different scripts needed to uDawn
Results
bm_short
1_64 (default on 1 node)
lover Version 1.000 MPI Version OpenMP Version Task Count 1 Thread Count: 64
Output file clover.out opened. All output will go there.
Clover Version 1.000 MPI Version OpenMP Version Task Count 1 Thread Count: 64
Clover will run from the following input:-
*clover Step 87 time 0.4971244 control sound timestep 5.85E-03 1, 1 x 5.21E-03 y 5.21E-03
Time 0.502972356941437826 Volume Mass Density Pressure Internal Energy Kinetic Energy Total Energy step: 87 0.1000E+03 0.2800E+02 0.2800E+00 0.1671E+00 0.4179E+02 0.1193E+01 0.4298E+02
Calculation complete Clover is finishing Wall clock 6.43371701240539551
Clover Version 1.000 MPI Version OpenMP Version Task Count 4 Thread Count: 16
Output file clover.out opened. All output will go there.
Clover Version 1.000 MPI Version OpenMP Version Task Count 4 Thread Count: 16
Clover will run from the following input:-
*clover Step 87 time 0.4971244 control sound timestep 5.85E-03 1, 1 x 5.21E-03 y 5.21E-03
Time 0.502972356941437826 Volume Mass Density Pressure Internal Energy Kinetic Energy Total Energy step: 87 0.1000E+03 0.2800E+02 0.2800E+00 0.1671E+00 0.4179E+02 0.1193E+01 0.4298E+02
Calculation complete Clover is finishing Wall clock 6.43540716171264648 Done Fri Feb 1 07:34:20 PST 2013
Clover Version 1.000 MPI Version OpenMP Version Task Count 64 Thread Count: 1
Output file clover.out opened. All output will go there.
Clover Version 1.000 MPI Version OpenMP Version Task Count 64 Thread Count: 1
Clover will run from the following input:-
*clover Step 87 time 0.4971244 control sound timestep 5.85E-03 1, 1 x 5.21E-03 y 5.21E-03
Time 0.502972356941437826 Volume Mass Density Pressure Internal Energy Kinetic Energy Total Energy step: 87 0.1000E+03 0.2800E+02 0.2800E+00 0.1671E+00 0.4179E+02 0.1193E+01 0.4298E+02
Calculation complete Clover is finishing Wall clock 6.72289299964904785 Done
Clover Version 1.000 MPI Version OpenMP Version Task Count 16 Thread Count: 4
Output file clover.out opened. All output will go there.
Clover Version 1.000 MPI Version OpenMP Version Task Count 16 Thread Count: 4
Clover will run from the following input:-
*clover
Step 87 time 0.4971244 control sound timestep 5.85E-03 1, 1 x 5.21E-03 y 5.21E-03
Time 0.502972356941437715 Volume Mass Density Pressure Internal Energy Kinetic Energy Total Energy step: 87 0.1000E+03 0.2800E+02 0.2800E+00 0.1671E+00 0.4179E+02 0.1193E+01 0.4298E+02
Calculation complete Clover is finishing Wall clock 6.45956182479858398 Done Fri Feb 1 07:45:24 PST 2013
Clover Version 1.000 MPI Version OpenMP Version Task Count 16 Thread Count: 1
Output file clover.out opened. All output will go there. 1
Clover Version 1.000 MPI Version OpenMP Version Task Count 16 Thread Count: 1
Clover will run from the following input:-
*clover
Step 87 time 0.4971244 control sound timestep 5.85E-03 1, 1 x 5.21E-03 y 5.21E-03
Time 0.502972356941437826 Volume Mass Density Pressure Internal Energy Kinetic Energy Total Energy step: 87 0.1000E+03 0.2800E+02 0.2800E+00 0.1671E+00 0.4179E+02 0.1193E+01 0.4298E+02
Calculation complete Clover is finishing Wall clock 10.2737510204315186 Done Fri Feb 1 08:40:01 PST 2013
bm
script: 8_64_1.cmd
Clover Version 1.000 MPI Version OpenMP Version Task Count 512 Thread Count: 1
Output file clover.out opened. All output will go there.
Clover Version 1.000 MPI Version OpenMP Version Task Count 512 Thread Count: 1
Clover will run from the following input:-
*clover
Step 2952 time 15.4956432 control sound timestep 5.11E-03 1, 1 x 5.21E-03 y 5.21E-03
Time 15.5007538630877200 Volume Mass Density Pressure Internal Energy Kinetic Energy Total Energy step: 2952 0.1000E+03 0.2800E+02 0.2800E+00 0.1604E+00 0.4009E+02 0.2590E+01 0.4268E+02
Calculation complete Clover is finishing Wall clock 37.7645969390869141 Done Fri Feb 1 08:55:03 PST 2013
8_16_4.cmd Clover Version 1.000 MPI Version OpenMP Version Task Count 128 Thread Count: 4
Output file clover.out opened. All output will go there.
Clover Version 1.000 MPI Version OpenMP Version Task Count 128 Thread Count: 4
Clover will run from the following input:-
*clover
Step 2952 time 15.4963148 control sound timestep 4.98E-03 1, 1 x 5.21E-03 y 5.21E-03
Time 15.5012992170708994 Volume Mass Density Pressure Internal Energy Kinetic Energy Total Energy step: 2952 0.1000E+03 0.2800E+02 0.2800E+00 0.1604E+00 0.4009E+02 0.2590E+01 0.4268E+02
Calculation complete Clover is finishing Wall clock 36.2393031120300293 Done Fri Feb 1 08:57:09 PST 2013
8_1_64.cmd
Clover Version 1.000 MPI Version OpenMP Version Task Count 8 Thread Count: 64
Output file clover.out opened. All output will go there.
Clover Version 1.000 MPI Version OpenMP Version Task Count 8 Thread Count: 64
Clover will run from the following input:-
*clover
Step 2953 time 15.4956906 control sound timestep 5.20E-03 1, 1 x 5.21E-03 y 5.21E-03
Time 15.5008953166272097 Volume Mass Density Pressure Internal Energy Kinetic Energy Total Energy step: 2953 0.1000E+03 0.2800E+02 0.2800E+00 0.1604E+00 0.4009E+02 0.2590E+01 0.4268E+02
Calculation complete Clover is finishing Wall clock 47.6886451244354248 Done Fri Feb 1 09:00:41 PST 2013
32_16_1.cmd
Clover Version 1.000 MPI Version OpenMP Version Task Count 512 Thread Count: 1
Output file clover.out opened. All output will go there.
Clover Version 1.000 MPI Version OpenMP Version Task Count 512 Thread Count: 1
Clover will run from the following input:-
*clover Step 2952 time 15.4956432 control sound timestep 5.11E-03 1, 1 x 5.21E-03 y 5.21E-03
Time 15.5007538630877200 Volume Mass Density Pressure Internal Energy Kinetic Energy Total Energy step: 2952 0.1000E+03 0.2800E+02 0.2800E+00 0.1604E+00 0.4009E+02 0.2590E+01 0.4268E+02
Calculation complete Clover is finishing Wall clock 25.7877268791198730 Done Sat Feb 2 03:52:11 PST 2013
16_16_2 Clover Version 1.000 MPI Version OpenMP Version Task Count 256 Thread Count: 2
Output file clover.out opened. All output will go there.
Clover Version 1.000 MPI Version OpenMP Version Task Count 256 Thread Count: 2
Clover will run from the following input:-
*clover Step 2955 time 15.4932598 control sound timestep 5.17E-03 1, 1 x 5.21E-03 y 5.21E-03
Time 15.4984344001438927 Volume Mass Density Pressure Internal Energy Kinetic Energy Total Energy step: 2955 0.1000E+03 0.2800E+02 0.2800E+00 0.1604E+00 0.4009E+02 0.2590E+01 0.4268E+02
Calculation complete Clover is finishing Wall clock 28.8500239849090576 Done Sat Feb 2 04:01:01 PST 2013
| Processor | Clock | Notes | Version | Build | Test | Nodes | MPI Tasks/ Node | OpenMP Threads per MPI Task | Time | | :-------------- |:--------: | :---: | :---: | :----------: | :----------: | :------------: | :---: | :---: | :-----: | :-----: | ----: | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm_short | 1 | 1 | 64 | 6.43 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm_short | 1 | 4 | 16 | 6.44 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm_short | 1 | 64 | 1 | 6.72 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm_short | 1 | 16 | 4 | 6.46 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm_short | 1 | 16 | 1 | 10.27 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm | 8 | 64 | 1 | 37.76 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm | 8 | 16 | 4 | 36.24 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm | 8 | 1 | 64 | 47.69 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm | 16 | 16 | 2 | 28.85 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 | 0.9 | Ref/F | bm | 32 | 16 | 1 | 25.79 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 8 | 64 | 1 | 37.74 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 8 | 16 | 4 | 34.38 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 8 | 1 | 64 | 35.60 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 16 | 16 | 2 | 28.34 | | Sequoia BG/Q | 1.6 | XLF 14.1.0.1 XLC 12.1.0.1 OMP_WAIT_POLICY=ACTIVE BG_SMP_FAST_WAKEUP=YES | 0.9 | Ref/F | bm | 32 | 16 | 1 | 25.83 |