Thu Jan 25 08:48:31 CET 2024 Currently Loaded Modules: 1) GCCcore/11.3.0 2) zlib/1.2.12-GCCcore-11.3.0 3) binutils/2.38-GCCcore-11.3.0 4) numactl/2.0.14-GCCcore-11.3.0 5) CUDA/11.7.0 6) NVHPC/22.7-CUDA-11.7.0 7) XZ/5.2.5-GCCcore-11.3.0 8) libxml2/2.9.13-GCCcore-11.3.0 9) libpciaccess/0.16-GCCcore-11.3.0 10) hwloc/2.7.1-GCCcore-11.3.0 11) OpenSSL/1.1 12) libevent/2.1.12-GCCcore-11.3.0 13) UCX/1.12.1-GCCcore-11.3.0 14) GDRCopy/2.3-GCCcore-11.3.0 15) UCX-CUDA/1.12.1-GCCcore-11.3.0-CUDA-11.7.0 16) libfabric/1.15.1-GCCcore-11.3.0 17) PMIx/4.1.2-GCCcore-11.3.0 18) UCC/1.0.0-GCCcore-11.3.0 19) NCCL/2.12.12-GCCcore-11.3.0-CUDA-11.7.0 20) UCC-CUDA/1.0.0-GCCcore-11.3.0-CUDA-11.7.0 21) OpenMPI/4.1.4-NVHPC-22.7-CUDA-11.7.0 22) nvompi/2022.07 23) Szip/2.1.1-GCCcore-11.3.0 24) HDF5/1.14.0-nvompi-2022.07 25) bzip2/1.0.8-GCCcore-11.3.0 26) ncurses/6.3-GCCcore-11.3.0 27) libreadline/8.1.2-GCCcore-11.3.0 28) Tcl/8.6.12-GCCcore-11.3.0 29) SQLite/3.38.3-GCCcore-11.3.0 30) GMP/6.2.1-GCCcore-11.3.0 31) libffi/3.4.2-GCCcore-11.3.0 32) Python/3.10.4-GCCcore-11.3.0 _ _ ___ _ | | _ \ \ Version : 5.0-58-gdeb419f-master / __| _ __ (_) | | ___ (_) | | \__ \ | ' \ _ | | / -_) _ | | |___/ |_|_|_| |_| |_| \___| |_| | | /_/ Reading the simulation parameters ------------------------------------------------------------------------------- HDF5 version 1.14.0 Python version 3.10.4 Parsing pyinit.py Parsing 5.0-58-gdeb419f-master Parsing pyprofiles.py Parsing input.py Parsing pycontrol.py Check for function preprocess() python preprocess function does not exist Calling python _smilei_check Calling python _prepare_checkpoint_dir Calling python _keep_python_running() : CAREFUL: Patches distribution: hilbertian  Smilei will run on GPU devices Geometry: 2Dcartesian ------------------------------------------------------------------------------- Interpolation order : 2 Maxwell solver : Yee simulation duration = 372.000000, total number of iterations = 3000 timestep = 0.124000 = 0.993936 x CFL, time resolution = 8.064516 Grid length: 320, 640 Cell length: 0.125, 2, 0 Number of cells: 2560, 320 Spatial resolution: 8, 0.5 Electromagnetic boundary conditions ------------------------------------------------------------------------------- xmin silver-muller, absorbing vector [1, 0] xmax silver-muller, absorbing vector [-1, -0] ymin silver-muller, absorbing vector [0, 1] ymax silver-muller, absorbing vector [-0, -1] Vectorization: ------------------------------------------------------------------------------- Mode: off Calling python writeInfo Initializing MPI ------------------------------------------------------------------------------- MPI_THREAD_MULTIPLE enabled Number of MPI processes: 1 OpenMP disabled OpenMP task parallelization not activated Number of patches: 4 x 2 Number of cells in one patch: 640 x 160 Dynamic load balancing: never Initializing the restart environment ------------------------------------------------------------------------------- Initializing species ------------------------------------------------------------------------------- Creating Species #0: electron > Pusher: boris > Boundary conditions: remove remove remove remove > Density profile: 2D built-in profile `trapezoidal` (value: 0.003000, xvacuum: 0.000000, yvacuum: 0.000000, xplateau: 1000000000.000000, yplateau: 640.000000, xslope1: 1000.000000, xslope2: 0.000000, yslope1: 0.000000, yslope2: 0.000000) Initializing laser parameters ------------------------------------------------------------------------------- Laser #0: separable profile omega : 1 chirp_profile : 1D built-in profile `tconstant` (0.000000) time envelope : 1D built-in profile `tgaussian` (start: 0.000000, duration: 372.000000, sigma: 577.078016, center: 120.000000, order: 2.000000) space envelope (y) : 1D user-defined function space envelope (z) : 1D user-defined function phase (y) : 1D user-defined function phase (z) : 1D user-defined function delay phase (y) : 0 delay phase (z) : 0 Initializing Patches ------------------------------------------------------------------------------- First patch created Approximately 10% of patches created Approximately 20% of patches created Approximately 30% of patches created Approximately 40% of patches created Approximately 50% of patches created Approximately 60% of patches created Approximately 70% of patches created All patches created Creating Diagnostics, antennas, and external fields ------------------------------------------------------------------------------- Created ParticleBinning #0: species electron Axis moving_x from 0 to 320 in 300 steps Axis ekin from 1 to 400 in 100 steps Probe diagnostic #0 2560 points origin : 0, 320 corner 0 : 320, 320 Probe diagnostic #1 2560x320 points (total = 819200) origin : 0, 0 corner 0 : 320, 0 corner 1 : 0, 640 Finalize MPI environment ------------------------------------------------------------------------------- Done creating diagnostics, antennas, and external fields Minimum memory consumption (does not include all temporary buffers) ------------------------------------------------------------------------------- Particles: Master 351 MB; Max 351 MB; Global 0.343 GB Fields: Master 84 MB; Max 84 MB; Global 0.0825 GB : Master 0 MB; Max 0 MB; Global 0 GB ParticleBinning0.h5: Master 0 MB; Max 0 MB; Global 0.000224 GB Probes0.h5: Master 0 MB; Max 0 MB; Global 0.000214 GB Probes1.h5: Master 70 MB; Max 70 MB; Global 0.0684 GB Initial fields setup ------------------------------------------------------------------------------- Applying external fields at time t = 0 Applying prescribed fields at time t = 0 Applying antennas at time t = 0 GPU allocation and copy of the fields and particles ------------------------------------------------------------------------------- Open files & initialize diagnostics ------------------------------------------------------------------------------- Running diags at time t = 0 ------------------------------------------------------------------------------- line 13: cudaLaunchKernel returned status 98: invalid device function [acn65:95349:0:95349] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x10) ==== backtrace (tid: 95349) ==== 0 0x0000000000039a70 __cudaPopCallConfiguration() ???:0 1 0x000000000003d1c4 __cudaPopCallConfiguration() ???:0 2 0x00000000000237bc __cudaPopCallConfiguration() ???:0 3 0x00000000000253ab __cudaPopCallConfiguration() ???:0 4 0x00000000000260ca __pgi_uacc_cuda_unregister_fat_binary() /proj/build/22A/Linux_x86_64/rte/accel-uni/build/Linux_x86_64/../../src/cuda_init.c:661 5 0x0000000000026092 __pgi_uacc_cuda_unregister_fat_binaries() /proj/build/22A/Linux_x86_64/rte/accel-uni/build/Linux_x86_64/../../src/cuda_init.c:651 6 0x0000000000039ce9 __run_exit_handlers() :0 7 0x0000000000039d37 __GI_exit() :0 8 0x0000000000045750 __pgi_uacc_exit() /proj/build/22A/Linux_x86_64/rte/accel-uni/build/Linux_x86_64/../../src/init.c:777 9 0x000000000001f451 __pgi_cuda_error_handler() /proj/build/22A/Linux_x86_64/rte/accel-uni/build/Linux_x86_64/../../src/cuda_launch.c:71 10 0x000000000002140a __pgi_uacc_cuda_launchk3() /proj/build/22A/Linux_x86_64/rte/accel-uni/build/Linux_x86_64/../../src/cuda_launch.c:559 11 0x000000000002262e __pgi_uacc_cuda_launch2() /proj/build/22A/Linux_x86_64/rte/accel-uni/build/Linux_x86_64/../../src/cuda_launche.c:157 12 0x00000000000227af __pgi_uacc_cuda_launch() /proj/build/22A/Linux_x86_64/rte/accel-uni/build/Linux_x86_64/../../src/cuda_launche.c:190 13 0x0000000000046ac3 __pgi_uacc_launch() /proj/build/22A/Linux_x86_64/rte/accel-uni/build/Linux_x86_64/../../src/launch.c:55 14 0x000000000076f401 Field::put_to() /scratch/project/open-28-44/smilei_GPU/Smilei/src/Field/Field.cpp:13 15 0x00000000005651da ElectroMagn::restartRhoJ() /scratch/project/open-28-44/smilei_GPU/Smilei/src/ElectroMagn/ElectroMagn.cpp:404 16 0x0000000000ad8094 VectorPatch::runAllDiags() /scratch/project/open-28-44/smilei_GPU/Smilei/src/Patch/VectorPatch.cpp:1506 17 0x0000000000ceb221 main() /scratch/project/open-28-44/smilei_GPU/Smilei/src/Smilei.cpp:429 18 0x0000000000022555 __libc_start_main() ???:0 19 0x000000000049c82f _start() ???:0 ================================= Stack trace (most recent call last): #20 Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in #19 Object "/scratch/project/open-28-44/smilei_GPU/Smilei/build/smilei", at 0x49c82e, in #18 Object "/lib64/libc.so.6", at 0x2ac4ebeb2554, in __libc_start_main #17 Object "/scratch/project/open-28-44/smilei_GPU/Smilei/build/smilei", at 0xceb220, in main #16 Object "/scratch/project/open-28-44/smilei_GPU/Smilei/build/smilei", at 0xad8093, in VectorPatch::runAllDiags(Params&, SmileiMPI*, unsigned int, Timers&, SimWindow*) #15 Object "/scratch/project/open-28-44/smilei_GPU/Smilei/build/smilei", at 0x5651d9, in ElectroMagn::restartRhoJ() #14 Object "/scratch/project/open-28-44/smilei_GPU/Smilei/build/smilei", at 0x76f400, in Field::put_to(double) #13 Object "/apps/all/NVHPC/22.7-CUDA-11.7.0/Linux_x86_64/22.7/compilers/lib/libacchost.so", at 0x2ac4e9d63ac2, in __pgi_uacc_launch #12 Object "/apps/all/NVHPC/22.7-CUDA-11.7.0/Linux_x86_64/22.7/compilers/lib/libacccuda.so", at 0x2ac4e85ec7ae, in __pgi_uacc_cuda_launch #11 Object "/apps/all/NVHPC/22.7-CUDA-11.7.0/Linux_x86_64/22.7/compilers/lib/libacccuda.so", at 0x2ac4e85ec62d, in #10 Object "/apps/all/NVHPC/22.7-CUDA-11.7.0/Linux_x86_64/22.7/compilers/lib/libacccuda.so", at 0x2ac4e85eb409, in __pgi_uacc_cuda_launchk3 #9 Object "/apps/all/NVHPC/22.7-CUDA-11.7.0/Linux_x86_64/22.7/compilers/lib/libacccuda.so", at 0x2ac4e85e9450, in #8 Object "/apps/all/NVHPC/22.7-CUDA-11.7.0/Linux_x86_64/22.7/compilers/lib/libacchost.so", at 0x2ac4e9d6274f, in __pgi_uacc_exit #7 Object "/lib64/libc.so.6", at 0x2ac4ebec9d36, in exit #6 Object "/lib64/libc.so.6", at 0x2ac4ebec9ce8, in #5 Object "/apps/all/NVHPC/22.7-CUDA-11.7.0/Linux_x86_64/22.7/compilers/lib/libacccuda.so", at 0x2ac4e85f0091, in #4 Object "/apps/all/NVHPC/22.7-CUDA-11.7.0/Linux_x86_64/22.7/compilers/lib/libacccuda.so", at 0x2ac4e85f00c9, in #3 Object "/apps/all/CUDA/11.7.0/lib/libcudart.so.11.0", at 0x2ac4e27503aa, in #2 Object "/apps/all/CUDA/11.7.0/lib/libcudart.so.11.0", at 0x2ac4e274e7bb, in #1 Object "/apps/all/CUDA/11.7.0/lib/libcudart.so.11.0", at 0x2ac4e27681c3, in #0 Object "/apps/all/CUDA/11.7.0/lib/libcudart.so.11.0", at 0x2ac4e2764a70, in Segmentation fault (Signal sent by tkill() [0x1e7300017475]) watf? exit -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[41340,1],0] Exit code: 1 -------------------------------------------------------------------------- Done