Different Optimization Results Between Single-Core and Multi-Core Parallel Calculations in Parallel Meep

[YosanChan]

Hello, I installed Parallel Meep, and when performing multi-core parallel computing using the command line [mpirun -np 4 python -m mpi4py mycode.py], I got different optimization results compared to running the same code in PyCharm for single-core computation. I would like to know if this is due to different optimization algorithms being used or if the logic of the program execution differs between the two? I designed a 3D splitter for average power splitting, based on this case (#2852). In the single-core calculation, the optimized results show transmission rates of 0.494/0.495 for the two ports. However, in the parallel calculation, the final optimized transmission rates for the two ports are 0.35/0.64. I’m not very familiar with the mpi4py execution logic, and I would appreciate any insights or suggestions you can provide. @smartalecH @oskooi

[smartalecH]

You're leaving out a lot of detail here (like what optimization algorithms you're using, what's inside of mycode.py, etc).

But speaking abstractly: if everything about your setup is deterministic, you should get the exact same results regardless of the number of cores you use. mpi4py/mpi/distributed computing won't affect this. Similarly, your IDE shouldn't affect things either.

If you have a random initial condition, and you aren't seeding your initial condition the same way on all processes, then they will be out of sync, and you'll get garbage.

If you are using a stochastic optimization algorithm, and you aren't seeding things the same on all processes, then likewise, you'll get garbage.

Similarly, if you are explicitly only calling certain routines on a single process (eg with mp.am_master()) then you could also get garbage.

[YosanChan]

Thank you for your response. I set up a uniform initial structure (rather than a random one), and the optimization algorithm used is a deterministic algorithm (LD_MMA). I did not explicitly call routines like mp.am_master(). Additionally, whether running on a single core or multiple cores, I get almost the same optimized structure after repeating the optimization (so it seems that this is not an issue caused by randomness). The attachment contains the main code (spl3D_clean.txt). I am relatively new to Meep, so there may be oversights or misunderstandings, and I would appreciate it if you could kindly review it.

spl3D_clean.txt

[YosanChan]

And based on the MEEP documentation, the strategy employed by parallel Meep for optimization problems is that all processes run the same optimization algorithm. Does this mean that during the forward simulation and adjoint simulation, the matrices are split and computed across different cores/processes/threads, and then each core/process/thread simultaneously collects and uses the computed results for gradient calculation and variable updating? If my understanding is correct, could the issue arise from different cores/processes/threads updating different variables, causing conflicts? (However, for deterministic algorithms, this doesn’t seem theoretically possible). If we switch to the strategy of having a single master process for optimization with the other processes synchronized, could this potentially lead to parallel computing results that are consistent with a single-core structure? Or are there any related examples that could help me better understand this issue? Thanks again!