Failed in plotting

[xiazhuozhao]

After installing the latest version of JAX, I used the pip install -e . command to install PESnet. The network runs successfully, but it seems there's no correct output for the plotting. I'm unable to pinpoint where the issue is. Could you help me, thanks!

2023-12-09 00:02:45 (INFO): Running command 'func'
2023-12-09 00:02:45 (INFO): Started
2023-12-09 00:02:46 (INFO): Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA
2023-12-09 00:02:46 (INFO): Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
2023-12-09 00:02:50.408321: W external/xla/xla/service/gpu/nvptx_compiler.cc:698] The NVIDIA driver's CUDA version is 12.0 which is older than the ptxas CUDA version (12.3.103). Because the driver is older than the ptxas version, XLA is disabling parallel compilation, which may slow down compilation. You should update your NVIDIA driver or use the NVIDIA-provided CUDA forward compatibility packages.  
2023-12-09 00:03:02 (INFO): creating RunStatusReporter for e9feee6d91754a878e2f5a31
2023-12-09 00:03:02 (INFO): starting from: {}
2023-12-09 00:03:02 (INFO): starting writer thread for <aim.sdk.reporter.RunStatusReporter object at 0x14714c03ca30>
log_dir: /home/shhgroup/shanghui/logs/pesnet/H2__09-12-23_00:02:50:760536
name: H2
aim_hash: e9feee6d91754a878e2f5a31
Initialization
CG precision: float32
converged SCF energy = -0.957081335258677
converged SCF energy = -1.06810203295004
converged SCF energy = -1.11904988746119
converged SCF energy = -1.12485012825092
converged SCF energy = -1.11685689709792
converged SCF energy = -1.09217978938209
converged SCF energy = -1.08209327618322
converged SCF energy = -1.04169183529242
converged SCF energy = -1.02684230384466
converged SCF energy = -0.993078569432703
converged SCF energy = -0.949031242350657
converged SCF energy = -0.924239239786162
converged SCF energy = -0.886451415542872
converged SCF energy = -0.86337080411078
converged SCF energy = -0.84716056352074
converged SCF energy = -0.834458913313593
MO initial loss: 0.011822241796392522; final loss: 0.011822241796392522
MO initial loss: 0.00943993208689221; final loss: 0.00943993208689221
MO initial loss: 0.006633517984115942; final loss: 0.006633517984115942
MO initial loss: 0.004138748404110831; final loss: 0.004138748404110831
MO initial loss: 0.00290062683801973; final loss: 0.00290062683801973
MO initial loss: 0.0012568684013349455; final loss: 0.0012568684013349455
MO initial loss: 0.0008599610176328052; final loss: 0.0008599610176328052
MO initial loss: 5.554254636501985e-05; final loss: 5.554254636501985e-05
MO initial loss: 0.00026591531105711684; final loss: 0.00026591531105711684
MO initial loss: 0.0013454877996926162; final loss: 0.0013454877996926162
MO initial loss: 0.0022861422131632654; final loss: 0.0022861422131632654
MO initial loss: 0.00414431091105927; final loss: 0.00414431091105927
MO initial loss: 0.0055136435914820685; final loss: 0.0055136435914820685
MO initial loss: 0.006573633496650197; final loss: 0.006573633496650197
MO initial loss: 0.007457328172754619; final loss: 0.007457328172754619
Pretraining
100%|██████████| 10000/10000 [04:16<00:00, 38.92it/s, MSE=2.2454947e-07, pmove=0.53344727]
Thermalizing
Training
  0%|          | 0/60000 [00:00<?, ?it/s]2023-12-09 00:09:57 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".   
  0%|          | 0/60000 [01:17<?, ?it/s, E=-.993, E_std=0.585, E_var=0.369, pmove=0.5372742][0] creating checkpoint
  0%|          | 100/60000 [03:10<16:11:07,  1.03it/s, E=-1.1, E_std=0.0255, E_var=0.000687, pmove=0.5363648]2023-12-09 00:11:52 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  0%|          | 200/60000 [04:50<16:16:43,  1.02it/s, E=-1.1, E_std=0.0181, E_var=0.00041, pmove=0.5311035]2023-12-09 00:13:31 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  0%|          | 300/60000 [06:29<16:11:30,  1.02it/s, E=-1.1, E_std=0.0158, E_var=0.000281, pmove=0.5317566]2023-12-09 00:15:11 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  1%|          | 400/60000 [08:09<16:12:11,  1.02it/s, E=-1.1, E_std=0.0173, E_var=0.000426, pmove=0.5332947]2023-12-09 00:16:50 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  1%|          | 500/60000 [09:48<16:09:12,  1.02it/s, E=-1.1, E_std=0.0277, E_var=0.00425, pmove=0.5339844]2023-12-09 00:18:29 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  1%|          | 600/60000 [11:27<16:09:12,  1.02it/s, E=-1.1, E_std=0.0102, E_var=0.000126, pmove=0.5315796]2023-12-09 00:20:09 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  1%|          | 700/60000 [13:12<15:54:04,  1.04it/s, E=-1.1, E_std=0.0191, E_var=0.00117, pmove=0.5337219]2023-12-09 00:21:53 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  1%|▏         | 800/60000 [14:51<16:02:14,  1.03it/s, E=-1.1, E_std=0.0129, E_var=0.000386, pmove=0.532489]2023-12-09 00:23:32 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  2%|▏         | 900/60000 [16:30<16:02:25,  1.02it/s, E=-1.1, E_std=0.0111, E_var=0.000151, pmove=0.5359986]2023-12-09 00:25:11 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  2%|▏         | 1000/60000 [18:08<16:00:20,  1.02it/s, E=-1.11, E_std=0.00919, E_var=0.000111, pmove=0.533606]2023-12-09 00:26:50 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  3%|▎         | 2000/60000 [32:34<4:10:41,  3.86it/s, E=-1.11, E_std=0.00491, E_var=2.67e-5, pmove=0.53570557]2023-12-09 00:41:14 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  5%|▌         | 3000/60000 [35:46<2:49:56,  5.59it/s, E=-1.1, E_std=0.00601, E_var=4.02e-5, pmove=0.53441167]2023-12-09 00:44:27 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  7%|▋         | 4000/60000 [38:48<2:47:37,  5.57it/s, E=-1.1, E_std=0.00541, E_var=3.36e-5, pmove=0.53153074]2023-12-09 00:47:28 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  8%|▊         | 5000/60000 [41:49<2:45:09,  5.55it/s, E=-1.1, E_std=0.00635, E_var=4.95e-5, pmove=0.5348511]2023-12-09 00:50:30 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
  8%|▊         | 5000/60000 [41:50<2:45:09,  5.55it/s, E=-1.11, E_std=0.00665, E_var=5.37e-5, pmove=0.5355103][5000] creating checkpoint
 10%|█         | 6000/60000 [44:52<2:42:03,  5.55it/s, E=-1.1, E_std=0.00576, E_var=3.7e-5, pmove=0.5304138]2023-12-09 00:53:33 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 12%|█▏        | 7000/60000 [47:57<2:38:48,  5.56it/s, E=-1.1, E_std=0.00658, E_var=7.58e-5, pmove=0.5318177]2023-12-09 00:56:37 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 13%|█▎        | 8000/60000 [51:03<2:36:34,  5.54it/s, E=-1.1, E_std=0.00487, E_var=2.86e-5, pmove=0.5316162]2023-12-09 00:59:44 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 15%|█▌        | 9000/60000 [54:07<2:32:31,  5.57it/s, E=-1.11, E_std=0.00504, E_var=4.03e-5, pmove=0.5337403]2023-12-09 01:02:48 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 17%|█▋        | 10000/60000 [57:14<2:30:25,  5.54it/s, E=-1.1, E_std=0.00567, E_var=4.15e-5, pmove=0.5334778]2023-12-09 01:05:54 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 17%|█▋        | 10000/60000 [57:14<2:30:25,  5.54it/s, E=-1.11, E_std=0.00633, E_var=8.39e-5, pmove=0.5336121][10000] creating checkpoint
 18%|█▊        | 11000/60000 [1:00:20<2:26:04,  5.59it/s, E=-1.11, E_std=0.00466, E_var=2.65e-5, pmove=0.5334228]2023-12-09 01:09:01 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 20%|██        | 12000/60000 [1:03:29<2:23:55,  5.56it/s, E=-1.1, E_std=0.00591, E_var=6.19e-5, pmove=0.5322876]2023-12-09 01:12:09 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 22%|██▏       | 13000/60000 [1:06:36<2:20:52,  5.56it/s, E=-1.09, E_std=0.0039, E_var=1.75e-5, pmove=0.53063357]2023-12-09 01:15:17 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 23%|██▎       | 14000/60000 [1:09:41<2:16:56,  5.60it/s, E=-1.1, E_std=0.00453, E_var=3.38e-5, pmove=0.53336185]2023-12-09 01:18:22 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 25%|██▌       | 15000/60000 [1:12:47<2:14:16,  5.59it/s, E=-1.1, E_std=0.00351, E_var=1.34e-5, pmove=0.5331665]2023-12-09 01:21:27 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 25%|██▌       | 15000/60000 [1:12:47<2:14:16,  5.59it/s, E=-1.1, E_std=0.00417, E_var=2.14e-5, pmove=0.53515625][15000] creating checkpoint
 27%|██▋       | 16000/60000 [1:15:57<2:12:11,  5.55it/s, E=-1.1, E_std=0.00386, E_var=1.8e-5, pmove=0.5310974]2023-12-09 01:24:37 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 28%|██▊       | 17000/60000 [1:18:58<2:09:28,  5.54it/s, E=-1.11, E_std=0.00328, E_var=1.14e-5, pmove=0.53376466]2023-12-09 01:27:39 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 30%|███       | 18000/60000 [1:22:03<2:05:13,  5.59it/s, E=-1.11, E_std=0.00448, E_var=3.08e-5, pmove=0.5347473]2023-12-09 01:30:44 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 32%|███▏      | 19000/60000 [1:25:06<2:02:21,  5.58it/s, E=-1.1, E_std=0.00396, E_var=2.06e-5, pmove=0.5323364]2023-12-09 01:33:47 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 33%|███▎      | 20000/60000 [1:28:10<1:59:44,  5.57it/s, E=-1.1, E_std=0.00335, E_var=1.31e-5, pmove=0.5312073]2023-12-09 01:36:51 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 33%|███▎      | 20000/60000 [1:28:11<1:59:44,  5.57it/s, E=-1.1, E_std=0.00401, E_var=2.12e-5, pmove=0.5314636][20000] creating checkpoint
 35%|███▌      | 21000/60000 [1:31:16<1:56:16,  5.59it/s, E=-1.1, E_std=0.00371, E_var=1.6e-5, pmove=0.5339905]2023-12-09 01:39:56 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 37%|███▋      | 22000/60000 [1:34:18<1:53:02,  5.60it/s, E=-1.1, E_std=0.00294, E_var=9.08e-6, pmove=0.53037715]2023-12-09 01:42:59 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 38%|███▊      | 23000/60000 [1:37:24<1:50:42,  5.57it/s, E=-1.1, E_std=0.00272, E_var=8.24e-6, pmove=0.53443605]2023-12-09 01:46:05 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 40%|████      | 24000/60000 [1:40:37<1:48:06,  5.55it/s, E=-1.1, E_std=0.00283, E_var=9.42e-6, pmove=0.53353274]2023-12-09 01:49:17 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 42%|████▏     | 25000/60000 [1:43:40<1:44:34,  5.58it/s, E=-1.1, E_std=0.00318, E_var=1.17e-5, pmove=0.53358155]2023-12-09 01:52:21 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 42%|████▏     | 25000/60000 [1:43:41<1:44:34,  5.58it/s, E=-1.1, E_std=0.00298, E_var=9.85e-6, pmove=0.5335877] [25000] creating checkpoint
 43%|████▎     | 26000/60000 [1:46:53<1:41:07,  5.60it/s, E=-1.1, E_std=0.00309, E_var=1.11e-5, pmove=0.53308105]2023-12-09 01:55:33 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 45%|████▌     | 27000/60000 [1:49:55<1:38:27,  5.59it/s, E=-1.1, E_std=0.00346, E_var=1.76e-5, pmove=0.53447264]2023-12-09 01:58:36 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 47%|████▋     | 28000/60000 [1:52:58<1:35:30,  5.58it/s, E=-1.09, E_std=0.0028, E_var=9.79e-6, pmove=0.53217167]2023-12-09 02:01:39 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 48%|████▊     | 29000/60000 [1:56:02<1:32:26,  5.59it/s, E=-1.1, E_std=0.00322, E_var=1.22e-5, pmove=0.53204346]2023-12-09 02:04:43 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 50%|█████     | 30000/60000 [1:59:06<1:29:23,  5.59it/s, E=-1.1, E_std=0.00244, E_var=6.71e-6, pmove=0.53277594]2023-12-09 02:07:47 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 50%|█████     | 30000/60000 [1:59:07<1:29:23,  5.59it/s, E=-1.1, E_std=0.00401, E_var=1.99e-5, pmove=0.5334656] [30000] creating checkpoint
 52%|█████▏    | 31000/60000 [2:02:10<1:26:24,  5.59it/s, E=-1.1, E_std=0.00501, E_var=5e-5, pmove=0.5336304]2023-12-09 02:10:50 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 53%|█████▎    | 32000/60000 [2:05:14<1:23:04,  5.62it/s, E=-1.1, E_std=0.00385, E_var=2.88e-5, pmove=0.53297734]2023-12-09 02:13:55 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 55%|█████▌    | 33000/60000 [2:08:18<1:20:15,  5.61it/s, E=-1.1, E_std=0.00263, E_var=7.37e-6, pmove=0.5326904]2023-12-09 02:16:58 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 57%|█████▋    | 34000/60000 [2:11:19<1:17:42,  5.58it/s, E=-1.1, E_std=0.00363, E_var=2.9e-5, pmove=0.53212893]2023-12-09 02:20:00 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 58%|█████▊    | 35000/60000 [2:14:24<1:14:33,  5.59it/s, E=-1.11, E_std=0.00338, E_var=1.83e-5, pmove=0.53356326]2023-12-09 02:23:04 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 58%|█████▊    | 35000/60000 [2:14:24<1:14:33,  5.59it/s, E=-1.11, E_std=0.00299, E_var=1.07e-5, pmove=0.53430784][35000] creating checkpoint
 60%|██████    | 36000/60000 [2:17:30<1:11:05,  5.63it/s, E=-1.1, E_std=0.00321, E_var=1.75e-5, pmove=0.5345215]2023-12-09 02:26:10 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 62%|██████▏   | 37000/60000 [2:20:31<1:08:33,  5.59it/s, E=-1.1, E_std=0.00332, E_var=1.98e-5, pmove=0.53378296]2023-12-09 02:29:12 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 63%|██████▎   | 38000/60000 [2:23:32<1:05:31,  5.60it/s, E=-1.1, E_std=0.00312, E_var=1.41e-5, pmove=0.53267825]2023-12-09 02:32:13 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 65%|██████▌   | 39000/60000 [2:26:35<1:02:43,  5.58it/s, E=-1.1, E_std=0.00303, E_var=1.25e-5, pmove=0.53477174]2023-12-09 02:35:16 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 67%|██████▋   | 40000/60000 [2:29:36<59:41,  5.58it/s, E=-1.1, E_std=0.00293, E_var=1.35e-5, pmove=0.53344727]2023-12-09 02:38:17 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 67%|██████▋   | 40000/60000 [2:29:37<59:41,  5.58it/s, E=-1.1, E_std=0.00346, E_var=2.41e-5, pmove=0.53345335][40000] creating checkpoint
 68%|██████▊   | 41000/60000 [2:32:42<56:40,  5.59it/s, E=-1.1, E_std=0.00282, E_var=9.83e-6, pmove=0.5316833]2023-12-09 02:41:23 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 70%|███████   | 42000/60000 [2:35:46<53:52,  5.57it/s, E=-1.1, E_std=0.00365, E_var=2.69e-5, pmove=0.53312993]2023-12-09 02:44:27 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 72%|███████▏  | 43000/60000 [2:38:48<50:51,  5.57it/s, E=-1.1, E_std=0.00229, E_var=5.73e-6, pmove=0.5336548]2023-12-09 02:47:29 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 73%|███████▎  | 44000/60000 [2:41:50<47:45,  5.58it/s, E=-1.1, E_std=0.00327, E_var=1.94e-5, pmove=0.53303224]2023-12-09 02:50:31 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 75%|███████▌  | 45000/60000 [2:44:51<44:46,  5.58it/s, E=-1.1, E_std=0.00242, E_var=7.22e-6, pmove=0.53007203]2023-12-09 02:53:32 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 75%|███████▌  | 45000/60000 [2:44:52<44:46,  5.58it/s, E=-1.1, E_std=0.00328, E_var=1.57e-5, pmove=0.53167117][45000] creating checkpoint
 77%|███████▋  | 46000/60000 [2:47:57<41:51,  5.57it/s, E=-1.1, E_std=0.00235, E_var=6.81e-6, pmove=0.53195804]2023-12-09 02:56:37 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 78%|███████▊  | 47000/60000 [2:51:00<38:59,  5.56it/s, E=-1.1, E_std=0.00217, E_var=5.32e-6, pmove=0.5342041]2023-12-09 02:59:40 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 80%|████████  | 48000/60000 [2:54:02<35:37,  5.61it/s, E=-1.1, E_std=0.0023, E_var=6.82e-6, pmove=0.5297302]2023-12-09 03:02:42 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 82%|████████▏ | 49000/60000 [2:57:05<32:54,  5.57it/s, E=-1.09, E_std=0.00212, E_var=5.36e-6, pmove=0.53096926]2023-12-09 03:05:46 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 83%|████████▎ | 50000/60000 [3:00:09<29:38,  5.62it/s, E=-1.1, E_std=0.00231, E_var=6.53e-6, pmove=0.53167725]2023-12-09 03:08:49 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 83%|████████▎ | 50000/60000 [3:00:09<29:38,  5.62it/s, E=-1.1, E_std=0.00287, E_var=1.2e-5, pmove=0.53486943] [50000] creating checkpoint
 85%|████████▌ | 51000/60000 [3:03:13<26:53,  5.58it/s, E=-1.1, E_std=0.00248, E_var=8.79e-6, pmove=0.5317627]2023-12-09 03:11:53 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 87%|████████▋ | 52000/60000 [3:06:15<30:37,  4.35it/s, E=-1.1, E_std=0.00212, E_var=5.34e-6, pmove=0.532367]2023-12-09 03:14:55 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 88%|████████▊ | 53000/60000 [3:09:17<20:59,  5.56it/s, E=-1.1, E_std=0.00304, E_var=2.22e-5, pmove=0.5289795]2023-12-09 03:17:58 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 90%|█████████ | 54000/60000 [3:12:21<17:56,  5.57it/s, E=-1.1, E_std=0.0021, E_var=4.94e-6, pmove=0.53134155]2023-12-09 03:21:01 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 92%|█████████▏| 55000/60000 [3:15:23<14:55,  5.58it/s, E=-1.1, E_std=0.00197, E_var=4.73e-6, pmove=0.5349182]2023-12-09 03:24:03 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 92%|█████████▏| 55000/60000 [3:15:23<14:55,  5.58it/s, E=-1.1, E_std=0.00236, E_var=9.55e-6, pmove=0.53149414][55000] creating checkpoint
 93%|█████████▎| 56000/60000 [3:18:26<12:00,  5.55it/s, E=-1.1, E_std=0.00369, E_var=2.58e-5, pmove=0.53092045]2023-12-09 03:27:07 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 95%|█████████▌| 57000/60000 [3:21:29<08:58,  5.57it/s, E=-1.1, E_std=0.00265, E_var=1.14e-5, pmove=0.53120124]2023-12-09 03:30:09 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 97%|█████████▋| 58000/60000 [3:24:30<06:01,  5.54it/s, E=-1.11, E_std=0.00233, E_var=6.36e-6, pmove=0.5334717]2023-12-09 03:33:11 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
 98%|█████████▊| 59000/60000 [3:27:32<02:58,  5.59it/s, E=-1.1, E_std=0.00273, E_var=1.37e-5, pmove=0.5321838]2023-12-09 03:36:13 (WARNING): Tracking a matplotlib object using "aim.Figure" might not behave as expected.In such cases, consider tracking with "aim.Image".
100%|██████████| 60000/60000 [3:30:37<00:00,  4.75it/s, E=-1.11, E_std=0.00244, E_var=8.71e-6, pmove=0.5347779]  
Evaluating final energy
Computing Energy: 100%|██████████| 123/123 [01:59<00:00,  1.03it/s, E=-1.012566(6)]
Computing Energy: 100%|██████████| 123/123 [01:52<00:00,  1.09it/s, E=-1.114847(7)]49, E_err=6.47e-6]
Computing Energy: 100%|██████████| 123/123 [01:53<00:00,  1.09it/s, E=-1.159060(4)]5, E_err=7.32e-6] 
Computing Energy: 100%|██████████| 123/123 [01:52<00:00,  1.09it/s, E=-1.173584(4)]7, E_err=3.86e-6]
Computing Energy: 100%|██████████| 123/123 [01:52<00:00,  1.09it/s, E=-1.1723436(32)] E_err=4.11e-6]
Computing Energy: 100%|██████████| 123/123 [01:52<00:00,  1.09it/s, E=-1.1626763(35)] E_err=3.22e-6]
Computing Energy: 100%|██████████| 123/123 [01:52<00:00,  1.09it/s, E=-1.148697(4)]E_err=3.49e-6]   
Computing Energy: 100%|██████████| 123/123 [01:52<00:00,  1.09it/s, E=-1.1328236(35)]err=3.79e-6]
Computing Energy: 100%|██████████| 123/123 [01:52<00:00,  1.09it/s, E=-1.1164778(31)]_err=3.49e-6]
Computing Energy: 100%|██████████| 123/123 [01:52<00:00,  1.09it/s, E=-1.1005371(29)]err=3.09e-6] 
2023-12-09 05:10:38 (INFO): Result: {'E_final': [-1.012566089630127, -1.11484694480896, -1.1590598821640015, -1.173583745956421, -1.1723438501358032, -1.1626760959625244, -1.148697018623352, -1.1328238248825073, -1.1164777278900146, -1.100536823272705, -1.0855448246002197, -1.071805715560913, -1.0594877004623413, -1.048685073852539, -1.039363980293274, -1.0314862728118896], 'E_final_std': [0.006494690198451281, 0.00734989857301116, 0.0038714748807251453, 0.004126369953155518, 0.0032370712142437696, 0.0035024748649448156, 0.0038031628355383873, 0.003507445566356182, 0.0031044657807797194, 0.0029472357127815485, 0.0026276602875441313, 0.0028136828914284706, 0.0028574015013873577, 0.002692325972020626, 0.002870911965146661, 0.004232349805533886], 'E_final_err': [6.470098795458199e-06, 7.322069021139649e-06, 3.856815983062527e-06, 4.110745924400338e-06, 3.224814413639831e-06, 3.4892131437163674e-06, 3.788762593636266e-06, 3.4941650241343023e-06, 3.0927110755111616e-06, 2.936076341215851e-06, 2.617710951842896e-06, 2.8030292023742407e-06, 2.846582276807499e-06, 2.68213178708745e-06, 2.8600415847381517e-06, 4.216324496166491e-06], 'E_gnn': [-1.0126067399978638, -1.1148062944412231, -1.1589748859405518, -1.17353355884552, -1.1723034381866455, -1.1626331806182861, -1.1486608982086182, -1.1327881813049316, -1.1164401769638062, -1.1004892587661743, -1.0854833126068115, -1.071752905845642, -1.0594615936279297, -1.0486533641815186, -1.0393187999725342, -1.0314669609069824], 'GNN_MAE': 4.3332576751708984e-05}
2023-12-09 05:10:38 (INFO): Completed after 5:07:53
2023-12-09 05:10:38 (ERROR): Traceback (most recent call last):
  File "/home/shhgroup/shanghui/miniconda3/envs/pesnet2/lib/python3.10/site-packages/sacred/run.py", line 429, in _final_call
    getattr(observer, method)(**kwargs)
  File "/home/shhgroup/shanghui/miniconda3/envs/pesnet2/lib/python3.10/site-packages/seml/observers.py", line 364, in completed_event
    requests.post(self.webhook_url, data=json.dumps(data), headers=headers)
  File "/home/shhgroup/shanghui/miniconda3/envs/pesnet2/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/home/shhgroup/shanghui/miniconda3/envs/pesnet2/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/shhgroup/shanghui/miniconda3/envs/pesnet2/lib/python3.10/site-packages/requests/sessions.py", line 575, in request
    prep = self.prepare_request(req)
  File "/home/shhgroup/shanghui/miniconda3/envs/pesnet2/lib/python3.10/site-packages/requests/sessions.py", line 486, in prepare_request
    p.prepare(
  File "/home/shhgroup/shanghui/miniconda3/envs/pesnet2/lib/python3.10/site-packages/requests/models.py", line 368, in prepare
    self.prepare_url(url, params)
  File "/home/shhgroup/shanghui/miniconda3/envs/pesnet2/lib/python3.10/site-packages/requests/models.py", line 439, in prepare_url
    raise MissingSchema(
requests.exceptions.MissingSchema: Invalid URL 'YOUR_WEBHOOK': No scheme supplied. Perhaps you meant https://YOUR_WEBHOOK?```

[n-gao]

HI @xiazhuozhao, thanks a lot for reporting this. :) It was an issue in seml_logger. I updated the package version, please tell me whether the bug is resolved now.

[xiazhuozhao]

Thank you! I noticed in the e2098a7 commit that you updated the version requirement for seml_logger from 5eb5f19 to c1c5761. However, I observed a discrepancy in the Logger.add_distribution_dict() method between the old version, which accepts 5 parameters, and the latest version, which only accepts 4, missing the n_bins parameter.
It will cause:

  0%|          | 0/10000 [00:21<?, ?it/s, MSE=0.05318243, pmove=0.9753418]
2023-12-12 11:35:18 (ERROR): Traceback (most recent call last):
  File "/home/shhgroup/shanghui/miniconda3/envs/pesnet/lib/python3.10/site-packages/seml_logger/__init__.py", line 66, in func
    result = fn(**kwargs, logger=logger)
  File "/home/shhgroup/shanghui/zhuozhao/pesnet/pesnet/train.py", line 188, in run
    electrons, hf_energies = pretrain(
  File "/home/shhgroup/shanghui/zhuozhao/pesnet/pesnet/train.py", line 110, in pretrain
    logger.add_distribution_dict(vmc.params['gnn'], 'pre_gnn', n_bins=100, step=step)
TypeError: Logger.add_distribution_dict() got an unexpected keyword argument 'n_bins'

2023-12-12 11:35:18 (ERROR): Failed after 0:01:07!
Traceback (most recent calls WITHOUT Sacred internals):
  File "/home/shhgroup/shanghui/miniconda3/envs/pesnet/lib/python3.10/site-packages/seml_logger/__init__.py", line 77, in func
    raise e
  File "/home/shhgroup/shanghui/miniconda3/envs/pesnet/lib/python3.10/site-packages/seml_logger/__init__.py", line 66, in func
    result = fn(**kwargs, logger=logger)
  File "/home/shhgroup/shanghui/zhuozhao/pesnet/pesnet/train.py", line 188, in run
    electrons, hf_energies = pretrain(
  File "/home/shhgroup/shanghui/zhuozhao/pesnet/pesnet/train.py", line 110, in pretrain
    logger.add_distribution_dict(vmc.params['gnn'], 'pre_gnn', n_bins=100, step=step)
TypeError: Logger.add_distribution_dict() got an unexpected keyword argument 'n_bins'

[xiazhuozhao]

I attempted to remove all n_bins parameters, and it seems the program is functioning correctly now.

[xiazhuozhao]

Hello, I tried removing the n_bins parameter, but the program still cannot save the image properly after evaluation.
python train.py with configs/systems/h2.yaml print_progress=True
The error log of the program is as follows:

Computing Energy: 100%|██████████| 123/123 [01:53<00:00,  1.09it/s, E=-1.031485(6)]t 
Evaluation:  94%|█████████▍| 15/16 [1:31:57<04:59, 299.54s/it, E=-1.03, E_std=0.00608, E_Evaluation: 100%|██████████| 16/16 [1:31:57<00:00, 299.06s/it, E=-1.03, E_std=0.00608, E_Evaluation: 100%|██████████| 16/16 [1:31:57<00:00, 344.86s/it, E=-1.03, E_std=0.00608, E_err=6.06e-6]
2023-12-12 18:04:58 (INFO): Plotting
/home/shhgroup/shanghui/miniconda3/envs/pesnet/lib/python3.10/site-packages/plotly/matplotlylib/renderer.py:571: UserWarning:

Dang! That path collection is out of this world. I totally don't know what to do with it yet! Plotly can only import path collections linked to 'data' coordinates

/home/shhgroup/shanghui/miniconda3/envs/pesnet/lib/python3.10/site-packages/plotly/matplotlylib/renderer.py:609: UserWarning:

I found a path object that I don't think is part of a bar chart. Ignoring.

2023-12-12 18:04:59 (INFO): Finished
2023-12-12 18:04:59 (INFO): Result: {'E_final': [-1.0125607252120972, -1.1148520708084106, -1.159051537513733, -1.1735835075378418, -1.1723477840423584, -1.162668228149414, -1.1487033367156982, -1.132829189300537, -1.1164772510528564, -1.100542664527893, -1.0855491161346436, -1.0718082189559937, -1.0594900846481323, -1.0486828088760376, -1.0393799543380737, -1.0314847230911255], 'E_final_std': [0.006816584151238203, 0.0047567058354616165, 0.004910124931484461, 0.0037949245888739824, 0.0036922767758369446, 0.004210345447063446, 0.0036227719392627478, 0.0032987305894494057, 0.003441084874793887, 0.00334502593614161, 0.003876573173329234, 0.004674193914979696, 0.0034526714589446783, 0.004583231173455715, 0.008094404824078083, 0.006080905441194773], 'E_final_err': [6.790773933540779e-06, 4.7386951118481525e-06, 4.891533303978456e-06, 3.7805555401523655e-06, 3.6782963913409678e-06, 4.194403454687349e-06, 3.609054726895685e-06, 3.2862403226603756e-06, 3.428055599754774e-06, 3.332360377307475e-06, 3.8618949715636995e-06, 4.656495613333455e-06, 3.4395983126274196e-06, 4.565877291845815e-06, 8.063756284280832e-06, 6.057880786946769e-06], 'E_gnn': [-1.0126467943191528, -1.114813208580017, -1.1589558124542236, -1.173477053642273, -1.172231912612915, -1.1625579595565796, -1.1485915184020996, -1.1327393054962158, -1.116417646408081, -1.1004831790924072, -1.0854827165603638, -1.0717618465423584, -1.0594898462295532, -1.0486887693405151, -1.0393292903900146, -1.0314857959747314], 'GNN_MAE': 6.529688835144043e-05}
2023-12-12 18:04:59 (INFO): Completed after 5:14:27

Did I forget to specify the saving directory?

[n-gao]

Thanks for catching this. I will update the repo! Regarding the data storage, by default, all data should be located in ~/logs/pesnet/. You can change this behavior by setting a folder variable. E.g., python train.py with ... folder=./logs.

[xiazhuozhao]

Thank you very much! I observed in the train.py file that Matplotlib is used to generate visualizations of the network's output, which are then handed over to the seml_logger for processing. Could you kindly confirm if, under normal circumstances, the plotted results are expected to be saved in the ~/logs/pesnet/ directory? Regrettably, in the most recent version of the code, I came across some files such as .chk, .checkpoints, .npz, .out, .json, and .pickle in that folder, but I couldn't locate any files related to the visualizations. Your assistance is greatly appreciated.