pyscf: 1.7.6.post1 -> 2.0.1 #144253

sheepforce · 2021-11-02T15:17:08Z

Motivation for this change

This updates PySCF to its latest release. The build system has changed somewhat and now uses a lot of CMake. I've enabled the tests, as inspired by an older commit of @drewrisinger .
The CPPE library for polarisable embedding has been added, which is an optional dependency of PySCF. I am not entirely sure if i did the python binding stuff correctly there. CPPE is at least not in the PYTHONPATH from within a nix-shell.

Things done

pkgs/top-level/python-packages.nix

pkgs/development/libraries/science/chemistry/cppe/default.nix

sheepforce · 2021-11-03T09:50:12Z

As written I've enabled the test suite. However, it is huge and runs for at least 2 h on my 10 cores and some single test are failing and need to be disabled. I guess it might be better to disable the full test suite? What do you think?

markuskowa · 2021-11-03T11:52:08Z

As written I've enabled the test suite. However, it is huge and runs for at least 2 h on my 10 cores and some single test are failing and need to be disabled. I guess it might be better to disable the full test suite? What do you think?

Let's disable the test suite for now. It is too big and not reliable. We still have a test in NixOS-QChem to catch basic errors.

pkgs/development/libraries/science/chemistry/cppe/default.nix

drewrisinger · 2021-11-03T15:30:13Z

As written I've enabled the test suite. However, it is huge and runs for at least 2 h on my 10 cores and some single test are failing and need to be disabled. I guess it might be better to disable the full test suite? What do you think?

Even PySCF mainline doesn't run all tests that they include in their source: https://github.com/pyscf/pyscf/blob/master/.github/workflows/run_tests.sh

Here are the tests that I excluded, feel free to use it as a starting point. I think tests typically ran for about 20-30 mins with all these disabled? https://github.com/drewrisinger/nur-packages/blob/77d8267ff1b432c76ae90fc57db9e9787640137f/pkgs/python-modules/pyscf/default.nix#L94-L255

drewrisinger

General comments about structure of cppe derivation, and a few details on the hash types & build variables

pkgs/development/libraries/science/chemistry/cppe/default.nix

pkgs/development/python-modules/pyscf/default.nix

drewrisinger · 2021-11-03T15:44:16Z

As written I've enabled the test suite. However, it is huge and runs for at least 2 h on my 10 cores and some single test are failing and need to be disabled. I guess it might be better to disable the full test suite? What do you think?

Let's disable the test suite for now. It is too big and not reliable. We still have a test in NixOS-QChem to catch basic errors.

That might be true, but the issue is that any packaging errors should ideally be caught before merge, not after. Do you expect a potential reviewer of future pyscf updates to know to go to NixOS-QChem, figure out how to run tests there against a PR, and then review? Vs the standard nixpkgs review process of nix-review pr ...?

pkgs/development/libraries/science/chemistry/cppe/default.nix

pkgs/top-level/python-packages.nix

markuskowa · 2021-11-03T16:39:38Z

As written I've enabled the test suite. However, it is huge and runs for at least 2 h on my 10 cores and some single test are failing and need to be disabled. I guess it might be better to disable the full test suite? What do you think?

Let's disable the test suite for now. It is too big and not reliable. We still have a test in NixOS-QChem to catch basic errors.

That might be true, but the issue is that any packaging errors should ideally be caught before merge, not after. Do you expect a potential reviewer of future pyscf updates to know to go to NixOS-QChem, figure out how to run tests there against a PR, and then review? Vs the standard nixpkgs review process of nix-review pr ...?

I agree with you, but unreliable test suites are worse than (or as good as) not having a test at all. I would not expect that some reviewer uses an external test. I look at the NixOS-QChem tests as a fallback option, which catches the error at some point later.

sheepforce · 2021-11-04T09:57:39Z

Thank you for the valuable feedback @drewrisinger , this solved some problems. I am doing final tunings on the test suite to get it running properly, and then it should be done.

drewrisinger · 2021-11-04T22:06:25Z

FWIW, release 2.0.1 was posted a few hours ago. Might want to roll that into this PR if possible, since it's not finalized? https://github.com/pyscf/pyscf/releases/tag/v2.0.1

drewrisinger

Few minor suggestions

pkgs/development/python-modules/cppe/default.nix

pkgs/development/python-modules/pyscf/default.nix

pkgs/development/python-modules/cppe/default.nix

pkgs/development/python-modules/pyscf/default.nix

sheepforce · 2021-11-11T21:44:30Z

I get a few errors when building pyscf, even when single-threaded. The most alarming is `` ... File "/build/source/pyscf/gto/basis/parse_nwchem.py", line 219, in search_seg with open(basisfile, 'r') as fin: OSError: [errno 24] Too many open files '/build/source/pyscf/gto/basis/ano.dat'

Rank 1820 tests in 2536.987 s FAILED (SKIP=4, errors=6)

I am afraid I cannot reproduce any of those 😐 I have it running now on all of my three different CPUs (i7-7700K, Xeon W2155 and the old Xeon E5420). Unfortunately I do not have access to others, such as the AMD Epyc machines of @markuskowa . But in general numerical effects this large are worrying. However, I don't see what we could change about it, except disabling those tests.

I am forcing the test to be always single core in the preCheck and the only parallel part is BLAS and FFT. Running them multi-core gives even less stable results.

Regarding your failing NWChem parser test; I suppose this might actually depend on how many other files you have open during your build? I don't think we can change this. ulimit -n $LARGENUMBER is not allowed and the number of open file handles would need to be set in the sysctl configuration. I've increased this number on my machines from the defaults some time ago.

Upstream builds with netlib BLAS in the CI apparently. I mean we could force blas to be blas-reference, but I guess this it not the way forward ...

sheepforce · 2021-11-12T11:17:40Z

@ofborg build python3.pkgs.pyscf

sheepforce · 2021-11-12T12:53:08Z

ofBorg built the current version just fine without any errors. 🙂

drewrisinger · 2021-11-12T15:11:45Z

Ok, I'm willing to chalk the errors up to a local build issue, I'd never seen them on my previous builds of my NUR package version.

pkgs/development/python-modules/fields/default.nix

pkgs/development/python-modules/polarizationsolver/default.nix

libcint: formatting and features libcint: platforms

libxc: formatting libxc: platforms

fields: expose package fields: formatting fields: platforms fields: platforms fields: remove redundant platform

polarizationsolver: expose polyrizationsolver: formatting polarizationsolver: platforms polarizationsolver: platforms polarizationsolver: license polarizationsolver: remove redundant platform

cppe: move pytestCheckHook to checkInputs cppe: hash cppe: license and hash cppe: formatting python3.pkgs.cppe: more tests cppe: formatting cppe: formatting cppe: platforms cppe: platforms

pyscf: hash pyscf: limit test suite to single core pyscf: adapting test suite pyscf: fix pythonpath for tests pyscf: formatting pyscf: platforms remove log pyscf: enable uadc module pyscf: platforms pyscf: formatting pyscf: disable instable N3 CI test pyscf: formating pyscf: increase ulimit pyscf: ulimit files pyscf: remove ulimit -n

markuskowa · 2021-11-24T12:47:13Z

I am still getting random errors from the test suite:

  MemoryError: Insufficient memory! MP2 memory usage 0 MB (currently available -1 MB)
  -------------------- >> begin captured stdout << ---------------------
  output file: /dev/null
  
  --------------------- >> end captured stdout << ----------------------
  
  ----------------------------------------------------------------------
  Ran 1820 tests in 2249.433s
  
  FAILED (SKIP=4, errors=2)

Note, that I ran this on a machine that had over 100 GB of free RAM!

markuskowa

We need to come up with a solution for the test suite.

markuskowa

I can't reproduce the errors anymore. Let's merge it and give it try.

github-actions · 2021-11-28T17:36:38Z

Successfully created backport PR #147740 for release-21.11.

markuskowa · 2021-11-30T14:31:14Z

Hydra can not build it at all (pyscf shows multiple test failures):
https://hydra.nixos.org/eval/1726603?filter=pyscf&compare=1726250&full=#tabs-now-fail
https://hydra.nixos.org/eval/1726676?filter=pyscf&compare=1726602&full=

sheepforce · 2021-11-30T14:41:19Z

Hm this is similar to this problem. Neither my machines, nor ofBorg had problems with those. The number of open files can unfortunately not be increased from within the job, but is a kernel setting. Interestingly they all fail by the NWChem parser: File "/build/source/pyscf/gto/basis/parse_nwchem.py", line 219, in search_seg

I will take a look if I can spot something there.

sheepforce · 2021-12-02T10:12:05Z

I've tried to figure out what's going on in the code there. The nwchem basis parser opens a file for each atom (or at least element) for each basis set, but it does so safely in a bracket pattern and closes them immediately. So there is no apparent reason why especially the nwchem basis set parser (with which one of the hydra jobs fails) should have too many open files. I've no idea why this is not working on Hydra and for this build.

The other Hydra jobs fails with numerical issues (huge ones), which very much looks like some faulty behaviour with respect to BLAS/LAPACK - numpy interaction or something like this. I still can't reproduce any of these errors on any of my Intel machines. Hydra has some AMD Epyc instances, correct? I've also built with mkl as the blas and lapack provider for all packages (global overlay) and can also not reproduce any of the errors with MKL.
@markuskowa , could you try building on your AMD machines with MKL for everything just once?
I've also tried to use blas-reference and lapack-reference (this is what upstream does) but numpy refuses to build with those, due to an attribute conflict in its derivation.

If MKL also does not solve the problem I will open an issue upstream and try to get an idea from the developers.

markuskowa · 2021-12-03T10:47:59Z

I can give it a try with MKL, but my suspicion is that this will not solve all the problems. It fails also on aarch64 (which we could deactivate?), where MKL does not work. I think the problem is somewhere else: pyscf-2.0.1 builds just fine on my internal Hydras on x86_64-linux with AMD Epyc as well as on Intel Xeon with the Openblas default.
Looking at logs of the build failures, it looks more like resource limit of Hydra's builders (it complains about too many open files).

markuskowa · 2021-12-03T12:04:53Z

FYI: building pyscf with MKL leads to the following errors. Note that they fail just above the test's threshold.

======================================================================
FAIL: test_c_ragf2 (test_c_agf2.KnownValues)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/source/pyscf/agf2/test/test_c_agf2.py", line 45, in test_c_ragf2
    self.assertAlmostEqual(np.max(np.absolute(vv1-vv2)), 0.0, 10)
AssertionError: 2.000888343900442e-10 != 0.0 within 10 places (2.000888343900442e-10 difference)

======================================================================
FAIL: test_c_uagf2 (test_c_agf2.KnownValues)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/source/pyscf/agf2/test/test_c_agf2.py", line 66, in test_c_uagf2
    self.assertAlmostEqual(np.max(np.absolute(vv1-vv2)), 0.0, 10)
AssertionError: 2.346496330574155e-10 != 0.0 within 10 places (2.346496330574155e-10 difference)

----------------------------------------------------------------------
Ran 1820 tests in 2344.443s

FAILED (SKIP=4, failures=2)
builder for '/nix/store/07675hcr0ylcm4x7nkjddcrwqba23r7q-python3.9-pyscf-2.0.1.drv' failed with exit code 1

sheepforce requested review from FRidh and jonringer as code owners November 2, 2021 15:17

sheepforce requested review from markuskowa and drewrisinger November 2, 2021 15:17

github-actions bot added the 6.topic: python label Nov 2, 2021

ofborg bot added 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Nov 2, 2021

jonringer reviewed Nov 2, 2021

View reviewed changes

pkgs/top-level/python-packages.nix Outdated Show resolved Hide resolved

markuskowa reviewed Nov 2, 2021

View reviewed changes

pkgs/development/libraries/science/chemistry/cppe/default.nix Outdated Show resolved Hide resolved

sheepforce force-pushed the pyscf branch from ae7a0c7 to f23d197 Compare November 3, 2021 09:48

SuperSandro2000 reviewed Nov 3, 2021

View reviewed changes

pkgs/development/libraries/science/chemistry/cppe/default.nix Outdated Show resolved Hide resolved

sheepforce force-pushed the pyscf branch from f23d197 to 630897c Compare November 3, 2021 13:21

drewrisinger suggested changes Nov 3, 2021

View reviewed changes

SuperSandro2000 reviewed Nov 3, 2021

View reviewed changes

pkgs/development/libraries/science/chemistry/cppe/default.nix Outdated Show resolved Hide resolved

SuperSandro2000 reviewed Nov 3, 2021

View reviewed changes

pkgs/top-level/python-packages.nix Outdated Show resolved Hide resolved

sheepforce force-pushed the pyscf branch 2 times, most recently from b7126e3 to f5d9be5 Compare November 4, 2021 09:56

ofborg bot requested a review from drewrisinger November 4, 2021 10:07

ofborg bot added 10.rebuild-darwin: 1-10 and removed 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin labels Nov 4, 2021

drewrisinger suggested changes Nov 4, 2021

View reviewed changes

sheepforce changed the title ~~pyscf: 1.7.6.post1 -> 2.0.0~~ pyscf: 1.7.6.post1 -> 2.0.1 Nov 8, 2021

SuperSandro2000 reviewed Nov 12, 2021

View reviewed changes

pkgs/development/python-modules/fields/default.nix Outdated Show resolved Hide resolved

SuperSandro2000 reviewed Nov 12, 2021

View reviewed changes

pkgs/development/python-modules/polarizationsolver/default.nix Outdated Show resolved Hide resolved

markuskowa approved these changes Nov 13, 2021

View reviewed changes

sheepforce added 6 commits November 22, 2021 12:28

libcint: 4.4.0 -> 4.4.6

dd7f587

libcint: formatting and features libcint: platforms

libxc: force 3rd and 4th derivatives compilation

2a9baed

libxc: formatting libxc: platforms

python3.pkgs.fields: init at 5.0.0

dbd7ba5

fields: expose package fields: formatting fields: platforms fields: platforms fields: remove redundant platform

python3.pkgs.polarizationsolver: init at 00424ac4

a6a5114

polarizationsolver: expose polyrizationsolver: formatting polarizationsolver: platforms polarizationsolver: platforms polarizationsolver: license polarizationsolver: remove redundant platform

cppe: init at 0.3.1

938a9e0

cppe: move pytestCheckHook to checkInputs cppe: hash cppe: license and hash cppe: formatting python3.pkgs.cppe: more tests cppe: formatting cppe: formatting cppe: platforms cppe: platforms

sheepforce force-pushed the pyscf branch from 3e216f8 to 21ca2de Compare November 22, 2021 11:29

ofborg bot requested a review from markuskowa November 22, 2021 11:41

markuskowa added the backport release-21.11 label Nov 23, 2021

markuskowa requested changes Nov 24, 2021

View reviewed changes

markuskowa approved these changes Nov 28, 2021

View reviewed changes

markuskowa merged commit c0d9398 into NixOS:master Nov 28, 2021

github-actions bot mentioned this pull request Nov 28, 2021

[Backport release-21.11] pyscf: 1.7.6.post1 -> 2.0.1 #147740

Merged

1 task

sheepforce mentioned this pull request Dec 8, 2021

Numerical Instabilities and other non-reproducible behaviour pyscf/pyscf#1123

Open

sheepforce self-assigned this Dec 10, 2021

sheepforce mentioned this pull request Dec 16, 2021

pyscf: disable parts of test suite and aarch64 #150952

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pyscf: 1.7.6.post1 -> 2.0.1 #144253

pyscf: 1.7.6.post1 -> 2.0.1 #144253

sheepforce commented Nov 2, 2021 •

edited

Loading

sheepforce commented Nov 3, 2021

markuskowa commented Nov 3, 2021

drewrisinger commented Nov 3, 2021

drewrisinger left a comment

drewrisinger commented Nov 3, 2021

markuskowa commented Nov 3, 2021

sheepforce commented Nov 4, 2021

drewrisinger commented Nov 4, 2021 •

edited

Loading

drewrisinger left a comment

sheepforce commented Nov 11, 2021 •

edited

Loading

sheepforce commented Nov 12, 2021

sheepforce commented Nov 12, 2021

drewrisinger commented Nov 12, 2021

markuskowa commented Nov 24, 2021

markuskowa left a comment

markuskowa left a comment

github-actions bot commented Nov 28, 2021

markuskowa commented Nov 30, 2021 •

edited

Loading

sheepforce commented Nov 30, 2021

sheepforce commented Dec 2, 2021

markuskowa commented Dec 3, 2021 •

edited

Loading

markuskowa commented Dec 3, 2021

pyscf: 1.7.6.post1 -> 2.0.1 #144253

pyscf: 1.7.6.post1 -> 2.0.1 #144253

Conversation

sheepforce commented Nov 2, 2021 • edited Loading

Motivation for this change

Things done

sheepforce commented Nov 3, 2021

markuskowa commented Nov 3, 2021

drewrisinger commented Nov 3, 2021

drewrisinger left a comment

Choose a reason for hiding this comment

drewrisinger commented Nov 3, 2021

markuskowa commented Nov 3, 2021

sheepforce commented Nov 4, 2021

drewrisinger commented Nov 4, 2021 • edited Loading

drewrisinger left a comment

Choose a reason for hiding this comment

sheepforce commented Nov 11, 2021 • edited Loading

sheepforce commented Nov 12, 2021

sheepforce commented Nov 12, 2021

drewrisinger commented Nov 12, 2021

markuskowa commented Nov 24, 2021

markuskowa left a comment

Choose a reason for hiding this comment

markuskowa left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 28, 2021

markuskowa commented Nov 30, 2021 • edited Loading

sheepforce commented Nov 30, 2021

sheepforce commented Dec 2, 2021

markuskowa commented Dec 3, 2021 • edited Loading

markuskowa commented Dec 3, 2021

sheepforce commented Nov 2, 2021 •

edited

Loading

drewrisinger commented Nov 4, 2021 •

edited

Loading

sheepforce commented Nov 11, 2021 •

edited

Loading

markuskowa commented Nov 30, 2021 •

edited

Loading

markuskowa commented Dec 3, 2021 •

edited

Loading