ci: Re-enable CI on GH200

[havogt]

No description provided.

[edopao]

Tödi seems to be back, we could try to resume this PR without any specific partition/account or time limit.

[edopao]

@FlorianDeconinck We got a failure in a test case:
tests/cartesian_tests/integration_tests/multi_feature_tests/test_code_generation.py::test_K_offset_write[dace:gpu]

Test log here:
https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/4455690602105886/4525297225819146/-/pipelines/1616373175

I see that there is a check on CUDA version:

gt4py/tests/cartesian_tests/integration_tests/multi_feature_tests/test_code_generation.py

Line 588 in 8040178

if cp.cuda.runtime.runtimeGetVersion() < 12000:

Maybe this check is not enough?

[FlorianDeconinck]

Hey Enrique,

I have a fix for that. Extent interval check is indeed broken for K and the test itself is bad. I can PR a quick fix to the test or you can change it as part of your PR

    def column_physics_conditional(A: Field[np.float64], B: Field[np.float64], scalar: np.float64):
-        with computation(BACKWARD), interval(1, None):
+        with computation(BACKWARD), interval(1, -1):
            if A > 0 and B > 0:
                A[0, 0, -1] = scalar
                B[0, 0, 1] = A
            lev = 1
            while A >= 0 and B >= 0:
                A[0, 0, lev] = -1
                B = -1
                lev = lev + 1

This should fix the test. The CUDA version test was a previous misunderstanding of where the race condition could come from

[edopao]

Hey Enrique,

I have a fix for that. Extent interval check is indeed broken for K and the test itself is bad. I can PR a quick fix to the test or you can change it as part of your PR

    def column_physics_conditional(A: Field[np.float64], B: Field[np.float64], scalar: np.float64):
-        with computation(BACKWARD), interval(1, None):
+        with computation(BACKWARD), interval(1, -1):
            if A > 0 and B > 0:
                A[0, 0, -1] = scalar
                B[0, 0, 1] = A
            lev = 1
            while A >= 0 and B >= 0:
                A[0, 0, lev] = -1
                B = -1
                lev = lev + 1

This should fix the test. The CUDA version test was a previous misunderstanding of where the race condition could come from

There is no hurry from our side, you can open a fix PR when you have time. I suspect that this test failure is flaky, it only happens sometimes.

[FlorianDeconinck]

PR open there: #1791

[havogt]

Looks like the dace problem is still there, even on CUDA 12: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/4455690602105886/4525297225819146/-/jobs/8815984483

[havogt]

cscs-ci run

[FlorianDeconinck]

Alright been enough failed attempt at fixing this - I'll PR a complete deactivation of the feature today and we will go back to the drawing board to figure out what we are clearly not understanding.

[edopao]

@FlorianDeconinck I have deactivated the test case this way. I was not sure whether to refer to issue #1684 or #1754.

[FlorianDeconinck]

Thanks, January has been idiotically overbooked and I keep falling off. I'll take it from there, sorry for the delay again

[havogt]

Thanks @edopao