[SYCL][E2E] Add a test to track UNSUPPORTED tests (#15849)

Similar to XFAIL tests (no-xfail-without-tracker.cpp) adding a requirement to follow UNSUPPORTED by UNSUPPORTED-TRACKER/UNSUPPORTED-INDENDED Adding a test to track the amount and the list of improperly UNSUPPORTED tests. Also update some tests to check the script.
intel · Oct 28, 2024 · e427e2f · e427e2f
1 parent 13e84cf
commit e427e2f
Show file tree

Hide file tree

Showing 9 changed files with 587 additions and 9 deletions.
diff --git a/sycl/test-e2e/Assert/assert_in_kernels.cpp b/sycl/test-e2e/Assert/assert_in_kernels.cpp
@@ -1,7 +1,7 @@
 // REQUIRES: linux
 
-// https://github.com/intel/llvm/issues/7634
 // UNSUPPORTED: hip
+// UNSUPPORTED-TRACKER: https://github.com/intel/llvm/issues/7634
 //
 // XFAIL: (opencl && gpu)
 // XFAIL-TRACKER: https://github.com/intel/llvm/issues/11364

diff --git a/sycl/test-e2e/Assert/assert_in_multiple_tus.cpp b/sycl/test-e2e/Assert/assert_in_multiple_tus.cpp
@@ -1,16 +1,16 @@
 // REQUIRES: linux
 
-// https://github.com/intel/llvm/issues/7634
 // UNSUPPORTED: hip
+// UNSUPPORTED-TRACKER: https://github.com/intel/llvm/issues/7634
 //
-// https://github.com/intel/llvm/issues/8832
 // UNSUPPORTED: cuda
+// UNSUPPORTED-TRACKER: https://github.com/intel/llvm/issues/8832
 //
 // XFAIL: (opencl && gpu)
 // XFAIL-TRACKER: https://github.com/intel/llvm/issues/11364
 
-// https://github.com/intel/llvm/issues/15029
 // UNSUPPORTED: gpu-intel-dg2
+// UNSUPPORTED-TRACKER: https://github.com/intel/llvm/issues/15029
 
 // RUN: %{build} -DSYCL_FALLBACK_ASSERT=1 -I %S/Inputs %S/Inputs/kernels_in_file2.cpp -o %t.out
 // RUN: %{run} %t.out &> %t.txt ; FileCheck %s --input-file %t.txt %if fpga %{ --check-prefix=CHECK-ACC %}

diff --git a/sycl/test-e2e/Assert/assert_in_multiple_tus_one_ndebug.cpp b/sycl/test-e2e/Assert/assert_in_multiple_tus_one_ndebug.cpp
@@ -1,10 +1,10 @@
 // REQUIRES: linux
 
-// https://github.com/intel/llvm/issues/7634
 // UNSUPPORTED: hip
+// UNSUPPORTED-TRACKER: https://github.com/intel/llvm/issues/7634
 //
-// https://github.com/intel/llvm/issues/8832
 // UNSUPPORTED: cuda
+// UNSUPPORTED-TRACKER: https://github.com/intel/llvm/issues/8832
 //
 // XFAIL: (opencl && gpu)
 // XFAIL-TRACKER: https://github.com/intel/llvm/issues/11364

diff --git a/sycl/test-e2e/Assert/assert_in_one_kernel.cpp b/sycl/test-e2e/Assert/assert_in_one_kernel.cpp
@@ -1,7 +1,7 @@
 // REQUIRES: linux
 
-// https://github.com/intel/llvm/issues/7634
 // UNSUPPORTED: hip
+// UNSUPPORTED-TRACKER: https://github.com/intel/llvm/issues/7634
 //
 // XFAIL: (opencl && gpu)
 // XFAIL-TRACKER: https://github.com/intel/llvm/issues/11364

diff --git a/sycl/test-e2e/Assert/assert_in_simultaneously_multiple_tus_one_ndebug.cpp b/sycl/test-e2e/Assert/assert_in_simultaneously_multiple_tus_one_ndebug.cpp
@@ -1,6 +1,6 @@
 // FIXME flaky fail on CUDA
-// FIXME HIP: https://github.com/intel/llvm/issues/7634
 // UNSUPPORTED: cuda, hip
+// UNSUPPORTED-TRACKER: https://github.com/intel/llvm/issues/7634
 //
 // XFAIL: (opencl && gpu)
 // XFAIL-TRACKER: https://github.com/intel/llvm/issues/11364

diff --git a/sycl/test-e2e/Basic/kernel_max_wg_size.cpp b/sycl/test-e2e/Basic/kernel_max_wg_size.cpp
@@ -5,6 +5,9 @@
 // https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/experimental/sycl_ext_intel_grf_size.asciidoc
 // REQUIRES: arch-intel_gpu_pvc || gpu-intel-dg2
 // UNSUPPORTED: cuda || hip
+// UNSUPPORTED-INTENDED: This extension is currently implemented in DPC++ only
+// for Intel GPU devices and only when using the Level Zero backend or OpenCL
+// backend.
 
 // clang-format off
 #include <sycl/detail/core.hpp>

diff --git a/sycl/test-e2e/README.md b/sycl/test-e2e/README.md
@@ -343,3 +343,29 @@ Once the issue is created, you can update the test by adding `XFAIL` and
 
 If you add `XFAIL` without `XFAIL-TRACKER` directive,
 `no-xfail-without-tracker.cpp` test will fail, notifying you about that.
+
+## Marking tests as unsupported
+
+Some tests may be considered unsupported, e.g.:
+* the test checks the feature that is not supported by some
+  backend / device / OS / etc.
+* the test is flaky or hangs, so it can't be marked with `XFAIL`.
+
+In these cases the test can be marked with `UNSUPPORTED`. This mark should be
+followed by either `UNSUPPORTED-INTENDED` or `UNSUPPORTED-TRACKER` depending on
+whether the test is not intended to be run with some feature at all or it was
+temporarily disabled due to some issue.
+```
+// UNSUPPORTED: cuda, hip
+// UNSUPPORTED-INTENDED: only supported by backends with SPIR-V IR
+
+// Sporadically fails on DG2.
+// UNSUPPORTED: gpu-intel-dg2
+// UNSUPPORTED-TRACKER: https://github.com/intel/llvm/issues/DDDDD
+// *OR*
+// UNSUPPORTED-TRACKER: PRJ-1234
+```
+
+If you add `UNSUPPORTED` without `UNSUPPORTED-TRACKER` or `UNSUPPORTED-INTENDED`
+directive, the `no-unsupported-without-tracker.cpp` test will fail, notifying
+you about that.
diff --git a/sycl/test/e2e_test_requirements/no-unsupported-without-info.cpp b/sycl/test/e2e_test_requirements/no-unsupported-without-info.cpp
diff --git a/sycl/test/e2e_test_requirements/no-xfail-without-tracker.cpp b/sycl/test/e2e_test_requirements/no-xfail-without-tracker.cpp
@@ -1,5 +1,6 @@
-// This test is intended to ensure that we have no trackers marked as XFAIL
+// This test is intended to ensure that we have no tests marked as XFAIL
 // without a tracker information added to a test.
+// For more info see: sycl/test-e2e/README.md
 //
 // The format we check is:
 // XFAIL: lit,features