-
Notifications
You must be signed in to change notification settings - Fork 12.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Clang] Add env var for nvptx-arch/amdgpu-arch timeout #102521
Conversation
When working on very busy systems, check-offload frequently fails many tests with this diagnostic: ``` clang: error: cannot determine amdgcn architecture: /tmp/llvm/build/bin/amdgpu-arch: Child timed out: ; consider passing it via '-march' ``` The timeout is 10 seconds. This patch accepts the environment variable `CLANG_TOOL_CHAIN_PROGRAM_WAIT` to increase it. It should be documented somewhere. Any suggestions on where?
@llvm/pr-subscribers-clang-driver @llvm/pr-subscribers-backend-amdgpu Author: Joel E. Denny (jdenny-ornl) ChangesWhen working on very busy systems, check-offload frequently fails many tests with this diagnostic:
The timeout is 10 seconds. This patch accepts the environment variable It should be documented somewhere. Any suggestions on where? Full diff: https://github.com/llvm/llvm-project/pull/102521.diff 4 Files Affected:
diff --git a/clang/lib/Driver/ToolChain.cpp b/clang/lib/Driver/ToolChain.cpp
index 2d50c2cbbc881c..04b281e1bb10cd 100644
--- a/clang/lib/Driver/ToolChain.cpp
+++ b/clang/lib/Driver/ToolChain.cpp
@@ -40,6 +40,7 @@
#include "llvm/Support/FileSystem.h"
#include "llvm/Support/FileUtilities.h"
#include "llvm/Support/Path.h"
+#include "llvm/Support/Process.h"
#include "llvm/Support/VersionTuple.h"
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/TargetParser/AArch64TargetParser.h"
@@ -105,7 +106,7 @@ ToolChain::ToolChain(const Driver &D, const llvm::Triple &T,
llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
ToolChain::executeToolChainProgram(StringRef Executable,
- unsigned SecondsToWait) const {
+ unsigned DefaultSecondsToWait) const {
llvm::SmallString<64> OutputFile;
llvm::sys::fs::createTemporaryFile("toolchain-program", "txt", OutputFile);
llvm::FileRemover OutputRemover(OutputFile.c_str());
@@ -116,6 +117,13 @@ ToolChain::executeToolChainProgram(StringRef Executable,
};
std::string ErrorMessage;
+ int SecondsToWait = DefaultSecondsToWait;
+ if (std::optional<std::string> Str =
+ llvm::sys::Process::GetEnv("CLANG_TOOL_CHAIN_PROGRAM_WAIT")) {
+ int Val = std::atoi(Str->c_str());
+ if (Val > 0)
+ SecondsToWait = Val;
+ }
if (llvm::sys::ExecuteAndWait(Executable, {}, {}, Redirects, SecondsToWait,
/*MemoryLimit=*/0, &ErrorMessage))
return llvm::createStringError(std::error_code(),
diff --git a/clang/lib/Driver/ToolChains/AMDGPU.cpp b/clang/lib/Driver/ToolChains/AMDGPU.cpp
index aa8f9197cfabc3..4ed366d21f5c43 100644
--- a/clang/lib/Driver/ToolChains/AMDGPU.cpp
+++ b/clang/lib/Driver/ToolChains/AMDGPU.cpp
@@ -899,7 +899,8 @@ AMDGPUToolChain::getSystemGPUArchs(const ArgList &Args) const {
else
Program = GetProgramPath("amdgpu-arch");
- auto StdoutOrErr = executeToolChainProgram(Program, /*SecondsToWait=*/10);
+ auto StdoutOrErr = executeToolChainProgram(Program,
+ /*DefaultSecondsToWait=*/10);
if (!StdoutOrErr)
return StdoutOrErr.takeError();
diff --git a/clang/lib/Driver/ToolChains/Cuda.cpp b/clang/lib/Driver/ToolChains/Cuda.cpp
index 17c952c808f725..104217eaf5d849 100644
--- a/clang/lib/Driver/ToolChains/Cuda.cpp
+++ b/clang/lib/Driver/ToolChains/Cuda.cpp
@@ -804,7 +804,8 @@ NVPTXToolChain::getSystemGPUArchs(const ArgList &Args) const {
else
Program = GetProgramPath("nvptx-arch");
- auto StdoutOrErr = executeToolChainProgram(Program, /*SecondsToWait=*/10);
+ auto StdoutOrErr = executeToolChainProgram(Program,
+ /*DefaultSecondsToWait=*/10);
if (!StdoutOrErr)
return StdoutOrErr.takeError();
diff --git a/llvm/utils/lit/lit/TestingConfig.py b/llvm/utils/lit/lit/TestingConfig.py
index eb9f8de2a7f960..06713429d06b4b 100644
--- a/llvm/utils/lit/lit/TestingConfig.py
+++ b/llvm/utils/lit/lit/TestingConfig.py
@@ -26,6 +26,7 @@ def fromdefaults(litConfig):
"SYSTEMROOT",
"TERM",
"CLANG",
+ "CLANG_TOOL_CHAIN_PROGRAM_WAIT",
"LLDB",
"LD_PRELOAD",
"LLVM_SYMBOLIZER_PATH",
|
@llvm/pr-subscribers-clang Author: Joel E. Denny (jdenny-ornl) ChangesWhen working on very busy systems, check-offload frequently fails many tests with this diagnostic:
The timeout is 10 seconds. This patch accepts the environment variable It should be documented somewhere. Any suggestions on where? Full diff: https://github.com/llvm/llvm-project/pull/102521.diff 4 Files Affected:
diff --git a/clang/lib/Driver/ToolChain.cpp b/clang/lib/Driver/ToolChain.cpp
index 2d50c2cbbc881c..04b281e1bb10cd 100644
--- a/clang/lib/Driver/ToolChain.cpp
+++ b/clang/lib/Driver/ToolChain.cpp
@@ -40,6 +40,7 @@
#include "llvm/Support/FileSystem.h"
#include "llvm/Support/FileUtilities.h"
#include "llvm/Support/Path.h"
+#include "llvm/Support/Process.h"
#include "llvm/Support/VersionTuple.h"
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/TargetParser/AArch64TargetParser.h"
@@ -105,7 +106,7 @@ ToolChain::ToolChain(const Driver &D, const llvm::Triple &T,
llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
ToolChain::executeToolChainProgram(StringRef Executable,
- unsigned SecondsToWait) const {
+ unsigned DefaultSecondsToWait) const {
llvm::SmallString<64> OutputFile;
llvm::sys::fs::createTemporaryFile("toolchain-program", "txt", OutputFile);
llvm::FileRemover OutputRemover(OutputFile.c_str());
@@ -116,6 +117,13 @@ ToolChain::executeToolChainProgram(StringRef Executable,
};
std::string ErrorMessage;
+ int SecondsToWait = DefaultSecondsToWait;
+ if (std::optional<std::string> Str =
+ llvm::sys::Process::GetEnv("CLANG_TOOL_CHAIN_PROGRAM_WAIT")) {
+ int Val = std::atoi(Str->c_str());
+ if (Val > 0)
+ SecondsToWait = Val;
+ }
if (llvm::sys::ExecuteAndWait(Executable, {}, {}, Redirects, SecondsToWait,
/*MemoryLimit=*/0, &ErrorMessage))
return llvm::createStringError(std::error_code(),
diff --git a/clang/lib/Driver/ToolChains/AMDGPU.cpp b/clang/lib/Driver/ToolChains/AMDGPU.cpp
index aa8f9197cfabc3..4ed366d21f5c43 100644
--- a/clang/lib/Driver/ToolChains/AMDGPU.cpp
+++ b/clang/lib/Driver/ToolChains/AMDGPU.cpp
@@ -899,7 +899,8 @@ AMDGPUToolChain::getSystemGPUArchs(const ArgList &Args) const {
else
Program = GetProgramPath("amdgpu-arch");
- auto StdoutOrErr = executeToolChainProgram(Program, /*SecondsToWait=*/10);
+ auto StdoutOrErr = executeToolChainProgram(Program,
+ /*DefaultSecondsToWait=*/10);
if (!StdoutOrErr)
return StdoutOrErr.takeError();
diff --git a/clang/lib/Driver/ToolChains/Cuda.cpp b/clang/lib/Driver/ToolChains/Cuda.cpp
index 17c952c808f725..104217eaf5d849 100644
--- a/clang/lib/Driver/ToolChains/Cuda.cpp
+++ b/clang/lib/Driver/ToolChains/Cuda.cpp
@@ -804,7 +804,8 @@ NVPTXToolChain::getSystemGPUArchs(const ArgList &Args) const {
else
Program = GetProgramPath("nvptx-arch");
- auto StdoutOrErr = executeToolChainProgram(Program, /*SecondsToWait=*/10);
+ auto StdoutOrErr = executeToolChainProgram(Program,
+ /*DefaultSecondsToWait=*/10);
if (!StdoutOrErr)
return StdoutOrErr.takeError();
diff --git a/llvm/utils/lit/lit/TestingConfig.py b/llvm/utils/lit/lit/TestingConfig.py
index eb9f8de2a7f960..06713429d06b4b 100644
--- a/llvm/utils/lit/lit/TestingConfig.py
+++ b/llvm/utils/lit/lit/TestingConfig.py
@@ -26,6 +26,7 @@ def fromdefaults(litConfig):
"SYSTEMROOT",
"TERM",
"CLANG",
+ "CLANG_TOOL_CHAIN_PROGRAM_WAIT",
"LLDB",
"LD_PRELOAD",
"LLVM_SYMBOLIZER_PATH",
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we should bump the default time up? I don't know what's reasonable in this area, but the GPU drivers have a bad habit of deadlocking, so this is to prevent that. I think you could document the env var in the error message itself, and -1
should mean effectively infinite.
I don't know the right value either. If no one else is complaining, maybe 10s is fine under normal circumstances.
Sure, I'll work on that. Maybe <=0 should be no timeout? |
Work for me. |
Also, just be aware that if you set this, it will apply to everything. So if you had a particularly long link job, probably would be a good idea to make it wait forever. |
Not sure I understand. grep found 2 uses of executeToolChainProgram: calling amdgpu-arch and nvptx-arch. Did I miss a use? |
Oh yeah, totally forgot these were separate from the commands, disregard. |
FWIW, on one of my test systems that's currently very busy, check-offload consistently fails with amdgpu-arch timeouts. However, the following so far makes it consistently succeed:
Previously, I've decreased lit parallelism to avoid the timeouts, but it hasn't worked consistently, and it slows down the test suite. |
Possibly worth increasing it to a minute then, this is basically just a last ditch effort to prevent builds from hanging forever. I've had it happen personally where either the CUDA or AMDGPU drivers were stuck, and when I tried to rebuild |
Should we also land the env var? If so, I'll just extend this patch. Either way, any objection to moving
I'm not sure how often the situation you describe occurs in practice. Hopefully a 60-second pause on a simple compilation doesn't prompt the same confusion and a premature Ctrl+C. |
You can just do both in the same patch, I don't think it's too important. |
Thanks for the review. |
* 'main' of https://github.com/llvm/llvm-project: (700 commits) [SandboxIR][NFC] SingleLLVMInstructionImpl class (llvm#102687) [ThinLTO]Clean up 'import-assume-unique-local' flag. (llvm#102424) [nsan] Make #include more conventional [SandboxIR][NFC] Use Tracker.emplaceIfTracking() [libc] Moved range_reduction_double ifdef statement (llvm#102659) [libc] Fix CFP long double and add tests (llvm#102660) [TargetLowering] Handle vector types in expandFixedPointMul (llvm#102635) [compiler-rt][NFC] Replace environment variable with %t (llvm#102197) [UnitTests] Convert a test to use opaque pointers (llvm#102668) [CodeGen][NFCI] Don't re-implement parts of ASTContext::getIntWidth (llvm#101765) [SandboxIR] Clean up tracking code with the help of emplaceIfTracking() (llvm#102406) [mlir][bazel] remove extra blanks in mlir-tblgen test [NVPTX][NFC] Update tests to use bfloat type (llvm#101493) [mlir] Add support for parsing nested PassPipelineOptions (llvm#101118) [mlir][bazel] add missing td dependency in mlir-tblgen test [flang][cuda] Fix lib dependency [libc] Clean up remaining use of *_WIDTH macros in printf (llvm#102679) [flang][cuda] Convert cuf.alloc for box to fir.alloca in device context (llvm#102662) [SandboxIR] Implement the InsertElementInst class (llvm#102404) [libc] Fix use of cpp::numeric_limits<...>::digits (llvm#102674) [mlir][ODS] Verify type constraints in Types and Attributes (llvm#102326) [LTO] enable `ObjCARCContractPass` only on optimized build (llvm#101114) [mlir][ODS] Consistent `cppType` / `cppClassName` usage (llvm#102657) [lldb] Move definition of SBSaveCoreOptions dtor out of header (llvm#102539) [libc] Use cpp::numeric_limits in preference to C23 <limits.h> macros (llvm#102665) [clang] Implement -fptrauth-auth-traps. (llvm#102417) [LLVM][rtsan] rtsan transform to preserve CFGAnalyses (llvm#102651) Revert "[AMDGPU] Move `AMDGPUAttributorPass` to full LTO post link stage (llvm#102086)" [RISCV][GISel] Add missing tests for G_CTLZ/CTTZ instruction selection. NFC Return available function types for BindingDecls. (llvm#102196) [clang] Wire -fptrauth-returns to "ptrauth-returns" fn attribute. (llvm#102416) [RISCV] Remove riscv-experimental-rv64-legal-i32. (llvm#102509) [RISCV] Move PseudoVSET(I)VLI expansion to use PseudoInstExpansion. (llvm#102496) [NVPTX] support switch statement with brx.idx (reland) (llvm#102550) [libc][newhdrgen]sorted function names in yaml (llvm#102544) [GlobalIsel] Combine G_ADD and G_SUB with constants (llvm#97771) Suppress spurious warnings due to R_RISCV_SET_ULEB128 [scudo] Separated committed and decommitted entries. (llvm#101409) [MIPS] Fix missing ANDI optimization (llvm#97689) [Clang] Add env var for nvptx-arch/amdgpu-arch timeout (llvm#102521) [asan] Switch allocator to dynamic base address (llvm#98511) [AMDGPU] Move `AMDGPUAttributorPass` to full LTO post link stage (llvm#102086) [libc][math][c23] Add fadd{l,f128} C23 math functions (llvm#102531) [mlir][bazel] revert bazel rule change for DLTITransformOps [msan] Support vst{2,3,4}_lane instructions (llvm#101215) Revert "[MLIR][DLTI][Transform] Introduce transform.dlti.query (llvm#101561)" [X86] pr57673.ll - generate MIR test checks [mlir][vector][test] Split tests from vector-transfer-flatten.mlir (llvm#102584) [mlir][bazel] add bazel rule for DLTITransformOps OpenMPOpt: Remove dead include [IR] Add method to GlobalVariable to change type of initializer. (llvm#102553) [flang][cuda] Force default allocator in device code (llvm#102238) [llvm] Construct SmallVector<SDValue> with ArrayRef (NFC) (llvm#102578) [MLIR][DLTI][Transform] Introduce transform.dlti.query (llvm#101561) [AMDGPU][AsmParser][NFC] Remove a misleading comment. (llvm#102604) [Arm][AArch64][Clang] Respect function's branch protection attributes. (llvm#101978) [mlir] Verifier: steal bit to track seen instead of set. (llvm#102626) [Clang] Fix Handling of Init Capture with Parameter Packs in LambdaScopeForCallOperatorInstantiationRAII (llvm#100766) [X86] Convert truncsat clamping patterns to use SDPatternMatch. NFC. [gn] Give two scripts argparse.RawDescriptionHelpFormatter [bazel] Add missing dep for the SPIRVToLLVM target [Clang] Simplify specifying passes via -Xoffload-linker (llvm#102483) [bazel] Port for d45de80 [SelectionDAG] Use unaligned store/load to move AVX registers onto stack for `insertelement` (llvm#82130) [Clang][OMPX] Add the code generation for multi-dim `num_teams` (llvm#101407) [ARM] Regenerate big-endian-vmov.ll. NFC [AMDGPU][AsmParser][NFCI] All NamedIntOperands to be of the i32 type. (llvm#102616) [libc][math][c23] Add totalorderl function. (llvm#102564) [mlir][spirv] Support `memref` in `convert-to-spirv` pass (llvm#102534) [MLIR][GPU-LLVM] Convert `gpu.func` to `llvm.func` (llvm#101664) Fix a unit test input file (llvm#102567) [llvm-readobj][COFF] Dump hybrid objects for ARM64X files. (llvm#102245) AMDGPU/NewPM: Port SIFixSGPRCopies to new pass manager (llvm#102614) [MemoryBuiltins] Simplify getCalledFunction() helper (NFC) [AArch64] Add invalid 1 x vscale costs for reductions and reduction-operations. (llvm#102105) [MemoryBuiltins] Handle allocator attributes on call-site LSV/test/AArch64: add missing lit.local.cfg; fix build (llvm#102607) Revert "Enable logf128 constant folding for hosts with 128bit floats (llvm#96287)" [RISCV] Add Syntacore SCR5 RV32/64 processors definition (llvm#102285) [InstCombine] Remove unnecessary RUN line from test (NFC) [flang][OpenMP] Handle multiple ranges in `num_teams` clause (llvm#102535) [mlir][vector] Add tests for scalable vectors in one-shot-bufferize.mlir (llvm#102361) [mlir][vector] Disable `vector.matrix_multiply` for scalable vectors (llvm#102573) [clang] Implement CWG2627 Bit-fields and narrowing conversions (llvm#78112) [NFC] Use references to avoid copying (llvm#99863) Revert "[mlir][ArmSME] Pattern to swap shape_cast(tranpose) with transpose(shape_cast) (llvm#100731)" (llvm#102457) [IRBuilder] Generate nuw GEPs for struct member accesses (llvm#99538) [bazel] Port for 9b06e25 [CodeGen][NewPM] Improve start/stop pass error message CodeGenPassBuilder (llvm#102591) [AArch64] Implement TRBMPAM_EL1 system register (llvm#102485) [InstCombine] Fixing wrong select folding in vectors with undef elements (llvm#102244) [AArch64] Sink operands to fmuladd. (llvm#102297) LSV: document hang reported in llvm#37865 (llvm#102479) Enable logf128 constant folding for hosts with 128bit floats (llvm#96287) [RISCV][clang] Remove bfloat base type in non-zvfbfmin vcreate (llvm#102146) [RISCV][clang] Add missing `zvfbfmin` to `vget_v` intrinsic (llvm#102149) [mlir][vector] Add mask elimination transform (llvm#99314) [Clang][Interp] Fix display of syntactically-invalid note for member function calls (llvm#102170) [bazel] Port for 3fffa6d [DebugInfo][RemoveDIs] Use iterator-inserters in clang (llvm#102006) ... Signed-off-by: Edwiin Kusuma Jaya <kutemeikito0905@gmail.com>
PR #102521, which landed as 1ea0865, implemented `CLANG_TOOLCHAIN_PROGRAM_TIMEOUT`, but the logic is obviously wrong. If the user-specified value is negative, it should become zero to mean infinite. Otherwise, it should be left as is. Thus, use `std::max` not `std::min`. This obvious fixup doesn't seem worth another pull request.
"CLANG_TOOLCHAIN_PROGRAM_TIMEOUT expected " | ||
"an integer, got '" + | ||
*Str + "'"); | ||
SecondsToWait = std::min(SecondsToWait, 0); // infinite |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So that negative becomes 0 to mean infinite, that's obviously supposed to be max not min. Pushed a fix as 7c4eb60.
When working on very busy systems, check-offload frequently fails many tests with this diagnostic:
This patch accepts the environment variable
CLANG_TOOLCHAIN_PROGRAM_TIMEOUT
to set the timeout. It also increases the timeout from 10 to 60 seconds.