Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deadlock in trace SDK shutdown #3890

Closed
jmacd opened this issue Mar 15, 2023 · 1 comment · Fixed by #3924
Closed

Deadlock in trace SDK shutdown #3890

jmacd opened this issue Mar 15, 2023 · 1 comment · Fixed by #3924
Labels
bug Something isn't working
Milestone

Comments

@jmacd
Copy link
Contributor

jmacd commented Mar 15, 2023

Description

Updated to v1.13.x and noticed SDKs deadlocked in shutdown in some of our applications.
Some of our applications call TracerProvider.Tracer() in a gRPC interceptor to trace the RPC.
Our interceptors have been applied to the OTel trace SDK (by mistake, I'd say, but FWIW this worked before 1.13.x)
The interceptors are now deadlocking shutdown because Shutdown() holds the lock, preventing Tracer() from being called.

Environment

  • OS: macOS
  • Architecture: ARM
  • Go Version: 1.19
  • opentelemetry-go version: 1.13.0

Steps To Reproduce

Call Tracer() from an interceptor from an ExportSpans() call during Shutdown() and it will deadlock.

Shutdown() (takes lock) -> calls final ExportSpans() -> calls interceptor -> calls Tracer() (tries to get lock) == deadlocked

This appears to be introduced by #3551.

Expected behavior

The Shutdown should proceed. It looks safe to release the lock, meaning to replace the

	p.mu.Lock()
	defer p.mu.Unlock()
	p.isShutdown = true

with something equivalent to

	p.mu.Lock()
	if p.isShutdown {
	    p.mu.Unlock()
            return
        }
        p.isShutdown = true
	p.mu.Unlock()
@jmacd jmacd added the bug Something isn't working label Mar 15, 2023
@dmathieu
Copy link
Member

It sounds like #3845 may be fixing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants