Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the yappi profiler for multithread profiling #6695

Closed
wants to merge 1 commit into from

Conversation

boxysean
Copy link
Contributor

@boxysean boxysean commented Jan 22, 2023

helps #6073 but does not resolve

Description

I am opening this PR for instructional purposes. Please advise if this is worthy change, or perhaps we just document how to profile with yappi.

Adds yappi for multi-threaded profiling. Warning: considerably slows down runtime, in my case 15 minute call turned into 3 hours.

(Alternative considered but not tried: could we avoid this change and just call dbt with yappi from the CLI ?)

Since dbt spawns 100+ threads (?!) this creates 100+ pstat files:

➜  pstat_models_7_with_change $ ls
thread-0.pstat        thread-116.pstat      thread-15.pstat       thread-33.pstat       thread-51.pstat       thread-7.pstat        thread-88.pstat
thread-1.pstat        thread-117.pstat      thread-16.pstat       thread-34.pstat       thread-52.pstat       thread-70.pstat       thread-89.pstat
thread-10.pstat       thread-118.pstat      thread-17.pstat       thread-35.pstat       thread-53.pstat       thread-71.pstat       thread-9.pstat
thread-100.pstat      thread-119.pstat      thread-18.pstat       thread-36.pstat       thread-54.pstat       thread-72.pstat       thread-90.pstat
thread-101.pstat      thread-12.pstat       thread-19.pstat       thread-37.pstat       thread-55.pstat       thread-73.pstat       thread-91.pstat
thread-102.pstat      thread-120.pstat      thread-2.pstat        thread-38.pstat       thread-56.pstat       thread-74.pstat       thread-92.pstat
thread-103.pstat      thread-121.pstat      thread-20.pstat       thread-39.pstat       thread-57.pstat       thread-75.pstat       thread-93.pstat
thread-104.pstat      thread-122.pstat      thread-21.pstat       thread-4.pstat        thread-58.pstat       thread-76.pstat       thread-94.pstat
thread-105.pstat      thread-123.pstat      thread-22.pstat       thread-40.pstat       thread-59.pstat       thread-77.pstat       thread-95.pstat
thread-106.pstat      thread-124.pstat      thread-23.pstat       thread-41.pstat       thread-6.pstat        thread-78.pstat       thread-96.pstat
thread-107.pstat      thread-125.pstat      thread-24.pstat       thread-42.pstat       thread-60.pstat       thread-79.pstat       thread-97.pstat
thread-108.pstat      thread-126.pstat      thread-25.pstat       thread-43.pstat       thread-61.pstat       thread-8.pstat        thread-98.pstat
thread-109.pstat      thread-127.pstat      thread-26.pstat       thread-44.pstat       thread-62.pstat       thread-80.pstat       thread-99.pstat
thread-11.pstat       thread-128.pstat      thread-27.pstat       thread-45.pstat       thread-63.pstat       thread-81.pstat       thread-combined.pstat
thread-110.pstat      thread-129.pstat      thread-28.pstat       thread-46.pstat       thread-64.pstat       thread-82.pstat
thread-111.pstat      thread-13.pstat       thread-29.pstat       thread-47.pstat       thread-65.pstat       thread-83.pstat
thread-112.pstat      thread-130.pstat      thread-3.pstat        thread-48.pstat       thread-66.pstat       thread-84.pstat
thread-113.pstat      thread-131.pstat      thread-30.pstat       thread-49.pstat       thread-67.pstat       thread-85.pstat
thread-114.pstat      thread-132.pstat      thread-31.pstat       thread-5.pstat        thread-68.pstat       thread-86.pstat
thread-115.pstat      thread-14.pstat       thread-32.pstat       thread-50.pstat       thread-69.pstat       thread-87.pstat

Checklist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant