Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

label-select option hang when using multiple labels #3400

Closed
amittar-lb opened this issue Nov 14, 2022 · 4 comments · Fixed by #3427
Closed

label-select option hang when using multiple labels #3400

amittar-lb opened this issue Nov 14, 2022 · 4 comments · Fixed by #3427
Assignees
@mdellweg
Labels
Issue prio-list

Comments

@amittar-lb
Copy link

Summary

when trying to fetch file distribution list with multiple labels (6 labels and above), the requests reaches timeout

Steps to reproduce

pulp file distribution list --label-select 'build_time=jf0l5j,builder_hostname=3qshx3,git_ver=foycpk,image_roles=otx8tg,parent_image_checksum=2a0ge1,parent_image_url=i9lug6'

these are just mock keys and values but the logic is the same

Expected behavior

need to get response from the pulp server in a reasonable time

Stacktrace/Error log

client (cli):

time pulp file distribution list --label-select 'build_time=jf0l5j,builder_hostname=3qshx3,git_ver=foycpk,image_roles=otx8tg,parent_image_checksum=2a0ge1,parent_image_url=i9lug6'
Error: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>


real    1m0.440s
user    0m0.272s
sys     0m0.028s

server (journal):

Nov 14 14:02:05 pulp01 gunicorn[765]: [2022-11-14 14:02:05 +0000] [765] [CRITICAL] WORKER TIMEOUT (pid:23553) 
Nov 14 14:02:06 pulp01 gunicorn[765]: [2022-11-14 14:02:06 +0000] [765] [WARNING] Worker with pid 23553 was terminated due to signal 9 
Nov 14 14:02:06 pulp01 gunicorn[23578]: [2022-11-14 14:02:06 +0000] [23578] [INFO] Booting worker with pid: 23578

Pulp and pulp-cli version info

pulp3 command line interface, version 0.15.0
3.21.4

Additonal context

the issue does not reproduce when using only a few labels (until 5 labels)
@mdellweg mdellweg transferred this issue from pulp/pulp-cli Nov 14, 2022
@ggainey
Copy link
Contributor

ggainey commented Nov 15, 2022

"Worker with pid 23553 was terminated due to signal 9" - makes me think OOMKiller may have come to visit. Need to reproduce and investigate what's going on here.

@amittar-lb
Copy link
Author

@ggainey came across a critical issue.
When running a command with more than 5 labels: pulp file distribution list --label-select 'builder_hostname=personal,git_ver=b2253e0c-dirty,image_roles=target,distribution_family=redhat,distribution_major_version=8,distribution_version=8.7'
I can see the core CPU is maxed out to 100% and it stays that way when using SELECT on postgres.
image

When running the same command again, another CPU is maxed out.
image

The CPU is back to normal only when killing the pid

@mdellweg
Copy link
Member

I could reproduce querying 10 distributions with 8 labels took five seconds when asking for 7 labels and lead to a gateway timeout after 30 seconds when querying all 8. postgres seemed to have calmed down after a few minutes though.

@mdellweg mdellweg self-assigned this Nov 25, 2022
@pulpbot pulpbot moved this to In Progress in RH Pulp Kanban board Nov 25, 2022
@mdellweg
Copy link
Member

An experiment shows, that switching from a GenericRelation to pgsql HStore fields solves the performance issues even without trying to do anything clever with the filter.

@pulpbot pulpbot moved this from In Progress to Free to take in RH Pulp Kanban board Nov 28, 2022
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 28, 2022
@mdellweg
Reimplemented labels as HStore field
cc261a0 
This solves the imminent performance issues.

fixes pulp#3400
@mdellweg mdellweg mentioned this issue Nov 28, 2022
Merged
@pulpbot pulpbot moved this from Free to take to Needs review in RH Pulp Kanban board Nov 28, 2022
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 28, 2022
@mdellweg
Reimplemented labels as HStore field
55825c7 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 28, 2022
@mdellweg
Reimplemented labels as HStore field
d5e3267 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 28, 2022
@mdellweg
Reimplemented labels as HStore field
43ab11e 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 28, 2022
@mdellweg
Reimplemented labels as HStore field
bdf0302 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 28, 2022
@mdellweg
Reimplemented labels as HStore field
dfb7475 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 28, 2022
@mdellweg
Reimplemented labels as HStore field
d133a10 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 28, 2022
@mdellweg
Reimplemented labels as HStore field
7c85e08 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 28, 2022
@mdellweg
Reimplemented labels as HStore field
796a5fb 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 28, 2022
@mdellweg
Reimplemented labels as HStore field
6a1a733 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 29, 2022
@mdellweg
Reimplemented labels as HStore field
ba43632 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 30, 2022
@mdellweg
Reimplemented labels as HStore field
a5051ff 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 30, 2022
@mdellweg
Reimplemented labels as HStore field
97cd925 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Nov 30, 2022
@mdellweg
Reimplemented labels as HStore field
0506d08 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Dec 2, 2022
@mdellweg
Reimplemented labels as HStore field
7c64ba9 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Dec 4, 2022
@mdellweg
Reimplemented labels as HStore field
1cff650 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Dec 4, 2022
@mdellweg
Reimplemented labels as HStore field
26c4a22 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Dec 6, 2022
@mdellweg
Reimplemented labels as HStore field
bbadee8 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit to mdellweg/pulpcore that referenced this issue Dec 7, 2022
@mdellweg
Reimplemented labels as HStore field
a3b8161 
This solves the imminent performance issues.

fixes pulp#3400
mdellweg added a commit that referenced this issue Dec 13, 2022
@mdellweg
Reimplemented labels as HStore field
4e25949 
This solves the imminent performance issues.

fixes #3400
@pulpbot pulpbot moved this from Needs review to Done in RH Pulp Kanban board Dec 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mdellweg mdellweg

Labels
Issue prio-list
Projects
No open projects
RH Pulp Kanban board
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Reimplemented labels as HStore field mdellweg/pulpcore
3 participants
@ggainey @mdellweg @amittar-lb