Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Launching a job with a large inventory is slow #3214

Closed
ryanpetrello opened this issue Feb 12, 2019 · 3 comments
Closed

Launching a job with a large inventory is slow #3214

ryanpetrello opened this issue Feb 12, 2019 · 3 comments
Labels
component:api type:bug

Comments

@ryanpetrello
Copy link
Contributor

ryanpetrello commented Feb 12, 2019 •
edited
Loading

ISSUE TYPE
  • Bug Report
COMPONENT NAME
  • API
SUMMARY
  1. Create an inventory w/ 10k hosts:
~ awx-manage shell_plus
v = 'ansible_connection: local'
for i in range(10000): Host(inventory_id=1, name='Host {}'.format(i), variables=v).save()
  1. Launch a simple (fast) job pointed at that inventory with a host limit set to one host (e.g., localhost)
  2. The job takes up to a minute to start printing stdout.

image

giphy

ENVIRONMENT
  • AWX version: 3.3.0+
  • AWX install method: any
  • Ansible version: any
@ryanpetrello
Copy link
Contributor Author

Before optimization:
image

After optimization:
image

ryanpetrello added a commit to ryanpetrello/awx that referenced this issue Feb 12, 2019
@ryanpetrello
optimize a slow query in inventory script generation
2a84350 
if we don't preload this column, Django needs it, and so it generates
one query per-host (!!!) to get it.  For large (10k+ host) inventories,
this is incredibly slow.

see: ansible#3214
@ryanpetrello ryanpetrello mentioned this issue Feb 12, 2019
Merged
ryanpetrello added a commit to ryanpetrello/awx that referenced this issue Feb 12, 2019
@ryanpetrello
optimize a slow query in inventory script generation
e245e50 
if we don't preload this column, Django needs it, and so it generates
one query per-host (!!!) to get it.  For large (10k+ host) inventories,
this is incredibly slow.

see: ansible#3214
e-tienne pushed a commit to e-tienne/awx that referenced this issue Feb 14, 2019
@ryanpetrello
optimize a slow query in inventory script generation
8719158 
if we don't preload this column, Django needs it, and so it generates
one query per-host (!!!) to get it.  For large (10k+ host) inventories,
this is incredibly slow.

see: ansible#3214
@ryanpetrello ryanpetrello mentioned this issue May 9, 2019
Closed
@coudenysj coudenysj mentioned this issue May 21, 2019
Closed
@towolf
Copy link

towolf commented Aug 12, 2019 •
edited
Loading

@ryanpetrello did you by any chance fix this for groups as well?

We have 10,000 groups ... :(

... and 3000 hosts ...

2019-08-12 19:03:20 CEST [5380-11052] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10822
2019-08-12 19:03:20 CEST [5380-11053] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10823
2019-08-12 19:03:20 CEST [5380-11054] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10820
2019-08-12 19:03:20 CEST [5380-11055] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10824
2019-08-12 19:03:20 CEST [5380-11056] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10825
2019-08-12 19:03:20 CEST [5380-11057] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10826
2019-08-12 19:03:20 CEST [5380-11058] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10827
2019-08-12 19:03:20 CEST [5380-11059] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10828
2019-08-12 19:03:20 CEST [5380-11060] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10829
2019-08-12 19:03:20 CEST [5380-11061] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10830
2019-08-12 19:03:20 CEST [5380-11062] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10831
2019-08-12 19:03:20 CEST [5380-11063] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10832
2019-08-12 19:03:20 CEST [5380-11064] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10834
2019-08-12 19:03:20 CEST [5380-11065] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10835
2019-08-12 19:03:20 CEST [5380-11066] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10836
2019-08-12 19:03:20 CEST [5380-11067] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10833
2019-08-12 19:03:20 CEST [5380-11068] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10837
2019-08-12 19:03:20 CEST [5380-11069] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10838
2019-08-12 19:03:20 CEST [5380-11070] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10839
2019-08-12 19:03:20 CEST [5380-11071] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10840
2019-08-12 19:03:20 CEST [5380-11072] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10841
2019-08-12 19:03:20 CEST [5380-11073] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10842
2019-08-12 19:03:20 CEST [5380-11074] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10843
2019-08-12 19:03:20 CEST [5380-11075] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10844
2019-08-12 19:03:20 CEST [5380-11076] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10845
2019-08-12 19:03:20 CEST [5380-11077] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10846
2019-08-12 19:03:20 CEST [5380-11078] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10847
2019-08-12 19:03:20 CEST [5380-11079] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10848
2019-08-12 19:03:20 CEST [5380-11080] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10849
2019-08-12 19:03:20 CEST [5380-11081] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10850
2019-08-12 19:03:20 CEST [5380-11082] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10851
2019-08-12 19:03:20 CEST [5380-11083] ansible_awx@awx LOG:  statement: SELECT "main_group"."id", "main_group"."inventory_id" FROM "main_group" WHERE "main_group"."id" = 10852

@towolf towolf mentioned this issue Aug 12, 2019
Closed
@towolf
Copy link

towolf commented Aug 12, 2019 •
edited
Loading

#4461

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component:api type:bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ryanpetrello @towolf @awxbot