Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DB queries >2K per view #471

Open
2 tasks
Tracked by #560
, #465 ,  #483 ,  #520 ,  #557
...
jon-nfc opened this issue Jan 14, 2025 · 3 comments
Open
2 tasks
Tracked by #560
, #465 ,  #483 ,  #520 ,  #557
...

DB queries >2K per view #471

jon-nfc opened this issue Jan 14, 2025 · 3 comments
Labels
task::refactor type::task Issue Type

Comments

@jon-nfc
Copy link
Member

jon-nfc commented Jan 14, 2025

As each view is run, authorization is conducted.

Details

Authorization is required, however when concurrent requests come in, the request time increases to beyond expectaations (>2s).

Screenshot from 2025-01-15 00-21-04

currently there are in excess of >2k database queries per view.

Tasks

  • Refactor DB queries to use prefetch_related as appropriate.

  • test amount of DB queries per view
    investigate if this can be tested
@jon-nfc jon-nfc added this to the Next Release milestone Jan 14, 2025
@jon-nfc jon-nfc moved this to Planning in Centurion ERP Jan 14, 2025
@jon-nfc jon-nfc self-assigned this Jan 16, 2025
@jon-nfc jon-nfc moved this from Planning to In progress in Centurion ERP Jan 16, 2025
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
refactor(access): prefetch team related fields
c40ecdd 
ref: #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
refactor(access): prefetch team related fields
a1f5f90 
ref: #471 #473
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
refactor(access): cache app settings
0d692c7 
ref: #469 #471
@jon-nfc jon-nfc mentioned this issue Jan 16, 2025
Closed
2 tasks
@jon-nfc
Copy link
Member Author

jon-nfc commented Jan 16, 2025

Post changes:

Image

  • left graph: pre changes
  • Right graph: post changes

Test ran was 1vu, 100x requests per url for each URL in alphabetical order.

jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
refactor(access): prefetch team related fields
033ac30 
ref: #471 #473
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
refactor(access): cache app settings
e4ecc07 
ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
test: Initial k6s individual page speed test cases
c1d9791 
ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
test: Initial k6s individual page speed test cases
c2ea176 
ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
test: Initial k6s individual page speed test cases
4314ec6 
ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
refactor(access): prefetch team related fields
5f5c911 
ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
refactor(access): cache app settings during perm check
3cee11a 
ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
refactor(access): cache app settings during model fetch
4e8bb88 
ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
refactor(access): prefetch team related fields
bd07c83 
ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
fix(access): cached orgs is an int list
fefde28 
ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
refactor(access): cache app settings during perm check
7f16a06 
ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
refactor(access): cache app settings during model fetch
9d776e1 
ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
chore(access): Add trace output for tests that pass locally and fail …
1b34bd7 
…on GH actions

ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
chore(access): Add trace output for tests that pass locally and fail …
76daec3 
…on GH actions

ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
chore(access): Add trace output for tests that pass locally and fail …
cb5eab3 
…on GH actions

ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
chore(access): Add trace output for tests that pass locally and fail …
e3b275c 
…on GH actions

ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
chore(access): Add trace output for tests that pass locally and fail …
e7f7800 
…on GH actions

ref: #469 #471
jon-nfc added a commit that referenced this issue Jan 16, 2025
@jon-nfc
docs(access): add page_speed test to contributing
aa69163 
ref: #469 #471
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
refactor(access): Permission Mixin has_permission to use view attribu…
9b7d7d3 
…te for object org

ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
refactor(core): When fetching a ticket, fetch related fields
17a2940 
ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
fix(core): ensure item_type exists before trying to get queryset
96565aa 
ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
feat(core): Cache ticket linked item queryset
e5b0ed4 
ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
refactor(access): Whilst attempting to get object org, check and use …
10b8b47 
…pk if exists

ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
refactor(access): Permission Mixin has_permission to use view attribu…
18437b3 
…te for object org

ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
chore(access): Remove extra calls to get_obj_permission from permis…
869f0d1 
…sion Mixin has_permission

ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
refactor(access): When fetching obj org, if pk exist attempt to fetch…
22b02dc 
… object

ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
chore(access): Remove extra calls to get_obj_permission from permis…
f9393a5 
…sion Mixin has_permission

ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
refactor(access): when checking obj permission use view cached obj or…
b6593c6 
…ganization

ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
fix(access): org mixin get_obj_org not to call get_object
a4788ab 
ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
test(access): during permission check function has_permission ensure …
5507c4a 
…`get_object` not called

ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
test(settings): regardless of permissions a user can change their own…
78afe7c 
… settings

ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
fix(access): when checking object permissions, dont cast obj to int u…
b7cdb21 
…ntill checking it exists

ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
test(settings): when checking if user can delete own settings, user m…
1ad370e 
…ust be owner of settings

ref: #471 #511
jon-nfc added a commit that referenced this issue Feb 1, 2025
@jon-nfc
fix(access): When conduting permission check for user settings, if us…
b419c03 
…er not owner of settings, deny access

ref: #471 #511
@jon-nfc
Copy link
Member Author

jon-nfc commented Feb 1, 2025

Image

graph of changes thus far.

peak at the start is the following views:

Name Last * Min Mean Max
v2:_api_v2_knowledge_base-list 248 ms 248 ms 248 ms 248 ms
v2:_api_v2_device-list 244 ms 244 ms 244 ms 245 ms

@jon-nfc jon-nfc mentioned this issue Jan 30, 2025
Closed
23 tasks
@jon-nfc jon-nfc mentioned this issue Feb 6, 2025
Closed
13 tasks
@jon-nfc jon-nfc moved this from In progress to Stale in Centurion ERP Feb 9, 2025
@jon-nfc jon-nfc removed their assignment Feb 9, 2025
@jon-nfc jon-nfc removed this from the Next Release milestone Feb 9, 2025
@jon-nfc jon-nfc removed the status in Centurion ERP Feb 9, 2025
@jon-nfc jon-nfc mentioned this issue Feb 10, 2025
Closed
21 tasks
@jon-nfc jon-nfc mentioned this issue Feb 10, 2025
Open
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
task::refactor type::task Issue Type
Projects
Centurion ERP
Status: No status
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jon-nfc