Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Performance optmization for Purchase Invoice submission (backport #40263) #41946

Merged
merged 16 commits into from
Jun 27, 2024

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Jun 19, 2024

Performance optimization for Purchase Invoice submission and cancellation

This PR mainly tries to optimize the Purchase Invoice where there are a large number of service items in the invoice and provisional accounting is enabled.

Reported Usecase:
Number of service items in Purchase Invoice: 2750
Submit Time: ~10 mins
Cancel Time: > 20 mins (cancelled the job)

After Optimization:
Submit Time: ~2 mins
Cancel Time: ~30 secs


This is an automatic backport of pull request #40263 done by [Mergify](https://mergify.com).

nabinhait added 14 commits June 19, 2024 05:28
(cherry picked from commit d7b738f)

# Conflicts:
#	erpnext/accounts/doctype/purchase_invoice/purchase_invoice.py
(cherry picked from commit f204d81)

# Conflicts:
#	erpnext/accounts/doctype/budget/budget.py
(cherry picked from commit acc0b2f)
(cherry picked from commit e4bd173)

# Conflicts:
#	erpnext/accounts/doctype/accounting_dimension_filter/accounting_dimension_filter.py
(cherry picked from commit 5cd9bf3)
(cherry picked from commit 49c7436)

# Conflicts:
#	erpnext/accounts/doctype/purchase_invoice/purchase_invoice.py
(cherry picked from commit 6ff9e6e)

# Conflicts:
#	erpnext/stock/get_item_details.py
(cherry picked from commit d279e23)

# Conflicts:
#	erpnext/stock/doctype/item_price/item_price.json
(cherry picked from commit b07769d)

# Conflicts:
#	erpnext/accounts/doctype/gl_entry/gl_entry.py
Copy link
Contributor Author

mergify bot commented Jun 19, 2024

Cherry-pick of d7b738f has failed:

On branch mergify/bp/version-15-hotfix/pr-40263
Your branch is up to date with 'origin/version-15-hotfix'.

You are currently cherry-picking commit d7b738ff61.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   erpnext/accounts/doctype/purchase_invoice/purchase_invoice.py

no changes added to commit (use "git add" and/or "git commit -a")

Cherry-pick of f204d81 has failed:

On branch mergify/bp/version-15-hotfix/pr-40263
Your branch is ahead of 'origin/version-15-hotfix' by 1 commit.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit f204d810bb.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   erpnext/accounts/doctype/budget_account/budget_account.json

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   erpnext/accounts/doctype/budget/budget.py

Cherry-pick of e4bd173 has failed:

On branch mergify/bp/version-15-hotfix/pr-40263
Your branch is ahead of 'origin/version-15-hotfix' by 5 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit e4bd173875.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   erpnext/accounts/doctype/accounting_dimension_filter/accounting_dimension_filter.py

no changes added to commit (use "git add" and/or "git commit -a")

Cherry-pick of 49c7436 has failed:

On branch mergify/bp/version-15-hotfix/pr-40263
Your branch is ahead of 'origin/version-15-hotfix' by 8 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit 49c74369a5.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   erpnext/accounts/doctype/purchase_invoice/purchase_invoice.py

no changes added to commit (use "git add" and/or "git commit -a")

Cherry-pick of 6ff9e6e has failed:

On branch mergify/bp/version-15-hotfix/pr-40263
Your branch is ahead of 'origin/version-15-hotfix' by 10 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit 6ff9e6ee84.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   erpnext/stock/get_item_details.py

no changes added to commit (use "git add" and/or "git commit -a")

Cherry-pick of d279e23 has failed:

On branch mergify/bp/version-15-hotfix/pr-40263
Your branch is ahead of 'origin/version-15-hotfix' by 11 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit d279e23623.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   erpnext/stock/doctype/item_price/item_price.json

no changes added to commit (use "git add" and/or "git commit -a")

Cherry-pick of b07769d has failed:

On branch mergify/bp/version-15-hotfix/pr-40263
Your branch is ahead of 'origin/version-15-hotfix' by 13 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit b07769d8d7.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   erpnext/accounts/doctype/gl_entry/gl_entry.py

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@ruthra-kumar
Copy link
Member

@nabinhait
Can you check the code in purchase_invoice.py? I've resolved the conflicts as much as possible, but it would be better for you to check that it matches the changes done to that file in #40263

@nabinhait nabinhait removed the request for review from ruthra-kumar June 27, 2024 12:01
@nabinhait nabinhait merged commit d396c18 into version-15-hotfix Jun 27, 2024
10 of 11 checks passed
@nabinhait nabinhait deleted the mergify/bp/version-15-hotfix/pr-40263 branch June 27, 2024 12:02
@vorasmit
Copy link
Collaborator

vorasmit commented Jul 1, 2024

Issue:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/runner/frappe-bench/apps/india_compliance/india_compliance/gst_india/overrides/test_ineligible_itc.py", line 770, in test_purchase_receipt_and_then_purchase_invoice_for_provisional_expense
    doc.submit()
  File "/home/runner/frappe-bench/apps/frappe/frappe/utils/typing_validations.py", line 31, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/frappe-bench/apps/frappe/frappe/model/document.py", line 1048, in submit
    return self._submit()
           ^^^^^^^^^^^^^^
  File "/home/runner/frappe-bench/apps/frappe/frappe/model/document.py", line 1031, in _submit
    return self.save()
           ^^^^^^^^^^^
  File "/home/runner/frappe-bench/apps/frappe/frappe/model/document.py", line 337, in save
    return self._save(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/frappe-bench/apps/frappe/frappe/model/document.py", line 3[59](https://github.com/resilient-tech/india-compliance/actions/runs/9741427737/job/26880617909?pr=2320#step:12:60), in _save
    return self.insert()
           ^^^^^^^^^^^^^
  File "/home/runner/frappe-bench/apps/frappe/frappe/model/document.py", line 315, in insert
    self.run_post_save_methods()
  File "/home/runner/frappe-bench/apps/frappe/frappe/model/document.py", line 1131, in run_post_save_methods
    self.run_method("on_submit")
  File "/home/runner/frappe-bench/apps/frappe/frappe/model/document.py", line 962, in run_method
    out = Document.hook(fn)(self, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/frappe-bench/apps/frappe/frappe/model/document.py", line 1322, in composer
    return composed(self, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/frappe-bench/apps/frappe/frappe/model/document.py", line 1304, in runner
    add_to_return_value(self, fn(self, *args, **kwargs))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/frappe-bench/apps/frappe/frappe/model/document.py", line 959, in fn
    return method_object(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/frappe-bench/apps/erpnext/erpnext/accounts/doctype/purchase_invoice/purchase_invoice.py", line 770, in on_submit
    self.make_gl_entries()
  File "/home/runner/frappe-bench/apps/erpnext/erpnext/accounts/doctype/purchase_invoice/purchase_invoice.py", line 801, in make_gl_entries
    gl_entries = self.get_gl_entries()
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/frappe-bench/apps/erpnext/erpnext/accounts/doctype/purchase_invoice/purchase_invoice.py", line 8[61](https://github.com/resilient-tech/india-compliance/actions/runs/9741427737/job/26880617909?pr=2320#step:12:62), in get_gl_entries
    self.make_item_gl_entries(gl_entries)
  File "/home/runner/frappe-bench/apps/erpnext/erpnext/accounts/doctype/purchase_invoice/purchase_invoice.py", line 1101, in make_item_gl_entries
    self.make_provisional_gl_entry(gl_entries, item)
  File "/home/runner/frappe-bench/apps/erpnext/erpnext/accounts/doctype/purchase_invoice/purchase_invoice.py", line 1244, in make_provisional_gl_entry
    provisional_account,
    ^^^^^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'provisional_account' where it is not associated with a value

Reference: https://github.com/resilient-tech/india-compliance/actions/runs/9741427737/job/26880617909?pr=2320

frappe-pr-bot pushed a commit that referenced this pull request Jul 3, 2024
# [15.29.0](v15.28.2...v15.29.0) (2024-07-03)

### Bug Fixes

* add auto-update for overdue status ([#42105](#42105)) ([317cc03](317cc03))
* add string for translation (backport [#41903](#41903)) ([#41963](#41963)) ([48dc24b](48dc24b))
* always post to tax account heads if LCV is booked ([706a6c1](706a6c1))
* batch picking in pick list based on Stock Settings (backport [#42021](#42021)) ([#42134](#42134)) ([a45f8ca](a45f8ca))
* batch reset while making SABB (backport [#42076](#42076)) ([#42123](#42123)) ([c3f5a49](c3f5a49))
* decimal issue in pick list (backport [#41972](#41972)) ([#41982](#41982)) ([9945a90](9945a90))
* **Delivery Note:** only show permitted actions ([cef6d0d](cef6d0d))
* do not show zero balance stock items in stock balance report (backport [#41958](#41958)) ([#41961](#41961)) ([c10b123](c10b123))
* expense account from item group not fetched (backport [#41957](#41957)) ([#41962](#41962)) ([760b2e2](760b2e2))
* fixed asset value in Fixed Asset Register (backport [#41930](#41930)) ([#42027](#42027)) ([f2feeaf](f2feeaf))
* handle none type object error ([b0aef9e](b0aef9e))
* incorrect against_account upon reposting ([a41577a](a41577a))
* incorrect Difference Amount (backport [#42008](#42008)) ([#42013](#42013)) ([838cc5b](838cc5b))
* incorrect discount on other item ([77f4199](77f4199))
* incorrect dr/cr on Adv Payment against Journals ([4e74257](4e74257))
* incorrect time period in asset depreciation schedule (backport [#41805](#41805)) ([#42043](#42043)) ([cf4d4ba](cf4d4ba))
* lead status filter (backport [#41816](#41816)) ([#42046](#42046)) ([3536a75](3536a75))
* manufacturing date issue in the batch (backport [#42034](#42034)) ([#42037](#42037)) ([a981633](a981633))
* move condition for shipment ([2180239](2180239))
* not able to make purchase return (backport [#42053](#42053)) ([#42055](#42055)) ([8a91bf3](8a91bf3))
* pricing rule with and without 'apply multiple' and priority ([f3aa885](f3aa885))
* provisional entry for non stock items ([d61dab8](d61dab8))
* Re-open allows SO's to be over credit limit ([7fcb0f5](7fcb0f5))
* refactor Asset Repair and Stock Entry linkage to resolve amendme… (backport [#41919](#41919)) ([#42058](#42058)) ([97c49b9](97c49b9))
* reload asset when creating asset depreciation ([7b5d504](7b5d504))
* reposting file attachment permission issue (backport [#42068](#42068)) ([#42075](#42075)) ([1f3374f](1f3374f))
* resolve gl entries duplication in asset purchase workflow (backport [#41845](#41845)) ([#42120](#42120)) ([58e18e2](58e18e2))
* **Sales Order:** only show permitted actions ([a0011c5](a0011c5))
* show zero stock items filter in the stock balance report (backport [#42147](#42147)) ([#42152](#42152)) ([11ebbf2](11ebbf2))
* stock qty validation in SCR (backport [#42124](#42124)) ([#42133](#42133)) ([d9e62fe](d9e62fe))
* Stock Reservation Entry was not getting created (backport [#42033](#42033)) ([#42035](#42035)) ([e278fc6](e278fc6))
* **test:** incorrect field for customer default billing currency ([3b15708](3b15708))
* this.frm.events.update_cost is not a function (backport [#41960](#41960)) ([#41965](#41965)) ([3b4d397](3b4d397))
* timeout error while submitting JV (backport [#42040](#42040)) ([#42099](#42099)) ([a0e06a4](a0e06a4))
* timeout while cancelling LCV (backport [#42030](#42030)) (backport [#42031](#42031)) ([#42032](#42032)) ([068de08](068de08))
* unhide serial no field (backport [#42045](#42045)) ([#42047](#42047)) ([482832f](482832f))
* valuation rate for the legacy batches (backport [#42011](#42011)) ([#42020](#42020)) ([f6be19c](f6be19c))
* Wrong Delete Batch on Purchase Receipt (backport [#42007](#42007)) ([#42012](#42012)) ([68b318a](68b318a))

### Features

* accounting dimension filters in gp report ([fe9dffb](fe9dffb))
* default account head for operating cost (backport [#41985](#41985)) ([#41987](#41987)) ([44c1671](44c1671))
* **gp:** group by cost center ([068ae87](068ae87))
* Turkish Chart Of Accounts (backport [#41756](#41756)) ([#42028](#42028)) ([63b26e6](63b26e6))

### Performance Improvements

* code optimization to handle large asset creation (backport [#42018](#42018)) ([#42025](#42025)) ([c27f272](c27f272))
* dont run queries unnecessarily, improved filters ([#41993](#41993)) ([b59c91a](b59c91a))
* Performance optmization for Purchase Invoice submission (backport [#40263](#40263)) ([#41946](#41946)) ([d396c18](d396c18))
@frappe-pr-bot
Copy link
Collaborator

🎉 This issue has been resolved in version 15.29.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 19, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants