Fix memory leaks and update PDF/A algorithm for non-document products #845
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
February 26, 2024 13:20
There were many. First, it turns out that hashing on a URL or URI is a bad idea. When doing profiling, the tables that hashed on these objects were tremendously big (thousands). Changeing them hash (key) on .toString() reduced the tables to 12 items. Seemed more reasonable. Second problem was the schematron were holding their past information in memory and then recycling them. resulted in 10s of MB per label processed being stored in each of the schematron validators. Doing a reset() on the validator when doen, cleared that problem. Third, the schematron validators were holding onto document frangments that translated into 1 MB per label (give or take as it dependend on the label size). To get rid of these, need to clean up (garbage collect) the schematron validators. This is going to cost in CPU cycles per label because the validator has to be created from scratch each time. However, long jobs will not fail because of out of memory. As part of (3), changed from caching the transform to caching the actual document. Hence it is loaded once from file or network and then used repeatedly to create new schematron validators. In this way, bet benefit of not reloadeing a million times, but pay to create validators. Not too much slower. Doing about 300 labels every 15 minutes currently. Think it was faster yesterday at maybe 400 every 15 minutes but not positive. Never wrote it down.
|
Ooops. Wrong branch for this work... I will add PDF when I have time. |
Quick check that container label is not Product_Document then press onward.
|
Ready for prime time. |
jordanpadams
approved these changes
Mar 2, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🗒️ Summary
Removed some caching that was causing excessive memory consumption. Also only do PDF valid for Product_Document.
⚙️ Test Data and/or Report
All tests pass. Previous behavior was 10s of MB per label. Current behavior from profiler is stable at 50 MB independent of number of labels.
♻️ Related Issues
Closes #824
Closes #826