Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elasticsuite_index_log_event cronjob could not handle huge DB data due to stop of execution #1110

Closed
rbayet opened this issue Oct 3, 2018 · 1 comment
Assignees

Comments

@rbayet
Copy link
Collaborator

rbayet commented Oct 3, 2018

It's more a performance enhancement more than a bug, because it's highly circumstantial, but the cronjob elasticsuite_index_log_event loads ALL records at once from DB table elasticsuite_tracker_log_event each time it is executed.

On an environment where somehow, for whatever reason, that cronjob has been locked as "running" for a few weeks (or months) and if the site in question has even a moderate traffic, there may be a situation where the cronjob fails systematically due to an OOM error or a max_execution_time.
In such a case, events logged in the DB might be indexed into ElasticSearch, but not removed from the DB.

Preconditions

Magento Version : EE 2.2.2

ElasticSuite Version : 2.5.12

Environment : Production

Third party modules : lots, none related to search or cronjob (except Smile_Cron).

Steps to reproduce

  1. Disable crons on a Magento installation
  2. Run a tool performing a high load on the Magento installation on many pages, generating a lot of records ( > 500K ) in the table elasticsuite_tracker_log_event
  3. Enable back Magento cronjobs

Expected result

  1. (Possiblity in several - or a lot - executions) the cronjob should be able to load, index and delete the events from DB by using a (config defined ?) hard-capped limit used for \Smile\ElasticsuiteTracker\Model\EventQueue::getEvents
  2. If the cronjob becomes locked, unlocking him should ensure that at least some of the older events have already been indexed and purged from DB

Actual result

  1. Not parameter is given to \Smile\ElasticsuiteTracker\Model\EventQueue::getEvents
  2. The cronjob loads all the events from DB before trying to index them and delete them
  3. If the cronjob comes locked, unlocking him only guaranties it will become locked again at the next execution
  4. The only thing to do is to purge the DB table elasticsuite_tracker_log_event
@romainruaud romainruaud self-assigned this Oct 4, 2018
rbayet added a commit to rbayet/elasticsuite that referenced this issue Oct 16, 2018
of 10K events at each cronjob run.
romainruaud added a commit that referenced this issue Oct 17, 2018
…_events_indexing

Resolves #1110 tracker log events indexed by chunk
@romainruaud
Copy link
Collaborator

Fixed by your PR #1138

Will be part of next minor release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants