Improve Manual Mode indexing

[colinmollenhour]

One of my dreams is for Magento to be able to:

1. Perform the heavy lifting of indexes in the background in near-real-time by using a separate daemon of sorts that uses a pub/sub method (e.g. Redis) to wake up the indexer instantly.
2. Clear the related cache tags only after the reindexing is complete. (otherwise the cache may be rewarmed before the reindexing is complete).
3. Not have potential issues with process statuses. Currently I think it would be possible for the status to end up as "pending" even though new events were added while some events were being processed. That is, the status should not be changed to "pending" if new events were added in the meantime (e.g.: prevent update if updated_at time changed)

[magento-team]

Hello Colin. Can you reach out to me at peter(at) magento.com? I'd like to discuss this in more details

[magento-team]

Hi. Could you please elaborate on #2? Do you mean that we should invalidate FPC entry for given page (product X) only after product X is already indexed? Our current approach was to use cache even the previous version and then refresh cache.

[colinmollenhour]

The 2nd point was not necessarily specifically related to FPC, but to cache in general (e.g. block_html). After (1) is implemented, if proper block rendering is dependent on indexes, then if indexes are rebuilt in the background but cache is cleared in the foreground (i.e. _afterSave method calls cleanCache($this->getTagIds())) then this would allow a situation where the cache is cleared and then re-populated before the background indexer completes reindexing which would mean you basically just refreshed stale cache entries. So, one solution might be to store the list of tags to be cleared in the indexer event so that the background indexer can clear the tags after the event is processed.

[tux-rampage]

This would require a Jobq implementation like ZendJobq or Gearman which is not available everywhere.

Imho this should be implemented as optional feature using a Jobq-Implementation only when it is available. The user should be able to specify a Jobq strategy in the system configuration.

I've implemented a Jobq module for Magento 1.x which supports ZendJobq, Gearman and a cli/cron based setup maybe I can port this to magento 2 core.

When implementing background jobs, you need to be aware of certain circumstances.
For example:

• When a product is saved a second time the indexer must not start before the first index run has finished.
• When triggering a Reindex for all products, it must not start before all individual indixers have completed and maybe it should cancel pending ones.

[colinmollenhour]

The indexer tables are already pretty close to acting as a job queue, just not a general purpose one. I think a general purpose job queue in Magento would be great, no almost a necessity..

[colinmollenhour]

These changes would never see CE anyway so closing..

[magento-engcom-team]

Hi @colinmollenhour. Thank you for your report.
The issue has been fixed in #27597 by @lbajsarowicz in 2.4-develop branch
Related commit(s):

• ca0e3a1b655fa0356e69d65bd163196562c72f14
• 1e226ffa0e382c73560eea23223d0c6a8e4d6f72

The fix will be available with the upcoming 2.4.0 release.