Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Learn how to fix the issue when events are not consumed or are consumed slowly #1820

Merged
merged 5 commits into from
May 3, 2023
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -43,30 +43,23 @@ related:

## Description

Events are not consumed or are consumed slowly. In RabbitMQ exception.log, the following exception can be found:
Events are not consumed or are consumed much slowler as normal. In RabbitMQ exception.log, the following exception can be found (often also with Broken Pipe reference)
VadymSachenko marked this conversation as resolved.
Show resolved Hide resolved

```
VadymSachenko marked this conversation as resolved.
Show resolved Hide resolved
Zed.CRITICAL: PhpAmqpLib\Exception\AMQPChannelClosedException - Channel connection is closed.
```

## Cause

The exact reason for the error is unknown, but we've noticed that it occurs predominantly in large Spryker environments with many product updates. We continue to explore the root cause.
There are several potential reasons for this error. The most common cause is that during P&S chunks take too much time to be processed and RabbitMQ is closing its TCP connection as it anticipates either a timeout or no futher connection to happen.
VadymSachenko marked this conversation as resolved.
Show resolved Hide resolved

## Solution

We have discovered two options that helped to resolve the error:
It is best to profile the job where you are experiencing this error to understand what exactly makes processing the chunks expensive. Temporary mitigation of the issue might be possible as explained below:

**1. Implementing RabbitMQ lazy queues**
One possible solution can be implementing RabbitMQ lazy queues to reduce RAM usage. This is already implemented for PaaS customers.
Configuring RabbitMQ to put messages on disk and load them into memory only when they are needed (lazy queuing) can help. For instructions on how to do this, see the CloudAMQ article [Stopping the stampeding herd problem with lazy queues](https://www.cloudamqp.com/blog/2017-07-05-solving-the-thundering-herd-problem-with-lazy-queues.html).

**2. Adjusting CHUNK_SIZE**
Using smaller chunk sizes for export and storage sync in RabbitMQ might also help to alleviate the issue. Include the following values in the config file used:
**Adjusting CHUNK_SIZE**
Using smaller chunk sizes might help to alleviate the issue as it reduces the time until a chunk is fully processed. If you are running a standard publishing setup you can adjust the following value in config_default.php.
VadymSachenko marked this conversation as resolved.
Show resolved Hide resolved

```
VadymSachenko marked this conversation as resolved.
Show resolved Hide resolved
$config[SynchronizationConstants::EXPORT_MESSAGE_CHUNK_SIZE] = 200;
$config[SynchronizationConstants::DEFAULT_SYNC_STORAGE_QUEUE_MESSAGE_CHUNK_SIZE] = 200;
$config[SynchronizationConstants::DEFAULT_SYNC_SEARCH_QUEUE_MESSAGE_CHUNK_SIZE] = 200;
$config[EventConstants::EVENT_CHUNK] = 200;
```