From 4e0774a977f604d324de0fe0f98d013858a4701c Mon Sep 17 00:00:00 2001 From: Thomas Lehner <67914368+ThomasLehnerSpryker@users.noreply.github.com> Date: Tue, 2 May 2023 12:12:46 +0200 Subject: [PATCH 1/3] Update rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md Corrected the contents of the article, as the SYNC setting normally are not related to the error --- ...sedexception-channel-connection-is-closed.md | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md b/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md index 2b5beb9d3cf..f31323c3661 100644 --- a/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md +++ b/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md @@ -43,7 +43,7 @@ related: ## Description -Events are not consumed or are consumed slowly. In RabbitMQ exception.log, the following exception can be found: +Events are not consumed or are consumed much slowler as normal. In RabbitMQ exception.log, the following exception can be found (often also with Broken Pipe reference) ``` Zed.CRITICAL: PhpAmqpLib\Exception\AMQPChannelClosedException - Channel connection is closed. @@ -51,22 +51,15 @@ Zed.CRITICAL: PhpAmqpLib\Exception\AMQPChannelClosedException - Channel connecti ## Cause -The exact reason for the error is unknown, but we've noticed that it occurs predominantly in large Spryker environments with many product updates. We continue to explore the root cause. +There are several potential reasons for this error. The most common cause is that during P&S chunks take too much time to be processed and RabbitMQ is closing its TCP connection as it anticipates either a timeout or no futher connection to happen. ## Solution -We have discovered two options that helped to resolve the error: +It is best to profile the job where you are experiencing this error to understand what exactly makes processing the chunks expensive. Temporary mitigation of the issue might be possible as explained below: -**1. Implementing RabbitMQ lazy queues** -One possible solution can be implementing RabbitMQ lazy queues to reduce RAM usage. This is already implemented for PaaS customers. -Configuring RabbitMQ to put messages on disk and load them into memory only when they are needed (lazy queuing) can help. For instructions on how to do this, see the CloudAMQ article [Stopping the stampeding herd problem with lazy queues](https://www.cloudamqp.com/blog/2017-07-05-solving-the-thundering-herd-problem-with-lazy-queues.html). - -**2. Adjusting CHUNK_SIZE** -Using smaller chunk sizes for export and storage sync in RabbitMQ might also help to alleviate the issue. Include the following values in the config file used: +**Adjusting CHUNK_SIZE** +Using smaller chunk sizes might help to alleviate the issue as it reduces the time until a chunk is fully processed. If you are running a standard publishing setup you can adjust the following value in config_default.php. ``` -$config[SynchronizationConstants::EXPORT_MESSAGE_CHUNK_SIZE] = 200; -$config[SynchronizationConstants::DEFAULT_SYNC_STORAGE_QUEUE_MESSAGE_CHUNK_SIZE] = 200; -$config[SynchronizationConstants::DEFAULT_SYNC_SEARCH_QUEUE_MESSAGE_CHUNK_SIZE] = 200; $config[EventConstants::EVENT_CHUNK] = 200; ``` From 6d45cc896d980950cd1d3d2e7597324b6a55e2a6 Mon Sep 17 00:00:00 2001 From: Vadym Sachenko Date: Wed, 3 May 2023 15:09:20 +0300 Subject: [PATCH 2/3] Apply suggestions from code review --- ...nelclosedexception-channel-connection-is-closed.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md b/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md index f31323c3661..4942bd86513 100644 --- a/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md +++ b/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md @@ -43,23 +43,24 @@ related: ## Description -Events are not consumed or are consumed much slowler as normal. In RabbitMQ exception.log, the following exception can be found (often also with Broken Pipe reference) +Events are not consumed or are consumed much slower than normal. In RabbitMQ exception.log, the following exception can be found (often also with Broken Pipe reference): -``` +```php Zed.CRITICAL: PhpAmqpLib\Exception\AMQPChannelClosedException - Channel connection is closed. ``` ## Cause -There are several potential reasons for this error. The most common cause is that during P&S chunks take too much time to be processed and RabbitMQ is closing its TCP connection as it anticipates either a timeout or no futher connection to happen. +There are several potential reasons for this error. The most common cause is that during P&S chunks take too much time to be processed and RabbitMQ is closing its TCP connection as it anticipates either a timeout or no further connection to happen. ## Solution It is best to profile the job where you are experiencing this error to understand what exactly makes processing the chunks expensive. Temporary mitigation of the issue might be possible as explained below: **Adjusting CHUNK_SIZE** -Using smaller chunk sizes might help to alleviate the issue as it reduces the time until a chunk is fully processed. If you are running a standard publishing setup you can adjust the following value in config_default.php. -``` +Using smaller chunk sizes might help to alleviate the issue because it reduces the time until a chunk is fully processed. If you are running a standard publishing setup, you can adjust the following value in `config_default.php`. + +```php $config[EventConstants::EVENT_CHUNK] = 200; ``` From f4cf0690f6ac992c5dfffe6f6fc4c465645d2cf9 Mon Sep 17 00:00:00 2001 From: Vadym Sachenko Date: Wed, 3 May 2023 15:09:46 +0300 Subject: [PATCH 3/3] Update rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md --- ...n-amqpchannelclosedexception-channel-connection-is-closed.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md b/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md index 4942bd86513..1f590d4d429 100644 --- a/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md +++ b/docs/scos/dev/troubleshooting/troubleshooting-general-technical-issues/rabbitmq-zed.critical-phpamqplib-exception-amqpchannelclosedexception-channel-connection-is-closed.md @@ -1,7 +1,7 @@ --- title: RabbitMQ- Zed.CRITICAL- PhpAmqpLib\Exception\AMQPChannelClosedException - Channel connection is closed description: Learn how to fix the issue when events are not consumed or are consumed slowly. -last_updated: Jun 16, 2021 +last_updated: May 3, 2023 template: troubleshooting-guide-template originalLink: https://documentation.spryker.com/2021080/docs/rabbitmq-zedcritical-phpamqplibexceptionamqpchannelclosedexception-channel-connection-is-closed originalArticleId: 39daf8b8-9e73-4c76-a6ee-372d3e5bbca6