[Service Bus] Stress/Spike Multiple Clients against Multiple Queues (T1 scenario within P0)

[ramya0820]

• Package Name: ramya0820/perf-tests-azure-js-sdk-service-bus
• Package Version: 1.0.0
• Operating system: Windows 10
• nodejs
  • version: v10.14.2
• typescript
  • version: ts-node v8.0.2

Describe the bug
On running multiple clients with single sender and receiver connecting to multiple queues, sending and receiving a single message and with receiver being closed after receiving the message, following observations were made:

• Start timestamp for reference

Time :  2019-03-21T00:08:29.750Z
Number of clients created so far :  2

• Encountered 2 errors

Time :  2019-03-21T00:46:09.965Z
Number of clients created so far :  998


(node:21904) UnhandledPromiseRejectionWarning: QuotaExceededError: The maximum number of '1000' tokens per connection has been reached. TrackingId:f5a65f5a-3932-4978-9abf-c87a3770cecd_G22, SystemTracker:perftestbasic.servicebus.windows.net:t1-queue-new-1000, Timestamp:2019-03-21T00:46:14
    at Object.translate (c:\workspace\perf\t0\node_modules\@azure\amqp-common\lib\errors.ts:537:13)
    at Receiver.messageCallback (c:\workspace\perf\t0\node_modules\@azure\amqp-common\lib\requestResponseLink.ts:142:25)
    at Receiver.emit (events.js:182:13)
    at Receiver.EventEmitter.emit (domain.js:441:20)
    at emit (c:\workspace\perf\t0\node_modules\rhea-promise\lib\util\utils.ts:183:20)
    at Object.emitEvent (c:\workspace\perf\t0\node_modules\rhea-promise\lib\util\utils.ts:195:5)
    at Receiver._link.on (c:\workspace\perf\t0\node_modules\rhea-promise\lib\link.ts:292:11)
    at Receiver.emit (events.js:182:13)
    at Receiver.EventEmitter.emit (domain.js:441:20)
    at Receiver.link.dispatch (c:\workspace\perf\t0\node_modules\rhea\lib\link.js:59:37)
(node:21904) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:21904) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Time :  2019-03-21T00:46:14.966Z
Number of clients created so far :  1000


Time :  2019-03-21T00:46:19.967Z
Number of clients created so far :  1001


Time :  2019-03-21T00:46:24.967Z
Number of clients created so far :  1001


Time :  2019-03-21T00:46:29.967Z
Number of clients created so far :  1001


Time :  2019-03-21T00:46:34.967Z
Number of clients created so far :  1001


Time :  2019-03-21T00:46:39.968Z
Number of clients created so far :  1001


Time :  2019-03-21T00:46:44.968Z
Number of clients created so far :  1001


Time :  2019-03-21T00:46:49.969Z
Number of clients created so far :  1001


Time :  2019-03-21T00:46:54.970Z
Number of clients created so far :  1001


Time :  2019-03-21T00:46:59.971Z
Number of clients created so far :  1001


(node:21904) UnhandledPromiseRejectionWarning: QuotaExceededError: The maximum number of '1000' tokens per connection has been reached. TrackingId:79e54b85-97c0-46f4-93f1-dd388ffe14ee_G22, SystemTracker:perftestbasic.servicebus.windows.net:t1-queue-new-1001, Timestamp:2019-03-21T00:46:16
    at Object.translate (c:\workspace\perf\t0\node_modules\@azure\amqp-common\lib\errors.ts:537:13)
    at Receiver.messageCallback (c:\workspace\perf\t0\node_modules\@azure\amqp-common\lib\requestResponseLink.ts:142:25)
    at Receiver.emit (events.js:182:13)
    at Receiver.EventEmitter.emit (domain.js:441:20)
    at emit (c:\workspace\perf\t0\node_modules\rhea-promise\lib\util\utils.ts:183:20)
    at Object.emitEvent (c:\workspace\perf\t0\node_modules\rhea-promise\lib\util\utils.ts:195:5)
    at Receiver._link.on (c:\workspace\perf\t0\node_modules\rhea-promise\lib\link.ts:292:11)
    at Receiver.emit (events.js:182:13)
    at Receiver.EventEmitter.emit (domain.js:441:20)
    at Receiver.link.dispatch (c:\workspace\perf\t0\node_modules\rhea\lib\link.js:59:37)
(node:21904) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)

To Reproduce
Steps to reproduce the behavior:

1. Run sample for more than 1000 clients.

Expected behavior
Investigate if threshold limit hit is correct and expected.
The message in error thrown should be inline with identified expectations.
Error handling or documentation on API in SDK to be appropriately updated.

Related T1 scenario issues - #1717 #1718

[ramya-rao-a]

@binzywu, @nemakam

Our Service Bus library uses AMQP, and we see the QuotaExceededError when we hit 1000 sender/receivers.

Below is an excerpt from https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-performance-improvements#queue-with-a-large-number-of-senders

Service Bus enables up to 1000 concurrent connections to a messaging entity (or 5000 using AMQP). This limit is enforced at the namespace level, and queues/topics/subscriptions are capped by the limit of concurrent connections per namespace. …….... If more than 1000 concurrent senders are required, the senders should send messages to the Service Bus protocol via HTTP.

Is (or 5000 using AMQP) supposed to have been (or 5000 using HTTP)? Because at the end, it says that if we need more than 1000, we should use HTTP.

[ramya-rao-a]

In #1757 and #5946 we determined that only upto 1000 concurrent connections are allowed when using AMQP.

I have created a PR to update the docs as linked in the comment above #1714 (comment)

Closing this issue as there is no further work regarding this in the client package