Slow Subscriber over udp

[bandicoot86]

Hi. Not sure whether it's an issue or not but will post here. Any feedback is welcome.
Using samples, sending 2Gb file locally, works fine - I managed to achieve almost 200MB per second. Once I move to the Digital ocean machine, the maximum rate I could achieve is 2 Megabyte per second.
Aeron init on server side

 MediaDriver driver = MediaDriver.launch();

FileReceiver equals to https://github.com/real-logic/aeron/blob/master/aeron-samples/src/main/java/io/aeron/samples/FileReceiver.java

FileSender is run in this fashion:

final String ch = "aeron:udp?endpoint=[PublicIp]:40123";
MediaDriver driver = MediaDriver.launch();
        try (Aeron aeron = Aeron.connect();
             Publication publication = aeron.addExclusivePublication(ch, 1);
        )
        {
            while (!publication.isConnected())
            {
                Thread.sleep(1);
            }

            final File file = new File(args[0]);
            final UnsafeBuffer buffer = new UnsafeBuffer(IoUtil.mapExistingFile(file, "sending"));
            final long correlationId = aeron.nextCorrelationId();

            sendFileCreate(publication, correlationId, buffer.capacity(), file.getName());
            streamChunks(publication, correlationId, buffer);
        }

Could anyone tell me why the rate is so drastically different from the local machine?

Receiver log:

19:13:02 - Aeron Stat (CnC v0.0.16), pid 7966, heartbeat age 194ms
======================================================================
  0:                    0 - Bytes sent
  1:           37,353,824 - Bytes received
  2:                    0 - Failed offers to ReceiverProxy
  3:                    0 - Failed offers to SenderProxy
  4:                    0 - Failed offers to DriverConductorProxy
  5:                  402 - NAKs sent
  6:                    0 - NAKs received
  7:                  885 - Status Messages sent
  8:                    0 - Status Messages received
  9:                    0 - Heartbeats sent
 10:                   26 - Heartbeats received
 11:                    0 - Retransmits sent
 12:                1,089 - Flow control under runs
 13:                    0 - Flow control over runs
 14:                    0 - Invalid packets
 15:                    0 - Errors
 16:                    0 - Short sends
 17:                    0 - Failed attempts to free log buffers
 18:                    0 - Sender flow control limits, i.e. back-pressure events
 19:                    0 - Unblocked Publications
 20:                    0 - Unblocked Control Commands
 21:                    0 - Possible TTL Asymmetry
 22:                    0 - ControllableIdleStrategy status
 23:                    0 - Loss gap fills
 24:                    0 - Client liveness timeouts
 25:                    1 - rcv-channel: aeron:udp?endpoint=0.0.0.0:40123
 26:    1,566,155,582,086 - client-heartbeat: 0
 27:           34,461,184 - sub-pos: 2 1293527099 1 aeron:udp?endpoint=0.0.0.0:40123 @0
 28:           34,461,184 - rcv-hwm: 3 1293527099 1 aeron:udp?endpoint=0.0.0.0:40123
 29:           34,461,184 - rcv-pos: 3 1293527099 1 aeron:udp?endpoint=0.0.0.0:40123

Sender log:

22:13:37 - Aeron Stat (CnC v0.0.16), pid 14961, heartbeat age 882ms
======================================================================
  0:          115,942,880 - Bytes sent
  1:                    0 - Bytes received
  2:                    0 - Failed offers to ReceiverProxy
  3:                    0 - Failed offers to SenderProxy
  4:                    0 - Failed offers to DriverConductorProxy
  5:                    0 - NAKs sent
  6:                  402 - NAKs received
  7:                    0 - Status Messages sent
  8:                3,163 - Status Messages received
  9:                   26 - Heartbeats sent
 10:                    0 - Heartbeats received
 11:                  400 - Retransmits sent
 12:                    0 - Flow control under runs
 13:                    0 - Flow control over runs
 14:                    0 - Invalid packets
 15:                    0 - Errors
 16:                    0 - Short sends
 17:                    0 - Failed attempts to free log buffers
 18:                    0 - Sender flow control limits, i.e. back-pressure events
 19:                    0 - Unblocked Publications
 20:                    0 - Unblocked Control Commands
 21:                    0 - Possible TTL Asymmetry
 22:                    0 - ControllableIdleStrategy status
 23:                    0 - Loss gap fills
 24:                    0 - Client liveness timeouts
 25:                    1 - snd-channel: aeron:udp?endpoint=165.22.87.37:40123
 26:          122,327,680 - pub-pos (sampled): 1 1293527099 1 aeron:udp?endpoint=165.22.87.37:40123
 27:          124,335,744 - pub-lmt: 1 1293527099 1 aeron:udp?endpoint=*:40123
 28:          115,947,136 - snd-pos: 1 1293527099 1 aeron:udp?endpoint=*:40123
 29:          115,947,264 - snd-lmt: 1 1293527099 1 aeron:udp?endpoint=*:40123
 30:                    0 - snd-bpe: 1 1293527099 1 aeron:udp?endpoint=*:40123
 31:    1,566,155,617,284 - client-heartbeat: 0

The machine is in Do, France. I am in Baltic.

[tmontgomery]

You are experiencing loss. See the NAKs sent/received and retransmits. You can look in the loss report to see more details.

[bandicoot86]

Thanks for the tip. Can you recommend something to mitigate the loss?

[tmontgomery]

If you are going over the general Internet, you will probably want to use Cubic Congestion Control so that you share correctly with other TCP connections. This will probably not drastically increase the throughput, but may help slightly. It will definitely help other competing connections, though. Which can have other effects to make more effective use of resources in the path.

If not doing congestion control, rate controlling the sending based rate to a known limit will reduce loss (as long as you pick a rate that can be maintained). Implementing a rate control is straight forward. Send some, wait for an interval, send some more, wait for an interval. etc.

Beyond that, if it is a dedicated network and not the general Internet, then you should look into capacity planning the network, etc. Once capacity planned, you will want to rate control the sending rate as mentioned above.

[bandicoot86]

Thank you Todd. I will try and play with the mentioned parameters. So, generally speaking, it's pretty hard(even impossible) to achieve high rates on the Internet?

[tmontgomery]

No. It's just a limitation on the general Internet. You are, essentially, blasting data out as fast as possible. The routers in the path will drop your data and as it isn't reacting to loss.... probably biasing against it.

Congestion control will reduce more appropriately and get more bandwidth overall. But it will not get you the 200MBps as that is just not achievable on that link.... if it were, you wouldn't have loss.

Using TCP-like congestion control (as Cubic will do) means you are limited to what TCP will achieve in the same situation. You won't go faster than that AND share bandwidth with competing TCP streams.

As you have control in Aeron and can implement your own congestion control, you can squeeze out some more bandwidth with a BBR type congestion control mechanism. We've not had sponsorship to implement it as of yet. But even BBR will not get what is just not there.

[bandicoot86]

Thank you.