Add a disruptor based async reporter

[codefromthecrypt]

We can calculate the size of a span without serializing it. We should be able to make an async reporter that defers serialization, but still allows us to drop spans that are too big to send. It should be fun work, if someone's up to it.

We could also consider making this disruptor based, or otherwise with a fancier ringbuffer opt-in given a dep like https://github.com/real-logic/agrona.

[wu-sheng]

In SkyWalking, we put the Spans(actually Segment) into a ringbuffer, then consume it, finally send it. So I agree with you, defer serialization is a good idea.

For the ringbuffer, LMAX-Exchange/disruptor is really a good project. For SkyWalking, we use a array based queue, implement by ourself.

[codefromthecrypt]

ignoring my other comment.. deferring is certainly worth doing, independent of this issue, which I've retitled to making things disruptor based #84

[codefromthecrypt]

The related change out in zipkin-reporter 2.2 had much more impact in benchmark vs microbenchmark. Here's an example spring-cloud/spring-cloud-sleuth#808

[tramchamploo]

Well, I don't see any size restriction or overflow strategy in disruptor. So maybe using disruptor before ByteBoundedQueue layer, or reimplement it to behave just like a ringbuffer?

[codefromthecrypt]

@tramchamploo I'm flexible, but before we make our own want to check something...

@mjpt777 are you aware of any size-based or otherwise bounded disruptor? We have means to calculate the size per item of things and would want to drop if what's currently unprocessed is at or above that

[mjpt777]

@adriancole In what dimensions are you referring to when you mention a "bounded disruptor"? Is this bounded backlog of items, or bounded size of individual entries, or some other bound? I think you are referring to bounded record size but want to be sure.

[codefromthecrypt]

@adriancole <https://github.com/adriancole> In what dimensions are you referring to when you mention a "bounded disruptor"? Is this bounded backlog of items, or bounded size of individual entries, or some other bound? I think you are referring to bounded record size but want to be sure.

yes so this would be the backlog. currently we have two (both) bounds, maximum count or records and total size of records in the backlog. the records are not uniform size.

[mjpt777]

If you are bounding in both backlog and record size then best to do that by limiting the overall bytes. There are a number of structures in Agrona for one-to-one and many-to-one ring buffers, and a broadcast buffer for one-to-many. You could also consider Aeron as it gives the best throughput and latency and can bound in both dimensions plus can can use in with shared memory or over the network. Aeron has many more useful features for this type of need with control over flow and congestion control if you are willing to investigate.

[codefromthecrypt]

Thanks for the tips! Both are worth exploring.

…

On 31 Jan 2018 8:49 am, "Martin Thompson" ***@***.***> wrote: If you are bounding in both backlog and record size then best to do that by limiting the overall bytes. There are a number of structures in Agrona <https://github.com/real-logic/agrona> for one-to-one and many-to-one ring buffers, and a broadcast buffer for one-to-many. You could also consider Aeron <https://github.com/real-logic/aeron> as it gives the best throughput and latency and can bound in both dimensions plus can can use in with shared memory or over the network. Aeron has many more useful features for this type of need with control over flow and congestion control if you are willing to investigate. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#81 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD613cIYQ0DkMeazj6a5RwsZ19Oazblks5tQJmMgaJpZM4Q583O> .

[tramchamploo]

Another choice is JCTools used by Netty.

[codefromthecrypt]

note: census uses a disruptor (they don't check bounds) it is prior art to consider

https://github.com/census-instrumentation/opencensus-java/blob/master/impl/src/main/java/io/opencensus/impl/internal/DisruptorEventQueue.java

I would caution that there have been a lot of small bugs about CPU which hopefully most are gone now. https://github.com/census-instrumentation/opencensus-java/search?q=disruptor&type=Issues

[codefromthecrypt]

ps anything like this should be carefully reviewed to not cause more support issues than it solves. Multiple folks should ok the impl in a way they are happy to help support bugs in it.