Provide codecs that support streaming

[travisbrown]

It's pretty easy to support streaming JSON parsing with Jawn and Spool:

import com.twitter.concurrent.{ Spool, SpoolSource }
import com.twitter.io.Buf
import com.twitter.util.Future
import jawn.{ AsyncParser, Facade, Parser }

class StreamingParser[J](implicit facade: Facade[J]) {
  private[this] val parser: AsyncParser[J] = Parser.async(mode = AsyncParser.UnwrapArray)
  private[this] val source: SpoolSource[J] = new SpoolSource[J]()

  def feedUtf8(s: String): Unit = parser.absorb(s)
  def feed(buf: Buf): Unit = parser.absorb(
    Buf.ByteBuffer.Owned.extract(buf)
  ).fold(source.raise, _.foreach(source.offer))

  def close(): Unit = source.close()
  def results: Future[Spool[J]] = source()
}

It'd be nice to have generic encoding and decoding type classes that support streaming. I started a draft of this last night, but it still needs a lot of work.

[sergeykolbasov]

By the way

It could be a little off topic, but.

We've discussed chunk upload/streaming in finch recently in our company and didnt came to any solution. AFAIK there is a finagle-stream which supports async HTTP codec.

Do we have some abstraction for it in finch?

[travisbrown]

@imliar Not off-topic at all! There's now support for streaming in finagle-httpx proper, so you don't need finagle-stream (which I believe may be on the chopping block)—see e.g. the Readers in Request and Response.

There's currently no abstraction for this in Finch, but it should be pretty straightforward to use Reader with Request and Response directly, and the streaming codecs I'm proposing here will make this even easier to use in conjunction with e.g. a streaming JSON parser.

[sergeykolbasov]

What about some uber abstraction? Iteratee for example. I think it would be perfect.

[travisbrown]

@imliar In my initial sketch Twitter Util's Spool plays a role similar to what an iteratee would provide.

[vkostyukov]

We should probably take a look at AsyncStream, which is a modern replacement for Spool.

[travisbrown]

@vkostyukov The nice thing about Spool is that SpoolSource makes it really easy to feed from Jawn's AsyncParser (and presumably from other similar asynchronous parsing interfaces). I'll try to take a look at what switching it over to AsyncStream would involve tonight.

[vkostyukov]

I vote we close this as we now have a first-class support for iteratee (our streaming story should be much mature now). Thanks again, @imliar, for the tremendous work!