Implement streams using kinesalite

[jbergknoff-rival]

Hi, here's an implementation of Dynamo streams which uses kinesalite as its backend (as suggested in #67). I initially tried to use #67 for my application, but ended up having issues with some data failing to show up in the stream. Nevertheless, huge thanks to @kiril-pirozenko-home24 for laying the groundwork. I was able to reuse or take inspiration from quite a bit of the code from the PR, particularly the listStreams implementation (for which Kinesis' functionality is substantially different) and some of the tests.

It's slightly awkward to reach into kinesalite and use its internals, which are not exported for library use, but this seemed like the most expedient thing to do. If there's interest in merging this PR, we can easily establish a cleaner interface between dynalite and kinesalite.

More tests would be nice, but the getRecords tests exercise essentially all of the "happy path" surface area for the streams API.

The PR also has a --verbose CLI option which has been very helpful for debugging.

[jbergknoff-rival]

This test was failing on the master branch.

[mhart]

This is very cool. Hopefully I'll have time over the next week to look through this in a little more detail, and should be able to get something merged sometime in the next couple of weeks.

Thanks for putting in the work! 👍

[jbergknoff-rival]

Great, sounds good, thanks.

[jbergknoff-rival]

I ran into an issue which isn't strictly related to this PR, but I threw a fix in here and added a test. There was a race condition that becomes obvious when createTableMs is set high: (1) create a table, (2) update the table while table is still in "CREATING" mode, (3) the update is processed, (4) the table creation finishes and runs its setTimeout, blowing away the updates because it assumes nothing has changed.

I checked against real Dynamo, and UpdateTable is rejected if the table is in the CREATING state, so I've updated it accordingly here.

[jbergknoff-rival]

@mhart I've just merged master into this PR. Please review when you have a chance.

[bwitt]

@mhart just checking if there's anything else that needs to happen here :)

[kosciak9]

Hello, is this being worked on? I would gladly help with bringing it back to mergable state and using this functionality.

[ryanblock]

This PR has gone quite stale (hardly a surprise given its age); I'm thinking this would probably wind up being better serviced generically through DynamoDB streams, which is definitely on the roadmap. Please let me know if y'all disagree!

[ryanblock]

Sorry, also seeing the PR feedback from #67 pointing you this direction @jbergknoff-rival; I'm thinking that since kinesalite is no longer maintained (and we aren't planning to maintain it) I'm wondering if perhaps the best approach would be to pull those internals completely out, where possible, and adopt them here? This is admittedly naive, but perhaps that will be a smoother ride than building from scratch?