feat(congestion): write stats to a csv file

[wacban]

This feature allows dumping the stats to a csv file. In a follow up PR I'll figure out how to plot this data conveniently. I implemented the framework and an example in the SimpleBackpressure strategy.

By default the model will add a single column round and populate it for each round.
Each strategy needs to populate the head in the init method and populate the values in the compute_chunk method.

In the simple backpressure strategy the first few rows look like follows:

round,shard_0_incoming_queue,shard_0_outgoing_queue,shard_1_incoming_queue,shard_1_outgoing_queue,shard_2_incoming_queue,shard_2_outgoing_queue,shard_3_incoming_queue,shard_3_outgoing_queue
1,0,0,0,0,0,0,0,0
2,20,0,0,0,0,0,0,0
3,40,0,0,0,0,0,0,0
4,57,0,3,0,0,0,0,0
5,73,0,2,0,2,0,0,0
6,89,0,0,0,3,0,1,0
7,105,0,0,0,0,0,4,0

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 102 lines in your changes are missing coverage. Please review.

Project coverage is 71.66%. Comparing base (13e573f) to head (55b9873).
Report is 1 commits behind head on master.

Files	Patch %	Lines
tools/congestion-model/src/main.rs	0.00%	50 Missing ⚠️
tools/congestion-model/src/evaluation/mod.rs	0.00%	40 Missing ⚠️
tools/congestion-model/src/model/queue.rs	0.00%	5 Missing ⚠️
tools/congestion-model/src/model/queue_bundle.rs	0.00%	5 Missing ⚠️
...ngestion-model/src/strategy/simple_backpressure.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10719      +/-   ##
==========================================
- Coverage   71.72%   71.66%   -0.07%     
==========================================
  Files         758      758              
  Lines      152593   152680      +87     
  Branches   152593   152680      +87     
==========================================
- Hits       109451   109418      -33     
- Misses      38191    38295     +104     
- Partials     4951     4967      +16     

Flag	Coverage Δ
backward-compatibility	0.24% <ø> (ø)
db-migration	0.24% <ø> (ø)
genesis-check	1.42% <ø> (ø)
integration-tests	36.87% <0.00%> (-0.08%)	⬇️
linux	70.42% <0.00%> (-0.03%)	⬇️
linux-nightly	71.15% <0.00%> (-0.09%)	⬇️
macos	53.05% <0.00%> (-1.71%)	⬇️
pytests	1.64% <ø> (ø)
sanity-checks	1.43% <ø> (ø)
unittests	67.47% <0.00%> (-0.06%)	⬇️
upgradability	0.29% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wacban]

Balanced Simple Backpressure

AllToOne SimpleBackpressure

[wacban]

A few more stats need to be added as well as stats for other strategies but I'll leave that for later.

[jakmeier]

I love this conceptually! Getting all the data out in a CSV is exactly what we need to solve the evaluation part cleanly. The summary table was a very crude first example of how we could get an output, this is miles better.

However, you are breaking several of the code architecture invariants. 😅 Not your fault, of course, you couldn't know it. But the way I had it in my mind, I wanted to keep the code base modular with minimal cross-over between the 4 main components (strategy, workload, model, evaluation). You can kind of see it how your changes forced you to pass around the stats writer to many files that shouldn't be affected by such a change.

I've tried to explain what my architectural idea is in in-line comments. Please take a look and let me know if you agree / want to follow my proposed architecture. Or if you think it's too much work or just not a good idea, we can can also work out a different architecture together, I'm open to your suggestions.

[wacban]

What the heck is mailbox? :)

[jakmeier]

Borrowed vocabulary from the actor model

https://www.brianstorti.com/the-actor-model/

The model basically treats each shard as an actor and the implicit incoming receipts queue is the actor's mailbox. And that's no coincidence. I always like to map problems to solutions that already exist.

Not only is the model implemented this way, I think even the real implementation can be thought of this way. Taking inspiration from Erlang, Akka, Actix, and other established actor frameworks could be useful. But it's also quite limited since they generally react to full queues by dropping messages / returning failures (which I still think we can avoid) and they don't operate on a global round-based clock that the blockchain has thanks to block heights. I think those are the main differences in constraints.

Anyway, sorry for the off-topic rambling... "mailbox" as the queue name makes sense in my mind but open to other names :)

[wacban]

new impl with new stats

simple backpressure, balanced - unbounded - waiting transactions is ever growing

simple backpressure, all to one - deadlocking - finished transactions goes flat about halfway the evaluation

[jakmeier]

Thanks @wacban for the changes! I personally like this much better, as it now fits in perfectly with the intended architecture.

This is going to be suuper helpful to analyse and compare different strategies!

[jakmeier]

Borrowed vocabulary from the actor model

https://www.brianstorti.com/the-actor-model/

The model basically treats each shard as an actor and the implicit incoming receipts queue is the actor's mailbox. And that's no coincidence. I always like to map problems to solutions that already exist.

Not only is the model implemented this way, I think even the real implementation can be thought of this way. Taking inspiration from Erlang, Akka, Actix, and other established actor frameworks could be useful. But it's also quite limited since they generally react to full queues by dropping messages / returning failures (which I still think we can avoid) and they don't operate on a global round-based clock that the blockchain has thanks to block heights. I think those are the main differences in constraints.

Anyway, sorry for the off-topic rambling... "mailbox" as the queue name makes sense in my mind but open to other names :)

[jakmeier]

Is this still needed, or just an artefact from previous code iterations? If it's not necessary, I'd prefer this to stay private.

Making it pub is okay for our little model if it makes things easier. But generally speaking, it breaks the intended type safety. While the value is private, we can guarantee that ShardId can only be created inside this file. So it's easy to check invariants on it. That's about halve the motivation for wrapping the usize in a new type. It's the same pattern I used for QueueId, TransactionId, and ReceiptId.

If you just need read-only access to the number, adding a simple getter method, or maybe impl From<ShardId> for usize {} or even impl Deref for ShardId { type Target = usize } could do the trick without breaking type properties but still allows reading the number with id.into() or *id, which seems just as convenient as id.0.