docs: Stream inspector design

[hariso]

Description

Quick checks:

• I have followed the Code Guidelines.
• There is no other pull request for the same update/change.
• I have written unit tests.
• I have made sure that the PR is of reasonable size and can be easily reviewed.

[lovromazgon]

I think we should dive just a bit deeper into the implementation options, specifically we should explore how we can extract records while they are flowing through the pipeline. I see at least two options:

• somehow wrap the connector Read and Write functions as well as the processor Process function
• or give the PubNode and SubNode the ability to somehow attach to it and inspect incoming/outgoing records.

The second option is more generic (we write it once and can attach to any node), but we currently can't address running nodes, they are spawned and run but not stored anywhere.

Maybe there's an even better solution, but in any case, I think it's worth discussing here.

[hariso]

I think we should dive just a bit deeper into the implementation options, specifically we should explore how we can extract records while they are flowing through the pipeline. I see at least two options:

Good callouts! I didn't mention those for two reasons: I wanted to make sure we're all on the same page w/r to the requirements. And I felt like it's more of a thing for the PRs and so I left the implementation in the design doc high-level. But I can mention it in this PR as well.

but we currently can't address running nodes, they are spawned and run but not stored anywhere.

How about this though?

[lovromazgon]

🚀