Router Prototyping/Critical Path

[sethp]

So we've gotten a prototype "pod" in-flight that will connect a GPY111 Ethernet PHY to the BeagleV-Fire's FPGA HSIO pins, which means we've learned a ton about:

• The role of an Ethernet PHY & SGMII/RGMII as ways to handle the "router" parts
• Power distribution networks & switching power supplies
• PCB trace impedance/length matching, and "signal integrity"

And we expect to have an opportunity to experiment with a full "SBC" that has:

• An FPGA SoC (i.e. hard processor cores, and a few hard interfaces like SGMII)
• External memory, hopefully accessible both from the CPU and FPGA "bits"
• Storage (at least enough to boot a linux distro)
• Tutorials on programming the FPGA and a path for loading in new code & bitstreams, etc.
• Other conveniences & niceties yet to be discovered

Which has brought us to also wonder what our next prototype should be, and what shape the hardware might have.

[sethp]

[space reserved for final details]

[sethp]

On Ethernet Ports

• How many ports will it have?
  • Definitely want at least three.
• Will they be all the same speed?
• Will the topology be fixed, or modular?
• What MTU will we support? Just 1500, or 1500 + 9000?

[sethp]

Topology

Is there a very cool super modular design that's got a big PCIe bus hanging off of an FPGA that just happens to implement a router? Yes. Will we do it now? No. We'll have a fixed layout of some number of ports.

Do we want to have a smaller number of higher speed ports? Probably not. Let's go for all GbE all the time for all ports.

How many ports?

How many ports ought we have? Having fewer than four RJ-45 connectors would require explaination as to why it's still a router, but four would match people's expectations, so let's set a minimum bar at four. In terms of I/O pins, we're in the high 70s for pin count for 4x GbE speaking RGMII (14 pins per), 15 or 16 for memory, and another 6 or so for SPI (storage). Another 2 for UART.

What about a single M.2 connector for e.g. WiFi? That would cost us approx. 11 I/O pins for e.g. https://developer.embeddedartists.com/docs-m2/integration/pinning/#implement-for-wi-fi-over-pcie .

So that's a minimum of of ~90 I/O pins with, some varied speed requirements, to act as a first-order filter on possible FPGAs.

[sethp]

re: MTUs and saturation

Under what conditions do we serve all ports at line rate?

• We certainly can't if there's a "fan-in" where N-1 of the ports all really want to send packets out the Nth port
• Usually there's a limit on packets per second as well as size of packet; it's a common choice to quote a device as being "gigabit capable" if it can handle max_pps * max_mtu > 1Gbps, which is a much easier target to hit than max_pps*minimum_useful_packet_size (64?) > 1Gbps
  • we confidently assert that frames contain exactly one packet (not necessarily true in general, but it probably is for IP & every useful thing we might do)
  • With an MTU of 1500 we'd need to handle 83kpps (packets per second) to qualify as "gigabit capable"; with an MTU of 9000 we'd only "need" ~14kpps, but almost certainly nobody's ISP sends/accepts "jumbo" frames (so we couldn't, say, max out the speedometer on a consumer internet speed test).
  • Hitting the minimum (routing 64 byte packets) would require handling 1945kpps (per interface)

For the purposes of setting an MTU, it's only useful to us to have those numbers if we think we're going to land in the 14-80kpps range and we're willing to declare that we're capable of gigabit-speeds-asterisk-with-1500-byte-packets.

How fast would our memory need to go?

[sethp]

On FPGAs

• Strong preference for a non-BGA package
• Strong preference for chips in the $15-$30 price range
• Strong preference for open toolchains (e.g. https://f4pga.readthedocs.io/en/latest/status.html )
• Will we have a CPU?
  • If we do, we'd want it to be RISC-V
  • SoC-style (baked into the silicon) or "soft" (part of the FPGA fabric)?
  • Will it run Linux? Or be more of an embedded-style minimalist situation?
• How much I/O do we need? Of what types?
  • We expect to have an Ethernet PHY per port; probably speaking RGMII? So we'll need at least 12 DDR-capable I/Os per port (or 14, if we can't figure out how to "bus" the MDIO control lines)
• How many LUTs will we need?

also, per @dougli1sqrd :

I just did some back of the napkin embedded memory calculations. I diagrammed out a basic idea of the Rx and Tx pipelines within the fpga (receiving and transmitting an ethernet frame) and it I'm thinking the Rx channel needs to hold N+1 frames (meaning it should always be able to hold at least one frame) in it's queuing: 1 for just the initial Raw bytes it's pulling in and then 1 for a staging area for whole validated frames that are waiting a queue to be placed in main memory accessible by the CPU core. The Tx channel is maybe simpler, maybe having just one main queue, holding M frames. So if a "size" of a frame is S, and we have P ports, then the number of bytes for queueing memory is PS(N + M + 1).

Picking some numbers, if we take S = 1024 (payload size) + 22 (headers), N,M = 1, for 4 ports that's a little over 100KBits of memory kinda minimally. That puts a pretty decent constraint on what FPGAs are avaliable to us acutally

Using an MTU of 1500 gets us to 192KBit minimum to hold a single "in-flight" packet at a time per port, which is kind of a lot (especially in non-BGA packages).

[sethp]

On External Memory

Given the need for packet buffers, we'll probably require some degree of external memory.

• What, uh, how it do?
• Go fast, zoom zoom?

[sethp]

pSRAM w/ Octal SPI: how many chips would we need to meet 1Gbps 2Gbps per port aggregate bandwidth (for just the RGMII side)? Each costs ~11-12 I/O pins (at whatever clock rate).

[sethp]

Memory bandwidth for packet buffers seems to represent a bottleneck for us here: we're assuming at this point that the FPGA internal memory will be fast enough and offer enough access ports that we can get 8Gbps of bandwidth to/from the RGMII links out of it (up to 10Gbps if/when we introduce an M.2 connector for an optional 5th port).

External RAM is a little bit of a different story, though. So far we've found ways to spend a very large sum of money on memory ICs (and ~hundreds of I/O pins to match), or we've talked about what it would mean to have the memory bandwidth bottleneck the throughput of the router.

Still in process: trying to find the right way to ask the geniemouser search to yield for us a chip that would balance cost, pins, and bandwidth to an acceptable level.

[sethp]

So raw-bandwidth-wise we're in pretty good shape in DDR3 land; see https://github.com/rustbox/router?tab=readme-ov-file#dram for some worked examples.

As for the "overhead," it seems like we'll have to account for that, since "CAS latency" is ~ a half dozen cycles long. Without some kind of pipelining, that means non-sequential reads (y'know, "Random Access") would maybe cut our throughput to less than 20%, meaning we'd actually need more than 50Gbps of raw bandwidth to keep up with our packet buffers. That's a much tighter constraint, requiring either a lot of parallel lanes or some of the most recent DDR4 chips to meet.

Probably there's a way to model our usage at one more level of detail that would let us at least SWAG what our overhead would be.

[sethp]

On Storage

We'll minimally need to semi-persistently store some level of router configuration. Maybe we can get away with non-volatile configuration memory in the FPGA for that? But if we want to have, say, Linux, and a whole filesystem, we'll need something more like.... eMMC?

[sethp]

It seems OK to have the storage be very slow.

Some possibilities:

• Built-in "flash" storage (maybe 5 or 6 pins for a QSPI implementation)
• MicroSD card (looks like ~8 pins?)

Some constraints:

• Boot-time? How does the system come up?

[sethp]

On "Break Glass" Access

We'll need of secondary access path to get into the router when things go wrong. Like, if it stops responding to pings, we'll need some sort of way to access the current state to debug and repair whatever went wrong. Something like an IMPI/serial port?

Ideally we'd make provisions for separating out "restoration of service" (i.e. turning it off and on again) from debugging, since those two things are in tension (make it work now vs. don't break again in the future).

A related quandary is how to handle recovery in case of a bad update: does a bug in the RGMII interface snowball into a "bricked" device? Once we do inevitably "brick" a device, how might we recover it?

[sethp]

On Less-Formed Thoughts

This post is less about things we're gonna run into getting the first version working, and more about the things we're going to run into getting it working well.

A partial list:

• updates & attestation
• tracing & faults
• power efficiency

[sethp]

re: power efficiency, my home GbE links are almost exclusively >90% idle, even while "heavily" used (a steady 1Mbps of traffic corresponds to a 99.9% idle gigabit link, and occasional spikes up to ~100Mbit still only uses 10% of the capacity of the link). At "idle" the traffic volume is on the order of tens of kbps—it sure would be nice to ramp our power usage relative to the utilization, if we can avoid e.g. switching the 125MHz clock during the periods when there's no signal on the link.