Investigate high frequency clock muxing as an alternative to an858, an909, an959

[martijnbastiaan]

A crucial part of a Bittide system is that each node can adjust its frequency based on its neighbors, thereby building a "synchronous enough" system. Eventually, we'll be able to use a DCO described in an858, an909, and an959 to very precisely step through frequencies. Off-the-shelve FPGA boards don't have this component though, so we'll either have to wait for a custom board or only test this behavior in simulation.

However, I think we might be able to implement the following clock muxing strategy, where BUFGMUX is a glitch-free clock mux available on Xilinx's 7-series FPGAs:

By selecting clk_L half of the time while selecting clk_H the other half, we'll achieve an effective frequency of (freq(clk_l) + freq(clk_h)) / 2. This ratio can be changed by the frequency manager to achieve an arbitrary frequency bounded by the frequencies of the slow and fast clock.

In practice this will be slightly more complicated due to the way a BUFGMUX obtains a glitch free clock on its output:

I.e., depending on the phase difference between clk_L and clk_H it might effectively skip a clock cycle. My gut feeling is that this doesn't really matter; if the mux introduces too many skips, this will picked up as a hint to run a little bit quicker. In fact; this is the whole point of this construction - to deal with clock drift.

---

A few variables affect this construction:

1. The minimum difference between freq(clk_L) and freq(clk_H). Although on average we'll reach some desired frequency, at any given point we'll either run quickly or slowly. This needs to accounted for in the size of our elastic buffers (or some other buffer).
2. How often the frequency manager can/will switch between clocks. Longer periods mean more fine-grained control over the average clock frequency, but like (1) this needs to be accounted for in buffer sizes.

---

Resources:

• 7 Series FPGAs Clocking Resources, UG472
• an858, an909, an959

[martijnbastiaan]

I just discussed this with Martin; the custom boards are unlikely to arrive any time soon, as no one has started working on them yet. For this year's implementation a very coarse clock correction mechanism with big buffers is perfectly fine.

[martijnbastiaan]

This would affect #42. Divergence could now only be measured across long periods.

[vmchale]

If I may ask, what's the story for simulating the custom boards? Or do we already have that.

[martijnbastiaan]

Related: https://docs.xilinx.com/v/u/13.1-English/pg057-fifo-generator

[martijnbastiaan]

If I may ask, what's the story for simulating the custom boards? Or do we already have that.

So for prosperity: this is one of the Clash SOWs. Basically we'd want to add a Signal to a Clock, indicating how long each period of a clock cycle is. This would allow things like unsafeSynchronizer to simulate the behavior of signals with a varying clock frequency (like clk in the diagram).

[martijnbastiaan]

It looks like around the 200 MHz mark we can expect a PLL/MCMM to generate clock frequencies differing by ~10 MHz. Thinking out loud a bit about the precision we might achieve and how it affects buffer sizes..:

• A precision of 1 Hz using a counter :: Index 10_000_000_000 and a threshold :: Index 10_000_000_000 and clk_select = counter <= threshold.
• To make the math a bit easier, let's use counter :: Unsigned 30 and threshold :: Unsigned 30.
• If we're unlucky, our threshold is exactly half, and two connected nodes switch their clocks in opposite direction. Thus, the FIFO would need to store 2^29 elements or 2^29 * 8 ~ 40000000000 bytes ~ 37 GiB when storing 64-bit elements. This is slightly 🤏 more than we have in blockrams.
• We can do better by switching four times every 2**30 cycles (instead of two times). Now we count twice with counter :: Unsigned 29:
  • We do clk_select = counter <= threshold / 2 until the counter saturates, then:
  • We do clk_select = counter <= (threshold / 2) + (threshold % 2) until the counter saturates
  • Repeat
• This would basically look like:

• The (threshold % 2) is there to make sure we don't lose accuracy. With even more counter "steps" we need to make sure we distribute this remainder over time (instead of appending it to the last counter cycle).
• If we want to get it down to a more reasonable 1 MiB, we'd need to half about ~7 times. That is 128 count "steps".

The math is all very approximate, I'm sure there's at least one off-by-one error in there.

[martijnbastiaan]

The diagram is supposed to read:

• switching thrice => switching four times
• switching four times => switching eight times

But drawio doesn't let me edit it anymore for some reason.. :-|

[martijnbastiaan]

We're not going to do this, given that we now have the clock multiplier boards :).