Lazy Sets

REST API code generation for Go describes several strategies a developer can adopt for implementing Sysl-defined services in Go. Stretching that theme, a speculative but very compelling idea is for SetOf… to be lazy.

func (*TransactionServiceServerImpl) GetTransactions(
    ctx context.Context,
    crn string,
    Backend1GetTxs func(ctx context.Context, crn string) SetOfTx,
    Backend2FetchTrans func(ctx context.Context, custid string) SetOfTx,
) SetOfTransaction {
    cctx, cancel := context.WithCancel(ctx)

    a := Backend1GetTxs(cctx, crn).Map(TxToTransaction)
    b := Backend2FechTrans(cctx, "crn:"+crn).Map(TxToTransaction)

    // Exclude b TxIDs already in a.
    return a.Union(b.ButNot(a.Project("TxID"))).OrCancel(cancel)
}

We've abandoned conventional error-handling because it is impossible to tell upfront whether a call will fail.

This model requires that operators Union, ButNot, etc. understand laziness and the possibility of failure at every point. Each operator returns a SetOf…, which has methods allowing access to elements of the set. The most basic access mode would be to stream the set via a channel:

• (SetOfTx).Stream() <-chan func() (Tx, error)

A core premise of this approach is that if we never actually need the data, it is meaningless to ponder whether the request succeeded or not. In fact, the implementation should not even make the call until it is clear that the result will be needed. For instance, B is not needed in A.Minus(B) when A is empty. In the rare case when you need to ensure that the call happens, you can call Any() just to trigger the behaviour.

An even more radical idea: Observable Sets

The above code is nominally designed to implement a request/response flow (REST or RPC, doesn't matter). However, there is nothing in this code that strictly demands this kind of flow. In fact, the very same code could be wired up to support a push flow. In this model:

1. The backends connect to the backend systems publishing the data, subscribing to transactions for the supplied crn.
2. The backends then return lazy sets that represent the data of interest, without necessarily containing any actual data.
3. In GetTransactions, the expression a.Union(b.ButNot(a.Project("TxID"))) wires these sets up to a computation graph that processes the incoming data into a result.
4. The returned lazy set exposes a subscription mechanism allowing the caller to consume the data asynchronously.
5. As actual data comes streaming in from the backends, the computation graph processes it and pushes the final processed data to the subscribed caller.
6. Once the full dataset is delivered, the intent could be either to end the stream and stop processing, or it could be to keep the channel open and wait for changes to the data. This takes us from lazy sets to observable sets.
7. When changes occur, the active subscription channel and compute graph will process the new information in order to determine what the new result should look like. This could either be simple re-evaluation of the entire data set or some kind of optimised delta algorithm that combines previously seen data and the new information into an update operation streamed to the caller to amend their view of the data.

A key point here is that everything described above assumes absolutely no change to the implementation of GetTransactions. In fact, the same running process could implement both flows off the very same function. It's simply a matter of how the sets are implemented under the hood: eager (conventional), lazy or observable.