Unified Event interface

[matejpavlovic]

Problem with the current approach

Mir currently uses Protocol Buffers to directly represent events. While this brings some advantage in terms of potential interoperability and, to a substantial extent, removes the need for explicit marshalling and unmarshalling, it requires the Mir framework itself to be aware of all the possible event types used by the modules (in order to be able to unmarshal all events). This is particularly problematic, since the goal of Mir is to be a stand-alone library independent of any protocol that is built on top of it and, in particular, all the different types of events that the protocol might define.

What we need

Events are all required to be serializable to allow for interoperability (e.g. using multiple programming languages), tracing, and debugging. At the same time, a very simple way of representing events (e.g. only in their serialized form) leads to inefficiencies when exchanging events within the same process (through shared memory). We should thus make sure that an event is only marshalled when necessary. (I.e., one module marshalling it and another module in the same process unmarshalling it, just because the Mir runtime cannot interpret it, is extremely wasteful).

We need a flexible yet efficient way of representing events in Mir. In particular, the approach should be general enough to allow a wide range of custom implementations of protocol-specific events while avoiding unnecessary marshalling and unmarshalling. In other words, Mir needs to be able to unmarshal (and perform basic operations on) an event without knowing its precise type. Thus, all events must be representable both by an object of the module-specific type and as the basic Mir event type, ideally without having to explicitly register module-specific event-processing handlers with the Mir runtime.

Proposed Event interface

We propose the following interface for a Mir event:

type Event interface {

	// Src returns the module that emitted the event.
	// While this information is not always necessary for the system operation,
	// it is useful for analyzing event traces and debugging.
	Src() t.ModuleID

	// Dest returns a list of destinations of the event, describing which modules should process this event.
	Dest() []t.EventDest

	// FollowUps returns a list of follow-up events.
	// Those are dependent events that must be processed only after this event has been fully processed.
	FollowUps() events.EventList

	// Marshal returns a serialized representation of the event
	// as a slice of bytes from which the event can be reconstructed by Mir.
	// Note that this means that the format of the output must be readable by Mir's event deserializer.
	// Also note that Marshal does not necessarily guarantee the output to be deterministic.
	// Even multiple subsequent calls to Marshal on the same Event object might return different byte slices.
	Marshal() []byte

	// IsRaw allows to determine whether this event is raw (returns true) or not (returns false).
	// The only difference between a raw event and one that is not raw is the output of the Data method.
	// - For a raw event, Data returns a byte slice with a serialized representation of the event's data.
	// - For an event that is not raw, Data returns an object that is expected to be directly understood by the caller.
	// This is useful for the module receiving the event, providing a cue on interpreting the return value of Data.
	IsRaw() bool

	// Data returns the event data, i.e., the event's payload.
	// If this is a raw event Data returns a byte slice representing the serialized form of the event data.
	// Otherwise, the type of the returned value is implementation-specific.
	Data() any
}

The set of methods allows for implementing events according to the requirements stated above.
The IsRaw and Data methods enable the encapsulation of any module-specific event payload in an opaque data slice that can be saved within a basic Mir-native implementation of the Event interface. Mir can then manipulate such a raw event, using only the basic functionality required to be implemented by every event (i.e., the methods defined by the Event interface). Once the raw event reaches a module that needs to process the event's data, the module can access the event data by calling Data, unmarshalling it, and so convert the raw module to a protocol-specific one. At the same time, the Event being an interface rather than a rigid struct, it can be handled by the Mir runtime without the need for marshalling the module-specific event data.

Note on t.EventDest: At the time of writing this text, the destination of an event can only be a single module. Thus, in the current implementation, t.EventDest could be replaced by t.ModuleID. However, we envision a generalization of the event dispatching mechanism, where an event's destination could be one or more queues / buffers / channels, to which modules could be registered / subscribed.

Note on the IsRaw method: One might argue that the IsRaw method is not necessary and the information it provides could be encoded directly in the return value of Data. E.g., in a programming language like Go, simply the actual type of the value could indicate whether it has to be unmarshalled or not - a byte slice is always unmarshalled, another type is used directly. There are, however, some issues with this approach.

1. Even in Go, this restricts the types that can be used for representing a module-specific event. In particular, the byte slice couldn't be used as a module-specific event type.
2. In a low-level language like C (our goal is to enable working with events in any language, not just Go), the function signature of Data would probably have to return a void* pointer that is later cast to the module-specific event type. Without further information, however, the caller would not know whether the pointer points to a byte array to be unmarshalled or to an already deserialized data structure ready for use.

[abread]

After some discussion with @matejpavlovic over Zoom, I think we are somewhat in agreement with something like this:

type Event interface {

	// Src returns the module that emitted the event.
	// While this information is not always necessary for the system operation,
	// it is useful for analyzing event traces and debugging.
	Src() t.ModuleID

	// Dest returns the destination module of the event.
	Dest() t.ModuleID

	// Marshal returns a serialized representation of the event
	// as a slice of bytes from which the event can be reconstructed by Mir.
	// Note that this means that the format of the output must be readable by Mir's event deserializer.
	// Also note that Marshal does not necessarily guarantee the output to be deterministic.
	// Even multiple subsequent calls to Marshal on the same Event object might return different byte slices.
	Marshal() []byte
	
	// DebugRepr returns a human-readable representation of the event.
	// While not used by the runtime itself, it can be used by debug tools.
	// Ideally, this representation should be JSON, to support advanced features of debug tools (like pretty printing).
	DebugRepr() string
}

Summary of changes:

• The Data() and IsRaw() methods are removed in favor of native Go fallible casts: every event is a unique Go type (so probably not []byte, but struct SerializedEvent { Ev []byte } is fine), and each module tries downcasting generic events to the specific kinds of events it can process.
  Implementing this in each module is similar to the current dispatch of protobuf events based on the Type field.
  Regarding CGO: it's cumbersome (if not impossible) to pass native go structs through an FFI boundary anyway, so it's best to keep a small amount of glue code on go's side to interact with external modules (dispatching events and calling specific handlers with a C calling convention, or serializing them and sending them to an external process).

• FollowUps() is removed, as it currently seems to serve no purpose. When reinstating the WAL module, it can work in a similar fashion to the current timer module: when

• Dest() is just a module ID again. Advanced pub/sub, queueing and other event redirection mechanisms can be realized with modules dedicated to these purposes.

That said, I'm not sure if it would be worth having the return type be []t.ModuleID, making the runtime capable of delivering an event to a static set of modules contained in the event.If a lot of events require delivery to multiple modules, it may be better to support this in the runtime, than to have modules emit the same event once per destination.

• A new method DebugRepr() (name was not discussed, suggestions are welcome!) is introduced to allow tools like mircat (and any future debugging tools) to display events.
  All events are serializable, which already constrains them to not contain potentially troublesome values such as closures. Still, they may contain recursive data structures which makes standardization on a format challenging.
  To keep it simple Matej proposed to leave the return type of this method as a string, and encourage all event libraries to use a JSON-encoded string by convention. Any debugging tools would try to deserialize JSON (to support a tree view/pretty printing/advanced querying/...), and fallback to displaying a regular string when deserialization fails.

[matejpavlovic]

Nice, thanks for the summary!
I'd suggest naming the DebugRepr() function just String(), as go automatically looks at this function when it is natively printing stuff. I'd also probably rename the Marshal() method to Bytes(), just as a weak personal preference. So it would look like this:

type Event interface {

	// Src returns the module that emitted the event.
	// While this information is not always necessary for the system operation,
	// it is useful for analyzing event traces and debugging.
	Src() t.ModuleID

	// Dest returns the destination module of the event.
	Dest() t.ModuleID

	// Bytes returns a serialized representation of the event
	// as a slice of bytes from which the event can be reconstructed by the module using the event.
	// Note that Bytes does not necessarily guarantee the output to be deterministic.
	// Even multiple subsequent calls to Bytes on the same event object might return different byte slices.
	Bytes() []byte
	
	// String returns a human-readable representation of the event.
	// While not used by the runtime itself, it can be used by asociated tools.
	// Ideally, this representation should be JSON, to support advanced features of debug tools (like pretty printing).
	String() string
}

Regarding the list of module IDs, I'm torn between having the possibility to (probably in rather rare cases) conveniently send the same event to multiple modules and simplifying code that deals with single destinations. E.g., answering the question "Is this event destined to this module?" turns into a simple comparison, as opposed to iterating over a slice. I have a slight tendency towards single destinations, but I'll think about it more.

[abread]

In my opinion, deciding on single vs multiple destinations should come down to how better/worse the runtime can perform.

I say that, because we can move the problem of ergonomics to DSL functions (and have the destination for loop under them). If we keep the codegen approach to create these functions, this is a fairly easy change.

Right now, I'm also inclined towards single destination. I'm skeptical on the possible savings in allocations of output events and event lists compensating the cost of the extra slice in each event.

[matejpavlovic]

Yes I agree. Good point with the allocation overhead. I do expect the vast majority of events to go to only one module, and it's probably better to create a few copies of the same event from time to time then creating a one-element slice for every single event.