Go Language Support – Milestone v0.2.0

[marcellanz]

Fixes #41

Features and changes for the "Go Support 0.2.0" release:

• CRDT Support
• Eventsoured re-implementation
• New API
• Integration Tests for the CRDT implementation, to be used for the TCK
• Integration Tests for the Eventsourced implementation
• Adoption of new Cloudstate Model based TCK tests
• Protobuf cleanup for go packages
• Eventsourced TCK Support
• Eventsourced effects and forward support
• Significant test coverage increase
• Dapr contrib model support with new CRDT support
• Chat example
• CRDT Shopping Cart example
• devcontainer.json Support
• Updated documentation
• CRDT and Effects & Forwards documentation

CRDT support has been added as the second state model supported. Together with the Eventsoured implementation, stream runners where introduced which enables multiple entities of the same id to be run at the same time. The first (kind of naive) implementation v0.1.x had a bug where solely one entity of the same id where able to run at the same time.

The Eventsourced support was re-implemented using the CRDT stream runner and its way to run a messages stream. Similar, error handing was adopted and also the new atomic command handling error behaviour with entity restart support has been added.

The Integration Tests for the CRDT implementation follow the new model-based approach of the Cloudstate TCK. They run with an in-memory gRPC Client and Server and are written in Go. These tests are yet not available in the Cloudstate TCK and complement them having local functional tests and also tests not done by the TCK itself.

The Protobuf cleanup for go packages where necessary because the go_package used so far, where incompatible and incorrect. This is fixed in this milestone and also fixed in the main Cloudstate repository.

With Eventsoured TCK Support also Eventsourced effects and forward support was correctly implemented and validated with the TCK.

A Significant test coverage increase was achieved with about 80 integration- and 150 unit-tests in this milestone.

Updated documentation and CRDT documentation is done with the new Antora based template.

The new API has been used in different contexts of available examples and public use of cloudstate. One is the Dapr contrib model support which uses a CRDT state model. This branch https://github.com/mrcllnz/components-contrib/tree/feature/cloudstate_crdt_support implements the Dapr Cloudstate component with the new Go API.

With the New API of this milestone, the API interfacing cloudstate instances as well as handling state model have changed slightly. For event sourcing, one is the removal of the cloudstate.EventEmitter type to be embedded with every entity, instead the eventsourced.EntityHandler has to be implemented providing HandleCommand and HandleEvent methods. Embedding a type as part of an API seemed a bit too awkward. With this change, the Go context.Context typed parameter of command and event handlers have been changed to eventsource.Context that embeds a context.Context. This simply prevents an unfortunate API like

func (sc *ShoppingCart) AddItem(ctx context.Context, c *eventsourced.Context, li *AddLineItem) (*empty.Empty, error)

where two context parameters are provided, one most of the time never used. Instead, the context.Context context is embedded and available through the API. Embedding context.Context is discouraged but not completely disallowed. The topic is well known in the Go community and there is work being done to relax recommendation against putting Contexts in structs. The narrative for context.Context is basically to discourage sharing state by embedding a context in structs and use it instead, to get the contexts passed along chains of function calls and use them with their purpose. We could have embedded the cloudstate eventsourced.Context within a context.Context with context.WithValue as other similar projects do it, but I found it too much of a stretch to get them again out again of a context.Context context.

Second, the Snapshotter and SnapshotHandler interfaces were combined into the optional eventsourced.Snapshooter interface.

Regarding error handling, the in Cloudstate otherwise well known context.Fail method to be used to signal the failure of a running context, was replaced with the in Go idiomatic return of an error as a value to be returned explicitly. This transformed the following kind of awkward code snipped:

func (sc *ShoppingCart) AddItem(ctx *eventsourced.Context, li *AddLineItem) (*empty.Empty, error) {
	if li.GetQuantity() <= 0 {
                ctx.Fail(fmt.Errorf("cannot add negative quantity of to item %q", li.GetProductId()))
		return nil, nil
	}
        // ...
}

into:

func (sc *ShoppingCart) AddItem(ctx *eventsourced.Context, li *AddLineItem) (*empty.Empty, error) {
	if li.GetQuantity() <= 0 {
		return nil, fmt.Errorf("cannot add negative quantity of to item %q", li.GetProductId())
	}
        // ...
}

which is much more explicit as it embraces the principle of Errors are values in Go.

Regarding the API of CRDT types, a decision was made how to represent, or better said encode, values in structures like maps, sets and registers. We follow the interface{} says nothing proverb, where we simply don't put empty interface typed values into a map, set or register. Until probably 2021, when Go gets generics-support, we put *any.Any values into maps, sets and registers. There is no magic of how to find out what it is in a map, the type used for ORMaps and ORSets as well as LWWRegisters. Go maps actually are generic types, where the compiler generates code for declared types like map[string]*shopping.Cart but this is not available in a dynamic way like we need it for our maps, sets and registers. While *any.Any types are kind of like empty interfaces they are more explicit that anything else in the context of Cloudstate, Cloudstate Go users know that they implement gRPC services and that their representation have a relationship to be*any.Any types.

Second, as we put *any.Any values into maps, sets and registers, we have to decide when to marshal (encode) values into *any.Any values. We could have chosen to encode values at times when cloudstate protocol replies where prepared to be sent (ActionReplies, CRDT State and Deltas) or right when they're inserted into CRDT types. I've chosen the approach to fail fast (thanks Joe) and put the burden to the API user. It's not that much of a hassle I think, it also shows how explicit we use and implement state-model as we don't hide them (actually we can't) inside an implementation of a Set or Map with then some implicit or explicit restrictions other Language Support libraries (rightfully in their field) have done.

There are occasions where this decision brings an additional decode and probably encode step like the getCart method in the CRTD shopping cart example:

if err := encoding.DecodeStruct(state.GetLwwregister().GetValue(), &item); err != nil {
     return nil, err
}

as we have to get values out of a CRDT type again and, to return them in the form of a gRCP service return type. If performance-critical, this can be explicitly optimized. The next iteration of the support library will show if that was the right decision or if it has to be adjusted. I did not want to optimize without empirical data. Also, any.Any types for many Cloudstate state model types are the encoded format that gets to be known for the proxy. This said, the second Go proverb chosen in this issue that says: Clear is better than clever is reflected and implemented in this PRs work.

[sleipnir]

Really great works @marcellanz

[pvlugter]

🎉 🚀 Very cool! Awesome work.

[codecov-io]

Codecov Report

Merging #43 into master will increase coverage by 21.87%.
The diff coverage is 67.00%.

@@             Coverage Diff             @@
##           master      #43       +/-   ##
===========================================
+ Coverage   44.24%   66.12%   +21.87%     
===========================================
  Files           9       32       +23     
  Lines         513     1417      +904     
===========================================
+ Hits          227      937      +710     
- Misses        241      354      +113     
- Partials       45      126       +81     

Impacted Files	Coverage Δ
cloudstate/discovery/protosupport.go	45.45% <ø> (ø)
cloudstate/crdt/ormap_getter.go	7.50% <7.50%> (ø)
cloudstate/crdt/context.go	23.52% <23.52%> (ø)
cloudstate/encoding/any_encoding.go	35.29% <35.29%> (ø)
cloudstate/crdt/lwwregister_clock.go	36.36% <36.36%> (ø)
cloudstate/crdt/runner.go	37.71% <37.71%> (ø)
cloudstate/crdt/cmdcontext.go	47.05% <47.05%> (ø)
cloudstate/cloudstate.go	39.39% <48.00%> (+8.46%)	⬆️
cloudstate/eventsourced/entity.go	50.00% <50.00%> (ø)
cloudstate/protocol/descriptor.go	50.00% <50.00%> (ø)
... and 50 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4169acf...4d3686f. Read the comment docs.

[ralphlaude]

Awesome @marcellanz!!

[jboner]

Wow. Well done, @marcellanz, well done.

[pvlugter]

🎉