Aerospike Connection Pooling

[dchoi-viant]

At Viant, we're leveraging this library on both the client and server.

There are some ideas that do not belong exclusively to this repository, mainly the Aerospike connection management system.

Complexities

1. Automated Configuration

The configuration and Aerospike connections are determined by server configurations.
Each mly client is instantiated separately.
Any connection pooling needs to be provided.

Solution 1 - Default global Aerospike connection pooler

Choices:

1. Default global
2. Default isolated
3. Explicitly provided

Concerns:

• Scaling issues: can a single connection support all the operations more opportunely than multiple connections? Will there be any throttling or queuing issues? Check if the Aerospike library provides any inherent configurations for load shedding.
• Goroutine based puts, and no use of singleflight: singleflight may provide a logically coherent form of load shedding. Note that singleflight uses first-in-as-truth (which makes sense given how singleflight returns a value), but in this case, since we are effectively modifying a state, it should be last-in-as-state. We can check if there is a lock-free form of concurrency control provided by Aerospike, that may be leveraged for this.

[dchoi-viant]

Noting some tie-ins
Issue #12 can also deduplicate Aerospike requests due to having less objects to cache.

[dchoi-viant]

Further digging shows potential existing mechanism for handling Aerospike connection pooling.

1. shared/client.WithDataStorer(...) - this provides an option to effectively "copy" the shared/datastore.Storer. This makes it such that callers and clients will need to have an initial instance, then will require a pull of the Storer instance, to be carried to future clients. This will introduce a dependency (only between the first and rest of the instantiations). Based on review, it seems this can be potentially brittle.
2. Further research into a more integrated connection management system for connection pooling. Possibly a new shared/client.Option.

[dchoi-viant]

Current shortest distance solution: rather than instantiating the map[string]*shared/datastore/client.Service in shared/datastore.NewStoresV2() (or similar), provide an argument such that the caller can control the lifetime of the map. Can be injected via usage of shared/client.Option.

[dchoi-viant]

Let's also review the Aerospike connection options passthru.

[dchoi-viant]

From tooling in 33832ba, we observe with 200 mly clients, without sharing, we peak at 500+ Aerospike client connections; with connection sharing, it peaks at about 250.

[dchoi-viant]

v0.16.1, while not ideal, is sufficient to mark this work as complete. Testing tools are included in 0d95dfc.

Will spin off usage and handling of CacheStatusFoundNoSuchKey to another issue.