Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a centralised cache manager #1005

Merged
merged 16 commits into from
Sep 2, 2022
Merged

Create a centralised cache manager #1005

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
6ea747a
add a cache manager
Chengxuan Aug 24, 2022
0d6de8e
remove deprecated keys
Chengxuan Aug 25, 2022
c49f39c
Merge branch 'config' of github.com:kaleido-io/firefly into cache-man…
Chengxuan Aug 25, 2022
7809337
clean docs and fixing tests
Chengxuan Aug 25, 2022
078e752
remove unused local config
Chengxuan Aug 25, 2022
77eb367
Merge branch 'main' of github.com:hyperledger/firefly into cache-mana…
Chengxuan Aug 26, 2022
37e57c8
add back support for cache size config
Chengxuan Aug 30, 2022
809afc9
Merge branch 'main' of github.com:hyperledger/firefly into cache-mana…
Chengxuan Aug 30, 2022
a1c3e9b
Merge branch 'main' of github.com:hyperledger/firefly into cache-mana…
Chengxuan Sep 1, 2022
e0aa9d7
using mockery mocks for all tests
Chengxuan Sep 1, 2022
48184c3
fixing merging error and add more tests
Chengxuan Sep 2, 2022
43d7c6e
adding tests for all cache initialisation
Chengxuan Sep 2, 2022
8fef4b2
update type for documentation
Chengxuan Sep 2, 2022
f0bc884
adding locking to avoid racing condition
Chengxuan Sep 2, 2022
0d1208e
Merge branch 'main' of github.com:hyperledger/firefly into cache-mana…
Chengxuan Sep 2, 2022
7f7f17b
fixing cyclomatic complexity
Chengxuan Sep 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ $(eval $(call makemock, internal/spievents, Manager, spievents
$(eval $(call makemock, internal/orchestrator, Orchestrator, orchestratormocks))
$(eval $(call makemock, internal/apiserver, FFISwaggerGen, apiservermocks))
$(eval $(call makemock, internal/apiserver, Server, apiservermocks))
$(eval $(call makemock, internal/cache, Manager, cachemocks))
$(eval $(call makemock, internal/metrics, Manager, metricsmocks))
$(eval $(call makemock, internal/operations, Manager, operationmocks))
$(eval $(call makemock, internal/multiparty, Manager, multipartymocks))
Expand Down
156 changes: 81 additions & 75 deletions docs/reference/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,6 @@ nav_order: 2
|---|-----------|----|-------------|
|keyNormalization|Mechanism to normalize keys before using them. Valid options are `blockchain_plugin` - use blockchain plugin (default) or `none` - do not attempt normalization (deprecated - use namespaces.predefined[].asset.manager.keyNormalization)|`string`|`<nil>`

## batch.cache

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cache items for batches|`string`|`<nil>`
|ttl|The time to live (TTL) for the cache|[`time.Duration`](https://pkg.go.dev/time#Duration)|`<nil>`

## batch.manager

|Key|Description|Type|Default Value|
Expand Down Expand Up @@ -108,13 +101,6 @@ nav_order: 2
|password|Password|`string`|`<nil>`
|username|Username|`string`|`<nil>`

## blockchain.ethereum.addressResolver.cache

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|size|The size of the cache|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`1000`
|ttl|The time to live (TTL) for the cache|[`time.Duration`](https://pkg.go.dev/time#Duration)|`24h`

## blockchain.ethereum.addressResolver.proxy

|Key|Description|Type|Default Value|
Expand Down Expand Up @@ -270,13 +256,6 @@ nav_order: 2
|readBufferSize|The size in bytes of the read buffer for the WebSocket connection|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`16Kb`
|writeBufferSize|The size in bytes of the write buffer for the WebSocket connection|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`16Kb`

## blockchainevent.cache

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cache items for blockchain events|`string`|`<nil>`
|ttl|The time to live (TTL) for the cache|[`time.Duration`](https://pkg.go.dev/time#Duration)|`<nil>`

## broadcast.batch

|Key|Description|Type|Default Value|
Expand All @@ -286,19 +265,95 @@ nav_order: 2
|size|The maximum number of messages that can be packed into a batch|`int`|`<nil>`
|timeout|The timeout to wait for a batch to fill, before sending|[`time.Duration`](https://pkg.go.dev/time#Duration)|`<nil>`

## cache

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|enabled|Enables caching, defaults to true|`boolean`|`<nil>`

## cache.addressresolver

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cached items for address resolver|`int`|`<nil>`
|ttl|Time to live of cached items for address resolver|`string`|`<nil>`

## cache.batch

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cached items for batches|`int`|`<nil>`
|ttl|Time to live of cache items for batches|`string`|`<nil>`

## cache.blockchain

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cache items for blockchain plugin cache|`string`|`<nil>`
|ttl|Time to live for blockchain cache items|`string`|`<nil>`
|limit|Max number of cached items for blockchain|`int`|`<nil>`
|ttl|Time to live of cached items for blockchain|`string`|`<nil>`

## cache.blockchainevent

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cached blockchain events for transactions|`int`|`<nil>`
|ttl|Time to live of cached blockchain events for transactions|`string`|`<nil>`

## cache.eventlistenertopic

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cached items for blockchain listener topics|`int`|`<nil>`
|ttl|Time to live of cached items for blockchain listener topics|`string`|`<nil>`

## cache.group

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cached items for groups|`int`|`<nil>`
|ttl|Time to live of cached items for groups|`string`|`<nil>`

## cache.identity

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cached identities for identity manager|`int`|`<nil>`
|ttl|Time to live of cached identities for identity manager|`string`|`<nil>`

## cache.message

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|size|Max size of cached messages for data manager|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`<nil>`
|ttl|Time to live of cached messages for data manager|`string`|`<nil>`

## cache.operations

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cache items for operations|`string`|`<nil>`
|ttl|Time to live for operation cache items|`string`|`<nil>`
|limit|Max number of cached items for operations|`int`|`<nil>`
|ttl|Time to live of cached items for operations|`string`|`<nil>`

## cache.signingkey

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cached signing keys for identity manager|`int`|`<nil>`
|ttl|Time to live of cached signing keys for identity manager|`string`|`<nil>`

## cache.transaction

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|size|Max size of cached transactions|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`<nil>`
|ttl|Time to live of cached transactions|`string`|`<nil>`

## cache.validator

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|size|Max size of cached validators for data manager|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`<nil>`
|ttl|Time to live of cached validators for data manager|`string`|`<nil>`

## cors

Expand Down Expand Up @@ -469,27 +524,13 @@ nav_order: 2
|initDelay|The initial retry delay|[`time.Duration`](https://pkg.go.dev/time#Duration)|`<nil>`
|maxDelay|The maximum retry delay|[`time.Duration`](https://pkg.go.dev/time#Duration)|`<nil>`

## event.listenerTopic.cache

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cache items for blockchain listener topics|`string`|`<nil>`
|ttl|The time to live (TTL) for the cache|[`time.Duration`](https://pkg.go.dev/time#Duration)|`<nil>`

## event.transports

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|default|The default event transport for new subscriptions|`string`|`<nil>`
|enabled|Which event interface plugins are enabled|`boolean`|`<nil>`

## group.cache

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|Max number of cache items for private group addresses|`string`|`<nil>`
|ttl|The time to live (TTL) for the cache|[`time.Duration`](https://pkg.go.dev/time#Duration)|`<nil>`

## histograms

|Key|Description|Type|Default Value|
Expand Down Expand Up @@ -529,13 +570,6 @@ nav_order: 2
|enabled|Enables or disables TLS on this API|`boolean`|`false`
|keyFile|The path to the private key file for TLS on this API|`string`|`<nil>`

## identity.manager.cache

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|limit|The identity manager cache limit in count of items|`int`|`<nil>`
|ttl|The time to live (TTL) for the cache|[`time.Duration`](https://pkg.go.dev/time#Duration)|`<nil>`

## log

|Key|Description|Type|Default Value|
Expand Down Expand Up @@ -568,13 +602,6 @@ nav_order: 2
|message|Configures the JSON key containing the log message|`string`|`<nil>`
|timestamp|Configures the JSON key containing the timestamp of the log|`string`|`<nil>`

## message.cache

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|size|The size of the cache|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`<nil>`
|ttl|The time to live (TTL) for the cache|[`time.Duration`](https://pkg.go.dev/time#Duration)|`<nil>`

## message.writer

|Key|Description|Type|Default Value|
Expand Down Expand Up @@ -764,13 +791,6 @@ nav_order: 2
|password|Password|`string`|`<nil>`
|username|Username|`string`|`<nil>`

## plugins.blockchain[].ethereum.addressResolver.cache

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|size|The size of the cache|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`1000`
|ttl|The time to live (TTL) for the cache|[`time.Duration`](https://pkg.go.dev/time#Duration)|`24h`

## plugins.blockchain[].ethereum.addressResolver.proxy

|Key|Description|Type|Default Value|
Expand Down Expand Up @@ -1361,23 +1381,9 @@ nav_order: 2
|readBufferSize|The size in bytes of the read buffer for the WebSocket connection|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`<nil>`
|writeBufferSize|The size in bytes of the write buffer for the WebSocket connection|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`<nil>`

## transaction.cache

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|size|The size of the cache|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`<nil>`
|ttl|The time to live (TTL) for the cache|[`time.Duration`](https://pkg.go.dev/time#Duration)|`<nil>`

## ui

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|enabled|Enables the web user interface|`boolean`|`<nil>`
|path|The file system path which contains the static HTML, CSS, and JavaScript files for the user interface|`string`|`<nil>`

## validator.cache

|Key|Description|Type|Default Value|
|---|-----------|----|-------------|
|size|The size of the cache|[`BytesSize`](https://pkg.go.dev/github.com/docker/go-units#BytesSize)|`<nil>`
|ttl|The time to live (TTL) for the cache|[`time.Duration`](https://pkg.go.dev/time#Duration)|`<nil>`
|path|The file system path which contains the static HTML, CSS, and JavaScript files for the user interface|`string`|`<nil>`
10 changes: 8 additions & 2 deletions internal/assets/manager_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,13 @@ package assets
import (
"context"
"testing"
"time"

"github.com/hyperledger/firefly/internal/cache"
"github.com/hyperledger/firefly/internal/coreconfig"
"github.com/hyperledger/firefly/internal/txcommon"
"github.com/hyperledger/firefly/mocks/broadcastmocks"
"github.com/hyperledger/firefly/mocks/cachemocks"
"github.com/hyperledger/firefly/mocks/databasemocks"
"github.com/hyperledger/firefly/mocks/datamocks"
"github.com/hyperledger/firefly/mocks/identitymanagermocks"
Expand Down Expand Up @@ -56,13 +59,16 @@ func newTestAssetsCommon(t *testing.T, metrics bool) (*assetManager, func()) {
mpm := &privatemessagingmocks.Manager{}
mti := &tokenmocks.Plugin{}
mm := &metricsmocks.Manager{}
cmi := &cachemocks.Manager{}
ctx := context.Background()
cmi.On("GetCache", mock.Anything).Return(cache.NewUmanagedCache(ctx, 100, 5*time.Minute), nil)
mom := &operationmocks.Manager{}
txHelper := txcommon.NewTransactionHelper("ns1", mdi, mdm)
txHelper, _ := txcommon.NewTransactionHelper(ctx, "ns1", mdi, mdm, cmi)
mm.On("IsMetricsEnabled").Return(metrics)
mm.On("TransferSubmitted", mock.Anything)
mom.On("RegisterHandler", mock.Anything, mock.Anything, mock.Anything)
mti.On("Name").Return("ut").Maybe()
ctx, cancel := context.WithCancel(context.Background())
ctx, cancel := context.WithCancel(ctx)
a, err := NewAssetManager(ctx, "ns1", "blockchain_plugin", mdi, map[string]tokens.Plugin{"magic-tokens": mti}, mim, msa, mbm, mpm, mm, mom, txHelper)
rag := mdi.On("RunAsGroup", mock.Anything, mock.Anything).Maybe()
rag.RunFn = func(a mock.Arguments) {
Expand Down
Loading