-
Notifications
You must be signed in to change notification settings - Fork 0
Adapters: Redis vs. WriteCached vs. CloudSearch
Redis::MetadataAdapter is a functional clone of Memory::MetadataAdapter with one extra method (#gone?) added to its query service. In all other respects, Redis can be used as a fully functional primary metadata store, provided you have a redis server that is properly configured to save and persist its database.
# Instantiate a Redis::MetadataAdapter whose keys never expire to use as
# a primary metadata store
adapter = Valkyrie::Persistence::Redis::MetadataAdapter.new(expiration: nil)
Redis::QueryService#gone?(id:) returns true if the store knows of a pending delete of the object identified by id. This allows the store to track unsettled deletes separate from objects that simply don't exist. This only works when using the Redis adapter as an expiring cache – if expiration is nil, the persister will just delete deleted keys instead of marking them as gone.
WriteCached::MetadataAdapter acts as a readback cache for a delayed-write primary backend. For example, if your primary backend takes 5-10 minutes to settle/commit new writes, wrapping it in a WriteCached adapter with a 15 minute expiration will provide a rolling cache of the last 15 minutes of writes (including deletes) to cover the delay.
# Instantiate a WriteCached::MetadataAdapter using a Redis::MetadataAdapter
# with a 15 minute expiration time to cover any commit/readback delays in
# the primary backend
adapter = Valkyrie::Persistence::WriteCached::MetadataAdapter.new(
primary_adapter: your_primary_backend_metadata_adapter,
cache_adapter: Valkyrie::Persistence::Redis::MetadataAdapter.new(
expiration: 15.minutes
)
)
When WriteCached::QueryService receives a query, it runs it against the primary backend as well as the cache. It then returns an iterator that will yield all of the results found in the backend, replacing each one with the copy in the cache if there is one, and skipping any for which the cache has delete markers. After the backend's results are exhausted, it will yield any newly created results from the cache that the backend hasn't settled yet. While this algorithm might not be perfect (yet), it passes all of Valkyrie's shared specs so far using both a delayed-commit solr instance and an AWS CloudSearch instance as the backend.
The reconciliation logic can be found in the WriteCached::QueryService::ReconcilingEnumerator class at the bottom of lib/valkyrie/persistence/write_cached/query_service.rb.
CloudSearch::MetadataAdapter is provided as an add-on gem for Valkyrie. It provides a primary Valkyrie metadata storage backend (CloudSearch::BasicMetadataAdapter), but depends on a WriteCached::MetadataAdapter to make up for CloudSearch's delayed commits. The default implementation in the gem uses a Redis::MetadataAdapter as the cache, but any adapter that fulfills the "shared specs + #gone?" contract will suffice.