Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create tiered artifact #12

Open
wants to merge 87 commits into
base: dummy-tiered-caching-base
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
fcd7476
Implemented and tested time took for QuerySearchResult objects for us…
Sep 28, 2023
87cbe2a
Moved setting to QueryPhase.execute()
Oct 2, 2023
cf18eb9
Moved setting to QueryPhase.execute()
Oct 2, 2023
c876de8
spotlessCheck
Oct 3, 2023
93a16d4
Added first draft of caching policy setup
Oct 4, 2023
8d6c8c4
Tiered caching framework draft changes
sgup432 Oct 9, 2023
83590e1
initial commit
Oct 10, 2023
aa0a5d7
fixed permissions
Oct 10, 2023
41dc035
Set up event listener to create RemovalNotification objects, also mis…
Oct 10, 2023
4ba49af
Added misc getter methods
Oct 11, 2023
dcb9fad
thread pool stuff
Oct 11, 2023
b912527
cleanup
Oct 11, 2023
44d5e61
Removed partial cache files from disk
Oct 11, 2023
43e2d1d
fixed double counting of on heap hit count
Oct 11, 2023
aa2d28d
first attempt at fixing cache closing issue in tests
Oct 11, 2023
354c311
misc stuff for testing
Oct 12, 2023
66915b6
Moved more stuff around for debugging issues
Oct 13, 2023
f985e63
fixed various serialization issues relating to silent ehcache put() a…
Oct 13, 2023
ae300e9
Cleanup
Oct 16, 2023
7a7b41f
More cleanup
Oct 16, 2023
ec729d0
Merged in Sagar's serialization changes
Oct 16, 2023
990ba9c
Part 1 of modifying ehcache to take serializable Key
Oct 16, 2023
d21af15
final attempts at doing it generally before learning that wasnt even …
Oct 17, 2023
d152478
Changed Ehcache disk tier implementation to not be generic, fixed per…
Oct 17, 2023
2d200fc
Fixed multiple disk tier instance issue, made spillover test pass
Oct 17, 2023
8f65db4
Removed persistence, cleaned up
Oct 17, 2023
7155afd
cleanup
Oct 17, 2023
c760594
added test entries metric for on heap
Oct 18, 2023
8510637
Fixed issue where multiple mock nodes on one JVM (in IT tests) would …
Oct 18, 2023
0b2e506
Merge branch 'framework-serialized' into update_node_stats
Oct 18, 2023
4680ea7
Added check for new entries field in IT test, added permissions allow…
Oct 18, 2023
c300d8b
Initial implementation for tiered node request cache stats
Oct 20, 2023
9d8d433
Added version checks to streaming functions
Oct 20, 2023
7d03562
Added UT for RequestCacheStats
Oct 23, 2023
c5099f1
Added IT for disk tier stats, plus spotlessApply
Oct 23, 2023
64b4bb3
cleaned up
Oct 23, 2023
8d13a17
Implements a memory-efficient roaring bitmap-based keystore for use i…
Sep 22, 2023
08a4f6d
Adds and tests took time for QuerySearchResult
Oct 9, 2023
dfd9128
Addressed Ankit's comments
Oct 19, 2023
a9ab327
Removed minimum version variable
Oct 20, 2023
4e57f4c
Changed type for tookTime from long to Long, to address Sagar's comment
Oct 26, 2023
1d47f38
Addressed dblock's comment
Oct 31, 2023
e39514b
Adds Serializer interface for use in ehcache disk tier and elsewhere;…
Nov 1, 2023
d45e6cf
Addressed Sagar's comments besides new counter/arraylist setup for re…
Nov 2, 2023
157ca6e
Addressed comments besides moving IT tests into IndicesRequestCacheIT…
Nov 6, 2023
4045616
Implemented/tested counter+removal list setup to allow more removals
Nov 6, 2023
0825316
Merge branch 'keystore-squashed' into feature/tiered-caching
Nov 6, 2023
8e51749
Integrated keystore
Nov 6, 2023
5099968
removed exceptions from interface signatures
Nov 6, 2023
207ca99
Merge branch 'tooktime-squashed' into feature/tiered-caching
Nov 6, 2023
21dab6f
Adds caching policy
Oct 4, 2023
e5cadb0
Merge branch 'caching-policy-squashed' into feature/tiered-caching
Nov 6, 2023
f149eec
Added first-draft integration for caching policy which will be change…
Nov 7, 2023
ee7301a
Added unit test for ehcache tier's took time policy integration
Nov 7, 2023
ee11ed8
Reworked caching policy to sit at tiered cache service level and use …
Nov 8, 2023
5123142
Fixed issue with eviction test for IRC tests
Nov 8, 2023
4865827
first parts of cachevalue rework (nonfunctional)
Nov 8, 2023
695521f
Merge commit '157ca6e9557cc5864c45c930be97efa6f17ea269' into cache-value
Nov 8, 2023
8fb86b7
Almost finished statsholder impl side, found and deleted a lot of old…
Nov 9, 2023
900230d
Finished cachevalue rework + disk stats implementation
Nov 10, 2023
07a2e4b
Simplified RBM size estimator
Nov 11, 2023
a9df136
Merge branch 'keystore-squashed' into feature/tiered-caching
Nov 11, 2023
2f884c2
addressed sagar's comments
Nov 13, 2023
9d88676
merged in serializer changes
Nov 13, 2023
4e8ba0b
Fixed broken test
Nov 13, 2023
41ae53e
Merged in cache value changes
Nov 13, 2023
09748eb
Re-added and tested serializer for IRC key
Nov 13, 2023
8cb83d7
added null check to BR deserializer
Nov 14, 2023
111c4e6
Fixed ehcache init issue in test cases
Nov 14, 2023
55b09fe
cleanup, fixed commented line
Nov 14, 2023
24a3cb9
Fixed issues with closing disk tier in tests, added IT for disk speci…
Nov 14, 2023
6a280c6
removed print statement
Nov 14, 2023
175c5c5
Removed incorrect line from test
Nov 16, 2023
af5178f
added required javadoc
Nov 17, 2023
7d1ce12
Optimized modulo calculation with bitmask
Nov 22, 2023
57e948f
Addressed Sagar's comments on caching policy/generics
Nov 22, 2023
c51e0f3
Addressed Sagar's other comments
Nov 22, 2023
100ca7b
removed dead code
Nov 22, 2023
c415ec8
removed more dead code
Nov 22, 2023
93fa4f8
Changed memory size estimator to reflect optimized modulo
Nov 27, 2023
433470e
Optimizes modulo using bitmask
Nov 27, 2023
e41321d
dummy commit
Nov 30, 2023
a51da38
Changed github action to run on my fork. REVERT THISgit add .
Nov 30, 2023
f0a26b9
added broken javadoc
Nov 30, 2023
01bcad7
added upload artifact section to workflow, spotlessApply
Nov 30, 2023
c6a93a4
Added notification for REMOVED case in ehcache event listener
Dec 1, 2023
ef019af
fixed artifact path
Dec 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added test entries metric for on heap
  • Loading branch information
Peter Alfonsi committed Oct 18, 2023

Unverified

This user has not yet uploaded their public signing key.
commit c760594789139a686cefc3e51269c1e40ff4fef6
Original file line number Diff line number Diff line change
@@ -52,6 +52,7 @@ public class RequestCacheStats implements Writeable, ToXContentFragment {
private long evictions;
private long hitCount;
private long missCount;
private long entries; // number of entries in the cache

public RequestCacheStats() {}

@@ -60,20 +61,23 @@ public RequestCacheStats(StreamInput in) throws IOException {
evictions = in.readVLong();
hitCount = in.readVLong();
missCount = in.readVLong();
entries = in.readVLong();
}

public RequestCacheStats(long memorySize, long evictions, long hitCount, long missCount) {
public RequestCacheStats(long memorySize, long evictions, long hitCount, long missCount, long entries) { //
this.memorySize = memorySize;
this.evictions = evictions;
this.hitCount = hitCount;
this.missCount = missCount;
this.entries = entries;
}

public void add(RequestCacheStats stats) {
this.memorySize += stats.memorySize;
this.evictions += stats.evictions;
this.hitCount += stats.hitCount;
this.missCount += stats.missCount;
this.entries += stats.entries;
}

public long getMemorySizeInBytes() {
@@ -96,12 +100,17 @@ public long getMissCount() {
return this.missCount;
}

public long getEntries() {
return this.entries;
}

@Override
public void writeTo(StreamOutput out) throws IOException {
out.writeVLong(memorySize);
out.writeVLong(evictions);
out.writeVLong(hitCount);
out.writeVLong(missCount);
out.writeVLong(entries);
}

@Override
@@ -111,6 +120,7 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
builder.field(Fields.EVICTIONS, getEvictions());
builder.field(Fields.HIT_COUNT, getHitCount());
builder.field(Fields.MISS_COUNT, getMissCount());
builder.field(Fields.ENTRIES, getEntries());
builder.endObject();
return builder;
}
@@ -127,5 +137,6 @@ static final class Fields {
static final String EVICTIONS = "evictions";
static final String HIT_COUNT = "hit_count";
static final String MISS_COUNT = "miss_count";
static final String ENTRIES = "entries";
}
}
Original file line number Diff line number Diff line change
@@ -65,7 +65,8 @@ public RequestCacheStats stats(TierType tierType) {
statsHolder.get(tierType).totalMetric.count(),
statsHolder.get(tierType).evictionsMetric.count(),
statsHolder.get(tierType).hitCount.count(),
statsHolder.get(tierType).missCount.count()
statsHolder.get(tierType).missCount.count(),
statsHolder.get(tierType).entries.count()
);
}

@@ -74,17 +75,20 @@ public RequestCacheStats overallStats() {
long totalEvictions = 0;
long totalHits = 0;
long totalMisses = 0;
long totalEntries = 0;
for (TierType tierType : TierType.values()) {
totalSize += statsHolder.get(tierType).totalMetric.count();
totalEvictions += statsHolder.get(tierType).evictionsMetric.count();
totalHits += statsHolder.get(tierType).hitCount.count();
totalMisses += statsHolder.get(tierType).missCount.count();
totalEntries += statsHolder.get(tierType).entries.count();
}
return new RequestCacheStats(
totalSize,
totalEvictions,
totalHits,
totalMisses
totalMisses,
totalEntries
);
}

@@ -98,6 +102,7 @@ public void onMiss(TierType tierType) {

public void onCached(Accountable key, BytesReference value, TierType tierType) {
statsHolder.get(tierType).totalMetric.inc(key.ramBytesUsed() + value.ramBytesUsed());
statsHolder.get(tierType).entries.inc();
}

public void onRemoval(Accountable key, BytesReference value, boolean evicted, TierType tierType) {
@@ -112,6 +117,7 @@ public void onRemoval(Accountable key, BytesReference value, boolean evicted, Ti
dec += value.ramBytesUsed();
}
statsHolder.get(tierType).totalMetric.dec(dec);
statsHolder.get(tierType).entries.dec();
}

static class StatsHolder implements Serializable {
@@ -120,5 +126,6 @@ static class StatsHolder implements Serializable {
final CounterMetric totalMetric = new CounterMetric();
final CounterMetric hitCount = new CounterMetric();
final CounterMetric missCount = new CounterMetric();
final CounterMetric entries = new CounterMetric();
}
}