chore: different periods between send and receive

[cmwylie19]

Description

We lowered the debounce period (to one second) in which our Store Subscribers receive updates from the Watch on the PeprStore resource, meaning they can react to changes faster.

The same debounce period was responsible for sending JSON patches to the PeprStore resource. The cache that holds these updates is updated and the items that were sent are deleted after the JSON patch succeeds. This meant that if we are sending events every second, and the kube-apiserver does not process the patch in a second, then the same events will be sent over multiple intervals. It is unclear if this would make the data incorrect as it is idempotent but it certainly introduces pressure on the network which could create a fallout of other network created problems.

This PR:

• Creates two different debounce periods, one for send and one for receive
• Sends a reason for the reject in SetItemAndWait/RemoveItemAndWait
• Cleans up the timeouts in SetItemAndWait/RemoveItemAndWait
• Adds an array item to the JSON patch to the Store to ensure a watch event occurs to resolve promises in Set/RemoveItemAndWait

e2e: defenseunicorns/pepr-excellent-examples#204

Related Issue

Fixes #1561

Relates to #

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Other (security config, docs update, etc)

Checklist before merging

• Unit, Journey, E2E Tests, docs, adr added or updated as needed
• Contributor Guide Steps followed

[codecov]

Codecov Report

Attention: Patch coverage is 57.14286% with 12 lines in your changes missing coverage. Please review.

Project coverage is 78.24%. Comparing base (65ea7d1) to head (1bc1a14).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/lib/storage.ts	50.00%	10 Missing ⚠️
src/lib/controller/store.ts	50.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1563      +/-   ##
==========================================
- Coverage   78.29%   78.24%   -0.06%     
==========================================
  Files          41       41              
  Lines        1880     1889       +9     
  Branches      444      417      -27     
==========================================
+ Hits         1472     1478       +6     
- Misses        376      409      +33     
+ Partials       32        2      -30     

Files with missing lines	Coverage Δ
src/lib/controller/storeCache.ts	100.00% <100.00%> (ø)
src/lib/controller/store.ts	12.00% <50.00%> (+1.04%)	⬆️
src/lib/storage.ts	83.56% <50.00%> (-1.95%)	⬇️

... and 6 files with indirect coverage changes

[samayer12]

Given the troubleshooting we performed yesterday, this seems like it should also come with a test in pepr-excellent-examples to verify that the store behaves as intended with many sequential writes. I think we can turn @cmwylie19's troubleshooting steps into something scripted so we can catch this issue without manual testing.

[cmwylie19]

Given the troubleshooting we performed yesterday, this seems like it should also come with a test in pepr-excellent-examples to verify that the store behaves as intended with many sequential writes. I think we can turn @cmwylie19's troubleshooting steps into something scripted so we can catch this issue without manual testing.

Here is one -- defenseunicorns/pepr-excellent-examples#204

It does a double SetItemAndWait and RemoveItemAndWait to ensure the promise is not rejected and the store stays idempotent

[samayer12]

E2E test added, good to go!