Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache recent blocks in memory to reduce load on the db #4215

Merged
merged 7 commits into from
Jan 19, 2023
Merged

Cache recent blocks in memory to reduce load on the db #4215

merged 7 commits into from
Jan 19, 2023

Conversation

neysofu
Copy link
Member

@neysofu neysofu commented Nov 30, 2022

Closes #4154.

@neysofu neysofu requested a review from lutter November 30, 2022 11:49
@neysofu neysofu marked this pull request as draft November 30, 2022 12:05
@neysofu
Copy link
Member Author

neysofu commented Nov 30, 2022

Marked as draft; I need to fix a few things before review.

@neysofu neysofu force-pushed the filippo/chain-head-cache branch from 328c9bc to 87bfadd Compare November 30, 2022 13:52
@neysofu neysofu marked this pull request as ready for review November 30, 2022 13:53
@neysofu neysofu force-pushed the filippo/chain-head-cache branch from 87bfadd to 2a1f6e7 Compare November 30, 2022 14:04
@neysofu
Copy link
Member Author

neysofu commented Dec 1, 2022

  1. Change the TimedCache into a HashMap or Vec.
  2. Make the capacity configurable.

@neysofu neysofu force-pushed the filippo/chain-head-cache branch 2 times, most recently from 12d613e to 7e7bc5c Compare December 5, 2022 13:57
@neysofu
Copy link
Member Author

neysofu commented Dec 5, 2022

@lutter The PR is ready... but remove-then-update fails, and I can't figure out why. Any idea?

lutter
lutter requested changes Dec 5, 2022
View reviewed changes
Copy link
Collaborator

@lutter lutter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks generally good. It just occurred to me since this touches a very sensitive part of the code, that we should run this change in the integration cluster before merging it.

store/postgres/src/chain_store.rs Outdated
// We invalidate the entire cache when the chain head changes.
let mut inner = self.inner.write();
inner.blocks.clear();
inner.blocks.insert(0, (chain_head, None));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is overly aggressive, and will reduce the effectiveness of the cache; it should be enough to remove any entry that is not within [chain_head - capacity + 1, chain_head] which would help reduce the churn from this cache. In normal operation, we'd only need to remove one block from the cache when the chain head advances.

I also think it would be simpler if the cache didn't require a call to set_chain_head and instead infer that from set_block; that makes it less likely that future code changes forget to call this and make the cache ineffective.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh .. I just realized that the hashmap is keyed off the distance from the chain head. If using a HashMap, why not just use the block number directly, even if that makes getting the current max block number a bit more expensive? If we want to go with 0 == chain_head, then a VecDeque might be a better data structure.

store/postgres/src/chain_store.rs Outdated
if offset_from_head >= inner.capacity as _ {
return;
}
// We only cache blocks if their ancestor is in the cache (chain head excluded).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The impact of this isn't entirely clear to me; could you add a comment why we do that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment wasn't really truthful, in retrospect. I tried to make a better job at addressing the issue in the comment and I'll probably add some others. We only want to add blocks to the cache if a child of that block is already present in the cache, because that's the easiest way to ensure the block is part of the main chain with the correct head. Otherwise, we might end up storing uncle blocks.

@neysofu neysofu force-pushed the filippo/chain-head-cache branch 2 times, most recently from fa9cd86 to cc866b6 Compare December 13, 2022 16:58
@neysofu neysofu force-pushed the filippo/chain-head-cache branch 2 times, most recently from c461326 to 7735a34 Compare January 3, 2023 10:45
neysofu added 4 commits January 5, 2023 12:39
@neysofu
store: cache recent blocks in memory
be5e9b6
@neysofu
store: refactor RecentBlocksCache's Inner methods
4d62f71
@neysofu
store: fix ancestor/child naming in RecentBlocksCache
2391091
@neysofu
store: use a BTreeMap for RecentCachedBlocks
98846fa 
This commit significatly alters the design of RecentCachedBlocks. Thes
most prominent changes are:

1. We don't require a `.set_chain_head` call anymore. Block insertion
   and chain head update attempt are now the same thing.
2. We don't evict all items in the cache anymore every time the chain
   head advances.
3. Unlike the previous data structure, we are now limited to storing a
   contiguous range of blocks in the cache. This is not really a
   drawback (as the cache contents will usually be identical, i.e. the
   last N blocks before the chain head), but it's worth pointing out.
@neysofu neysofu force-pushed the filippo/chain-head-cache branch from 7735a34 to 98846fa Compare January 5, 2023 11:39
lutter
lutter requested changes Jan 6, 2023
View reviewed changes
Copy link
Collaborator

@lutter lutter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good in general; I would mostly like to understand the logic in insert_block a little better before approving

@neysofu
Copy link
Member Author

neysofu commented Jan 9, 2023

Integration testing went well - it's on fraction anybody wants to take a look.

@neysofu
store: fix RecentBlocksCache for NEAR
6d46bac 
This commit removes the requirement of continuous block number ranges
for RecentBlocksCache. Two reasons for that:

- NEAR doesn't have continuous block number ranges, so the previous
  cache design would have limited effectiveness for NEAR chains.
- Parent hash comparisons are actually enough to uphold all invariants,
  so block number checks were unnecessary in the first place.

The actual code changes are small and mostly limited to `insert_block`.
@neysofu
Copy link
Member Author

neysofu commented Jan 17, 2023

Addressed all comments.

lutter
lutter approved these changes Jan 19, 2023
View reviewed changes
Copy link
Collaborator

@lutter lutter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work. Curious to see it in action.

@neysofu neysofu merged commit 5945af0 into master Jan 19, 2023
@neysofu neysofu deleted the filippo/chain-head-cache branch January 19, 2023 12:42
vutran1710 added a commit to KyberNetwork/graph-node that referenced this pull request Feb 1, 2023
@vutran1710 @dependabot
@neysofu @lutter @saihaj @azf20 @tilacog @mangas @evaporei @leoyvens @maoueh @islishude @dotansimha @mudabbirk @Theodus @rattrap @jeffywu @sduchesneau
Sync fork (#37)
f3ecf35 
* build(deps): bump blake3 from 1.3.1 to 1.3.2 (graphprotocol#4194)

Bumps [blake3](https://github.com/BLAKE3-team/BLAKE3) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/BLAKE3-team/BLAKE3/releases)
- [Commits](BLAKE3-team/BLAKE3@1.3.1...1.3.2)

---
updated-dependencies:
- dependency-name: blake3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump cid from 0.8.6 to 0.9.0 (graphprotocol#4193)

Bumps [cid](https://github.com/multiformats/rust-cid) from 0.8.6 to 0.9.0.
- [Release notes](https://github.com/multiformats/rust-cid/releases)
- [Commits](multiformats/rust-cid@v0.8.6...v0.9.0)

---
updated-dependencies:
- dependency-name: cid
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump indexmap from 1.9.1 to 1.9.2 (graphprotocol#4192)

Bumps [indexmap](https://github.com/bluss/indexmap) from 1.9.1 to 1.9.2.
- [Release notes](https://github.com/bluss/indexmap/releases)
- [Changelog](https://github.com/bluss/indexmap/blob/master/RELEASES.md)
- [Commits](indexmap-rs/indexmap@1.9.1...1.9.2)

---
updated-dependencies:
- dependency-name: indexmap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* all: fix some Clippy warnings (graphprotocol#4178)

* node, store: Add `graphman stats target`

* store: Make resolve_column_names return SqlName

* node, store: Allow analyzing all tables with `graphman analyze`

* node, store: Add `graphman stats set-target`

* store/postgres: insist on a database with C locale

* store: remove unused lazy_static! (graphprotocol#4200)

* Update docker-compose.yml

* Update docker-compose.yml

* Fix: run locale check after migrations (graphprotocol#4201)

* store/postgres: run locale check after migrations

* store/postgres: place succes log message after locale check

* Update docker-compose.yml

* Update docker-compose.yml

* ci: fix Postgres' locale in CI

* integration-tests: createdb with C locale

Co-authored-by: tilacog <tilacog@gmail.com>
Co-authored-by: Adam Fuller <azf20@users.noreply.github.com>
Co-authored-by: Filippo Costa <filippo@neysofu.me>
Co-authored-by: Filippo Costa <filippo@edgeandnode.com>

* update locale for parallel tests (graphprotocol#4205)

* store: Unfail subgraph status properly after rewind

This is being done by changing `revert_subgraph_errors` to change both
the `failed` and `status` columns in the case of both being `failed`.

Without this change, rewinding failed subgraphs past the failed
block would result in the `failed` status staying the same.

* tests: Test that rewind resets subgraph health

Ports the `fatal-error` test from the legacy test suite to the new test suite,
and extends it to check that the subgraph health is reset to `Healthy` after a rewind.

* enable firehose eth block ingestor (graphprotocol#4204)

* firehose ingestor transforms (graphprotocol#4216)

* Updated `Substreams` to latest version of Protobuf definition and activated `ProductionMode` by default (graphprotocol#4211)

The production mode is required to benefits from automatic backprocessing and downloading of block scoped data message as they are produced. This will drastically improve the ingestion speed of a substreams (time to gather some metrics!).

Updated also the instructions to re-generate the `substreams.proto` file with more instructions of how we do it.

Co-authored-by: Matthieu Vachon <matt@streamingfast.io>

* Remove unused deps from `graph-server-metrics` (graphprotocol#4224)

* server-metrics: remove unused deps

* metrics-server: remove MetricsServer trait

* build(deps): bump openssl from 0.10.42 to 0.10.43 (graphprotocol#4208)

Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.42 to 0.10.43.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](sfackler/rust-openssl@openssl-v0.10.42...openssl-v0.10.43)

---
updated-dependencies:
- dependency-name: openssl
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* The `SubstreamsBlockStream` was using `request.clone()` side stepping latest cursor value (graphprotocol#4228)

The `request.clone()` does not correctly use the `latest_cursor` value which is the valid up to date in memory cursor to use on re-connection. This led to poisining error in `graph-node` where the same block was processed multiple time because the cursor was not correctly used.

Fixed by moving the request creation directly where it's needed which will use the correct up to date `latest_cursor` value now.

Co-authored-by: Matthieu Vachon <matt@streamingfast.io>

* fail when ens rainbow not present (graphprotocol#4219)

* docker: use exec form for CMD directive (graphprotocol#4217)

* graph, graphql: expose metrics for GraphQL validations failures (graphprotocol#4230)

* `EmptyNodeCapabilities` for chains that don't discriminate nodes (graphprotocol#3985)

* chain(all), graph: EmptyNodeCapabilities

* chain-ethereum: fix PartialOrd for NodeCapabilities

The Ord and PartialOrd implementations of NodeCapabilities currently
break trait rules: `NodeCapabilities {true, false}` is both greater
than and less than (at the same time) `NodeCapabilities {false, true}`.
It'd be nice to get this fixed before we get some obscure bug during
comparisons, or a panic due to std assuming transitivity during
sorts.

* build(deps): bump express in /tests/integration-tests (graphprotocol#4235)

Bumps [express](https://github.com/expressjs/express) from 4.17.1 to 4.18.2.
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](expressjs/express@4.17.1...4.18.2)

---
updated-dependencies:
- dependency-name: express
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build: remove cargo-chef, revert to stable rust (graphprotocol#4187)

* build(deps): bump base64 from 0.13.1 to 0.20.0 (graphprotocol#4241)

Bumps [base64](https://github.com/marshallpierce/rust-base64) from 0.13.1 to 0.20.0.
- [Release notes](https://github.com/marshallpierce/rust-base64/releases)
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md)
- [Commits](marshallpierce/rust-base64@v0.13.1...v0.20.0)

---
updated-dependencies:
- dependency-name: base64
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* graph, node: Move regex into the graph crate

* node, store: Parse index definitions

* node, store: Allow suppressing standard indexes in 'graphman index list'

* graph, node: Colorize the output of 'graphman index list'

* node, store: Use index::Method when creating manual indexes

* node, store: Add SQL printing to `graphman index list`

* node: Fix output from `graphman index list`

* node, store: Parse the index method using FromStr

* graphql: Simplify the arguments for prefetch::fetch

* store: Simplify arguments for Layout.query

* graph, graphql, store: Pass trace flag for query

* all: Allow tracing GraphQL queries via HTTP

* graph, graphql: Include query_id and variables in trace

* graph, node: Print execution errors from `graphman query`

* graph, graphql: Include block number in query trace

* build(deps): bump openssl from 0.10.43 to 0.10.45 (graphprotocol#4267)

Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.43 to 0.10.45.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](sfackler/rust-openssl@openssl-v0.10.43...openssl-v0.10.45)

---
updated-dependencies:
- dependency-name: openssl
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump proc-macro2 from 1.0.47 to 1.0.49 (graphprotocol#4259)

Bumps [proc-macro2](https://github.com/dtolnay/proc-macro2) from 1.0.47 to 1.0.49.
- [Release notes](https://github.com/dtolnay/proc-macro2/releases)
- [Commits](dtolnay/proc-macro2@1.0.47...1.0.49)

---
updated-dependencies:
- dependency-name: proc-macro2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump quote from 1.0.20 to 1.0.23 (graphprotocol#4258)

Bumps [quote](https://github.com/dtolnay/quote) from 1.0.20 to 1.0.23.
- [Release notes](https://github.com/dtolnay/quote/releases)
- [Commits](dtolnay/quote@1.0.20...1.0.23)

---
updated-dependencies:
- dependency-name: quote
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump anyhow from 1.0.66 to 1.0.68 (graphprotocol#4257)

Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.66 to 1.0.68.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](dtolnay/anyhow@1.0.66...1.0.68)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump semver from 1.0.14 to 1.0.16 (graphprotocol#4256)

Bumps [semver](https://github.com/dtolnay/semver) from 1.0.14 to 1.0.16.
- [Release notes](https://github.com/dtolnay/semver/releases)
- [Commits](dtolnay/semver@1.0.14...1.0.16)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump num_cpus from 1.14.0 to 1.15.0 (graphprotocol#4265)

Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](seanmonstar/num_cpus@v1.14.0...v1.15.0)

---
updated-dependencies:
- dependency-name: num_cpus
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump termcolor from 1.1.2 to 1.1.3 (graphprotocol#4264)

Bumps [termcolor](https://github.com/BurntSushi/termcolor) from 1.1.2 to 1.1.3.
- [Release notes](https://github.com/BurntSushi/termcolor/releases)
- [Commits](BurntSushi/termcolor@1.1.2...1.1.3)

---
updated-dependencies:
- dependency-name: termcolor
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Updated link to graph-cli networks list link (graphprotocol#4246)

* Update protobuf-related dependencies (graphprotocol#4272)

* ci: run code coverage stats every 3 days (graphprotocol#4188)

* fix cloud build (graphprotocol#4279)

* Fix attestability of ValueParseError, EnumCoercionError, ScalarCoercionError (graphprotocol#4278)

* Fix attestability of ValueParseError

* Fix attestability of EnumCoercionError, ScalarCoercionError

* build(deps): bump git-testament from 0.2.0 to 0.2.2 (graphprotocol#4275)

Bumps [git-testament](https://github.com/kinnison/git-testament) from 0.2.0 to 0.2.2.
- [Release notes](https://github.com/kinnison/git-testament/releases)
- [Commits](kinnison/git-testament@0.2.0...0.2.2)

---
updated-dependencies:
- dependency-name: git-testament
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: Update ipfs api (graphprotocol#4283)

* build(deps): bump blake3 from 1.3.2 to 1.3.3 (graphprotocol#4209)

Bumps [blake3](https://github.com/BLAKE3-team/BLAKE3) from 1.3.2 to 1.3.3.
- [Release notes](https://github.com/BLAKE3-team/BLAKE3/releases)
- [Commits](BLAKE3-team/BLAKE3@1.3.2...1.3.3)

---
updated-dependencies:
- dependency-name: blake3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: Include graft ipfs lookup for raw_yaml in infinite retries (graphprotocol#4284)

* substreams: Fix spurious builds (graphprotocol#4286)

* graphql: Allow gt/lt comparisons with `Bytes` fields

Fixes graphprotocol#4282

* graph, store: Avoid a bind variable for fulltext queries

We control the string we pass as the language, and therefore do not need to
use a bind variable. We just need to make sure we are passing a valid SQL
string constant

* store: Properly account for fulltext columns in inserts

Fixes graphprotocol#2330

* docker: adds support for ipfs auth urls (graphprotocol#4252)

* `v0.29.0` release notes, `NEWS.md` and Cargo package version (graphprotocol#4298)

* Release v0.29.0

* news: release v0.29.0

* build(deps): bump proc-macro2 from 1.0.49 to 1.0.50 (graphprotocol#4295)

Bumps [proc-macro2](https://github.com/dtolnay/proc-macro2) from 1.0.49 to 1.0.50.
- [Release notes](https://github.com/dtolnay/proc-macro2/releases)
- [Commits](dtolnay/proc-macro2@1.0.49...1.0.50)

---
updated-dependencies:
- dependency-name: proc-macro2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump cid from 0.9.0 to 0.10.1 (graphprotocol#4289)

Bumps [cid](https://github.com/multiformats/rust-cid) from 0.9.0 to 0.10.1.
- [Release notes](https://github.com/multiformats/rust-cid/releases)
- [Changelog](https://github.com/multiformats/rust-cid/blob/master/CHANGELOG.md)
- [Commits](multiformats/rust-cid@v0.9.0...v0.10.1)

---
updated-dependencies:
- dependency-name: cid
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Cache recent blocks in memory to reduce load on the db (graphprotocol#4215)

* store: cache recent blocks in memory

* store: refactor RecentBlocksCache's Inner methods

* store: fix ancestor/child naming in RecentBlocksCache

* store: use a BTreeMap for RecentCachedBlocks

This commit significatly alters the design of RecentCachedBlocks. The
most prominent changes are:

1. We don't require a `.set_chain_head` call anymore. Block insertion
   and chain head update attempt are now the same thing.
2. We don't evict all items in the cache anymore every time the chain
   head advances.
3. Unlike the previous data structure, we are now limited to storing a
   contiguous range of blocks in the cache. This is not really a
   drawback (as the cache contents will usually be identical, i.e. the
   last N blocks before the chain head), but it's worth pointing out.

* rust: upgrade to 1.66

* store: readability improvements to RecentBlocksCache

* store: fix RecentBlocksCache for NEAR

This commit removes the requirement of continuous block number ranges
for RecentBlocksCache. Two reasons for that:

- NEAR doesn't have continuous block number ranges, so the previous
  cache design would have limited effectiveness for NEAR chains.
- Parent hash comparisons are actually enough to uphold all invariants,
  so block number checks were unnecessary in the first place.

The actual code changes are small and mostly limited to `insert_block`.

* build(deps): bump tower-test from `c9d84cd` to `b01bb12` (graphprotocol#4296)

Bumps [tower-test](https://github.com/tower-rs/tower) from `c9d84cd` to `b01bb12`.
- [Release notes](https://github.com/tower-rs/tower/releases)
- [Commits](tower-rs/tower@c9d84cd...b01bb12)

---
updated-dependencies:
- dependency-name: tower-test
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump prost from 0.11.5 to 0.11.6 (graphprotocol#4297)

Bumps [prost](https://github.com/tokio-rs/prost) from 0.11.5 to 0.11.6.
- [Release notes](https://github.com/tokio-rs/prost/releases)
- [Commits](tokio-rs/prost@v0.11.5...v0.11.6)

---
updated-dependencies:
- dependency-name: prost
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump termcolor from 1.1.3 to 1.2.0 (graphprotocol#4294)

Bumps [termcolor](https://github.com/BurntSushi/termcolor) from 1.1.3 to 1.2.0.
- [Release notes](https://github.com/BurntSushi/termcolor/releases)
- [Commits](BurntSushi/termcolor@1.1.3...1.2.0)

---
updated-dependencies:
- dependency-name: termcolor
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump atomic_refcell from 0.1.8 to 0.1.9 (graphprotocol#4290)

Bumps [atomic_refcell](https://github.com/bholley/atomic_refcell) from 0.1.8 to 0.1.9.
- [Release notes](https://github.com/bholley/atomic_refcell/releases)
- [Commits](https://github.com/bholley/atomic_refcell/commits)

---
updated-dependencies:
- dependency-name: atomic_refcell
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump bumpalo from 3.7.0 to 3.12.0 (graphprotocol#4306)

* Env. vars. for forking (graphprotocol#4308)

* adding env params for forking opts

* adding documentation

Co-authored-by: Jeff Wu <jeffywu@pm.me>

* build(deps): bump prost-types from 0.11.5 to 0.11.6 (graphprotocol#4312)

* build(deps): bump git-testament from 0.2.2 to 0.2.4 (graphprotocol#4313)

* Add causality region column and implement isolation rules (graphprotocol#4162)

* feature(offchain): Add `causality_region` column to entity tables

For now this just tracks the tables that need the column and adds the
column to the DDL, but still unconditionally inserts 0. Inserting the
correct causality region is follow up work.

* store: Move `has_causality_region` to manifest, rename to `entities_with_causality_region`

* *: Add `causality_region` to EntityKey

The tricky part was changing `get_many` to return the entity key.

* store: Insert the causality region

* store: Read isolation between causality regions

It was just necessary to make sure that `find` and `find_many` use
the causality region in their where clause.

* fix: Fix release build

* provider: Make stop idempotent

Callers wanted that anyways, and it helps tests.

* tests: Refactor file ds test to use events

* tests: Test conflict between onchain and offchain

* tests: Test conflict between offchain and offchain

* test: Fix unit test

* tests: Improve tests and comments to address review

* fix: Change migration to add column 'if not exists'

* store: Add a materialized view 'info.chain_sizes'

* config: Bump default ipfs timeout to 60 seconds (graphprotocol#4324)

It used to be that 30 seconds was sufficient, but now we see ipfs
requests taking minutes to find a file in the DHT. So 60 seconds
seems like a reasonable step.

* strip all null bytes from utf8 strings that come from substreams, like it's done on other sources (graphprotocol#4328)

* fix test

* add protobuf

* fix database setup for unit test on kyber-ci

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Filippo Costa <filippocosta.italy@gmail.com>
Co-authored-by: David Lutterkort <lutter@watzmann.net>
Co-authored-by: Saihajpreet Singh <saihajpreet.singh@gmail.com>
Co-authored-by: Adam Fuller <azf20@users.noreply.github.com>
Co-authored-by: Tiago Guimarães <tilacog@gmail.com>
Co-authored-by: Filippo Costa <filippo@neysofu.me>
Co-authored-by: Filippo Costa <filippo@edgeandnode.com>
Co-authored-by: Filipe Azevedo <filipe@azevedo.io>
Co-authored-by: Eva Pace <eba.pachi@gmail.com>
Co-authored-by: Leonardo Yvens <leoyvens@gmail.com>
Co-authored-by: Matthieu Vachon <matthieu.o.vachon@gmail.com>
Co-authored-by: Matthieu Vachon <matt@streamingfast.io>
Co-authored-by: Matthieu Vachon <matthieu.vachon@dfuse.io>
Co-authored-by: Shude Li <islishude@gmail.com>
Co-authored-by: Dotan Simha <dotansimha@gmail.com>
Co-authored-by: mudabbirk <59587832+mudabbirk@users.noreply.github.com>
Co-authored-by: Theo Butler <theodusbutler@gmail.com>
Co-authored-by: Leo <leonard.mocanu@ymail.com>
Co-authored-by: Jeff Wu <jeffywu@pm.me>
Co-authored-by: Stéphane Duchesneau <stephane.duchesneau@gmail.com>
Co-authored-by: Vu Tran <vutran@Vus-MacBook-Pro.lan>
Co-authored-by: Vu Tran <vutran@Vus-MacBook-Pro.local>
@lutter lutter mentioned this pull request Apr 19, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maoueh maoueh maoueh left review comments

@lutter lutter lutter approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Cache recent blocks in memory
3 participants
@neysofu @lutter @maoueh