Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: chain sync projection primitives #520

Merged
merged 16 commits into from
Nov 18, 2022
Merged

feat: chain sync projection primitives #520

merged 16 commits into from
Nov 18, 2022

Conversation

mkazlauskas
Copy link
Member

@mkazlauskas mkazlauskas commented Nov 13, 2022 •
edited
Loading

Context

In order to create a database that is optimized for our provider queries, we'll need to project events from Chain Sync protocol

Proposed Solution

See README of the new projection package and the last commit (cc4bcea)

Important Changes Introduced

See the commit log.

Follow-up work

projection:

  • Create an abstraction for "store projectors" and implementation for stake pools and stake keys
  • Implement postgres store/repository for stake pools and stake keys

ogmios:

  • Complete the Ogmios transaction to core mapping. This PR only added mapping of certificates, the rest of transaction fields are currently not mapped.
  • Create a source observable from Ogmios ChainSync client. Also, create an abstraction to initialize evtCache$ for withRolledBackEvents

golden-test-generator:

  • Write tests for existing functionality
  • Start Ogmios ChainSync from some block (not necessarily from origin)

@mkazlauskas mkazlauskas requested review from rhyslbw and a team November 13, 2022 18:29
@mkazlauskas mkazlauskas force-pushed the feat/generalized-block-handler branch from f390ac1 to cc4bcea Compare November 13, 2022 20:57
rhyslbw
rhyslbw requested changes Nov 14, 2022
View reviewed changes
Copy link
Member

@rhyslbw rhyslbw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really nice so far @mkazlauskas!

  1. 8a41a2f seems like a fix.
  2. Let's drop cae91f0 given there's no implementation to give us test coverage.
  3. I think we should be keeping the projection package free from Ogmios implementations, as it makes sense for those to be scoped to the ogmios package.

yarn-project.nix Outdated Show resolved Hide resolved
packages/core/src/Provider/Provider.ts Show resolved Hide resolved
packages/projection/package.json Outdated Show resolved Hide resolved
packages/projection/src/operators/withRolledBackEvents.ts Outdated Show resolved Hide resolved
@mkazlauskas mkazlauskas force-pushed the feat/generalized-block-handler branch 2 times, most recently from ba7b405 to b548aec Compare November 14, 2022 14:48
@mkazlauskas
Copy link
Member Author

Addressed requested changes and added 1 of my own: b548aec replaced blockHeight with slot in CertificatePointer because this type is very similar to the pointer used in 'pointer addresses', and it uses slots. Better keep them aligned.

Ready for re-review.

@mkazlauskas mkazlauskas requested review from rhyslbw and a team November 14, 2022 15:00
@mkazlauskas mkazlauskas force-pushed the feat/generalized-block-handler branch from 31bd0d3 to 4524a62 Compare November 15, 2022 15:51
mirceahasegan
mirceahasegan previously approved these changes Nov 16, 2022
View reviewed changes
Copy link
Collaborator

@mirceahasegan mirceahasegan left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My current review is a bit shallow, but it looks good to me 👽.
I would prefer a more in-depth re-review after the squash.

@mkazlauskas mkazlauskas force-pushed the feat/generalized-block-handler branch from 4524a62 to 0205fbe Compare November 16, 2022 14:33
@mkazlauskas mkazlauskas force-pushed the feat/generalized-block-handler branch from 0205fbe to 35c22a4 Compare November 16, 2022 14:41
rhyslbw
rhyslbw requested changes Nov 17, 2022
View reviewed changes
Copy link
Member

@rhyslbw rhyslbw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now would be a good time to remove the ignore for the GTG package if it's able to be made compliant without to much effort.

packages/wallet/package.json Show resolved Hide resolved
packages/projection/package.json Show resolved Hide resolved
packages/projection/README.md Show resolved Hide resolved
packages/projection/README.md Show resolved Hide resolved
packages/core/src/CardanoNode/types/CardanoNode.ts Show resolved Hide resolved
packages/golden-test-generator/src/index.ts Outdated Show resolved Hide resolved
rhyslbw
rhyslbw previously approved these changes Nov 18, 2022
View reviewed changes
@mkazlauskas
fix(golden-test-generator): correct path of package.json
aa02fec
@mkazlauskas
feat(golden-test-generator)!: add chain-sync support
c69cdba 
Rename the existing 'get-blocks' command to 'chain-sync'
add extend it to support simulating rollbacks.

It can still be used to simply get blocks without
specifying any rollbacks, but command name and
the resulting data structure has changed.
@mkazlauskas
feat(golden-test-generator): add block range support to chain-sync
66626a3
@mkazlauskas
fix(golden-test-generator)!: preserve BigInts by replacing serializer…
d951df2 
… to use toSerializableObj util

add metadata for supplied blockHeights option

export getBlockHeaderAndHash
@mkazlauskas
feat(util-rxjs): add blockingWithLatestFrom and toEmpty operators
d2e84a9
@mkazlauskas
refactor(core): make 28 and 32 byte hash types non-generic
a7f90b7 
tsc was implicitly casting to an invalid type in some situations
@mkazlauskas
fix(core): correct hash types in genesis key delegation certificate
10d1f2b 
implement coreToCsl genesis key delegation conversion
@mkazlauskas
feat!: implement ogmiosToCore certificates mapping
aef2e8d 
feat(core): add addressNetworkId and createRewardAccount utils

BREAKING CHANGE: rename block types
- CompactBlock -> BlockInfo
- Block -> ExtendedBlockInfo

BREAKING CHANGE: hoist ogmiosToCore to ogmios package
@mkazlauskas
fix(ogmios): map prevHash='genesis' to undefined
ad47bbb
@mkazlauskas
feat(ogmios): add ogmiosToCore.blockHeader
08fc7dc 
translating the entire block is a lot more expensive than just header
@mkazlauskas mkazlauskas force-pushed the feat/generalized-block-handler branch from 881751f to 6df9255 Compare November 18, 2022 07:26
@rhyslbw rhyslbw self-requested a review November 18, 2022 07:38
rhyslbw
rhyslbw approved these changes Nov 18, 2022
View reviewed changes
Copy link
Member

@rhyslbw rhyslbw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stellar work @mkazlauskas 🚀

mirceahasegan
mirceahasegan approved these changes Nov 18, 2022
View reviewed changes
Copy link
Collaborator

@mirceahasegan mirceahasegan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! 🚀

@rhyslbw rhyslbw merged commit 91b0fc2 into master Nov 18, 2022
@rhyslbw rhyslbw deleted the feat/generalized-block-handler branch November 18, 2022 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mirceahasegan mirceahasegan mirceahasegan approved these changes

@rhyslbw rhyslbw rhyslbw approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mkazlauskas @rhyslbw @mirceahasegan