Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Access] Index tx results and events #4772

Merged
merged 6 commits into from
Sep 29, 2023

Conversation

peterargue
Copy link
Contributor

Closes: #4739, #4674

This PR adds support for indexing flow.LightTransactionResults and flow.Events from ExecutionData as part of the data indexing.

@peterargue peterargue self-assigned this Sep 27, 2023
@peterargue peterargue changed the title [Access] Index tx results and events from exec data [Access] Index tx results and events Sep 27, 2023
@peterargue peterargue force-pushed the petera/4739-index-tx-results-events branch 2 times, most recently from 339679f to ebc6c8b Compare September 28, 2023 23:43
@peterargue peterargue force-pushed the petera/4739-index-tx-results-events branch from ebc6c8b to eb2e242 Compare September 29, 2023 00:22
Base automatically changed from gregor/index-exeuction-data to master September 29, 2023 01:43
@codecov-commenter
Copy link

codecov-commenter commented Sep 29, 2023

Codecov Report

Attention: 16 lines in your changes are missing coverage. Please review.

Comparison is base (603e204) 55.79% compared to head (743c770) 55.80%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4772      +/-   ##
==========================================
+ Coverage   55.79%   55.80%   +0.01%     
==========================================
  Files         939      939              
  Lines       86860    86902      +42     
==========================================
+ Hits        48462    48498      +36     
+ Misses      34750    34749       -1     
- Partials     3648     3655       +7     
Flag Coverage Δ
unittests 55.80% <67.34%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
storage/badger/batch.go 86.95% <100.00%> (ø)
...dule/state_synchronization/indexer/indexer_core.go 86.56% <84.21%> (-1.67%) ⬇️
utils/unittest/fixtures.go 0.00% <0.00%> (ø)

... and 10 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

log zerolog.Logger
db *badger.DB
Copy link
Contributor

@sideninja sideninja Sep 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish we could avoid passing an instance to the specific badger db, but I guess batch requires it? I don't like having a dependency on a specific implementation. I know this may call for a refactor, but my take on this would be to have an interface for batch (this is now even more true with implementing pebble since the same pattern will have to be established) and then have methods optionally supporting batch. Something like pseudo code:

batch := batcher.New()

s.events.Store(blockID, []flow.EventsList{result.AllEvents()}, storage.WithBatch(batch))
s.serviceEvents.Store(blockID, result.AllServiceEvents(), storage.WithBatch(batch))

batch.Flush()

And then batcher instance is passed in as:

indexer.New(log, badger.NewBatcher(), registers, headers, events, results)

I feel this would be a nice pattern because it will also allow the same for Pebble, but now the Batch type assumes only badger and is very implementation-specific. Just a thought.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good idea. There are other instances in the code where we do this so I'm reluctant to fold a larger refactor into this PR.

How about updating storage.NewBatch to take an interface instead of the whole *badger.DB instance? then all this needs is that interface?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added here: f8c0c34

@@ -227,6 +255,152 @@ func TestExecutionState_IndexBlockData(t *testing.T) {
assert.True(t, testRegisterFound)
})

t.Run("Index Events", func(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice

Copy link
Contributor

@sideninja sideninja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is good to go, but had a design concern that might be out of the scope even if deemed justifiable

@sideninja sideninja self-requested a review September 29, 2023 17:46
@peterargue peterargue added this pull request to the merge queue Sep 29, 2023
Merged via the queue into master with commit f7a3a89 Sep 29, 2023
@peterargue peterargue deleted the petera/4739-index-tx-results-events branch September 29, 2023 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[State Sync] Add transaction results to the execution data indexer
4 participants