Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor auto_refresh cleanup #5813

Merged
merged 2 commits into from
Jan 15, 2025
Merged

Conversation

Sovietaced
Copy link
Contributor

@Sovietaced Sovietaced commented Oct 5, 2024

Why are the changes needed?

This pull request is some follow up work from #5767 to clean up auto refresh so that it has a constructor that returns a pointer to the implementation instead of an interface. This gives the user of the constructor more flexibility (ie. for intra-package testing) while still adhering to the auto_refresh interface.

What changes were proposed in this pull request?

The default auto_refresh cache implementation was updated to the InMemoryAutoRefresh and moved into its own file.
Some additional fields were added to improve assertions given the nature of the asynchronous enqueue/syncing that occurs with the auto refresh cache.

Lastly, the new constructor for the in memory auto refresh cache was updated to use the options pattern so that test-only values or non-default options only have to be added in those situations.

The preexisting helper functions to create refresh caches have been left in place for backwards compatibility.

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Summary by Bito

Refactoring of auto-refresh cache implementation with core functionality moved to InMemoryAutoRefresh type. Introduces flexible constructor pattern using options while maintaining backward compatibility. Adds new fields for improved testing of async operations. Changes are structural without modifying core functionality.

Unit tests added: False

Estimated effort to review (1-5, lower is better): 3

@Sovietaced Sovietaced force-pushed the auto-refresh-cleanup branch from f6c93f4 to eeabd91 Compare October 5, 2024 22:49
Copy link

codecov bot commented Oct 5, 2024

Codecov Report

Attention: Patch coverage is 88.55932% with 27 lines in your changes missing coverage. Please review.

Project coverage is 37.05%. Comparing base (05c85a8) to head (857af1b).
Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
flytestdlib/cache/in_memory_auto_refresh.go 88.55% 18 Missing and 9 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #5813   +/-   ##
=======================================
  Coverage   37.05%   37.05%           
=======================================
  Files        1318     1318           
  Lines      132583   132604   +21     
=======================================
+ Hits        49122    49139   +17     
- Misses      79211    79215    +4     
  Partials     4250     4250           
Flag Coverage Δ
unittests-datacatalog 51.58% <ø> (ø)
unittests-flyteadmin 54.33% <ø> (ø)
unittests-flytecopilot 30.99% <ø> (ø)
unittests-flytectl 62.29% <ø> (-0.05%) ⬇️
unittests-flyteidl 7.23% <ø> (ø)
unittests-flyteplugins 53.85% <ø> (ø)
unittests-flytepropeller 42.66% <ø> (ø)
unittests-flytestdlib 55.29% <88.55%> (+0.15%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@pvditt pvditt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for cleaning this up. Apologies on my end for letting this PR sit for a while. Could you resolve the merge conflicts and then we can get this + #5940 merged?

@Sovietaced
Copy link
Contributor Author

Thank you for cleaning this up. Apologies on my end for letting this PR sit for a while. Could you resolve the merge conflicts and then we can get this + #5940 merged?

Sure

@Sovietaced Sovietaced force-pushed the auto-refresh-cleanup branch from b0d678a to 1ae220c Compare January 15, 2025 08:27
Signed-off-by: Jason Parraga <sovietaced@gmail.com>
@Sovietaced Sovietaced force-pushed the auto-refresh-cleanup branch from 1ae220c to 9a40f0a Compare January 15, 2025 08:28
@Sovietaced Sovietaced requested a review from pvditt January 15, 2025 08:30
@Sovietaced
Copy link
Contributor Author

Should be good for another look @pvditt

Signed-off-by: Jason Parraga <sovietaced@gmail.com>
@flyte-bot
Copy link
Collaborator

flyte-bot commented Jan 15, 2025

Code Review Agent Run #0468bf

Actionable Suggestions - 2
  • flytestdlib/cache/in_memory_auto_refresh.go - 2
Review Details
  • Files reviewed - 3 · Commit Range: 9a40f0a..857af1b
    • flytestdlib/cache/auto_refresh.go
    • flytestdlib/cache/auto_refresh_example_test.go
    • flytestdlib/cache/in_memory_auto_refresh.go
  • Files skipped - 0
  • Tools
    • Golangci-lint (Linter) - ✖︎ Failed
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful

AI Code Review powered by Bito Logo

@flyte-bot
Copy link
Collaborator

Changelist by Bito

This pull request implements the following key changes.

Key Change Files Impacted
Feature Improvement - Auto-refresh Cache Implementation Refactoring

auto_refresh.go - Moved implementation to new file and cleaned up interface

in_memory_auto_refresh.go - Created new file with InMemoryAutoRefresh implementation and options pattern

auto_refresh_example_test.go - Updated example status constant name

Comment on lines +155 to +156
go func(ctx context.Context) {
err := w.sync(ctx)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing panic recovery in worker goroutines

Consider adding error handling for goroutine panics using defer recover(). The current implementation may lead to unhandled panics in worker goroutines.

Code suggestion
Check the AI-generated fix before applying
Suggested change
go func(ctx context.Context) {
err := w.sync(ctx)
go func(ctx context.Context) {
defer func() {
if r := recover(); r != nil {
logger.Errorf(ctx, "Panic in worker goroutine: %v\n%s", r, debug.Stack())
}
}()
err := w.sync(ctx)

Code Review Run #0468bf


Is this a valid issue, or was it incorrectly flagged by the Agent?

  • it was incorrectly flagged

Comment on lines +349 to +356
updatedBatch, err := w.syncCb(ctx, newBatch)

if err != nil {
w.metrics.SyncErrors.Inc()
logger.Errorf(ctx, "failed to get latest copy of a batch. Error: %v", err)
t.Stop()
continue
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider consistent timer stop placement

The sync latency metric timer t is stopped in the error path but continues timing in the success path until line 370. Consider stopping the timer immediately after the sync callback to ensure accurate latency measurements.

Code suggestion
Check the AI-generated fix before applying
Suggested change
updatedBatch, err := w.syncCb(ctx, newBatch)
if err != nil {
w.metrics.SyncErrors.Inc()
logger.Errorf(ctx, "failed to get latest copy of a batch. Error: %v", err)
t.Stop()
continue
}
updatedBatch, err := w.syncCb(ctx, newBatch)
t.Stop()
if err != nil {
w.metrics.SyncErrors.Inc()
logger.Errorf(ctx, "failed to get latest copy of a batch. Error: %v", err)
continue
}

Code Review Run #0468bf


Is this a valid issue, or was it incorrectly flagged by the Agent?

  • it was incorrectly flagged

@pvditt pvditt merged commit e0833b7 into flyteorg:master Jan 15, 2025
53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants