Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Add indices to Scraper database so that queries are quicker #5039

Merged
merged 2 commits into from
Jan 22, 2025

Conversation

ameten
Copy link
Contributor

@ameten ameten commented Dec 18, 2024

Description

Add indices to Scraper database so that queries are quicker.

Scraper with SeaORM version 1.1.1 (currently reverted) reported that some queries are running slow. These indices will improve it.

Related issues

Backward compatibility

Yes

Testing

Tested queries with indices on a separate database where full production database backup was restored

Copy link

changeset-bot bot commented Dec 18, 2024

⚠️ No Changeset found

Latest commit: fbcebc3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link

codecov bot commented Dec 18, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.53%. Comparing base (85e16de) to head (fbcebc3).
Report is 11 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5039   +/-   ##
=======================================
  Coverage   77.53%   77.53%           
=======================================
  Files         103      103           
  Lines        2110     2110           
  Branches      190      190           
=======================================
  Hits         1636     1636           
  Misses        453      453           
  Partials       21       21           
Components Coverage Δ
core 87.80% <ø> (ø)
hooks 79.39% <ø> (ø)
isms 83.68% <ø> (ø)
token 91.27% <ø> (ø)
middlewares 79.80% <ø> (ø)

@ameten ameten force-pushed the ameten/scraper-add-indices branch from e8b5607 to d308ad1 Compare January 21, 2025 13:17
Copy link
Contributor

@daniel-savu daniel-savu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any chance you can test these changes, along with SeaORM v1.1.1, on the RC scraper to make sure all slow queries are being fixed by this - or at least understand why remaining slow queries are happening

@ameten
Copy link
Contributor Author

ameten commented Jan 22, 2025

any chance you can test these changes, along with SeaORM v1.1.1, on the RC scraper to make sure all slow queries are being fixed by this - or at least understand why remaining slow queries are happening

The changes in this particular PR were tested when they were done in production database some time ago.

In general, we need to bring back SeaORM 1.1.1 (or the latest version) and test it as well. It would be a separate task, though.

@daniel-savu
Copy link
Contributor

The changes in this particular PR were tested when they were done in production database some time ago.
In general, we need to bring back SeaORM 1.1.1 (or the latest version) and test it as well. It would be a separate task, though.

Just saying that if the purpose of this PR is to fix seaorm 1.1.1 issues, it should be evaluated against seaorm 1.1.1 to make sure they're no longer issues. Otherwise there's no urgency in releasing this

@ameten ameten enabled auto-merge January 22, 2025 17:17
@ameten ameten added this pull request to the merge queue Jan 22, 2025
Merged via the queue into main with commit 6c631d8 Jan 22, 2025
42 checks passed
@ameten ameten deleted the ameten/scraper-add-indices branch January 22, 2025 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants