fix(semver): Backfill missed semver releases #28798
Conversation
|
This PR has a migration; here is the generated SQL --
-- MIGRATION NOW PERFORMS OPERATION THAT CANNOT BE WRITTEN AS SQL:
-- Raw Python operation
-- |
| queryset = RangeQuerySetWrapperWithProgressBar( | ||
| Release.objects.filter( | ||
| version__contains="@", | ||
| version__regex=SEMVER_REGEX, | ||
| major=None, | ||
| date_added__gte=datetime(2021, 7, 21, tzinfo=pytz.utc), | ||
| ).values_list( |
There was a problem hiding this comment.
I'd recommend we just iterate over the whole table like the previous backfills do. 2021/07/21 is only applicable to getsentry and not to on-prem/st users, and that regex is pretty expensive to run, and will run for each batch that RangeQuerySetWrapperWithProgressBar processes.
It'd be fine to just entirely duplicate src/sentry/migrations/0223_semver_backfill_2.py as is and just have ops run that again.
There was a problem hiding this comment.
@wedamija what if we narrow it down just by keeping major=None filter, surely it is not needed to re-process the already parsed ones .. that brings us down to about 6.5 mill releases
There was a problem hiding this comment.
Scratch that, that's incorrect.. difference is not that massive
There was a problem hiding this comment.
@ahmedetefy The other thing is that when we use RangeQuerySetWrapperWithProgressBar it orders by id so that we can iterate through in batches. So typically the index used will be on pk, and we won't be able to make use of other indexes.
In practice, for this table it's not a big deal. This backfill takes < 1 day to run from what I remember, and it's written fairly efficiently since it only makes an update if the parsed fields change, and batches updates in groups of 100.
Migration to backfill semver columns that were not populated incorrectly as a result of bug that did not run the semver version parser on Release creation
efc228a to
6830983
Compare
Migration to backfill semver columns that were not populated incorrectly as a result of bug that did not run the semver version parser on Release creation
Migration to backfill semver columns for releases because some releases were not populated incorrectly from July 21st of 2021 (when the last backfill occurred) as a result of bug that did not run the semver version parser on Release creation
Basically a replica of #27518
Related Ticket: https://getsentry.atlassian.net/browse/WOR-1373?atlOrigin=eyJpIjoiZmVjOWY1N2NiN2M2NDBiNDgzYzM3ZDgyM2FhNjI3NjIiLCJwIjoiamlyYS1zbGFjay1pbnQifQ
Bug fix: #28722