Add upgrade notes entry on optional field data migration utility script #1071
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cognifloyd
approved these changes
Jun 1, 2021
Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>
Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>
Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>
Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>
Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>
arm4b
reviewed
Jun 11, 2021
Comment on lines
+20
to
+72
| * The underlying database field type for storing large values such as action execution result has | ||
| changed for various database models (ActionExecutionDB, LiveActionDB, WorkflowExecutionDB, | ||
| TaskExecutionDB, TriggerInstanceDB). | ||
|
|
||
| For most users this change will result in 8-20x speed up when working with (reading and writing) | ||
| large values from / to the database. | ||
|
|
||
| The change is fully transparent to the end user and new objects created after upgrade to |st2| | ||
| v3.5 will automatically utilize this new field type. | ||
|
|
||
| Existing objects in the database will continue to utilize old field type. | ||
|
|
||
| If you want to migrate them to the new field type, you can use | ||
| ``st2-migrate-db-dict-field-values`` migration script which ships with |st2| v3.5. The script | ||
| only operates on "finalized" objects (i.e. finished executions) and it's idempotent which means | ||
| you can re-run it on failures or similar. | ||
|
|
||
| It's worth noting that running this script is optional - in most cases users primarily care about | ||
| performance for recent / new objects (e.g. viewing recent executions) so if you don't migrate | ||
| existing database field values this means retrieving those objects will still be slow, but it | ||
| doesn't affect newly created objects post v3.5 upgrade which will utilize new field type and | ||
| as such, exhibit much better performance. | ||
|
|
||
| By default the script will run in an interactive mode and display a prompt with a warning which needs | ||
| to be acknowledged before continuing. If you want to run script in an non-interactive mode, pass | ||
| ``--yes`` command line argument to it. | ||
|
|
||
| The script also defaults to migrating data for the past 30 days. You can migrate objects from | ||
| a different time period using ``--start-dt`` and ``--end-dt`` arguments as shown below. | ||
|
|
||
| The script currently doesn't support batching so in case you have many objects in the database | ||
| (especially trigger instances) you may need to migrate things in smaller chunks and call this | ||
| script multiple time (e.g. using a day long intervals or shorter). | ||
|
|
||
| Before running this script, you may also want to purge some old operational data. For information | ||
| on that, please refer to :doc:`Purging Old Operational Data </troubleshooting/purging_old_data>` | ||
| documentation page. | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| # Migrate objects with creation date between April 20th, 2021 and April 25th, 2021 | ||
| /opt/stackstorm/st2/bin/st2-migrate-db-dict-field-values --start-dt "2021-04-20T19:16:55Z" --end-dt "2021-04-25T19:26:55Z" | ||
|
|
||
| # Migrate object between April 20th and "now" | ||
| /opt/stackstorm/st2/bin/st2-migrate-db-dict-field-values --start-dt "2021-04-20T19:16:55Z" --end-dt "now" | ||
|
|
||
| .. note:: | ||
|
|
||
| You are strongly recommended to create a full database backup before running this script. | ||
|
|
||
| If you run this migration script and a need arises, you won't be able to rollback back to a | ||
| previous version (v3.4) because code in previous version doesn't include support for this new | ||
| field type (in such case you would need to restore the database backup). |
There was a problem hiding this comment.
@Kami The upgrade note looked "little big" though :)
There was a problem hiding this comment.
Yeah, I think it's better to give more context than less.
I always hate when there is a regression / issue with a new version of some project / library and changelog / upgrade notes don't give much clue where to begin and then you need start going over the commits immediately which means more wasted time...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Upgrade notes entry for optional migration script in StackStorm/st2#5255.