From c51cab44c3c7c0807cf657deb7be4ce959ffc0de Mon Sep 17 00:00:00 2001 From: Tomaz Muraus Date: Sun, 9 May 2021 20:12:54 +0200 Subject: [PATCH 01/11] Add docs on optional field data migration utility. --- docs/source/upgrade_notes.rst | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index 573e2537..1611ef33 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -3,10 +3,36 @@ Upgrade Notes ============= +.. _ref-upgrade-notes-v3-5: + +* Underlying database field type for storing large values such as action execution results has + changed for various models (ActionExecutionDB, LiveActionDB, etc.). + + For most users this change will result in 8-20x speed up when working with (reading and writing) + large values from / to the database. + + The change is fully transparent to the end user and new objects created after upgrade to |st2| + v3.5 will automatically utilize this new field type. + + Existing objects in the database will continue to utilize old field type. + + If you want to migrate them to the new field type, you can use + ``st2-migrate-db-dict-field-values`` migration script which ships with |st2| v3.5. The script + only operates on "finalized" objects (e.g. finished executions) and it's idempotent which means + you can re-run it on failures or similar. + + It's worth noting that running this script is optional - in most cases users care about + performance for recent / new objects (e.g. viewing recent executions) so if you don't migrate + existing database field values this means retrieving those objects will still be slow, but it + doesn't affect newly created objects post v3.5 upgrade. + +|st2| v3.5 +---------- + .. _ref-upgrade-notes-v3-4: |st2| v3.4 -------------- +---------- * Python 2 support was removed. Any packs that only support python 2 will need to be migrated to python 3. From 5098e286e933b59f845bf30c988538a743459472 Mon Sep 17 00:00:00 2001 From: Tomaz Muraus Date: Sun, 9 May 2021 20:22:09 +0200 Subject: [PATCH 02/11] Upgrade entry, add note on the no rollback scenario. --- docs/source/upgrade_notes.rst | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index 1611ef33..ba2888ec 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -6,7 +6,8 @@ Upgrade Notes .. _ref-upgrade-notes-v3-5: * Underlying database field type for storing large values such as action execution results has - changed for various models (ActionExecutionDB, LiveActionDB, etc.). + changed for various database models (ActionExecutionDB, LiveActionDB, WorkflowExecutionDB, + TaskExecutionDB, TriggerInstanceDB, etc.). For most users this change will result in 8-20x speed up when working with (reading and writing) large values from / to the database. @@ -18,13 +19,20 @@ Upgrade Notes If you want to migrate them to the new field type, you can use ``st2-migrate-db-dict-field-values`` migration script which ships with |st2| v3.5. The script - only operates on "finalized" objects (e.g. finished executions) and it's idempotent which means + only operates on "finalized" objects (i.e. finished executions) and it's idempotent which means you can re-run it on failures or similar. - It's worth noting that running this script is optional - in most cases users care about + It's worth noting that running this script is optional - in most cases users primary care about performance for recent / new objects (e.g. viewing recent executions) so if you don't migrate existing database field values this means retrieving those objects will still be slow, but it - doesn't affect newly created objects post v3.5 upgrade. + doesn't affect newly created objects post v3.5 upgrade which will utilize new field type and + such exhibit much better performance. + + .. note:: + + If you run this migration script and a need arises, you won't be able to rollback back to a + previous version (v3.4) because code in previous version doesn't include support for this new + field type. |st2| v3.5 ---------- From 322432043c9686643273939cf9f6a5145a216d28 Mon Sep 17 00:00:00 2001 From: Tomaz Muraus Date: Tue, 11 May 2021 20:52:33 +0200 Subject: [PATCH 03/11] Add a note on backup, interactive mode. --- docs/source/upgrade_notes.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index ba2888ec..50f88ded 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -28,11 +28,17 @@ Upgrade Notes doesn't affect newly created objects post v3.5 upgrade which will utilize new field type and such exhibit much better performance. + By default script will run in an interactive mode and display a prompt with warning which needs + to be confirmed before continuing. If you want to run script in an non-interactive mode, pass + ``--yes`` command line argument to it. + .. note:: + You are strongly recommended to create full database backup before running this script. + If you run this migration script and a need arises, you won't be able to rollback back to a previous version (v3.4) because code in previous version doesn't include support for this new - field type. + field type (in such case you would need to restore the database backup). |st2| v3.5 ---------- From ea9dad1aad59c59d6e1808cc62a9d33b42c6f976 Mon Sep 17 00:00:00 2001 From: Tomaz Muraus Date: Wed, 12 May 2021 21:39:54 +0200 Subject: [PATCH 04/11] Add a note on new script arguments. --- docs/source/upgrade_notes.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index 50f88ded..c81bf704 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -32,6 +32,17 @@ Upgrade Notes to be confirmed before continuing. If you want to run script in an non-interactive mode, pass ``--yes`` command line argument to it. + Script also defaults to migrating data for the past 30 days. You can control time period for + objects which will be migrated using ``--start-dt`` and ``--end-dt`` argument as shown below. + + The script currently doesn't support batching so in case you have many objects in the database + (especially trigger instances) you may need to migrate things in smaller chunks and call this + script multiple time (e.g. using a day long intervals or shorter). + + .. code-block:: bash + + /opt/stackstorm/st2/bin/st2-migrate-db-dict-field-values --start-dt "2021-04-20T19:16:55Z" --end-dt "2021-04-25T19:26:55Z" + .. note:: You are strongly recommended to create full database backup before running this script. From 649ececf85e0bdddbd01d1bf1e826282690e4a14 Mon Sep 17 00:00:00 2001 From: Tomaz Muraus Date: Wed, 12 May 2021 22:37:04 +0200 Subject: [PATCH 05/11] Update upgrade notes. --- docs/source/upgrade_notes.rst | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index c81bf704..8cf90db6 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -3,6 +3,9 @@ Upgrade Notes ============= +|st2| v3.5 +---------- + .. _ref-upgrade-notes-v3-5: * Underlying database field type for storing large values such as action execution results has @@ -39,10 +42,18 @@ Upgrade Notes (especially trigger instances) you may need to migrate things in smaller chunks and call this script multiple time (e.g. using a day long intervals or shorter). + Before running this script, you may also want to purge some old operational data. For information + on that, please refer to :doc:`Purging Old Operational Data `. + documentation page. + .. code-block:: bash + # Migrate objects with creation date between April 20th, 2021 and April 25th, 2021 /opt/stackstorm/st2/bin/st2-migrate-db-dict-field-values --start-dt "2021-04-20T19:16:55Z" --end-dt "2021-04-25T19:26:55Z" + # Migrate object between April 20th and "now" + /opt/stackstorm/st2/bin/st2-migrate-db-dict-field-values --start-dt "2021-04-20T19:16:55Z" --end-dt "now" + .. note:: You are strongly recommended to create full database backup before running this script. @@ -51,9 +62,6 @@ Upgrade Notes previous version (v3.4) because code in previous version doesn't include support for this new field type (in such case you would need to restore the database backup). -|st2| v3.5 ----------- - .. _ref-upgrade-notes-v3-4: |st2| v3.4 From fa7057cde86b095b832e820eb497ef5e8e33f4cf Mon Sep 17 00:00:00 2001 From: Tomaz Muraus Date: Wed, 12 May 2021 23:15:41 +0200 Subject: [PATCH 06/11] Update upgrade notes. --- docs/source/upgrade_notes.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index 8cf90db6..991c3abd 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -8,9 +8,9 @@ Upgrade Notes .. _ref-upgrade-notes-v3-5: -* Underlying database field type for storing large values such as action execution results has +* Underlying database field type for storing large values such as action execution result has changed for various database models (ActionExecutionDB, LiveActionDB, WorkflowExecutionDB, - TaskExecutionDB, TriggerInstanceDB, etc.). + TaskExecutionDB, TriggerInstanceDB). For most users this change will result in 8-20x speed up when working with (reading and writing) large values from / to the database. @@ -29,10 +29,10 @@ Upgrade Notes performance for recent / new objects (e.g. viewing recent executions) so if you don't migrate existing database field values this means retrieving those objects will still be slow, but it doesn't affect newly created objects post v3.5 upgrade which will utilize new field type and - such exhibit much better performance. + as such, exhibit much better performance. By default script will run in an interactive mode and display a prompt with warning which needs - to be confirmed before continuing. If you want to run script in an non-interactive mode, pass + to be acknowledged before continuing. If you want to run script in an non-interactive mode, pass ``--yes`` command line argument to it. Script also defaults to migrating data for the past 30 days. You can control time period for @@ -43,7 +43,7 @@ Upgrade Notes script multiple time (e.g. using a day long intervals or shorter). Before running this script, you may also want to purge some old operational data. For information - on that, please refer to :doc:`Purging Old Operational Data `. + on that, please refer to :doc:`Purging Old Operational Data ` documentation page. .. code-block:: bash From 4b96588568da8b20fe5e3981e14138e24e45cc0b Mon Sep 17 00:00:00 2001 From: Eugen Cusmaunsa Date: Fri, 11 Jun 2021 23:11:47 +0100 Subject: [PATCH 07/11] Update docs/source/upgrade_notes.rst Co-authored-by: Jacob Floyd --- docs/source/upgrade_notes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index 991c3abd..e24ad5c0 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -56,7 +56,7 @@ Upgrade Notes .. note:: - You are strongly recommended to create full database backup before running this script. + You are strongly recommended to create a full database backup before running this script. If you run this migration script and a need arises, you won't be able to rollback back to a previous version (v3.4) because code in previous version doesn't include support for this new From e66ca38bf1484644692a83cca4efc752f16eef1b Mon Sep 17 00:00:00 2001 From: Eugen Cusmaunsa Date: Fri, 11 Jun 2021 23:12:27 +0100 Subject: [PATCH 08/11] Update docs/source/upgrade_notes.rst Co-authored-by: Jacob Floyd --- docs/source/upgrade_notes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index e24ad5c0..8c2bf0fd 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -31,7 +31,7 @@ Upgrade Notes doesn't affect newly created objects post v3.5 upgrade which will utilize new field type and as such, exhibit much better performance. - By default script will run in an interactive mode and display a prompt with warning which needs + By default the script will run in an interactive mode and display a prompt with a warning which needs to be acknowledged before continuing. If you want to run script in an non-interactive mode, pass ``--yes`` command line argument to it. From 5ed676c60946466f1901450d1c5e5e5438245dce Mon Sep 17 00:00:00 2001 From: Eugen Cusmaunsa Date: Fri, 11 Jun 2021 23:12:36 +0100 Subject: [PATCH 09/11] Update docs/source/upgrade_notes.rst Co-authored-by: Jacob Floyd --- docs/source/upgrade_notes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index 8c2bf0fd..3a28fa53 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -25,7 +25,7 @@ Upgrade Notes only operates on "finalized" objects (i.e. finished executions) and it's idempotent which means you can re-run it on failures or similar. - It's worth noting that running this script is optional - in most cases users primary care about + It's worth noting that running this script is optional - in most cases users primarily care about performance for recent / new objects (e.g. viewing recent executions) so if you don't migrate existing database field values this means retrieving those objects will still be slow, but it doesn't affect newly created objects post v3.5 upgrade which will utilize new field type and From 6739b841f6d58af77f179941a14eda81b468415c Mon Sep 17 00:00:00 2001 From: Eugen Cusmaunsa Date: Fri, 11 Jun 2021 23:12:42 +0100 Subject: [PATCH 10/11] Update docs/source/upgrade_notes.rst Co-authored-by: Jacob Floyd --- docs/source/upgrade_notes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index 3a28fa53..458a2774 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -8,7 +8,7 @@ Upgrade Notes .. _ref-upgrade-notes-v3-5: -* Underlying database field type for storing large values such as action execution result has +* The underlying database field type for storing large values such as action execution result has changed for various database models (ActionExecutionDB, LiveActionDB, WorkflowExecutionDB, TaskExecutionDB, TriggerInstanceDB). From a745c2518766bc9d638d837f55a848d2038acbd5 Mon Sep 17 00:00:00 2001 From: Eugen Cusmaunsa Date: Fri, 11 Jun 2021 23:13:07 +0100 Subject: [PATCH 11/11] Update docs/source/upgrade_notes.rst Co-authored-by: Jacob Floyd --- docs/source/upgrade_notes.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index 458a2774..48d462c4 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -35,8 +35,8 @@ Upgrade Notes to be acknowledged before continuing. If you want to run script in an non-interactive mode, pass ``--yes`` command line argument to it. - Script also defaults to migrating data for the past 30 days. You can control time period for - objects which will be migrated using ``--start-dt`` and ``--end-dt`` argument as shown below. + The script also defaults to migrating data for the past 30 days. You can migrate objects from + a different time period using ``--start-dt`` and ``--end-dt`` arguments as shown below. The script currently doesn't support batching so in case you have many objects in the database (especially trigger instances) you may need to migrate things in smaller chunks and call this