From 14da4f521c76b2f92f61cc6e67902db73c1389d2 Mon Sep 17 00:00:00 2001 From: W Chan Date: Thu, 10 Jun 2021 20:04:51 +0000 Subject: [PATCH 1/6] Add coordination service requirement for single node and HA install The coordination service is required to support multiple workflow use cases. Update docs to let users know that redis is installed by default in single node installation script and that redis/zookeeper needs to be included in a HA deployment. --- docs/source/install/overview.rst | 7 ++-- docs/source/reference/ha.rst | 59 ++++++++++++++++++-------------- 2 files changed, 38 insertions(+), 28 deletions(-) diff --git a/docs/source/install/overview.rst b/docs/source/install/overview.rst index 0a874275..7eb0b1b0 100644 --- a/docs/source/install/overview.rst +++ b/docs/source/install/overview.rst @@ -94,10 +94,13 @@ ChatOps can be also enabled by installing `hubot-stackstorm plugin Dependencies ------------ -The required dependencies are RabbitMQ, and MongoDB. The optional dependencies are: +The required dependencies are RabbitMQ, MongoDB, and Redis (or Zookeeper). + +The coordination service is required for workflows that has multiple branches and tasks with items. Previously, the coordination service is optional to support concurrency policies. The backend for the coordination service can be configured to use Redis, Zookeeper, or other. Since v3.5, redis server is installed as part of the single node installation script. The python redis client is also installed into the |st2| virtualenv. If using Zookeeper, the kazoo module needs to be manually installed into the |st2| virtualenv. + +The optional dependencies are: - nginx for SSL termination, reverse-proxying API endpoints and serving static HTML. - - Redis or Zookeeper for concurrency policies (see :doc:`/reference/policies`). - LDAP authentication. diff --git a/docs/source/reference/ha.rst b/docs/source/reference/ha.rst index 63e8500f..70fbc98d 100644 --- a/docs/source/reference/ha.rst +++ b/docs/source/reference/ha.rst @@ -232,10 +232,13 @@ not affect functionality. See :ref:`here` for how to configure |st2| to connect to a RabbitMQ cluster. -Zookeeper/Redis -^^^^^^^^^^^^^^^ -Various |st2| features rely on a proper co-ordination backend in a distributed deployment to work -correctly. +Coordination +^^^^^^^^^^^^ +Support of workflows with concurrent task executions and concurrency policies for action executions +rely on a proper co-ordination backend in a distributed deployment to work correctly. + +The coordination service can be configured to use different backends such as redis or zookeeper. For +the single node installation script, redis is installed and configured by default. `This `__ shows how to run a replicated Zookeeper setup. (Note: Make sure to refer to the documentation in the @@ -300,13 +303,14 @@ Controller Box ^^^^^^^^^^^^^^ This box runs all the shared required dependencies and some |st2| components: -* Nginx as load balancer -* MongoDB -* RabbitMQ -* st2chatops -* st2web + * Nginx as load balancer + * MongoDB + * RabbitMQ + * Redis/Zookeeper + * st2chatops + * st2web -In practice ``MongoDB`` ``RabbitMQ`` will usually be on standalone clusters +In practice ``MongoDB``, ``RabbitMQ``, and ``Redis/Zookeeper`` will usually be on standalone clusters managed outside of |st2|. The two shared components (``st2chatops`` and ``st2web``) are placed here for the sake of convenience. They could be placed anywhere with the right configuration. @@ -323,32 +327,35 @@ Follow these steps to provision a controller box on Ubuntu 16.04: Install Required Dependencies ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -1. Install ``MongoDB`` and ``RabbitMQ``: +1. Install ``MongoDB``, ``RabbitMQ``, and ``Redis``: + + The python redis client is already included in the |st2| virtualenv. If using Zookeeper, the + kazoo module needs to be installed into the |st2| virtualenv. .. code-block:: bash - $ sudo apt-get install -y mongodb-server rabbitmq-server + $ sudo apt-get install -y mongodb-server rabbitmq-server redis-server 2. Fix ``bind_ip`` in ``/etc/mongodb.conf`` to bind MongoDB to an interface that has an IP address reachable from ``st2-multi-node-1`` and ``st2-multi-node-2``. -4. Restart MongoDB: +3. Restart MongoDB: .. code-block:: bash $ sudo service mongodb restart -5. Add stable |st2| repos: +4. Add stable |st2| repos: .. code-block:: bash $ curl -s https://packagecloud.io/install/repositories/StackStorm/stable/script.deb.sh | sudo bash -6. Setup ``st2web`` and SSL termination. Follow :ref:`install webui and setup +5. Setup ``st2web`` and SSL termination. Follow :ref:`install webui and setup ssl`. You will need to stop after removing the default Nginx config file. -7. A sample configuration for Nginx as load balancer for the controller box is provided below. +6. A sample configuration for Nginx as load balancer for the controller box is provided below. With this configuration Nginx will load balance all requests between the two blueprint boxes ``st2-multi-node-1`` and ``st2-multi-node-2``. This includes requests to ``st2api`` and ``st2auth``. Nginx also serves as the webserver for ``st2web``. @@ -356,14 +363,14 @@ Install Required Dependencies .. literalinclude:: /../../st2/conf/HA/nginx/st2.conf.controller.sample :language: none -8. Create the st2 logs directory and the st2 user: +7. Create the st2 logs directory and the st2 user: .. code-block:: bash mkdir -p /var/log/st2 useradd st2 -12. Install ``st2chatops`` following :ref:`setup chatops`. +8. Install ``st2chatops`` following :ref:`setup chatops`. Blueprint box ^^^^^^^^^^^^^ @@ -390,13 +397,13 @@ also be made to offer different services. $ sudo apt-get install -y nginx -8. Replace ``/etc/st2/st2.conf`` with the sample ``st2.conf`` provided below. This config points to - the controller node or configuration values of ``database`` and ``messaging``. +4. Replace ``/etc/st2/st2.conf`` with the sample ``st2.conf`` provided below. This config points to + the controller node or configuration values of ``database``, ``messaging``, and ``coordination``. .. literalinclude:: /../../st2/conf/HA/st2.conf.sample :language: ini -9. Generate a certificate: +5. Generate a certificate: .. code-block:: bash @@ -405,19 +412,19 @@ also be made to offer different services. -days XXX -nodes -subj "/C=US/ST=California/L=Palo Alto/O=StackStorm/OU=Information \ Technology/CN=$(hostname)" -10. Configure users & authentication as per :ref:`this documentation`. Make +6. Configure users & authentication as per :ref:`this documentation`. Make sure that user configuration on all boxes running ``st2auth`` is identical. This ensures consistent authentication from the entire |st2| install since the request to authenticate a user can be forwarded by the load balancer to any of the ``st2auth`` processes. -11. Use the sample Nginx config that is provided below for the blueprint boxes. In this config +7. Use the sample Nginx config that is provided below for the blueprint boxes. In this config Nginx will act as the SSL termination endpoint for all the REST endpoints exposed by ``st2api`` and ``st2auth``: .. literalinclude:: /../../st2/conf/HA/nginx/st2.conf.blueprint.sample :language: nginx -12. To use Timer triggers, enable them on only one server. Make this change in +8. To use Timer triggers, enable them on only one server. Make this change in ``/etc/st2/st2.conf``: .. code-block:: yaml @@ -426,8 +433,8 @@ also be made to offer different services. enable = False -14. See :doc:`/reference/sensor_partitioning` to decide how to partition sensors to suit your +9. See :doc:`/reference/sensor_partitioning` to decide how to partition sensors to suit your requirements. -15. All content should be synced by choosing a suitable strategy as outlined above. This is crucial +10. All content should be synced by choosing a suitable strategy as outlined above. This is crucial to obtain predictable outcomes. From f2788887a37247708e2bcb848bd99aa4eaa64923 Mon Sep 17 00:00:00 2001 From: W Chan Date: Thu, 10 Jun 2021 21:07:30 +0000 Subject: [PATCH 2/6] Add upgrade notes on coordination service requirement --- docs/source/install/upgrades.rst | 6 ++++++ docs/source/upgrade_notes.rst | 5 +++++ 2 files changed, 11 insertions(+) diff --git a/docs/source/install/upgrades.rst b/docs/source/install/upgrades.rst index b4b29b3e..9eb6f844 100644 --- a/docs/source/install/upgrades.rst +++ b/docs/source/install/upgrades.rst @@ -193,6 +193,12 @@ v3.5 sudo rpm -e --nodeps nodejs sudo yum upgrade st2chatops +* Redis server is installed and configured as backend for the coordination service + by default in the single node installation script to support workflows with multiple + branches and tasks with items. Upgrade requires coordination service to be setup + manually, For workflows to be executed properly, setup the coordination service + accordingly. See :doc:`../reference/policies` for setup instruction. + v3.4 '''' diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index b2813aa5..b42dbafb 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -10,6 +10,11 @@ Upgrade Notes * Node was upgraded from v10 to v14. Node 14 repository will be required to be setup, prior to upgrade of st2chatops. +* Redis server is installed and configured as backend for the coordination service + by default in the single node installation script to support workflows with multiple + branches and tasks with items. Upgrade requires coordination service to be setup + manually, For workflows to be executed properly, setup the coordination service + accordingly. .. _ref-upgrade-notes-v3-4: From acac919aa8dcf1dfe07ece3e665998f40f8aa28d Mon Sep 17 00:00:00 2001 From: W Chan Date: Thu, 10 Jun 2021 21:10:52 +0000 Subject: [PATCH 3/6] Add redis port to system requirements --- docs/source/install/system_requirements.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/install/system_requirements.rst b/docs/source/install/system_requirements.rst index e5e77fa1..91106f63 100644 --- a/docs/source/install/system_requirements.rst +++ b/docs/source/install/system_requirements.rst @@ -57,6 +57,7 @@ By default, |st2| and related services use these TCP ports: * nginx (80, 443) * mongodb (27017) * rabbitmq (4369, 5672, 25672) +* redis (6379) or zookeeper (2181, 2888, 3888) * st2auth (9100) * st2api (9101) * st2stream (9102) From 35667dc54bb282c879feb5f62f9e685e77dce4c7 Mon Sep 17 00:00:00 2001 From: W Chan Date: Thu, 10 Jun 2021 21:20:06 +0000 Subject: [PATCH 4/6] Add redis to uninstall procedures --- docs/source/install/uninstall.rst | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/source/install/uninstall.rst b/docs/source/install/uninstall.rst index 7f12c5ec..528483c3 100644 --- a/docs/source/install/uninstall.rst +++ b/docs/source/install/uninstall.rst @@ -48,6 +48,7 @@ below. Only execute the instructions for your distribution. sudo service nginx stop sudo service mongod stop sudo service rabbitmq-server stop + sudo service redis-server stop * RHEL/CentOS 7.x/8.x: @@ -57,6 +58,7 @@ below. Only execute the instructions for your distribution. sudo systemctl stop nginx sudo systemctl stop mongod sudo systemctl stop rabbitmq-server + sudo systemctl stop redis .. note:: @@ -119,13 +121,13 @@ below. Only execute the instructions for your distribution. .. sourcecode:: bash - sudo apt-get purge mongodb-org* rabbitmq-server erlang* nginx nodejs + sudo apt-get purge mongodb-org* rabbitmq-server erlang* nginx nodejs redis-server * RHEL/CentOS: .. sourcecode:: bash - sudo yum erase mongodb-org* rabbitmq-server erlang* nginx nodejs + sudo yum erase mongodb-org* rabbitmq-server erlang* nginx nodejs redis .. note:: @@ -164,6 +166,8 @@ last pieces. sudo rm -rf /etc/st2 /opt/stackstorm sudo rm -rf /var/log/st2 /var/log/mongodb sudo rm -rf /var/lib/mongodb /var/run/mongodb.pid + sudo rm -rf /etc/redis/redis.conf /var/lib/redis + sudo userdel -r redis * RHEL/CentOS: @@ -172,7 +176,8 @@ last pieces. sudo rm -rf /etc/st2 /etc/mongod* /etc/rabbitmq /etc/nginx /opt/stackstorm sudo rm -rf /var/log/st2 /var/log/mongodb /var/log/rabbitmq /var/log/nginx sudo rm -rf /var/lib/rabbitmq /var/lib/mongo - + sudo rm -rf /etc/redis/redis.conf /var/lib/redis + sudo userdel -r redis At this point, your system is no longer running any |st2|-related services, and all the main dependencies have been removed. You can either re-install |st2|, or use this system for other From e8fa66a7cbdbe954edf1d67081880c26b913f6ff Mon Sep 17 00:00:00 2001 From: W Chan Date: Thu, 10 Jun 2021 21:26:54 +0000 Subject: [PATCH 5/6] Add instruction to manually install redis for each distros --- docs/source/install/common/configure_components.rst | 3 ++- docs/source/install/rhel7.rst | 7 ++++--- docs/source/install/rhel8.rst | 7 ++++--- docs/source/install/u18.rst | 3 ++- docs/source/install/u20.rst | 3 ++- 5 files changed, 14 insertions(+), 9 deletions(-) diff --git a/docs/source/install/common/configure_components.rst b/docs/source/install/common/configure_components.rst index dfd580c9..5c6466af 100644 --- a/docs/source/install/common/configure_components.rst +++ b/docs/source/install/common/configure_components.rst @@ -1,7 +1,8 @@ -If you are not running RabbitMQ or MongoDB on the same system, or have changed the +If you are not running RabbitMQ, MongoDB, or Redis on the same system, or have changed the defaults, please adjust these settings: * RabbitMQ connection at ``/etc/st2/st2.conf`` * MongoDB at ``/etc/st2/st2.conf`` +* Redis connection at coordination section of ``/etc/st2/st2.conf`` See the :doc:`Configuration documentation ` for more information. diff --git a/docs/source/install/rhel7.rst b/docs/source/install/rhel7.rst index e9ed910d..142811e8 100644 --- a/docs/source/install/rhel7.rst +++ b/docs/source/install/rhel7.rst @@ -55,7 +55,7 @@ Install Dependencies .. include:: __mongodb_note.rst -Install MongoDB, and RabbitMQ: +Install MongoDB, RabbitMQ, and Redis .. code-block:: bash @@ -75,8 +75,9 @@ Install MongoDB, and RabbitMQ: sudo yum -y install crudini sudo yum -y install mongodb-org sudo yum -y install rabbitmq-server - sudo systemctl start mongod rabbitmq-server - sudo systemctl enable mongod rabbitmq-server + sudo yum -y install redis + sudo systemctl start mongod rabbitmq-server redis + sudo systemctl enable mongod rabbitmq-server redis The default python on CentOS/RHEL 7.x is python 2, |st2| uses python3 and requires the python3-devel package. The installation of the st2 package will automatically install python3-devel if it is available in an enabled repository. On CentOS distributions the relevant repository is typically enabled however on RHEL distributions it is provided by the rhel-7-server-optional-rpms repository (repository name dependant on RHEL distribution). diff --git a/docs/source/install/rhel8.rst b/docs/source/install/rhel8.rst index 3d953bbc..05326192 100644 --- a/docs/source/install/rhel8.rst +++ b/docs/source/install/rhel8.rst @@ -56,7 +56,7 @@ Install Dependencies .. include:: __mongodb_note.rst -Install MongoDB, RabbitMQ: +Install MongoDB, RabbitMQ, and Redis: .. code-block:: bash @@ -80,8 +80,9 @@ Install MongoDB, RabbitMQ: sudo yum makecache -y --disablerepo='*' --enablerepo='rabbitmq_rabbitmq-server' sudo yum -y install erlang sudo yum -y install rabbitmq-server - sudo systemctl start mongod rabbitmq-server - sudo systemctl enable mongod rabbitmq-server + sudo yum -y install redis + sudo systemctl start mongod rabbitmq-server redis + sudo systemctl enable mongod rabbitmq-server redis Setup Repositories diff --git a/docs/source/install/u18.rst b/docs/source/install/u18.rst index 88e3642b..dd9dfbc4 100644 --- a/docs/source/install/u18.rst +++ b/docs/source/install/u18.rst @@ -22,7 +22,7 @@ Minimal Installation Install Dependencies ~~~~~~~~~~~~~~~~~~~~ -Install MongoDB, and RabbitMQ: +Install MongoDB, RabbitMQ, and Redis: .. code-block:: bash @@ -39,6 +39,7 @@ Install MongoDB, and RabbitMQ: sudo apt-get install -y crudini sudo apt-get install -y mongodb-org sudo apt-get install -y rabbitmq-server + sudo apt-get install -y redis-server For Ubuntu ``Bionic`` you may need to enable and start MongoDB. diff --git a/docs/source/install/u20.rst b/docs/source/install/u20.rst index b3a6e704..1ae9acd8 100644 --- a/docs/source/install/u20.rst +++ b/docs/source/install/u20.rst @@ -21,7 +21,7 @@ Minimal Installation Install Dependencies ~~~~~~~~~~~~~~~~~~~~ -Install MongoDB, and RabbitMQ: +Install MongoDB, RabbitMQ, and Redis: .. code-block:: bash @@ -38,6 +38,7 @@ Install MongoDB, and RabbitMQ: sudo apt-get install -y crudini sudo apt-get install -y mongodb-org sudo apt-get install -y rabbitmq-server + sudo apt-get install -y redis-server For Ubuntu ``Focal`` you may need to enable and start MongoDB. From 35e6c9be37add98279495b527e34121474da75c0 Mon Sep 17 00:00:00 2001 From: W Chan Date: Thu, 10 Jun 2021 22:29:12 +0000 Subject: [PATCH 6/6] Add a section for the coordination service Add a section for the coordination service and how to configure the backend. Previously, this is documented in the policies section. Refactor the policies section to reference this coordination section. --- docs/source/coordination.rst | 51 ++++++++++++++++++++++++ docs/source/index.rst | 1 + docs/source/install/upgrades.rst | 2 +- docs/source/reference/policies.rst | 62 +++--------------------------- 4 files changed, 58 insertions(+), 58 deletions(-) create mode 100644 docs/source/coordination.rst diff --git a/docs/source/coordination.rst b/docs/source/coordination.rst new file mode 100644 index 00000000..dec0e455 --- /dev/null +++ b/docs/source/coordination.rst @@ -0,0 +1,51 @@ +Coordination +============ + +Coordination backend is required to support workflows with multiple branches or tasks with items and +actions with concurrency policies defined. + +StackStorm utilizes the ``OpenStack Tooz`` library for communicating with the coordination backend. +The coordination backend must support the ``Locking`` functionality as defined by the ``Tooz`` +interface. Please refrence the `OpenStack Tooz compatability page `_ +for more information what interfaces are implemented by various backends. + +The following is a list of backends that can be configured for the coordination service. For the +full list of the supported backends and how to configure them, please visit +`OpenStack Tooz documentation `_. + + * Redis + * Zookeeper + * consul + * etcd + * file (for testing when all the services are running on a single host) + +The configuration of the coordination service is done in the ``coordination`` section +of ``/etc/st2/st2.conf``. The following are configuration examples for Redis and Zookeeper. + +Redis: + +.. code-block:: ini + + [coordination] + url = redis://:password@host:port + +ZooKeeper: + +.. code-block:: ini + + [coordination] + url = kazoo://username:password@host:port + +Some of these coordination backends also require corresponding client libraries to be installed +in |st2| virtualenv. We do not ship these libraries by default. As an example, to install the client +library in |st2| virtualenv, run: + +.. sourcecode:: bash + + sudo su + + # Example when using redis backend + /opt/stackstorm/st2/bin/pip install redis + + # Example when using consul backend + /opt/stackstorm/st2/bin/pip install consul diff --git a/docs/source/index.rst b/docs/source/index.rst index af3b39a1..63b043a4 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -30,6 +30,7 @@ Contents: authentication rbac + coordination inquiries reference/index troubleshooting/index diff --git a/docs/source/install/upgrades.rst b/docs/source/install/upgrades.rst index 9eb6f844..ab4950de 100644 --- a/docs/source/install/upgrades.rst +++ b/docs/source/install/upgrades.rst @@ -197,7 +197,7 @@ v3.5 by default in the single node installation script to support workflows with multiple branches and tasks with items. Upgrade requires coordination service to be setup manually, For workflows to be executed properly, setup the coordination service - accordingly. See :doc:`../reference/policies` for setup instruction. + accordingly. See :doc:`../coordination` for setup instruction. v3.4 '''' diff --git a/docs/source/reference/policies.rst b/docs/source/reference/policies.rst index 893c271f..4f3bf2df 100644 --- a/docs/source/reference/policies.rst +++ b/docs/source/reference/policies.rst @@ -22,6 +22,11 @@ can specify that new executions are canceled, rather than delayed. There are two forms of concurrency policy: ``action.concurrency`` and ``action.concurrency.attr``. +.. note:: + + The concurrency policy type is not enabled by default and requires a backend coordination + service such as ZooKeeper or Redis to work. See :doc:`../coordination` for setup instruction. + action.concurrency ~~~~~~~~~~~~~~~~~~ @@ -83,63 +88,6 @@ host. attributes: - hostname -.. note:: - - The concurrency policy type is not enabled by default and requires a backend coordination - service such as ZooKeeper or Redis to work. - -If you have installed a coordination service in your network, you need to configure |st2| to -connect to that backend service. This is done in the ``coordination`` section of -``/etc/st2/st2.conf``. - -The following are examples for ZooKeeper and Redis: - -ZooKeeper: - -.. code-block:: ini - - [coordination] - url = kazoo://username:password@host:port - - -Redis: - -.. code-block:: ini - - [coordination] - url = redis://:password@host:port - -Other supported coordination backends include: - -* consul -* etcd -* MySQL -* PostgreSQL -* file (for testing when all the services are running on a single host) - -StackStorm utilizes the ``OpenStack Tooz`` library for communicating with the -coordination backend. The coordination backend must support the ``Locking`` -functionality as defined by the ``Tooz`` interface. Please refrence the -`OpenStack Tooz compatability page `_ -for more information what interfaces are implemented by various backends. - -For the full list of the supported backends and how to configure them, please visit -`OpenStack Tooz documentation `_. - -Some of these coordination backends also require corresponding client libraries to be installed -in |st2| virtualenv. We do not ship these libraries by default. As an example, to install the client -library in |st2| virtualenv, run: - -.. sourcecode:: bash - - sudo su - - # Example when using redis backend - /opt/stackstorm/st2/bin/pip install redis - - # Example when using consul backend - /opt/stackstorm/st2/bin/pip install consul - Retry ----- action.retry in st2 actions