Merge pull request #1075 from StackStorm/redis

Add coordination service requirement for single node and HA install

docs/source/coordination.rst

@@ -0,0 +1,51 @@
+Coordination
+============
+
+Coordination backend is required to support workflows with multiple branches or tasks with items and
+actions with concurrency policies defined.
+
+StackStorm utilizes the ``OpenStack Tooz`` library for communicating with the coordination backend.
+The coordination backend must support the ``Locking`` functionality as defined by the ``Tooz`` 
+interface. Please refrence the `OpenStack Tooz compatability page <https://docs.openstack.org/tooz/latest/user/compatibility.html>`_
+for more information what interfaces are implemented by various backends.
+
+The following is a list of backends that can be configured for the coordination service. For the
+full list of the supported backends and how to configure them, please visit
+`OpenStack Tooz documentation <https://docs.openstack.org/tooz/latest/>`_.
+
+ * Redis
+ * Zookeeper
+ * consul
+ * etcd
+ * file (for testing when all the services are running on a single host)
+
+The configuration of the coordination service is done in the ``coordination`` section
+of ``/etc/st2/st2.conf``. The following are configuration examples for Redis and Zookeeper.
+
+Redis:
+
+.. code-block:: ini
+
+    [coordination]
+    url = redis://:password@host:port
+
+ZooKeeper:
+
+.. code-block:: ini
+
+    [coordination]
+    url = kazoo://username:password@host:port
+
+Some of these coordination backends also require corresponding client libraries to be installed
+in |st2| virtualenv. We do not ship these libraries by default. As an example, to install the client
+library in |st2| virtualenv, run:
+
+.. sourcecode:: bash
+
+    sudo su
+
+    # Example when using redis backend
+    /opt/stackstorm/st2/bin/pip install redis
+
+    # Example when using consul backend
+    /opt/stackstorm/st2/bin/pip install consul

docs/source/index.rst

@@ -30,6 +30,7 @@ Contents:
 
     authentication
     rbac
+    coordination
     inquiries
     reference/index
     troubleshooting/index

docs/source/install/common/configure_components.rst

@@ -1,7 +1,8 @@
-If you are not running RabbitMQ or MongoDB on the same system, or have changed the
+If you are not running RabbitMQ, MongoDB, or Redis on the same system, or have changed the
 defaults, please adjust these settings:
 
 * RabbitMQ connection at ``/etc/st2/st2.conf``
 * MongoDB at ``/etc/st2/st2.conf``
+* Redis connection at coordination section of ``/etc/st2/st2.conf``
 
 See the :doc:`Configuration documentation </install/config/config>` for more information.

docs/source/install/overview.rst

@@ -94,10 +94,13 @@ ChatOps can be also enabled by installing `hubot-stackstorm plugin
 
 Dependencies
 ------------
-The required dependencies are RabbitMQ, and MongoDB. The optional dependencies are:
+The required dependencies are RabbitMQ, MongoDB, and Redis (or Zookeeper).
+
+The coordination service is required for workflows that has multiple branches and tasks with items. Previously, the coordination service is optional to support concurrency policies. The backend for the coordination service can be configured to use Redis, Zookeeper, or other. Since v3.5, redis server is installed as part of the single node installation script. The python redis client is also installed into the |st2| virtualenv. If using Zookeeper, the kazoo module needs to be manually installed into the |st2| virtualenv.
+
+The optional dependencies are:
 
   - nginx for SSL termination, reverse-proxying API endpoints and serving static HTML.
-  - Redis or Zookeeper for concurrency policies (see :doc:`/reference/policies`).
   - LDAP authentication.
 
 

docs/source/install/rhel7.rst

@@ -55,7 +55,7 @@ Install Dependencies
 
 .. include:: __mongodb_note.rst
 
-Install MongoDB, and RabbitMQ:
+Install MongoDB, RabbitMQ, and Redis
 
 .. code-block:: bash
 
@@ -75,8 +75,9 @@ Install MongoDB, and RabbitMQ:
   sudo yum -y install crudini
   sudo yum -y install mongodb-org
   sudo yum -y install rabbitmq-server
-  sudo systemctl start mongod rabbitmq-server
-  sudo systemctl enable mongod rabbitmq-server
+  sudo yum -y install redis
+  sudo systemctl start mongod rabbitmq-server redis
+  sudo systemctl enable mongod rabbitmq-server redis
 
 The default python on CentOS/RHEL 7.x is python 2, |st2| uses python3 and requires the python3-devel package. The installation of the st2 package will automatically install python3-devel if it is available in an enabled repository. On CentOS distributions the relevant repository is typically enabled however on RHEL distributions it is provided by the rhel-7-server-optional-rpms repository (repository name dependant on RHEL distribution).
 

docs/source/install/rhel8.rst

@@ -56,7 +56,7 @@ Install Dependencies
 
 .. include:: __mongodb_note.rst
 
-Install MongoDB, RabbitMQ:
+Install MongoDB, RabbitMQ, and Redis:
 
 .. code-block:: bash
 
@@ -80,8 +80,9 @@ Install MongoDB, RabbitMQ:
   sudo yum makecache -y --disablerepo='*' --enablerepo='rabbitmq_rabbitmq-server'
   sudo yum -y install erlang
   sudo yum -y install rabbitmq-server
-  sudo systemctl start mongod rabbitmq-server
-  sudo systemctl enable mongod rabbitmq-server
+  sudo yum -y install redis
+  sudo systemctl start mongod rabbitmq-server redis
+  sudo systemctl enable mongod rabbitmq-server redis
 
 
 Setup Repositories

docs/source/install/system_requirements.rst

@@ -57,6 +57,7 @@ By default, |st2| and related services use these TCP ports:
 * nginx (80, 443)
 * mongodb (27017)
 * rabbitmq (4369, 5672, 25672)
+* redis (6379) or zookeeper (2181, 2888, 3888)
 * st2auth (9100)
 * st2api (9101)
 * st2stream (9102)

docs/source/install/u18.rst

@@ -22,7 +22,7 @@ Minimal Installation
 Install Dependencies
 ~~~~~~~~~~~~~~~~~~~~
 
-Install MongoDB, and RabbitMQ:
+Install MongoDB, RabbitMQ, and Redis:
 
 .. code-block:: bash
 
@@ -39,6 +39,7 @@ Install MongoDB, and RabbitMQ:
   sudo apt-get install -y crudini
   sudo apt-get install -y mongodb-org
   sudo apt-get install -y rabbitmq-server
+  sudo apt-get install -y redis-server
 
 For Ubuntu ``Bionic`` you may need to enable and start MongoDB.
 

docs/source/install/u20.rst

@@ -21,7 +21,7 @@ Minimal Installation
 Install Dependencies
 ~~~~~~~~~~~~~~~~~~~~
 
-Install MongoDB, and RabbitMQ:
+Install MongoDB, RabbitMQ, and Redis:
 
 .. code-block:: bash
 
@@ -38,6 +38,7 @@ Install MongoDB, and RabbitMQ:
   sudo apt-get install -y crudini
   sudo apt-get install -y mongodb-org
   sudo apt-get install -y rabbitmq-server
+  sudo apt-get install -y redis-server
 
 For Ubuntu ``Focal`` you may need to enable and start MongoDB.
 

docs/source/install/uninstall.rst

@@ -48,6 +48,7 @@ below. Only execute the instructions for your distribution.
     sudo service nginx stop
     sudo service mongod stop
     sudo service rabbitmq-server stop
+    sudo service redis-server stop
 
 * RHEL/CentOS 7.x/8.x:
 
@@ -57,6 +58,7 @@ below. Only execute the instructions for your distribution.
     sudo systemctl stop nginx
     sudo systemctl stop mongod
     sudo systemctl stop rabbitmq-server
+    sudo systemctl stop redis
 
 .. note::
 
@@ -119,13 +121,13 @@ below. Only execute the instructions for your distribution.
 
   .. sourcecode:: bash
 
-    sudo apt-get purge mongodb-org* rabbitmq-server erlang* nginx nodejs
+    sudo apt-get purge mongodb-org* rabbitmq-server erlang* nginx nodejs redis-server
 
 * RHEL/CentOS:
 
   .. sourcecode:: bash
 
-    sudo yum erase mongodb-org* rabbitmq-server erlang* nginx nodejs
+    sudo yum erase mongodb-org* rabbitmq-server erlang* nginx nodejs redis
 
 .. note::
 
@@ -164,6 +166,8 @@ last pieces.
     sudo rm -rf /etc/st2 /opt/stackstorm
     sudo rm -rf /var/log/st2 /var/log/mongodb
     sudo rm -rf /var/lib/mongodb /var/run/mongodb.pid 
+    sudo rm -rf /etc/redis/redis.conf /var/lib/redis
+    sudo userdel -r redis
 
 * RHEL/CentOS:
 
@@ -172,7 +176,8 @@ last pieces.
     sudo rm -rf /etc/st2 /etc/mongod* /etc/rabbitmq /etc/nginx /opt/stackstorm
     sudo rm -rf /var/log/st2 /var/log/mongodb /var/log/rabbitmq /var/log/nginx
     sudo rm -rf /var/lib/rabbitmq /var/lib/mongo
-
+    sudo rm -rf /etc/redis/redis.conf /var/lib/redis
+    sudo userdel -r redis
 
 At this point, your system is no longer running any |st2|-related services, and all the main
 dependencies have been removed. You can either re-install |st2|, or use this system for other

docs/source/install/upgrades.rst

@@ -193,6 +193,12 @@ v3.5
      sudo rpm -e --nodeps nodejs
      sudo yum upgrade st2chatops
 
+* Redis server is installed and configured as backend for the coordination service
+  by default in the single node installation script to support workflows with multiple
+  branches and tasks with items. Upgrade requires coordination service to be setup
+  manually, For workflows to be executed properly, setup the coordination service
+  accordingly. See :doc:`../coordination` for setup instruction.
+
 v3.4
 ''''
 

docs/source/reference/ha.rst

@@ -232,10 +232,13 @@ not affect functionality.
 See :ref:`here<ref-rabbitmq-cluster-config>` for how to configure |st2| to connect to a RabbitMQ
 cluster.
 
-Zookeeper/Redis
-^^^^^^^^^^^^^^^
-Various |st2| features rely on a proper co-ordination backend in a distributed deployment to work
-correctly.
+Coordination
+^^^^^^^^^^^^
+Support of workflows with concurrent task executions and concurrency policies for action executions 
+rely on a proper co-ordination backend in a distributed deployment to work correctly.
+
+The coordination service can be configured to use different backends such as redis or zookeeper. For
+the single node installation script, redis is installed and configured by default.
 
 `This <https://zookeeper.apache.org/doc/r3.5.7/zookeeperStarted.html#sc_RunningReplicatedZooKeeper>`__
 shows how to run a replicated Zookeeper setup. (Note: Make sure to refer to the documentation in the
@@ -300,13 +303,14 @@ Controller Box
 ^^^^^^^^^^^^^^
 This box runs all the shared required dependencies and some |st2| components:
 
-* Nginx as load balancer
-* MongoDB
-* RabbitMQ
-* st2chatops
-* st2web
+ * Nginx as load balancer
+ * MongoDB
+ * RabbitMQ
+ * Redis/Zookeeper
+ * st2chatops
+ * st2web
 
-In practice ``MongoDB`` ``RabbitMQ`` will usually be on standalone clusters
+In practice ``MongoDB``, ``RabbitMQ``, and ``Redis/Zookeeper`` will usually be on standalone clusters
 managed outside of |st2|. The two shared components (``st2chatops`` and ``st2web``) are placed here
 for the sake of convenience. They could be placed anywhere with the right configuration.
 
@@ -323,47 +327,50 @@ Follow these steps to provision a controller box on Ubuntu 16.04:
 Install Required Dependencies
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-1. Install ``MongoDB`` and ``RabbitMQ``:
+1. Install ``MongoDB``, ``RabbitMQ``, and ``Redis``:
+
+   The python redis client is already included in the |st2| virtualenv. If using Zookeeper, the
+   kazoo module needs to be installed into the |st2| virtualenv.
 
   .. code-block:: bash
 
-      $ sudo apt-get install -y mongodb-server rabbitmq-server
+      $ sudo apt-get install -y mongodb-server rabbitmq-server redis-server
 
 2. Fix ``bind_ip`` in ``/etc/mongodb.conf`` to bind MongoDB to an interface that has an IP address
    reachable from ``st2-multi-node-1`` and ``st2-multi-node-2``.
 
-4. Restart MongoDB:
+3. Restart MongoDB:
 
    .. code-block:: bash
 
       $ sudo service mongodb restart
 
-5. Add stable |st2| repos:
+4. Add stable |st2| repos:
 
   .. code-block:: bash
 
       $ curl -s https://packagecloud.io/install/repositories/StackStorm/stable/script.deb.sh | sudo bash
 
-6. Setup ``st2web`` and SSL termination. Follow :ref:`install webui and setup
+5. Setup ``st2web`` and SSL termination. Follow :ref:`install webui and setup
    ssl<ref-install-webui-ssl-deb>`. You will need to stop after removing the default Nginx config
    file.
 
-7. A sample configuration for Nginx as load balancer for the controller box is provided below.
+6. A sample configuration for Nginx as load balancer for the controller box is provided below.
    With this configuration Nginx will load balance all requests between the two blueprint boxes
    ``st2-multi-node-1`` and ``st2-multi-node-2``. This includes requests to ``st2api`` and
    ``st2auth``. Nginx also serves as the webserver for ``st2web``.
 
   .. literalinclude:: /../../st2/conf/HA/nginx/st2.conf.controller.sample
      :language: none
 
-8. Create the st2 logs directory and the st2 user:
+7. Create the st2 logs directory and the st2 user:
 
   .. code-block:: bash
 
         mkdir -p /var/log/st2
         useradd st2
 
-12. Install ``st2chatops`` following :ref:`setup chatops<ref-setup-chatops-deb>`.
+8. Install ``st2chatops`` following :ref:`setup chatops<ref-setup-chatops-deb>`.
 
 Blueprint box
 ^^^^^^^^^^^^^
@@ -390,13 +397,13 @@ also be made to offer different services.
 
       $ sudo apt-get install -y nginx
 
-8. Replace ``/etc/st2/st2.conf`` with the sample ``st2.conf`` provided below. This config points to
-   the controller node or configuration values of ``database`` and ``messaging``.
+4. Replace ``/etc/st2/st2.conf`` with the sample ``st2.conf`` provided below. This config points to
+   the controller node or configuration values of ``database``, ``messaging``, and ``coordination``.
 
   .. literalinclude:: /../../st2/conf/HA/st2.conf.sample
      :language: ini
 
-9. Generate a certificate:
+5. Generate a certificate:
 
   .. code-block:: bash
 
@@ -405,19 +412,19 @@ also be made to offer different services.
         -days XXX -nodes -subj "/C=US/ST=California/L=Palo Alto/O=StackStorm/OU=Information \
         Technology/CN=$(hostname)"
 
-10. Configure users & authentication as per :ref:`this documentation<ref-config-auth-deb>`. Make
+6. Configure users & authentication as per :ref:`this documentation<ref-config-auth-deb>`. Make
     sure that user configuration on all boxes running ``st2auth`` is identical. This ensures
     consistent authentication from the entire |st2| install since the request to authenticate a
     user can be forwarded by the load balancer to any of the ``st2auth`` processes.
 
-11. Use the sample Nginx config that is provided below for the blueprint boxes. In this config
+7. Use the sample Nginx config that is provided below for the blueprint boxes. In this config
     Nginx will act as the SSL termination endpoint for all the REST endpoints exposed by
     ``st2api`` and ``st2auth``:
 
   .. literalinclude:: /../../st2/conf/HA/nginx/st2.conf.blueprint.sample
      :language: nginx
 
-12. To use Timer triggers, enable them on only one server. Make this change in
+8. To use Timer triggers, enable them on only one server. Make this change in
     ``/etc/st2/st2.conf``:
 
     .. code-block:: yaml
@@ -426,8 +433,8 @@ also be made to offer different services.
         enable = False
 
 
-14. See :doc:`/reference/sensor_partitioning` to decide how to partition sensors to suit your
+9. See :doc:`/reference/sensor_partitioning` to decide how to partition sensors to suit your
     requirements.
 
-15. All content should be synced by choosing a suitable strategy as outlined above. This is crucial
+10. All content should be synced by choosing a suitable strategy as outlined above. This is crucial
     to obtain predictable outcomes.