docs: Add a topic about record/replay with ovsdb-server.

Also added a NEWS entry.

Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Acked-by: Dumitru Ceara <dceara@redhat.com>

Documentation/automake.mk

@@ -54,6 +54,7 @@ DOC_SOURCE = \
 	Documentation/topics/ovs-extensions.rst \
 	Documentation/topics/ovsdb-replication.rst \
 	Documentation/topics/porting.rst \
+	Documentation/topics/record-replay.rst \
 	Documentation/topics/tracing.rst \
 	Documentation/topics/userspace-tso.rst \
 	Documentation/topics/windows.rst \

Documentation/topics/index.rst

@@ -48,6 +48,7 @@ OVS
    dpdk/index
    windows
    language-bindings
+   record-replay
    testing
    tracing
    userspace-tso

Documentation/topics/record-replay.rst

@@ -0,0 +1,138 @@
+..
+      Copyright 2021, Red Hat, Inc.
+
+      Licensed under the Apache License, Version 2.0 (the "License"); you may
+      not use this file except in compliance with the License. You may obtain
+      a copy of the License at
+
+          http://www.apache.org/licenses/LICENSE-2.0
+
+      Unless required by applicable law or agreed to in writing, software
+      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+      License for the specific language governing permissions and limitations
+      under the License.
+
+      Convention for heading levels in Open vSwitch documentation:
+
+      =======  Heading 0 (reserved for the title in a document)
+      -------  Heading 1
+      ~~~~~~~  Heading 2
+      +++++++  Heading 3
+      '''''''  Heading 4
+
+      Avoid deeper levels because they do not render well.
+
+============================
+Debugging with Record/Replay
+============================
+
+The ``ovs-replay`` library provides a set of internal functions for recording
+certain events for later replay.  This library is integrated into the
+``stream`` and some other modules to record all incoming data across all
+streams (ssl, tcp, unixctl) of applications based on Open vSwitch libraries
+and play these streams later for debugging or performance testing purposes.
+
+Support for this feature is currently integrated into the ``ovsdb-server`` and
+``ovsdb-client`` applications.  As a result, this allows to record lifecycle
+of the ``ovsdb-server`` process in large OVN deployments.  Later, by using only
+the recorded data, the user can replay transactions and connections that
+occurred in a large deployment on their local PC.  At the same time it is
+possible to tweak various log levels, run a process under debugger or tracer,
+measure performance with ``perf``, and so on.
+
+ .. note::
+
+    The current version of record/replay engine does not work correctly with
+    internal time-based events that leats to communications with other
+    processes.  For this reason it can not be used with clustered databases
+    (RAFT implementation is heavily time dependent).
+    In addition, recording automatically disables inactivity probes on
+    JSONRPC connections and updates for the Manager status in a _Server
+    database.
+
+High-level feature overview was presented on Open vSwitch and OVN 2020 Fall
+Conference: `Debugging OVSDB with stream record/replay`__
+
+__ https://www.openvswitch.org/support/ovscon2020/slides/Debugging-OVSDB-with-stream-record_replay.pdf
+
+Recording ovsdb-server events
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To start recording events for the ``ovsdb-server`` process, there is a special
+command line argument ``--record``.  Before starting the database server, make
+sure that you have a copy of a database file, so you can use it for replay
+later.  Here are the general steps to take:
+
+1. Create a directory where the replay files will be stored::
+
+    $ mkdir replay-dir
+    $ REPLAY_DIR=$(pwd)/replay-dir
+
+2. Copy the current database file for later use::
+
+    $ cp my_database $REPLAY_DIR/
+
+3. Run ``ovsdb-server`` with recording enabled::
+
+    $ ovsdb-server --record=$REPLAY_DIR <other arguments> my_database
+
+4. Work with the database as usual.
+
+5. Stop the ``ovsdb-server`` process at the end (it is important to send an
+   ``exit`` command so that during replay the process will exit in the end
+   too)::
+
+    $ ovs-appctl -t ovsdb-server exit
+
+After that ``$REPLAY_DIR`` should contain replay files with recorded data.
+
+Replay of recorded session
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+During replay, the ``ovsdb-server`` will receive all the same connections,
+transactions and commands as it had at the time of recording, but it will not
+create any actual network/socket connections and will not communicate with
+any other process.  Everything will be read from the replay files.
+
+Since there is no need to wait for IPC, all events will be received one by one
+without any delays, so the application will process them as quickly as
+possible. This can be used as a performance test where the user can measure how
+quickly the ``ovsdb-server`` can handle some workload recorded in a real
+deployment.
+
+The command line argument to start a replay session is ``--replay``.  The steps
+will look like this:
+
+1. Restore the database file from a previous copy::
+
+    $ cp $REPLAY_DIR/my_database my_database
+
+2. Start ``ovsdb-server`` with the same set of arguments as in the recording
+   stage, except for ``--record``::
+
+    $ ovsdb-server --replay=$REPLAY_DIR <other arguments> my_database
+
+3. The process should exit in the end when the ``exit`` command is replayed.
+
+On step 2 it is possible to add extra logging arguments to debug some recorded
+issue, or run the process under debugger.  It's also possible to replay with
+a different version of ``ovsdb-server`` binary as long as this does not affect
+the data that goes in and out of the process, e.g. pure performance
+optimizations.
+
+~~~~~~~~~~~
+Limitations
+~~~~~~~~~~~
+
+The record/replay engine has the following limitations:
+
+1. Record/Replay of clustered databases is not supported.
+
+2. Inactivity probes on JSONRPC connections are suppressed.
+
+3. Manager status updates suppressed in ``ovsdb-server``.
+
+To remove above limitations, it is necessary to implement correct handling of
+internally generated time-based events. (possibly by recording of time and
+subsequent time warping).

NEWS

@@ -1,5 +1,9 @@
 Post-v2.15.0
 ---------------------
+   - OVSDB:
+     * New command line options --record/--replay for ovsdb-server and
+       ovsdb-client to record and replay all the incoming transactions,
+       monitors, etc.  More datails in Documentation/topics/record-replay.rst.
    - In ovs-vsctl and vtep-ctl, the "find" command now accept new
      operators {in} and {not-in}.
    - Userspace datapath: