From 3b7a72ae5b9c4cf45da94ca18b3697d51edb6116 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=A9r=C3=B4me=20Lafr=C3=A9choux?= Date: Fri, 18 Nov 2022 15:49:56 +0100 Subject: [PATCH] Update install/services doc --- source/installation.rst | 80 ++++++++++++++++++++++++++++++----------- source/services.rst | 2 ++ 2 files changed, 61 insertions(+), 21 deletions(-) diff --git a/source/installation.rst b/source/installation.rst index 40e4e91..887e64f 100644 --- a/source/installation.rst +++ b/source/installation.rst @@ -57,35 +57,48 @@ The instructions that follow are tested on a Debian 11 (Bullseye) system. Prerequisites ------------- -BEMServer relies on PostgreSQL and the TimescaleDB extension. +PostgreSQL +^^^^^^^^^^ + +BEMServer uses PostgreSQL. First, install PostgreSQL 13:: $ aptitude install postgresql -TimescaleDB is not in the Debian repositories. Follow instructions in -https://docs.timescale.com/latest/getting-started/setup +Create a database. Assuming the user has database creation permission::: -.. note:: - There is no need to install PostgreSQL from external repositories as - suggested in TimescaleDB docs. + $ createdb bemserver -Create a database and install TimescaleDB extension (should be done after each -TimescaleDB update). +Install prerequisites for psycopg2 compilation (used by BEMServer Core):: -Assuming the user has database creation permission::: + $ aptitude install python3-dev libpq-dev - $ createdb bemserver - $ psql -U $USER -d bemserver -c "CREATE EXTENSION IF NOT EXISTS timescaledb" +TimescaleDB +^^^^^^^^^^^ + +TimescaleDB is a PostgreSQL extension specialized in timeseries data. It may be +used to improve the performance of read-write operations in the table storing +timeseries data. + +Installation instructions are detailed in +`TimescaleDB documentation `_: .. note:: - Later, when updating timescaledb, the extension in the database should be updated too:: + There is no need to install PostgreSQL from external repositories as + suggested in TimescaleDB docs. - $ psql -U $USER -d bemserver -c "ALTER EXTENSION timescaledb UPDATE;" +Install TimescaleDB as described there, then setup the extension in the database +(this should be done after each TimescaleDB update):: -Install prerequisites for psycopg2 compilation (used by BEMServer Core):: +$ psql -U $USER -d bemserver -c "CREATE EXTENSION IF NOT EXISTS timescaledb" - $ aptitude install python3-dev libpq-dev +After the tables are created (see below), create hypertables in timeseries data table:: + +$ psql -U $USER -d bemserver -c "SELECT create_hypertable('ts_data', 'timestamp')" + +Redis +^^^^^ The task scheduler uses Redis database as broker:: @@ -99,7 +112,7 @@ BEMServer API Install bemserver_api:: - $ pip install bemserver_core + $ pip install bemserver_api Database setup ^^^^^^^^^^^^^^ @@ -108,9 +121,9 @@ Set DB URI in an environment variable (adapt line with actual values):: $ export SQLALCHEMY_DATABASE_URI="postgresql+psycopg2://user:password@localhost:5432/bemserver" -Use `setup_db` command to initialize a clean database:: +Use `bemserver_db_upgrade` command to setup the database:: - $ bemserver_setup_db + $ bemserver_db_upgrade Create an admin user:: @@ -152,12 +165,17 @@ Install bemserver_ui:: UI setup ^^^^^^^^ -Create a configuration file (e.g. bemserver-ui.cfg). Add UI parameters: +Create a configuration file (e.g. bemserver-ui.cfg). Add UI parameters and a +random secret key: .. code-block:: BEMSERVER_API_HOST = "localhost:5000" BEMSERVER_API_USE_SSL = False + SECRET_KEY = "XXX" + +..warning:: + The secret key should be a long random string kept secret. Set an environment variable to point to that file:: @@ -178,12 +196,32 @@ At this point, the web UI can be launched from the command line:: Scheduled Tasks --------------- -BEMServer uses `Celery`_ to manage asynchronous tasks. +BEMServer uses `Celery`_ to manage asynchronous tasks. It needs workers to +execute tasks, and another process, called beat, to trigger scheduled tasks + +Celery can be configured with a Python file. + +Create a Celery configuration file (e.g. bemserver-celery-settings.py). It +should contain the scheduling parameters of the services to run:: + + CELERYBEAT_SCHEDULE = { + "service_id": { # Unique identifier of your choice + "task": "ServiceName", # Task name of the service + "schedule": 3600, # Scheduling interval in seconds + "args": (arg_1, args_2), # Task arguments + "kwargs": {"kwarg": val} # Task keyword arguments + }, + } + +For details about how to define entries in the schedule, see +`Celery documentation `_. +Schedules may also be passed in crontab form. Open two shells in an environment where bemerver-core is installed, and in each -shell, define an environment variable with the DB URI:: +shell, define required environment variables:: $ export SQLALCHEMY_DATABASE_URI="postgresql+psycopg2://user:password@localhost:5432/bemserver" + $ export BEMSERVER_CELERY_SETTINGS_FILE="/path/to/bemserver-celery-settings.py" In a shell, start Celery workers to execute the tasks:: diff --git a/source/services.rst b/source/services.rst index cb4e776..d1ff8a6 100644 --- a/source/services.rst +++ b/source/services.rst @@ -22,6 +22,8 @@ The folling sections describe the internal services shipped with BEMServer. Cleanup Service =============== +Task name: "Cleanup" + The Cleanup service drops outliers from timeseries. It runs on a 5 minutes schedule. Once enabled for a campaign, it reads data from all timeseries in the campaign in "Raw" state, drops outliers and writes to the "Clean" state.