[ADD][16.0] fs_storage_backup

Run black

[ADD] tests

[FIX] Tests

[FIX] Tests

[ADD] Misconfig tests

fixup

fs_storage_backup/README.rst

@@ -0,0 +1,103 @@
+=========================
+Filesystem Storage Backup
+=========================
+
+.. 
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! This file is generated by oca-gen-addon-readme !!
+   !! changes will be overwritten.                   !!
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! source digest: sha256:58d2fd55ddd0a80685cbd1df4e905793535d2214d39e43e8922b582888e61763
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
+.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png
+    :target: https://odoo-community.org/page/development-status
+    :alt: Beta
+.. |badge2| image:: https://img.shields.io/badge/licence-AGPL--3-blue.png
+    :target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
+    :alt: License: AGPL-3
+.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fstorage-lightgray.png?logo=github
+    :target: https://github.com/OCA/storage/tree/16.0/fs_storage_backup
+    :alt: OCA/storage
+.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
+    :target: https://translation.odoo-community.org/projects/storage-16-0/storage-16-0-fs_storage_backup
+    :alt: Translate me on Weblate
+.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png
+    :target: https://runboat.odoo-community.org/builds?repo=OCA/storage&target_branch=16.0
+    :alt: Try me on Runboat
+
+|badge1| |badge2| |badge3| |badge4| |badge5|
+
+With this module you can configure one or more database backup
+locations.
+
+**Table of contents**
+
+.. contents::
+   :local:
+
+Configuration
+=============
+
+1. Go to Settings > Technical > FS Storage > FS Storage
+2. Select a filesystem you want to use for backups. **NOTE: Make sure
+   you don't use the filestore as backup location otherwise it's
+   possible you'll back up the backup**
+3. Enable ``Use For Backups``
+4. Follow it (using the chatter) if you want to get notified when a
+   backup fails
+5. To know if the backup is working correctly you can run the scheduled
+   action (``Backup database and delete old backups``) manually to test
+   it.
+
+Usage
+=====
+
+The backup is done automatically by a scheduled action
+(``Backup database and delete old backups``).
+
+Known issues / Roadmap
+======================
+
+-  **Configurable backup frequency**: e.g. backup every 7 days in s3 and
+   every 4 hours on a FTP server.
+
+Bug Tracker
+===========
+
+Bugs are tracked on `GitHub Issues <https://github.com/OCA/storage/issues>`_.
+In case of trouble, please check there if your issue has already been reported.
+If you spotted it first, help us to smash it by providing a detailed and welcomed
+`feedback <https://github.com/OCA/storage/issues/new?body=module:%20fs_storage_backup%0Aversion:%2016.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
+
+Do not contact contributors directly about support or help with technical issues.
+
+Credits
+=======
+
+Authors
+-------
+
+* Onestein
+
+Contributors
+------------
+
+-  Dennis Sluijk d.sluijk@onestein.nl (https://onestein.nl)
+
+Maintainers
+-----------
+
+This module is maintained by the OCA.
+
+.. image:: https://odoo-community.org/logo.png
+   :alt: Odoo Community Association
+   :target: https://odoo-community.org
+
+OCA, or the Odoo Community Association, is a nonprofit organization whose
+mission is to support the collaborative development of Odoo features and
+promote its widespread use.
+
+This module is part of the `OCA/storage <https://github.com/OCA/storage/tree/16.0/fs_storage_backup>`_ project on GitHub.
+
+You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.

fs_storage_backup/__init__.py

@@ -0,0 +1 @@
+from . import models

fs_storage_backup/__manifest__.py

@@ -0,0 +1,14 @@
+{
+    "name": "Filesystem Storage Backup",
+    "category": "Technical",
+    "version": "16.0.1.0.0",
+    "license": "AGPL-3",
+    "author": "Onestein, Odoo Community Association (OCA)",
+    "website": "https://github.com/OCA/storage",
+    "depends": ["fs_storage", "mail"],
+    "data": [
+        "data/ir_cron_data.xml",
+        "data/mail_message_subtype_data.xml",
+        "views/fs_storage_view.xml",
+    ],
+}

fs_storage_backup/data/ir_cron_data.xml

@@ -0,0 +1,18 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<odoo noupdate="1">
+    <record model="ir.cron" id="cron_backup_db">
+        <field name="name">Backup database and delete old backups</field>
+        <field name="interval_number">1</field>
+        <field name="interval_type">days</field>
+        <field name="numbercall">-1</field>
+        <field name="active">True</field>
+        <field name="doall" eval="False" />
+        <field name="model_id" ref="fs_storage.model_fs_storage" />
+        <field name="state">code</field>
+        <field name="code">model.cron_backup_db()</field>
+        <field
+            name="nextcall"
+            eval="(datetime.now() + timedelta(days=1)).strftime('%Y-%m-%d 00:00:00')"
+        />
+    </record>
+</odoo>

fs_storage_backup/data/mail_message_subtype_data.xml

@@ -0,0 +1,15 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<odoo noupdate="1">
+    <record id="message_subtype_backup_failed" model="mail.message.subtype">
+        <field name="name">Backup Failed</field>
+        <field name="description">Backup failed</field>
+        <field name="res_model">fs.storage</field>
+        <field name="default" eval="True" />
+    </record>
+    <record id="message_subtype_cleanup_failed" model="mail.message.subtype">
+        <field name="name">Backup Cleanup Failed</field>
+        <field name="description">Failed to clean up old backups</field>
+        <field name="res_model">fs.storage</field>
+        <field name="default" eval="True" />
+    </record>
+</odoo>

fs_storage_backup/models/__init__.py

@@ -0,0 +1 @@
+from . import fs_storage

fs_storage_backup/models/fs_storage.py

@@ -0,0 +1,116 @@
+import logging
+import traceback
+from datetime import timedelta, timezone
+from os import path
+
+from odoo import _, api, fields, models, tools
+from odoo.exceptions import ValidationError
+from odoo.service import db
+
+_logger = logging.getLogger(__name__)
+
+
+class FSStorage(models.Model):
+    _name = "fs.storage"
+    _inherit = ["fs.storage", "mail.thread"]  # Use queue_job_cron instead?
+
+    use_for_backup = fields.Boolean(string="Use For Backups")
+    backup_include_filestore = fields.Boolean(
+        string="Include Filestore In Backup",
+    )
+    backup_filename_format = fields.Char(
+        string="Backup Filename", default="backup-%(db)s-%(dt)s.%(ext)s"
+    )
+    backup_keep_time = fields.Integer(string="Keep backups of (in days)", default=7)
+    backup_dir = fields.Char(string="Backup Directory", default="backups")
+
+    @property
+    def _server_env_fields(self):
+        env_fields = super()._server_env_fields
+        env_fields.update(
+            {
+                "use_for_backup": {},
+                "backup_include_filestore": {},
+                "backup_filename_format": {"no_default_field": False},
+                "backup_keep_time": {"no_default_field": False},
+                "backup_dir": {"no_default_field": False},
+            }
+        )
+        return env_fields
+
+    @api.constrains("backup_keep_time")
+    def _constrain_backup_keep_time(self):
+        if self.backup_keep_time < 1:
+            raise ValidationError(
+                _("Keep backups of (in days) must be greater or than 0.")
+            )
+
+    def _get_backup_format(self):
+        self.ensure_one()
+        return self.backup_include_filestore and "dump.zip" or "dump"
+
+    def _get_backup_path(self):
+        self.ensure_one()
+        file_ext = self._get_backup_format()
+        current_datetime = fields.Datetime.now().strftime("%Y%m%d%H%M%S")
+        return path.join(
+            self.backup_dir,
+            self.backup_filename_format
+            % {"db": self.env.cr.dbname, "dt": current_datetime, "ext": file_ext},
+        )
+
+    def backup_db(self):
+        self.ensure_one()
+        try:
+            backup_path = self._get_backup_path()
+            self.fs.makedirs(self.backup_dir, exist_ok=True)
+            if self.fs.exists(backup_path):
+                raise Exception("File already exists (%s)." % backup_path)
+            backup_file = self.fs.open(backup_path, "w")
+            list_db = tools.config["list_db"]
+            if not list_db:
+                tools.config["list_db"] = True
+            db.dump_db(
+                self.env.cr.dbname,
+                backup_file.buffer,
+                backup_format=self._get_backup_format(),
+            )
+            tools.config["list_db"] = list_db
+        except Exception as e:
+            _logger.exception("Database backup failed: %s", e)
+            self.message_post(
+                subject=_("Database backup failed"),
+                body="<pre>%s</pre>" % tools.html_escape(traceback.format_exc()),
+                subtype_id=self.env.ref(
+                    "fs_storage_backup.message_subtype_backup_failed"
+                ).id,
+            )
+
+    def cleanup_old_backups(self):
+        self.ensure_one()
+        expiry_date = fields.Datetime.now() - timedelta(days=self.backup_keep_time)
+        try:
+            files = self.fs.ls(self.backup_dir, detail=False)
+            for file_path in files:
+                file_dt = self.fs.modified(file_path)
+                file_dt = file_dt.astimezone(timezone.utc)
+                file_dt = file_dt.replace(tzinfo=None)
+                if file_dt < expiry_date:
+                    self.fs.rm(file_path)
+        except Exception as e:
+            _logger.exception("Failed to clean up old backups: %s", e)
+            self.message_post(
+                subject=_("Failed to clean up old backups"),
+                body="<pre>%s</pre>" % tools.html_escape(traceback.format_exc()),
+                subtype_id=self.env.ref(
+                    "fs_storage_backup.message_subtype_cleanup_failed"
+                ).id,
+            )
+
+    @api.model
+    def cron_backup_db(self):
+        # use_for_backup is not searchable
+        storages = self.search([])
+        for storage in storages.filtered(lambda s: s.use_for_backup):
+            storage.backup_db()
+            storage.cleanup_old_backups()

fs_storage_backup/readme/CONFIGURE.md

@@ -0,0 +1,7 @@
+1. Go to Settings > Technical > FS Storage > FS Storage
+2. Select a filesystem you want to use for backups. 
+   **NOTE: Make sure you don't use the filestore as backup location otherwise it's possible you'll back up the backup**
+3. Enable `Use For Backups`
+4. Follow it (using the chatter) if you want to get notified when a backup fails
+5. To know if the backup is working correctly you can run the scheduled action 
+   (`Backup database and delete old backups`) manually to test it.

fs_storage_backup/readme/CONTRIBUTORS.md

@@ -0,0 +1 @@
+- Dennis Sluijk <d.sluijk@onestein.nl> (https://onestein.nl)

fs_storage_backup/readme/DESCRIPTION.md

@@ -0,0 +1 @@
+With this module you can configure one or more database backup locations.

fs_storage_backup/readme/ROADMAP.md

@@ -0,0 +1 @@
+- **Configurable backup frequency**: e.g. backup every 7 days in s3 and every 4 hours on a FTP server.

fs_storage_backup/readme/USAGE.md

@@ -0,0 +1 @@
+The backup is done automatically by a scheduled action (`Backup database and delete old backups`).