Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various Backup and restore improvements #174

Merged
merged 32 commits into from
Jan 5, 2021
Merged

Various Backup and restore improvements #174

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
b395c15
squash commits
NikTJ777 Aug 13, 2020
0689fb6
rebase on master
NikTJ777 Nov 5, 2020
50d4d65
update database restore test
NikTJ777 Nov 6, 2020
82fc1c5
Merge branch 'master' into ntj/restore-improvements-merged
sivanov-nuodb Dec 21, 2020
8ba8c66
Remove commented out and unused code
sivanov-nuodb Dec 21, 2020
6775243
Undo unused configmap and test changes
sivanov-nuodb Dec 21, 2020
5a75a38
Uno changes in testlib
sivanov-nuodb Dec 21, 2020
536765d
Undo new constants.go file
sivanov-nuodb Dec 21, 2020
15e777c
Fix some review comments in #PR80
sivanov-nuodb Dec 21, 2020
4ed8e25
Fix merge problems
sivanov-nuodb Dec 21, 2020
537376d
Unconditionally log messages from nuosm
sivanov-nuodb Dec 21, 2020
8613569
[HELM-181] nuosm should handle invalid parameters better, e.g. invali…
sivanov-nuodb Dec 23, 2020
8796aa2
Various logging fixes
sivanov-nuodb Dec 23, 2020
65a997d
Minor logging fixes in nuosm
sivanov-nuodb Dec 23, 2020
095f73d
Die if nuodocker fails; fix bug in curl download path
sivanov-nuodb Dec 23, 2020
2d7977a
Clear restore request only on successfull restore
sivanov-nuodb Dec 23, 2020
2b09a1b
Remove NUOCMD_API_SERVER references; fix problem with autoRestore whe…
sivanov-nuodb Dec 23, 2020
495ca59
Add tests for autoImport
sivanov-nuodb Dec 29, 2020
5dcacaf
Fix autoRestart value
sivanov-nuodb Dec 29, 2020
513d1ae
Add multiple SMs restore tests
sivanov-nuodb Dec 29, 2020
ce929c9
Merge branch 'master' into ntj/restore-improvements-merged
sivanov-nuodb Dec 29, 2020
c7dd6d1
Remove unused functions in nuobackup
sivanov-nuodb Dec 29, 2020
5640a38
Address comments on shell script linting test
sivanov-nuodb Dec 30, 2020
6fd4d7a
Re-add fix done by original change
sivanov-nuodb Dec 30, 2020
4826b33
Bamboo build hosts have dash shell which doesn't like function keyword
sivanov-nuodb Dec 30, 2020
610bc2a
Address POSIX compliant shell script issues in database/files/readine…
sivanov-nuodb Dec 30, 2020
0a94328
Fix test dependencies
sivanov-nuodb Dec 30, 2020
ae8f292
Address review comments
sivanov-nuodb Dec 31, 2020
ed07dd3
[ci skip] Add default value for DB_NAME
sivanov-nuodb Dec 31, 2020
b1b8d83
Remove nuosm exit codes
sivanov-nuodb Dec 31, 2020
fd73670
Add autoRestore test automation
sivanov-nuodb Dec 31, 2020
e1b6d9d
Add timestamp to nuosm and nuorestore logs
sivanov-nuodb Dec 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,7 @@ github.com/gruntwork-io/gruntwork-cli v0.6.1/go.mod h1:jp6Z7NcLF2avpY8v71fBx6hds
github.com/gruntwork-io/terratest v0.28.10 h1:4K3idJK5atavZ7iwY+L+RPCA7lFwPu87dVnoFOaoPzI=
github.com/gruntwork-io/terratest v0.28.10/go.mod h1:PkVylPuUNmItkfOTwSiFreYA4FkanK8AluBuNeGxQOw=
github.com/gruntwork-io/terratest v0.30.15 h1:/OAnHKEERSsy03wYi8dyEwnpYy2XV8wpzwq0c4BHvoc=
github.com/gruntwork-io/terratest v0.31.2 h1:xvYHA80MUq5kx670dM18HInewOrrQrAN+XbVVtytUHg=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.3/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
Expand Down
2 changes: 1 addition & 1 deletion stable/admin/files/nuoadmin
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/sh
#!/bin/bash

. ${NUODB_HOME}/etc/nuodb_setup.sh

Expand Down
23 changes: 21 additions & 2 deletions stable/database/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,12 @@ The following tables list the configurable parameters of the `database` chart an
| `configFilesPath` | Directory path where `configFiles.*` are found | `/etc/nuodb/` |
| `configFiles.*` | See below. | `{}` |
| `podAnnotations` | Annotations to pass through to the SM an TE pods | `nil` |
| `autoImport.*` | Enable and configure the automatic initializing of the initial database state | `disabled` |
| `autoImport.source` | The source - typically a URL - of the database copy to import | `""` |
| `autoImport.credentials` | Authentication for the download of `source` in the form `user`:`password` | '""'|
| `autoImport.stripLevels` | The number of levels to strip off pathnames when unpacking a TAR file of an archive | `1` |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assumes that the entire contents of the TAR file are contained in a path of that length. Otherwise it would be ambiguous which path within the TAR file contains the archive. I think it would be better if we took a pathname within the TAR file that contains the archive. Also, it is possible that the archive and journal are in different directories, but I'm not sure how this parameter would account for that.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This value of stripLevels is transparently passed to tar as --strip-components option. The typical use case is that the source tarball contains only one archive directory (including the journal).
Currently, an external journal directory is not supported.

| `autoImport.type` | Type of content in `source`. One of `stream` -> exact copy of an archive; or `backupset` -> a NuoDB hotcopy backupset | 'backupset' |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the name "stream" instead of "archive"?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the name originally chosen. I'm not sure about the reason. Effectively stream is a tar.gz of NuoDB archive which is ready to be downloaded.

| `autoRestore.*` | Enable and configure the automatic re-initialization of a single archive in a running database - see the options in `autoImport` | `disabled` |
| `sm.logPersistence.enabled` | Whether to enable persistent storage for logs | `false` |
| `sm.logPersistence.overwriteBackoff.copies` | How many copies of the crash directory to keep within windowMinutes | `3` |
| `sm.logPersistence.overwriteBackoff.windowMinutes` | The window within which to keep the number of crash copies | `120` |
Expand All @@ -242,7 +248,7 @@ The following tables list the configurable parameters of the `database` chart an
| `sm.hotcopy.timeout` | Timeout for a started hotcopy job to complete (seconds) | `1800` |
| `sm.hotcopy.successHistory` | Number of successful Jobs to keep | `5` |
| `sm.hotcopy.failureHostory` | Number of failed jobs to keep | `5` |
| `sm.hotcopy.backupDir` | Directory path where backiupsets will be stored | `/var/opt/nuodb/backup` |
| `sm.hotcopy.backupDir` | Directory path where backupsets will be stored | `/var/opt/nuodb/backup` |
| `sm.hotcopy.backupGroup` | Name of the backup group | `{{ .Values.cloud.cluster.name }}` |
| `sm.hotcopy.fullSchedule` | cron schedule for FULL hotcopy jobs | `35 22 * * 6` |
| `sm.hotcopy.incrementalSchedule` | cron schedule for INCREMENTAL hotcopy jobs | `35 22 * * 0-5` |
Expand Down Expand Up @@ -303,7 +309,20 @@ Any file located in `database.configFilesPath` can be replaced; the YAML key cor
| ----- | ----------- | ------ |
| `nuodb.config` | [NuoDB database options][6] | `nil` |

#### database.serviceSuffix
#### restore.*

The following tables list the configurable parameters of the restore chart and their default values.

| Parameter | Description | Default |
| ----- | ----------- | ------ |
| `restore.type` | What type of restore to perform: [ "database" | "archive" ]. A "database" restore restarts the entire database at a previous state. An "archive" restore restores/repairs a SINGLE archive in a RUNNING database. | `"database"` |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the database have to be RUNNING? Can you restore a single archive with a DB that is down? I know that it virtually becomes a database restore, but will the flag work?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The database doesn't have to be running, however, the admin layer should. nuorestore just sets a flag in KV store which indicates that restore needs to be performed on the next database restart.

| `restore.target` | Where to restore `TO` | `{{ .Values.database.name }}` |
| `restore.source` | Where to restore `FROM` [ backupset | url | `:latest` | `:group-latest` ] | `:latest` |
| `restore.credentials` | Credentials to use for a URL source (user:password) | `""` |
| `restore.autoRestart` | Whether to automatically restart the database and trigger the restore (true/false). Only valid for a "database" restore | `true` |


### Running

The purpose of this section is to allow customisation of the names of the clusterIP and balancer database services (load-balancers).

Expand Down
4 changes: 2 additions & 2 deletions stable/database/files/nuobackup
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# A backup can be specific to a backup-group, and the backup-groups can have different schedules.
#
# An optional semaphore can be specified - if the semaphore is specified, and not set in the KV store, then
# NO backupis performed.
# NO backup is performed.
# This allows a cronjob to run `nuobackup` on a schedule, and conditionally perform a backup if the semaphore has been set.
#
# The information of recent backups in the KV store is organised as a ringbuffer.
Expand All @@ -32,7 +32,7 @@
# --wait-timeout - timeout (in seconds) to wait for specified amount of SM processes

# for debugging...
[ -n "$NUODB_DEBUG" ] && set -x
[ "$NUODB_DEBUG" = "verbose" ] && set -x

: ${NUODB_MAX_BACKUP_HISTORY:=10}
: ${NUODB_BACKUP_KEY:=/nuodb/nuobackup}
Expand Down
Loading