Account for gap between snapshot preparation and archive creation

[adriansuarez]

For SMs created from snapshots, we currently we rely on the absence of the info.json file and presence of the backup.txt file to detect that the archive has to be prepared from a snapshot. But there is a window of time until the info.json file for the newly-prepared archive is created in which any failure would leave the nuosm script unable to detect if it should perform a normal restart.

This change adds a restore.txt file which is created at the same time the old info.json and backup.txt file are deleted and is used to detect that the SM already performed snapshot preparation and should continue with normal startup.

[sivanov-nuodb]

Looks good.

[kontaras]

Don't forget to delete restored.txt at the end. It will break an SM being restored, snapshot, and restored.

[adriansuarez]

Don't forget to delete restored.txt at the end. It will break an SM being restored, snapshot, and restored.

Great point. The info.json file is created by nuodocker start sm, so the nuosm script cannot delete restored.txt after info.json is created, because we have handed control of the startup lifecycle to nuodocker start sm by that point. Since nuodocker start sm is implemented in the container image, putting that behavior there would create a dependency on a particular image version for backup to work.

I will put the deletion inside the backup_hooks.py script in the pre-backup hook, before the new backup.txt is created.

[sivanov-nuodb]

Don't forget to delete restored.txt at the end. It will break an SM being restored, snapshot, and restored.

Great point. The info.json file is created by nuodocker start sm, so the nuosm script cannot delete restored.txt after info.json is created, because we have handed control of the startup lifecycle to nuodocker start sm by that point. Since nuodocker start sm is implemented in the container image, putting that behavior there would create a dependency on a particular image version for backup to work.

I will put the deletion inside the backup_hooks.py script in the pre-backup hook, before the new backup.txt is created.

What is the issue by not deleting the restore.txt file? I see that only the absence of the file is checked in combination with the absence of info.json.

[adriansuarez]

What is the issue by not deleting the restore.txt file? I see that only the absence of the file is checked in combination with the absence of info.json.

After thinking a little bit more about it, I actually do not think it is an issue. A stale restore.txt can only be found in the case where there is a stale info.json, i.e. restore with same domain and database. The presence of the backup.txt would still prevent us from skipping the deletion of these files.

Either way, I think it makes sense to remove the stale restore.txt, since it is irrelevant which backup the database that was backed up came from.