Bug Report: Tablet becomes unhealthy during incremental backup

[dishank-infurnia]

Overview of the Issue

While taking an incremental backup using vtctlclient, the target tablet demotes to a BACKUP type tablet during the backup.

I looked through the code (release-17.0 branch) and found that:
On go/vt/vttablet/tabletmanager/rpc_backup.go:83, engine.ShouldDrainForBackup() is called to check whether tablet should be drained or not, but builtin engine implements ShouldDrainForBackup as true always. Incremental backups are not catered for in this function.

Reproduction Steps

Take an incremental backup using:

vtctlclient Backup -- --incremental_from_pos=auto <tablet alias> 

Binary Version

Version: 17.0.0 (Git revision 70a94669826071e31f4ac14047de911753f4af5d branch 'heads/v17.0.0') built on Tue Jun 27 13:10:52 UTC 2023 by vitess@buildkitsandbox using go1.20.5 linux/amd64

Operating System and Environment details

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Linux 5.10.162+

x86_64

Log Fragments

No response

[mattlord]

/cc @shlomi-noach

[shlomi-noach]

Confirmed as described, thank you for the report and for the analysis. Fixed in #13773, to be backported to release-17.0.

[shlomi-noach]

Backport to v17: #13789