DDL: should we support to rollback from v7.1.0 to v6.5.1?

[AndreMouche]

Enhancement

I accidentally upgraded tidb(only tidb) from v6.5.1 to v7.1.0(master) successfully, with tikv and pd are still v6.5.1.

Now I would like to rollback tidb(only tidb) from v7.1.0 to v6.5.1 , however, it failed with error

[2023/07/25 11:13:50.818 -07:00] [FATAL] [terror.go:300] ["unexpected error"] [error="[meta:1050]table already exists"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/Users/pingcap/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:300\nmain.createStoreAndDomain\n\t/Users/pingcap/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:315\nmain.main\n\t/Users/pingcap/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:214\nruntime.main\n\t/usr/local/go1.19.3/src/runtime/proc.go:250"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/Users/pingcap/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:300\nmain.createStoreAndDomain\n\t/Users/pingcap/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:315\nmain.main\n\t/Users/pingcap/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:214\nruntime.main\n\t/usr/local/go1.19.3/src/runtime/proc.go:250"]

it cames from the SQL: https://github.com/pingcap/tidb/blob/v6.5.1/session/session.go#L3090
The error was throwed by this function InitMDLTable https://github.com/pingcap/tidb/blob/v6.5.1/session/session.go#L3147-L3176

tidb/session/session.go

Lines 3147 to 3176 in 4084b07

	func InitMDLTable(store kv.Storage) error {
	return kv.RunInNewTxn(kv.WithInternalSourceType(context.Background(), kv.InternalTxnDDL), store, true, func(ctx context.Context, txn kv.Transaction) error {
	t := meta.NewMeta(txn)
	exists, err := t.CheckMDLTableExists()
	if err != nil || exists {
	return errors.Trace(err)
	}
	dbID, err := t.CreateMySQLDatabaseIfNotExists()
	if err != nil {
	return err
	}
	splitAndScatterTable(store, []int64{ddl.MDLTableID})
	p := parser.New()
	stmt, err := p.ParseOneStmt(mdlTable, "", "")
	if err != nil {
	return errors.Trace(err)
	}
	tblInfo, err := ddl.BuildTableInfoFromAST(stmt.(*ast.CreateTableStmt))
	if err != nil {
	return errors.Trace(err)
	}
	tblInfo.State = model.StatePublic
	tblInfo.ID = ddl.MDLTableID
	tblInfo.UpdateTS = t.StartTS
	err = t.CreateTableOrView(dbID, tblInfo)
	if err != nil {
	return errors.Trace(err)
	}
	return t.SetMDLTables()

in the transaction:

1. it returns false when CheckMDLTableExist
2. TiDB believes that this tidb_mdl_info not exist, so create it by CreateTableOrView
3. CreateTableOrView run the sql to create the tidb-mdl-info and meet already exist error, which is supposed not exist by step 1

here comes the function CheckMDLTableExists https://github.com/pingcap/tidb/blob/v6.5.1/meta/meta.go#L616-L623

tidb/meta/meta.go

Lines 616 to 623 in 4084b07

	// CheckMDLTableExists check if the tables related to concurrent DDL exists.
	func (m *Meta) CheckMDLTableExists() (bool, error) {
	v, err := m.txn.Get(mDDLTableVersion)
	if err != nil {
	return false, errors.Trace(err)
	}
	return bytes.Equal(v, []byte("2")), nil
	}

which means v6.5.1 the mDDLTableVersion should be []byte("2") when the mdl-table exists. So I believe v7.1.0 may changed the value of mDDLTableVersion. And I checked the same function on v7.1.0 and I think I've found the answer
https://github.com/pingcap/tidb/blob/v7.1.0/session/session.go#L3175-L3182

tidb/session/session.go

Lines 3175 to 3183 in 635a436

	// InitMDLTable is to create tidb_mdl_info, which is used for metadata lock.
	func InitMDLTable(store kv.Storage) error {
	return kv.RunInNewTxn(kv.WithInternalSourceType(context.Background(), kv.InternalTxnDDL), store, true, func(ctx context.Context, txn kv.Transaction) error {
	t := meta.NewMeta(txn)
	ver, err := t.CheckDDLTableVersion()
	if err != nil || ver >= meta.MDLTableVersion {
	return errors.Trace(err)
	}
	dbID, err := t.CreateMySQLDatabaseIfNotExists()

Which means v7.1.0 thinks if mDDLTableVersion >=2, mdl-table should exist.
And the value of mDDLTableVersion should be seted to 3 by InitDDLJobTables(store, meta.BackfillTableVersion) with BackfillTableVersion=3 on v7.1.0 function BootstrapSession

tidb/session/session.go

Lines 3270 to 3290 in 635a436

	func BootstrapSession(store kv.Storage) (*domain.Domain, error) {
	ctx := kv.WithInternalSourceType(context.Background(), kv.InternalTxnBootstrap)
	cfg := config.GetGlobalConfig()
	if len(cfg.Instance.PluginLoad) > 0 {
	err := plugin.Load(context.Background(), plugin.Config{
	Plugins: strings.Split(cfg.Instance.PluginLoad, ","),
	PluginDir: cfg.Instance.PluginDir,
	})
	if err != nil {
	return nil, err
	}
	}
	err := InitDDLJobTables(store, meta.BaseDDLTableVersion)
	if err != nil {
	return nil, err
	}
	err = InitMDLTable(store)
	if err != nil {
	return nil, err
	}
	err = InitDDLJobTables(store, meta.BackfillTableVersion)

Now let's get back to the most fundamental issue, how to rollback to v6.5.1 from v7.1.0 successfully? it seems hard to change the value of key DDLTableVersion to []byte("2"). And drop table tidb_mdl_info itself is a DDL job.
And should we provide a workaround for this case?

[bb7133]

I don't think currently we support rollback(downgrade, more precisely) for any version/component.