Skip to content

Commit

Permalink
Allow ignoring unknown options when loading options from a file
Browse files Browse the repository at this point in the history 
Summary:
Added a flag, `ignore_unknown_options`, to skip unknown options when loading an options file (using `LoadLatestOptions`/`LoadOptionsFromFile`) or while verifying options (using `CheckOptionsCompatibility`). This will help in downgrading the db to an older version.

Also added `--ignore_unknown_options` flag to ldb

**Example Use case:**
In MyRocks, if copying from newer version to older version, it is often impossible to start because of new RocksDB options that don't exist in older version, even though data format is compatible.
MyRocks uses these load and verify functions in [ha_rocksdb.cc::check_rocksdb_options_compatibility](https://github.com/facebook/mysql-5.6/blob/e004fd9f416821d043ccc8ad4a345c33ac9953f0/storage/rocksdb/ha_rocksdb.cc#L3348-L3401).

**Test Plan:**
Updated the unit tests.
`make check`

ldb:
$ ./ldb --db=/tmp/test_db --create_if_missing put a1 b1
OK

Now edit /tmp/test_db/<OPTIONS-file> and add an unknown option.

Try loading the options now, and it fails:
$ ./ldb --db=/tmp/test_db --try_load_options get a1
Failed: Invalid argument: Unrecognized option DBOptions:: abcd

Passes with the new --ignore_unknown_options flag
$ ./ldb --db=/tmp/test_db --try_load_options --ignore_unknown_options get a1
b1
Closes #2423

Differential Revision: D5212091

Pulled By: sagar0

fbshipit-source-id: 2ec17636feb47dc0351b53a77e5f15ef7cbf2ca7
  • Loading branch information
@sagar0 @facebook-github-bot
sagar0 authored and facebook-github-bot committed Jun 13, 2017
1 parent 6b5a5dc commit 89ad9f3
Show file tree
Hide file tree
Showing 11 changed files with 190 additions and 56 deletions.
20 changes: 15 additions & 5 deletions include/rocksdb/convenience.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -161,12 +161,15 @@ namespace rocksdb {
// @param input_strings_escaped when set to true, each escaped characters
// prefixed by '\' in the values of the opts_map will be further converted
// back to the raw string before assigning to the associated options.
// @param ignore_unknown_options when set to true, unknown options are ignored
// instead of resulting in an unknown-option error.
// @return Status::OK() on success. Otherwise, a non-ok status indicating
// error will be returned, and "new_options" will be set to "base_options".
Status GetColumnFamilyOptionsFromMap(
const ColumnFamilyOptions& base_options,
const std::unordered_map<std::string, std::string>& opts_map,
ColumnFamilyOptions* new_options, bool input_strings_escaped = false);
ColumnFamilyOptions* new_options, bool input_strings_escaped = false,
bool ignore_unknown_options = false);

// Take a default DBOptions "base_options" in addition to a
// map "opts_map" of option name to option value to construct the new
Expand All @@ -189,12 +192,15 @@ Status GetColumnFamilyOptionsFromMap(
// @param input_strings_escaped when set to true, each escaped characters
// prefixed by '\' in the values of the opts_map will be further converted
// back to the raw string before assigning to the associated options.
// @param ignore_unknown_options when set to true, unknown options are ignored
// instead of resulting in an unknown-option error.
// @return Status::OK() on success. Otherwise, a non-ok status indicating
// error will be returned, and "new_options" will be set to "base_options".
Status GetDBOptionsFromMap(
const DBOptions& base_options,
const std::unordered_map<std::string, std::string>& opts_map,
DBOptions* new_options, bool input_strings_escaped = false);
DBOptions* new_options, bool input_strings_escaped = false,
bool ignore_unknown_options = false);

// Take a default BlockBasedTableOptions "table_options" in addition to a
// map "opts_map" of option name to option value to construct the new
Expand Down Expand Up @@ -229,14 +235,16 @@ Status GetDBOptionsFromMap(
// @param input_strings_escaped when set to true, each escaped characters
// prefixed by '\' in the values of the opts_map will be further converted
// back to the raw string before assigning to the associated options.
// @param ignore_unknown_options when set to true, unknown options are ignored
// instead of resulting in an unknown-option error.
// @return Status::OK() on success. Otherwise, a non-ok status indicating
// error will be returned, and "new_table_options" will be set to
// "table_options".
Status GetBlockBasedTableOptionsFromMap(
const BlockBasedTableOptions& table_options,
const std::unordered_map<std::string, std::string>& opts_map,
BlockBasedTableOptions* new_table_options,
bool input_strings_escaped = false);
bool input_strings_escaped = false, bool ignore_unknown_options = false);

// Take a default PlainTableOptions "table_options" in addition to a
// map "opts_map" of option name to option value to construct the new
Expand All @@ -250,14 +258,16 @@ Status GetBlockBasedTableOptionsFromMap(
// @param input_strings_escaped when set to true, each escaped characters
// prefixed by '\' in the values of the opts_map will be further converted
// back to the raw string before assigning to the associated options.
// @param ignore_unknown_options when set to true, unknown options are ignored
// instead of resulting in an unknown-option error.
// @return Status::OK() on success. Otherwise, a non-ok status indicating
// error will be returned, and "new_table_options" will be set to
// "table_options".
Status GetPlainTableOptionsFromMap(
const PlainTableOptions& table_options,
const std::unordered_map<std::string, std::string>& opts_map,
PlainTableOptions* new_table_options,
bool input_strings_escaped = false);
PlainTableOptions* new_table_options, bool input_strings_escaped = false,
bool ignore_unknown_options = false);

// Take a string representation of option names and values, apply them into the
// base_options, and return the new options as a result. The string has the
Expand Down
3 changes: 3 additions & 0 deletions include/rocksdb/utilities/ldb_cmd.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ class LDBCommand {
static const std::string ARG_TTL_END;
static const std::string ARG_TIMESTAMP;
static const std::string ARG_TRY_LOAD_OPTIONS;
static const std::string ARG_IGNORE_UNKNOWN_OPTIONS;
static const std::string ARG_FROM;
static const std::string ARG_TO;
static const std::string ARG_MAX_KEYS;
Expand Down Expand Up @@ -149,6 +150,8 @@ class LDBCommand {
// If true, try to construct options from DB's option files.
bool try_load_options_;

bool ignore_unknown_options_;

bool create_if_missing_;

/**
Expand Down
13 changes: 10 additions & 3 deletions include/rocksdb/utilities/options_util.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ namespace rocksdb {
// casting the return value of TableFactoroy::GetOptions() to
// BlockBasedTableOptions and making necessary changes.
//
// ignore_unknown_options can be set to true if you want to ignore options
// that are from a newer version of the db, esentially for forward
// compatibility.
//
// examples/options_file_example.cc demonstrates how to use this function
// to open a RocksDB instance.
//
Expand All @@ -53,15 +57,17 @@ namespace rocksdb {
// @see LoadOptionsFromFile
Status LoadLatestOptions(const std::string& dbpath, Env* env,
DBOptions* db_options,
std::vector<ColumnFamilyDescriptor>* cf_descs);
std::vector<ColumnFamilyDescriptor>* cf_descs,
bool ignore_unknown_options = false);

// Similar to LoadLatestOptions, this function constructs the DBOptions
// and ColumnFamilyDescriptors based on the specified RocksDB Options file.
//
// @see LoadLatestOptions
Status LoadOptionsFromFile(const std::string& options_file_name, Env* env,
DBOptions* db_options,
std::vector<ColumnFamilyDescriptor>* cf_descs);
std::vector<ColumnFamilyDescriptor>* cf_descs,
bool ignore_unknown_options = false);

// Returns the latest options file name under the specified db path.
Status GetLatestOptionsFileName(const std::string& dbpath, Env* env,
Expand All @@ -80,7 +86,8 @@ Status GetLatestOptionsFileName(const std::string& dbpath, Env* env,
// * merge_operator
Status CheckOptionsCompatibility(
const std::string& dbpath, Env* env, const DBOptions& db_options,
const std::vector<ColumnFamilyDescriptor>& cf_descs);
const std::vector<ColumnFamilyDescriptor>& cf_descs,
bool ignore_unknown_options = false);

} // namespace rocksdb
#endif // !ROCKSDB_LITE
68 changes: 48 additions & 20 deletions options/options_helper.cc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1008,7 +1008,8 @@ Status ParseDBOption(const std::string& name,
std::string ParseBlockBasedTableOption(const std::string& name,
const std::string& org_value,
BlockBasedTableOptions* new_options,
bool input_strings_escaped = false) {
bool input_strings_escaped = false,
bool ignore_unknown_options = false) {
const std::string& value =
input_strings_escaped ? UnescapeOptionString(org_value) : org_value;
if (!input_strings_escaped) {
Expand Down Expand Up @@ -1042,7 +1043,11 @@ std::string ParseBlockBasedTableOption(const std::string& name,
}
const auto iter = block_based_table_type_info.find(name);
if (iter == block_based_table_type_info.end()) {
return "Unrecognized option";
if (ignore_unknown_options) {
return "";
} else {
return "Unrecognized option";
}
}
const auto& opt_info = iter->second;
if (opt_info.verification != OptionVerificationType::kDeprecated &&
Expand All @@ -1056,12 +1061,17 @@ std::string ParseBlockBasedTableOption(const std::string& name,
std::string ParsePlainTableOptions(const std::string& name,
const std::string& org_value,
PlainTableOptions* new_options,
bool input_strings_escaped = false) {
bool input_strings_escaped = false,
bool ignore_unknown_options = false) {
const std::string& value =
input_strings_escaped ? UnescapeOptionString(org_value) : org_value;
const auto iter = plain_table_type_info.find(name);
if (iter == plain_table_type_info.end()) {
return "Unrecognized option";
if (ignore_unknown_options) {
return "";
} else {
return "Unrecognized option";
}
}
const auto& opt_info = iter->second;
if (opt_info.verification != OptionVerificationType::kDeprecated &&
Expand All @@ -1075,12 +1085,14 @@ std::string ParsePlainTableOptions(const std::string& name,
Status GetBlockBasedTableOptionsFromMap(
const BlockBasedTableOptions& table_options,
const std::unordered_map<std::string, std::string>& opts_map,
BlockBasedTableOptions* new_table_options, bool input_strings_escaped) {
BlockBasedTableOptions* new_table_options, bool input_strings_escaped,
bool ignore_unknown_options) {
assert(new_table_options);
*new_table_options = table_options;
for (const auto& o : opts_map) {
auto error_message = ParseBlockBasedTableOption(
o.first, o.second, new_table_options, input_strings_escaped);
o.first, o.second, new_table_options, input_strings_escaped,
ignore_unknown_options);
if (error_message != "") {
const auto iter = block_based_table_type_info.find(o.first);
if (iter == block_based_table_type_info.end() ||
Expand All @@ -1090,7 +1102,8 @@ Status GetBlockBasedTableOptionsFromMap(
(iter->second.verification != OptionVerificationType::kByName &&
iter->second.verification !=
OptionVerificationType::kByNameAllowNull &&
iter->second.verification != OptionVerificationType::kDeprecated)) {
iter->second.verification !=
OptionVerificationType::kDeprecated)) {
// Restore "new_options" to the default "base_options".
*new_table_options = table_options;
return Status::InvalidArgument("Can't parse BlockBasedTableOptions:",
Expand All @@ -1117,7 +1130,8 @@ Status GetBlockBasedTableOptionsFromString(
Status GetPlainTableOptionsFromMap(
const PlainTableOptions& table_options,
const std::unordered_map<std::string, std::string>& opts_map,
PlainTableOptions* new_table_options, bool input_strings_escaped) {
PlainTableOptions* new_table_options, bool input_strings_escaped,
bool ignore_unknown_options) {
assert(new_table_options);
*new_table_options = table_options;
for (const auto& o : opts_map) {
Expand All @@ -1132,11 +1146,12 @@ Status GetPlainTableOptionsFromMap(
(iter->second.verification != OptionVerificationType::kByName &&
iter->second.verification !=
OptionVerificationType::kByNameAllowNull &&
iter->second.verification != OptionVerificationType::kDeprecated)) {
iter->second.verification !=
OptionVerificationType::kDeprecated)) {
// Restore "new_options" to the default "base_options".
*new_table_options = table_options;
return Status::InvalidArgument("Can't parse PlainTableOptions:",
o.first + " " + error_message);
o.first + " " + error_message);
}
}
}
Expand Down Expand Up @@ -1229,16 +1244,19 @@ Status GetMemTableRepFactoryFromString(const std::string& opts_str,
Status GetColumnFamilyOptionsFromMap(
const ColumnFamilyOptions& base_options,
const std::unordered_map<std::string, std::string>& opts_map,
ColumnFamilyOptions* new_options, bool input_strings_escaped) {
ColumnFamilyOptions* new_options, bool input_strings_escaped,
bool ignore_unknown_options) {
return GetColumnFamilyOptionsFromMapInternal(
base_options, opts_map, new_options, input_strings_escaped);
base_options, opts_map, new_options, input_strings_escaped, nullptr,
ignore_unknown_options);
}

Status GetColumnFamilyOptionsFromMapInternal(
const ColumnFamilyOptions& base_options,
const std::unordered_map<std::string, std::string>& opts_map,
ColumnFamilyOptions* new_options, bool input_strings_escaped,
std::vector<std::string>* unsupported_options_names) {
std::vector<std::string>* unsupported_options_names,
bool ignore_unknown_options) {
assert(new_options);
*new_options = base_options;
if (unsupported_options_names) {
Expand All @@ -1258,6 +1276,8 @@ Status GetColumnFamilyOptionsFromMapInternal(
// Note that we still return Status::OK in such case to maintain
// the backward compatibility in the old public API defined in
// rocksdb/convenience.h
} else if (s.IsInvalidArgument() && ignore_unknown_options) {
continue;
} else {
// Restore "new_options" to the default "base_options".
*new_options = base_options;
Expand All @@ -1284,16 +1304,19 @@ Status GetColumnFamilyOptionsFromString(
Status GetDBOptionsFromMap(
const DBOptions& base_options,
const std::unordered_map<std::string, std::string>& opts_map,
DBOptions* new_options, bool input_strings_escaped) {
return GetDBOptionsFromMapInternal(
base_options, opts_map, new_options, input_strings_escaped);
DBOptions* new_options, bool input_strings_escaped,
bool ignore_unknown_options) {
return GetDBOptionsFromMapInternal(base_options, opts_map, new_options,
input_strings_escaped, nullptr,
ignore_unknown_options);
}

Status GetDBOptionsFromMapInternal(
const DBOptions& base_options,
const std::unordered_map<std::string, std::string>& opts_map,
DBOptions* new_options, bool input_strings_escaped,
std::vector<std::string>* unsupported_options_names) {
std::vector<std::string>* unsupported_options_names,
bool ignore_unknown_options) {
assert(new_options);
*new_options = base_options;
if (unsupported_options_names) {
Expand All @@ -1313,6 +1336,8 @@ Status GetDBOptionsFromMapInternal(
// Note that we still return Status::OK in such case to maintain
// the backward compatibility in the old public API defined in
// rocksdb/convenience.h
} else if (s.IsInvalidArgument() && ignore_unknown_options) {
continue;
} else {
// Restore "new_options" to the default "base_options".
*new_options = base_options;
Expand Down Expand Up @@ -1360,12 +1385,14 @@ Status GetOptionsFromString(const Options& base_options,
Status GetTableFactoryFromMap(
const std::string& factory_name,
const std::unordered_map<std::string, std::string>& opt_map,
std::shared_ptr<TableFactory>* table_factory) {
std::shared_ptr<TableFactory>* table_factory, bool ignore_unknown_options) {
Status s;
if (factory_name == BlockBasedTableFactory().Name()) {
BlockBasedTableOptions bbt_opt;
s = GetBlockBasedTableOptionsFromMap(BlockBasedTableOptions(), opt_map,
&bbt_opt, true);
&bbt_opt,
true, /* input_strings_escaped */
ignore_unknown_options);
if (!s.ok()) {
return s;
}
Expand All @@ -1374,7 +1401,8 @@ Status GetTableFactoryFromMap(
} else if (factory_name == PlainTableFactory().Name()) {
PlainTableOptions pt_opt;
s = GetPlainTableOptionsFromMap(PlainTableOptions(), opt_map, &pt_opt,
true);
true, /* input_strings_escaped */
ignore_unknown_options);
if (!s.ok()) {
return s;
}
Expand Down
9 changes: 6 additions & 3 deletions options/options_helper.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ Status GetMutableDBOptionsFromStrings(
Status GetTableFactoryFromMap(
const std::string& factory_name,
const std::unordered_map<std::string, std::string>& opt_map,
std::shared_ptr<TableFactory>* table_factory);
std::shared_ptr<TableFactory>* table_factory,
bool ignore_unknown_options = false);

Status GetStringFromTableFactory(std::string* opts_str, const TableFactory* tf,
const std::string& delimiter = "; ");
Expand Down Expand Up @@ -129,7 +130,8 @@ Status GetDBOptionsFromMapInternal(
const DBOptions& base_options,
const std::unordered_map<std::string, std::string>& opts_map,
DBOptions* new_options, bool input_strings_escaped,
std::vector<std::string>* unsupported_options_names = nullptr);
std::vector<std::string>* unsupported_options_names = nullptr,
bool ignore_unknown_options = false);

// In addition to its public version defined in rocksdb/convenience.h,
// this further takes an optional output vector "unsupported_options_names",
Expand All @@ -138,7 +140,8 @@ Status GetColumnFamilyOptionsFromMapInternal(
const ColumnFamilyOptions& base_options,
const std::unordered_map<std::string, std::string>& opts_map,
ColumnFamilyOptions* new_options, bool input_strings_escaped,
std::vector<std::string>* unsupported_options_names = nullptr);
std::vector<std::string>* unsupported_options_names = nullptr,
bool ignore_unknown_options = false);

static std::unordered_map<std::string, OptionTypeInfo> db_options_type_info = {
/*
Expand Down
Loading

0 comments on commit 89ad9f3

Please sign in to comment.