CLI toml-test decoder

[moorereason]

Is your feature request related to a problem? Please describe.

I've been doing differential fuzzing against a few TOML decoders to see where they may handle inputs differently. I would like to add tomlplusplus into the mix.

Describe the solution you'd like

Can tomlplusplus provide a CLI toml-test decoder? The decoder takes a given TOML input via STDIN and outputs annotated JSON to STDOUT. Details are in the toml-test README.

Additional context

I've filed several bug reports in other projects and submitted new test cases to toml-test based upon this fuzzing.

Thanks for considering.

[marzer]

There's no toml-test decoder for toml++ currently, though my test suite consumes the test inputs from toml-test and generates C++ unit tests to validate using the test suite, e.g.: conformance_burntsushi_valid.cpp

I update the conformance tests every time I do any work on the library so it's generally up-to-date.

"Why did you bother writing a C++ code generator and not just provide an encoder/decoder?", I hear you ask.

It made sense at the time. Third-party libs in C++ can be very annoying, with or without CMake and friends, so in the early days of toml++ this approach seemed easier. I am going to do another batch of toml++ work at some point in January so if I get time I'll revisit this.

[moorereason]

I'm not advocating that you stop generating unit tests. Every(?) TOML package I've looked at generates local unit tests from the suite. However, I've only found 5 packages that have a toml-test decoder implementation.

I find a toml-test decoder useful for differential fuzzing and submitting bug reports. For example, I'm not a C++ programmer, but I can use a Go fuzzer to find and report bugs in a C++ decoder such as toml11 (e.g. ToruNiina/toml11#180 & ToruNiina/toml11#181). No C++ skills required! 😄

[marzer]

I'm not advocating that you stop generating unit tests.

Oh, certainly. I didn't think you were. Apologies if I gave you that impression. Was just explaining why this repo does the seemingly-harder thing first, instead of the probably-easier-thing, heh.

I do see the appeal. There's even a few tests I have to manually skip the code generation for because various C++ compilers don't like the in-code equivalent (e.g. GCC doesn't like whitespace after a trailing \ in a string literal, and there's a few tests in toml-test that have those). I'll get around to it eventually, hopefully as part of the next release.

[marzer]

@moorereason I've implemented this in the v3 branch - see v3/toml-test for setup info.

Every(?) TOML package I've looked at generates local unit tests from the suite.

Just re-read this. How do you figure that? It certainly isn't true for the C++ TOML ecosystem. In fact toml++ is the only one that chose that route, as far as I can tell - the others (the still-maintained ones, at least) appear to have done the sensible in the first place and built toml-test decoders 😄

In any case, toml++ has now joined the club and provided both an encoder and decoder for toml-test. Let me know how you get on with it.

[moorereason]

Sweet! I'm trying to get through my existing corpus, and I'm getting a UTF-8 error on a case.

Given an input of 0="\u0800", the tt_decoder returns:

terminate called after throwing an instance of 'nlohmann::detail::type_error'
  what():  [json.exception.type_error.316] invalid UTF-8 byte at index 1: 0x80

Not sure what to do with that.

[marzer]

O_o

Are you using any custom tests that aren't currently a part of toml-test? I built it from master today and didn't experience that.

If so, can you share the toml/json files?

[moorereason]

Are you using any custom tests that aren't currently a part of toml-test?

Yes, but I'm doing differential fuzzing against other toml-test decoder implementations via STDIN and comparing the results.

The TOML document in question is 10 characters long:

0="\u0800"

I've tested that input with 6 other decoders (in Go, C99, C++, Fortran, and Dart), and they all return valid and equal results.

[marzer]

Oh, right, fuzzing. Ok well I'll do some digging and see what's going on.

[marzer]

Ah, I see, it's the JSON library I'm using. Doesn't like the JSON generated from that snippet. Guess I need to RTFM to figure out why 😅

[marzer]

@moorereason That's fixed now in the v3 branch.

[marzer]

Oh, since you're adding new upstream tests to toml-test, you might like to know that the unicode-related crash in the JSON lib was actually caused by my lib converting the unicode scalar escape sequence \u0800 into the wrong utf-8 byte sequence.

I had some unit tests in place for those conversions but they weren't broad enough. I've added a bunch more now:

tomlplusplus/tests/user_feedback.cpp

Lines 196 to 211 in b41e12f

	SECTION("github/issues/125") // https://github.com/marzer/tomlplusplus/issues/125
	{
	parse_expected_value(FILE_LINE_ARGS, R"("\u0800")"sv, "\xE0\xA0\x80"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\u7840")"sv, "\xE7\xA1\x80"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\uAA23")"sv, "\xEA\xA8\xA3"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\uA928")"sv, "\xEA\xA4\xA8"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\u9CBF")"sv, "\xE9\xB2\xBF"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\u2247")"sv, "\xE2\x89\x87"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\u13D9")"sv, "\xE1\x8F\x99"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\u69FC")"sv, "\xE6\xA7\xBC"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\u8DE5")"sv, "\xE8\xB7\xA5"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\u699C")"sv, "\xE6\xA6\x9C"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\u8CD4")"sv, "\xE8\xB3\x94"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\u4ED4")"sv, "\xE4\xBB\x94"sv);
	parse_expected_value(FILE_LINE_ARGS, R"("\u2597")"sv, "\xE2\x96\x97"sv);
	}

- might be worth adding a similar set to toml-test (I don't know if it would fit with the test format though)