Custom type registration : instrusive API

[pfeatherstone]

Currently, the API for registering custom types is as follows:

using nlohmann::json;

namespace ns {
    void to_json(json& j, const person& p) {
        j = json{{"name", p.name}, {"address", p.address}, {"age", p.age}};
    }

    void from_json(const json& j, person& p) {
        j.at("name").get_to(p.name);
        j.at("address").get_to(p.address);
        j.at("age").get_to(p.age);
    }

It would be great if there was a MACRO-style registration a bit like what msgpack-c uses:

struct person 
{
   std::string name;
   std::string address;
   int age;
   MSGPACK_DEFINE_MAP(name, address, age);
};

or yas:

struct   person
{
   std::string name;
   std::string address;
   int age;
    YAS_DEFINE_STRUCT_SERIALIZE("person", name, address, age);
};

or

struct   person
{
   std::string name;
   std::string address;
   int age;
};
YAS_DEFINE_INTRUSIVE_SERIALIZE("person", name, address, age);

[pfeatherstone]

By the way, if you look at those implementations, they are doing some crazy macro expansions which i'm not brave enough to undertake. But the end result is great, as you can see above. Super clean and easy. It's almost providing compile time reflection in C++. I know this repo isn't a free code service, but if one is looking for suggestions, here is one.

[nlohmann]

I understand the idea. Do I assume correctly that you would assume a "canonic" serialization to

{
    "name": "a name value",
    "address": "an address value",
    "age": 42
}

[pfeatherstone]

I'm not sure what you mean by "canonic" serialization, but in case of JSON serialisation, then yes i would assume a 'person' object would serialise to your example above. What would be great is if there were casting functions like:
nlohmann::json::cast_to : nlohmann::json -> person
nlohmann::json::cast_from: person -> nlohmann::json
I don't know if that's possible. You have the first one already as mentioned here https://github.com/nlohmann/json#arbitrary-types-conversions though the object type has to be in the ns namespace. Is that strictly necessary?

[nlohmann]

With "canonic" I meant that

1. We always serialize to a JSON object.
2. The names of the variables are the names of the keys in that object.

[nlohmann]

If I understand the C++ macro magic correctly, the library needs to make an assumption on the upper bound of member variables.

Assuming three members, a possible implementation would be:

#include <iostream>
#include "json.hpp"

using json = nlohmann::json;

struct person
{
   std::string name;
   std::string address;
   int age;
};

#define YAS_DEFINE_INTRUSIVE_SERIALIZE(type, var1, var2, var3) \
   void to_json(nlohmann::json& j, const type& t)\
   {\
       j[#var1] = t.var1;\
       j[#var2] = t.var2;\
       j[#var3] = t.var3;\
   }\
   void from_json(const nlohmann::json& j, type& t)\
   {\
       j.at(#var1).get_to(t.var1);\
       j.at(#var2).get_to(t.var2);\
       j.at(#var3).get_to(t.var3);\
   }

YAS_DEFINE_INTRUSIVE_SERIALIZE(person, name, address, age)

int main() {
    person p;
    p.name = "Ned Flanders";
    p.age = 65;
    p.address = "Springfield";
    
    std::cout << json(p) << std::endl;

    json j;
    j["name"] = "Homer Simpson";
    j["address"] = "Springfield";
    j["age"] = 42;

    person s = j;

    std::cout << json(s) << std::endl;
}

Output:

{"address":"Springfield","age":65,"name":"Ned Flanders"}
{"address":"Springfield","age":42,"name":"Homer Simpson"}

Is this what you have in mind?

[pfeatherstone]

Yes exactly. I hadn't realised the macro expansion had an upper bound on the number of arguments. I thought it used VA_ARGS and worked automagically :)
I guess it would be nice if it worked with nested custom types (or whatever the correct term is). For example:

struct child
{
   std::string name;
   int age;
};
NLOHMANN_SERIALISE(child, string, age);

struct parent
{
   std::vector<child> children;
};
NLOHMANN_SERIALISE(parent, children);

If that makes sense.
And having conversion functions to_nlohmann and from_nlohmann (instead of to_json and from_json) means it still supports the other binary serialisation protocols (bson, msgpack, etc)
This would be an insanely powerful feature.

[nlohmann]

• I have to look into VA_ARGS, but what I read seems that it only works in printf-like calls. Anything else assumes an upper bound of arguments, but at least the user does not need to be aware of what's going on behind the curtains.
• Your example with nested types works without adjustments.
• What's the point of changing the names of the functions? You can still generate MessagePack from a JSON value by calling json::to_msgpack.

[pfeatherstone]

I have no real opinions on function names. Whatever works and Whatever you think is best. I was just clarifying that conversion functions were to and from the json object, not a json serialised string. What do you think overall ? Do you like the macro expansion API for registering types ?

[FrancoisChabot]

One of the big drawback here is that since json, as a format, is meant to be human-readable, that syntax only looks nice if you have a 1-to-1 match between variable names and the serialized tag, which is inherently precluded by many organizations' coding standards. If such a feature went in, I would expect an immediate followup request to be able to remap names.

At that point, it's worth pointing out that there is a middle ground possible between the current approach and the full macro-based approach:

Something along these lines (with better names):

struct person 
{
   std::string name;
   std::string address;
   int age_;
   
   void as_json(nlohmann::Visitor& v) { 
      v("name", name); 
      v("address", address); 
      v("age", age_); 
   }
};

I think this strikes a good balance between not having to essentially write the same code twice, while remaining debugable and flexible.

[pfeatherstone]

@FrancoisChabot I like your suggestion. It's a good middle ground. I would maybe go a tiny bit further and use something like https://github.com/Neargye/nameof so you don't have to specify the variable name (though please don't use something that depends on C++17 and beyond)

[pfeatherstone]

I still think the macro api is awesome. But looking at how yas and msgpack-c implemented it, it doesn't look debug-friendly that's for sure.

[KonanM]

namespace detail{
    constexpr size_t nArgs(std::string_view str) {
        //constexpr count_if is C++20
        size_t nargs = 1;
        for (auto c : str) if (c == ',') ++nargs;
        return nargs;
    }
    template<typename T, size_t... Is>
    void to_json_impl(nlohmann::json::json& j, const T& t, std::index_sequence<Is...>) {
        ((j[std::get<Is>(t._memberNames)] = std::get<Is>(t.members())), ...);
    }
    template<typename T, size_t... Is>
    void from_json_impl(const nlohmann::json::json& j, T& t, std::index_sequence<Is...>) {
        (j.at(std::get<Is>(t._memberNames)).get_to(std::get<Is>(t.members())), ...);
    }
}

#define NLOHMANN_DEFINE_STRUCT_SERIALIZE(Type, ...) \
constexpr inline decltype(auto) members() const { return std::tie(__VA_ARGS__); } \
constexpr inline decltype(auto) members() { return std::tie(__VA_ARGS__); }  \
static constexpr std::string_view _typeName = #Type;\
static constexpr std::array<std::string_view, detail::nArgs(#__VA_ARGS__)> _memberNames = \
[](){ std::array<std::string_view, detail::nArgs(#__VA_ARGS__)> out{}; \
size_t off = 0, next = 0; std::string_view ini(#__VA_ARGS__); \
for(auto& strV : out){ next = ini.find_first_of(',', off); strV = ini.substr(off, next - off); off = next + 1;} return out;}();\
friend void to_json(nlohmann::json::json& j, const Type& t) { detail::to_json_impl(j, t, std::make_index_sequence<_memberNames.size()>{}); } \
friend void from_json(const nlohmann::json::json& j, const Type& t) { detail::from_json_impl(j, t, std::make_index_sequence<_memberNames.size()>{}); } 

struct person{
    std::string name;
    std::string address;
    int age;
    NLOHMANN_DEFINE_STRUCT_SERIALIZE(person, name, address, age)
};

If you don't like macro heavy implementations, something along this lines might be an option (this is implemented using C++17, but should be easy to port to C++11). If something like this is viable then I could also try to rewrite it for C++11.

[pfeatherstone]

Yeah that looks great but very much a C++17 implementation. std::string_view is available in the nonstd libraries, std::tie isn't constexpr in C++11 and std::index_sequence is C++14. Do you think it's possible as a C++11 implementation?

[pfeatherstone]

As a user i don't care about the implementation, that's hidden away from me. So long as it works i'm happy. Personally though, i am restricted to C++11 due to what i develop for. But that's just me, maybe this API could be enabled in C++17 only. Tough if you can't use it.

[nlohmann]

@KonanM Thanks for the code! It would be really nice to have a C++11 equivalent for this.

[pfeatherstone]

@KonanM I see you've used the same api in your tser library. Great job.

[KonanM]

@pfeatherstone yeah I wrote that basically for my own library and I'm happy to share that code. @nlohmann I converted the code to C++11 and hope I have not overseen anything:
I replaced std::string_view with const char* by using a small little trick where I initialize an extra array with the whole #VA_ARGS characters and replace the comma ',' by '\0'.
Using the example above it boils down to:
person::_memberNameData = {{'n', 'a', 'm', 'e', '\0', 'a', 'd', 'd', 'r', 'e', 's', 's', '\0', 'a', 'g', 'e', '\0'}}
person::_memberNames = {&_memberNameData[0], &_memberNameData[5], &_memberNameData[13] }
I had to drop the constexpr for the 'members()' due to the missing constexpr std::tie support for C++11, but I don't think it's actually necessary.
Then I did the little trick with std::initializer_list instead of folding over the comma operator and lastly you already have a custom implementation for 'index_sequence' in the cpp_future.hpp header.
Feel free to take that snippet and modify however you want and let me know if you need further help.

namespace nlohmann {
    namespace detail {
        //#include <nlohmann/detail/meta/cpp_future.hpp> for index_sequence and make_index_sequence
        template<size_t...I >
        using index_sequence = std::index_sequence<I...>;
        template <size_t N>
        using make_index_sequence = std::make_index_sequence<N>;

        constexpr size_t n_args(const char* str) {
            size_t nargs = 1;
            for (char const* c = str; *c; ++c) if (*c == ',') ++nargs;
            return nargs;
        }
        constexpr size_t str_size(const char* str) {
            size_t strSize = 1;
            for (char const* c = str; *c; ++c, ++strSize);
            return strSize;
        }
        template<typename T, size_t... Is>
        void to_json_impl(nlohmann::json::json& j, const T& t, nlohmann::detail::index_sequence<Is...>) {
            (void)std::initializer_list<int>{(j[std::get<Is>(t._memberNames)] = std::get<Is>(t.members()), 0)...};
        }
        template<typename T, size_t... Is>
        void from_json_impl(const nlohmann::json::json& j, T& t, nlohmann::detail::index_sequence<Is...>) {
            (void)std::initializer_list<int>{(j.at(std::get<Is>(t._memberNames)).get_to(std::get<Is>(t.members())), 0)...};
        }
    }
}

#define NLOHMANN_DEFINE_STRUCT_SERIALIZE(Type, ...)                                                                     \
inline decltype(auto) members() const { return std::tie(__VA_ARGS__); }                                                 \
inline decltype(auto) members() { return std::tie(__VA_ARGS__); }                                                       \
static constexpr std::array<char, nlohmann::detail::str_size(#__VA_ARGS__)> _memberNameData = [](){                     \
std::array<char, nlohmann::detail::str_size(#__VA_ARGS__)> chars{}; size_t idx = 0; constexpr auto* ini(#__VA_ARGS__);  \
for (char const* c = ini; *c; ++c, ++idx) if(*c != ',') chars[idx] = *c;  return chars;}();                             \
static constexpr std::array<const char*, nlohmann::detail::n_args(#__VA_ARGS__)> _memberNames =                         \
[](){ std::array<const char*, nlohmann::detail::n_args(#__VA_ARGS__)> out{ {Type::_memberNameData.data()} };            \
for(size_t i = 0, nArgs = 0; i < Type::_memberNameData.size() - 1; ++i)                                                 \
if (Type::_memberNameData[i] == '\0'){++nArgs; out[nArgs] = &Type::_memberNameData[i] + 1;} return out;}();             \
friend void to_json(nlohmann::json::json& j, const Type& t) {                                                           \
nlohmann::detail::to_json_impl(j, t, nlohmann::detail::make_index_sequence<_memberNames.size()>{}); }                   \
friend void from_json(const nlohmann::json::json& j, const Type& t) {                                                   \
nlohmann::detail::from_json_impl(j, t, nlohmann::detail::make_index_sequence<_memberNames.size()>{}); }

[pfeatherstone]

decltype(auto) is C++14 isn't it?

[pfeatherstone]

my compiler complains that the lambda for _memberNameData is not constexpr

[pfeatherstone]

by the way, i'm using g++ 8.4.0 with CXXFLAGS -std=c++11

[nlohmann]

I tried to make the code work with C++11 and the current version of the library. I still struggle to find a constexpr version of the n_args and str_size function. Here is what I got so far:

namespace nlohmann {
    namespace detail {
        constexpr size_t n_args(const char* str) {
            size_t nargs = 1;
            for (char const* c = str; *c; ++c) if (*c == ',') ++nargs;
            return nargs;
        }
        constexpr size_t str_size(const char* str) {
            size_t strSize = 1;
            for (char const* c = str; *c; ++c, ++strSize);
            return strSize;
        }
        template<typename T, size_t... Is>
        void to_json_impl(nlohmann::json& j, const T& t, nlohmann::detail::index_sequence<Is...>) {
            (void)std::initializer_list<int>{(j[std::get<Is>(t._memberNames)] = std::get<Is>(t.members()), 0)...};
        }
        template<typename T, size_t... Is>
        void from_json_impl(const nlohmann::json& j, T& t, nlohmann::detail::index_sequence<Is...>) {
            (void)std::initializer_list<int>{(j.at(std::get<Is>(t._memberNames)).get_to(std::get<Is>(t.members())), 0)...};
        }
    }
}

#define NLOHMANN_DEFINE_STRUCT_SERIALIZE(Type, ...)                                                                     \
inline auto members() const -> decltype(std::tie(__VA_ARGS__)) { return std::tie(__VA_ARGS__); }                        \
inline auto members() -> decltype(std::tie(__VA_ARGS__)) { return std::tie(__VA_ARGS__); }                              \
static constexpr std::array<char, nlohmann::detail::str_size(#__VA_ARGS__)> _memberNameData = [](){                     \
std::array<char, nlohmann::detail::str_size(#__VA_ARGS__)> chars{}; size_t idx = 0; constexpr auto* ini(#__VA_ARGS__);  \
for (char const* c = ini; *c; ++c, ++idx) if(*c != ',') chars[idx] = *c;  return chars;}();                             \
static constexpr std::array<const char*, nlohmann::detail::n_args(#__VA_ARGS__)> _memberNames =                         \
[](){ std::array<const char*, nlohmann::detail::n_args(#__VA_ARGS__)> out{ {Type::_memberNameData.data()} };            \
for(size_t i = 0, nArgs = 0; i < Type::_memberNameData.size() - 1; ++i)                                                 \
if (Type::_memberNameData[i] == '\0'){++nArgs; out[nArgs] = &Type::_memberNameData[i] + 1;} return out;}();             \
friend void to_json(nlohmann::json& j, const Type& t) {                                                                 \
nlohmann::detail::to_json_impl(j, t, nlohmann::detail::make_index_sequence<_memberNames.size()>{}); }                   \
friend void from_json(const nlohmann::json& j, Type& t) {                                                               \
nlohmann::detail::from_json_impl(j, t, nlohmann::detail::make_index_sequence<_memberNames.size()>{}); }

Any ideas?

[pfeatherstone]

are constexpr lambdas allowed in C++11?

[nlohmann]

are constexpr lambdas allowed in C++11?

No, that's C++17.

[pfeatherstone]

So I don’t think the above will work. Unless implicit constexpr lambdas are ok in c++11 but I don’t think so. Maybe macro is the only way

[KonanM]

We can still write a normal struct with an constexpr operator() inside the macro. I will give it another try tomorrow. Otherwise we could still drop constexpr support for C++1 and replace it by const.

[pfeatherstone]

The lambdas aren’t capturing anything so I don’t think you need the structs, just static functions.

[KonanM]

Ok I gave it another try and made it work with C++14 with msvc, gcc and clang, but I completely forgot that I can't use any control statements in constexpr functions in C++11... like you can't use a for loop at all. https://godbolt.org/z/Zdc9Mz

That doesn't make the implementation impossible, but at that point the macro based approach is probably just easier (you would have to replace the for loop by index sequence template parameters...). I will give the macro based approach a try to see how that looks like

[KonanM]

Ok here is a super simply macro based solution, which works for up to 9 parameters (can be expanded) - that should work for basically every compiler. There might be more elegant solution, but this one is dead simple and doesn't require any great macro magic.

https://godbolt.org/z/bgfpz2

[pfeatherstone]

I had a look at the msgpack-c implementation and it looks like a lot of the MACROS are inspired by boost.preprocessor. The number of arguments is limited to 256. I think that is reasonable. Could get it to work with boost.preprocessor, if it does then pinch all the macros we need.

[pfeatherstone]

Actually one more thing. What i really like about this library as opposed to msgpack-c say, is that the intermediate representation between type and serialized buffer, i.e. the nlohmann::json object is usable, and as such, would be nice if that could be a member variable of a struct and then serialisable.

Now currently, if you have:

struct person2
{
    std::string name;
    std::string address;
    int age;
    nlohmann::json j;
};
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(person, name, address, age, j)

this won't work because of the line

j.at("j").get_to(t.j)

But if this were replaced with:

t.j = j.at("j");

then it would be fine.
@nlohmann is there any particular reason to use get_to? Would it be ok to replace:

#define JSON_FROM(v1) j.at(#v1).get_to(t.v1);

with

#define JSON_FROM(v1) t.v1 =  j.at(#v1);

Are there any performance issues?

[pfeatherstone]

Really the problem with using

j.at("j").get_to(t.j)

when j is a nlohmann::json object is you get the compilation error:

error: no matching function for call to ‘nlohmann::basic_json<>::get_to(nlohmann::json&) const’

Maybe the function nlohmann::basic_json<>::get_to(nlohmann::json&) const could be trivially defined to keep the compiler quiet.

[pfeatherstone]

Hmm, actually #define JSON_FROM(v1) t.v1 = j.at(#v1); won't always work

[pfeatherstone]

Maybe the solution is dealing with this error:
error: no matching function for call to ‘nlohmann::basic_json<>::get_to(nlohmann::json&) const’

[nlohmann]

I would not like to mix this issue with adding an intermediate value. I will prepare a merge request for #2175 (comment).

[nlohmann]

@pfeatherstone @KonanM I finally had time to open a PR for this issue, see #2225. It would be great if you could provide some feedback.

[nlohmann]

Thanks a lot everybody!

[pfeatherstone]

Last thing: I want to do something like this:

struct person
{
    std::string name;
    std::string address;
    int age;
    nlohmann::json j;
};

NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(person, name, address, age, j)

Which spits out error:

error: no matching function for call to ‘nlohmann::basic_json<>::get_to(nlohmann::json&) const’

Is there a way of dealing with this compiler error?
I tried doing the trivial conversion:

namespace nlohmann {
    void to_json(json& j, const json& t) {j = t;}
    void from_json(const json& j, json& t) {t = j;}
}

which opens Pandora's box of errors. Is there a way of doing this?

[pfeatherstone]

That would be the icing on the cake

[pfeatherstone]

So instead of :

#define NLOHMANN_JSON_FROM(v1) j.at(#v1).get_to(t.v1);

I propose something like

#define NLOHMANN_JSON_FROM(v1) nlohmann_json_from(#v1, j, t.v1)

where

template<typename T>
inline void nlohmann_json_from(const char* name, const nlohmann::json& j, T& obj)
{
    j.at(name).get_to(obj);
}

template<>
inline void nlohmann_json_from(const char* name, const nlohmann::json& j, nlohmann::json& obj)
{
    obj = j[name];
}

I'm not a huge fan of this, but it avoids having to suppress the error:

error: no matching function for call to ‘nlohmann::basic_json<>::get_to(nlohmann::json&) const’

Which could require some refactoring inside the basic_json class using more type_traits.
@nlohmann @KonanM What do you guys think?
I'm really sorry, I should have brought this up when you requested some feedback. I should have remembered that i raised this a few days/weeks ago.

[nlohmann]

It seems to be sufficient to add

template<typename ValueType,
         detail::enable_if_t <
             detail::is_basic_json<ValueType>::value,
             int> = 0>
ValueType & get_to(ValueType& v) const
{
    v = *this;
    return v;
}

(the other function explicitly require not detail::is_basic_json<ValueType>::value).

Then the following code works:

#include <iostream>
#include "json.hpp"

struct person
{
    std::string name;
    std::string address;
    int age;
    nlohmann::json j;
};

NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(person, name, address, age, j)

int main()
{
    person p;
    p.name = "Homer Simpson";
    p.address = "Springfield";
    p.age = 42;
    p.j = {{"foo", "bar"}};
    
    nlohmann::json j = p;
    std::cout << j << std::endl;
}

Output:

{"address":"Springfield","age":42,"j":{"foo":"bar"},"name":"Homer Simpson"}

Any thoughts on this?

[pfeatherstone]

Yep that looks better than what I suggested. Ship it. Thanks! :)

[KonanM]

Sorry that I'm that late, but yeah the approach by @nlohmann is much better. The other approach would also fail when being used with more than one 'NLOHMANN_DEFINE_TYPE'. I was thinking about if it's possible to drop the SFINAE and only specialize 'nlohmann::json', but I guess not.

[m4ce]

#include <nlohmann/json.hpp>
#include <iostream>

namespace nl = nlohmann;

enum class Schema {
  HTTP,
  HTTPS
};

NLOHMANN_JSON_SERIALIZE_ENUM(Schema, {
    {Schema::HTTP, "http"},
    {Schema::HTTPS, "https"},
})

struct Foo {
  int a;
  NLOHMANN_DEFINE_TYPE_INTRUSIVE(Foo, a)
};

struct Request {
  Schema schema;
  Foo foo;
  std::vector<int> bar;
  NLOHMANN_DEFINE_TYPE_INTRUSIVE(Request, schema, foo, bar)
};

int main() {
  std::string s = "{\"schema\":\"https\",\"x\":{\"a\":11},\"bar\":[1,2,3,4]}";
  Request r = nl::json::parse(s);
}

the example above fails with:

libc++abi.dylib: terminating with uncaught exception of type nlohmann::detail::out_of_range: [json.exception.out_of_range.403] key 'foo' not found

is it possible to just skip fields that are not present?

[nlohmann]

Not when using the NLOHMANN_DEFINE_TYPE_INTRUSIVE. You would need to define a from_json method yourself.

[ilsme]

Hi guys, what about inheritance?

struct parent
{
    std::string surname;
};
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(parent, surname);

struct child : public parent
{
   std::string name;
};
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(child, name);

It would be great if something similar would be possible.

[pfeatherstone]

Yep, that should be handled.

[pfeatherstone]

Or maybe not. It would complicate NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE and maybe confuse users. You're better off defining your custom functions to_json and from_json in this case.

[pfeatherstone]

For reference, here i discuss an alternative implementation of NLOHMANN_DEFINE_TYPE_INTRUSIVE, that doesn't have an upper bound on the number of member variables and only uses 1 macro, the rest is variadic templates

[nlohmann]

See #2404 (reply in thread).