serde: support for deserializing named tracing types

[hawkw]

Feature Request

Crates

tracing-serde

Motivation

Currently, tracing-serde supports serializing tracing types, such as Metadata, Event, span::Attributes and span::Record, etc. However, it does not support deserializing those types from a serialized representation. It would be good to support this.

Proposal

We should add structs to tracing-serde that represent the deserialized form of tracing types that can currently be serialized. These would have public accessors returning the fields of those types, and implement the serde::Deserialize trait.

Alternatives

From the outside, it might seem like we should just implement serde::Deserialize for tracing-core types like Metadata, Event, et cetera, rather than adding new structs that those types can be deserialized as. However, there are some serious issues with this. tracing structs like Event and Metadata consist primarily of references to other data for performance and space optimization reasons. Similarly to how we don't serialize the internal representations of types like ValueSet and instead use SerializeMap etc, we wouldn't want to deserialize internal representations that don't make sense outside of the address space of the process where they originated.

Another option is to not implement Deserialize for any structs, and just change all Serialize implementations to serialize data structures like maps and sets. This would mean removing all uses of SerializeStruct and replacing it with unstructured data. However, this makes using deserialized tracing data more difficult for users, as rather than having a nice structured API for well-known fields, everything is represented as unstructured maps that may or may not contain particular values or types.

[kellpossible]

I'm looking at a way to send tracing data between Rust processes via a serde bridge, this would be very useful. Will probably end up implementing a stripped down proof of concept in the meantime.

[kellpossible]

Okay so I got most of the way through the implementation of this feature but was blocked on trying to create a value set with tracing_core::field::value_set(), being very difficult, perhaps easier when const generics arrives.

[Ekleog]

@kellpossible Did you get to anything viable for the serde bridge? Do you have a PR up somewhere with a base code I could maybe try to complete? I'm looking for basically the same thing, except I'm doing that between a Rust program and the Rust wasm module it executes using wasmtime.

[kellpossible]

@Ekleog it's not perfect due to lifetime issues I ran into, but here's what I've got so far https://github.com/AleoHQ/tracing-bridge
I'm also using it between a rust executable and a rust wasm module running in a browser

[Ekleog]

Hmm… I feel like I'm missing something, but… I can't see how this code makes it possible to reconstruct the various tracing structures on the deserialization side to be able to pass them to the outer instance? OTOH, reading the tracing API makes it look like it's maybe just not possible with the methods that are currently made available, so… maybe that's just a use case that tracing currently does not support, and I'd be better off using slog or log in the wasm blob, as there are APIs there that can reasonably be plugged over a serialization domain?

Anyway, I've filed #1170 to ask for the missing API in tracing, let's see how that evolves!

Thank you anyway for your feedback :)

[kellpossible]

@Ekleog yes it's not perfect, it seems like the API methods available don't allow this to be done properly with tracing. I just tried to hash together something as well as I could at the time.

[jamesmunns]

So, I'm looking into this for getting tracing data off of a no_std platflorm, using some kind of serial stream.

I'd prefer not to use JSON (it's not particularly compact, higher serialization cost than something like postcard), and ran into the "no deserialization" limitation mentioned here.

I had a brief chat with @hawkw, and I think I am going to try and do roughly the following:

• Fork the tracing-serde crate (temporarily)
• Change the internal representation of the Serialize* types to something that contains the structured relevant data that can be serialized (more or less - the fields that get put into JSON already today), in a low-cost way (borrowing the original data), rather than just a borrow of the tracing types
• Implement Serialize in a similar "cheap" method as is used today
• ALSO implement Deserialize of the structured data, likely borrowing from the deserialization buffer
• ALSO implement a to_owned() method (only with std/alloc feature enabled), that makes String/Vec, etc versions of the contained data.

This approach won't be suitable for fully "round tripping" tracing types like Event, to allow you to then report it to some other Subscriber somewhere else, but will be suitable for transmitting a serialized version of that data somewhere else, without a loss of data.

A bit more graphically, today you can do (only) this:

Event
-> AsSerde
-> Serialize
-> "JSON Data that is more or less an "Event Report"
-> Send this to some reporting system, like Honeycomb, etc.

What I'm proposing would allow you to do this:

Event
-> AsSerde (converts it to a structured "EventReport" type, currently named "SerializeEvent")
-> Serialize (to a binary format, like postcard)
-> Transmit over some wire...
-> Deserialize (from a binary format, back into "SerializeEvent")
-> Serialize (to JSON)
-> Send this to some reporting system, like Honeycomb, etc.

What this would NOT allow you to do:

Event
-> AsSerde (converts it to a structured "EventReport" type, currently named "SerializeEvent")
-> Serialize (to a binary format, like postcard)
-> Transmit over some wire...
-> Deserialize (from a binary format, back into "SerializeEvent")
-> Feed this into another subscriber on another PC (Not possible! We have a `SerializeEvent`, not an `Event`!)

Edit: One thing I forgot to mention is that even with these changes, if today you only use SerializeEvent to spit out JSON, the goal is to have the output format be the same (if you use serde-json or some other self-describing format). I'm hoping everything I described above could be done in a back-compat way, so this could be merged as part of tracing-serde. If this ends up not being possible, I'll make it a separate crate, and tracing folks can decide to switch to this if/when it makes sense at a breaking change point in the future.

[jamesmunns]

I'm now working on this on the serde-structs branch of my fork of tracing.

I'm now pretty convinced this is possible, though there's a couple things I'm not sure about how to do in no_std land without alloc or heapless (in particular, collecting &[&str] on the serialize/deserialize end).

I'm also hunting down some serializations problems, as well, but progress is good.

https://github.com/jamesmunns/tracing/blob/serde-structs/tracing-serde-structs/src/lib.rs