tarpc client in other language than Rust

[adam0z]

Hello,
I have done a small research looking for a Rust RPC library and yours looks the most mature and pro.
I successfully run a Rust server and client. Could you tell me how I can call a method in other language like Python or JS? Is there any client lib example?

May there be some description how the protocol looks like?
Regards

[tikue]

Hey, thanks for your interest! So the main blocker to implementations in other languages is that the rust implementation relies on a Rust-specific serialization format. That's something I'd like to change but haven't had time to do so. With prost! -- the protobuf library for rust -- now available I might give that a shot. Though that means moving away from serde -- which may end up being the right decision.

In terms of protocol, it's not explicitly documented yet and relies on implementation details of bincode. Basically just id{8}length{8}blob{length}, and the blob is a rust enum representing the RPC variant and its args. It's subject to change in the future though.

[adam0z]

Hello, thanks for your reply. Right, so you are going to make possible to change serialization format that other languages could connect to the server. So why would you like to move away from serde?

May be able to elastically change also a transport layer? Now it's TCP, but what if somebody would like to use ZeroMQ? Are you going to support it in the future?

I need some extended framework to communicate my search engine in Rust with a web server and a browser, so looking forward to your elastic solution, I would use paritytech/jsonrpc. Do you now any similar solutions?

[tikue]

I wouldn't say I want to move away from Serde, but I do like the idea of using protocol buffers, and if the best protobuf library for rust isn't Serde-compatible then maybe it's worth considering.

Regarding the transport layer, there's some previous discussion in #38. I like the idea of the transport being generic but I'd want to do it in a way that doesn't cause ergonomics to suffer.

[Boscop]

Would it be possible to not hardcode the specific on-wire format but let users decide which serialize and deserialize function they want to use? E.g. if someone wants to compress the data more.

[tikue]

@Boscop that's covered by issue #83. I'd recommend discussing it there. I agree it's useful, but coming up with a good design hasn't been a priority.

[ysimonson]

It seems to me that bincode has a really simple specification - one that would be easily amenable to porting to other languages. The problem then would be that, while calling code might be able to parse the bincode-encoded content, it would not be able to deserialize it, because it doesn't have the struct/enum/etc. specifications handy.

This is where you would typically instead lean on either something like JSON (or one of its binary analogues) that also encodes things like field names, or protobufs. But the former has substantial performance overhead, and the latter requires a lot of scaffolding work that tarpc avoids by design.

What if, instead of making the encoding swappable, tarpc had some means of introspection - including methods and their type signatures? Other languages could then introspect on a service to autogenerate client stubs. That way you get to keep the big ergonomic and performance benefits of tarpc, but still facilitate the ability to call tarpc services from other languages. The downside of this of course is that it's a lot of work, but maybe it could be streamlined with some FFI-friendly library.

[tikue]

@ysimonson that's a pretty interesting idea. One problem, though, is that you'd still need an implementation of the service inspector for each language you're interested in! And after you have the service definition, the process of generating the stubs still requires a lot of language-specific decisions to be made.

I have no opposition to any of this. I think if I were to attempt to support other languages, I'd start by more formally defining the tarpc language specification. Maybe in lalrpop? Then, I'd probably rewrite tarpc into a procedural macro -- which would be backwards compatible, but I'd be able to reuse the lalrpop-generated parser in the procedural macro as well as any compilers for other languages.

[ysimonson]

If the introspective methods were provided via the same RPC mechanisms, maybe it'd be possible to implement in another language by hard-coding the bincode deserialization for those specific methods? Admittedly that sounds pretty hack-ey. It is kinda-sorta what we did for 0rpc though it was much easier there because the languages involved are dynamically typed and so necessarily can't use something as efficient as bincode.

Wouldn't defining the language specification like that bring it much closer to gRPC/protobuf? I guess what interests me about tarpc is that it avoids much of the ceremony associated with autogenerated code. I've been seeing if it's possible to shoehorn something like this into the existing tarpc design, but don't have a solid enough grasp on the codebase (yet?)

[tikue]

As long as tarpc is still invoked by a macro, I don't think the user will care if it's a macro defined via macro_rules! or a proper procedural macro. Indeed, tarpc already uses procedural macros for a few things, where some macro_rules! hygiene limitations prevented me from introducing new idents.

[tikue]

Also, writing it in a procedural macro would facilitate much better error messages!

[ysimonson]

Ah okay, I didn't realize lalrpop could be used with procedural macros. I have some reading to do :)

[tikue]

With the latest release, tarpc is now transport-agnostic, so in a sense it's really become a rust-only framework, even moreso than it was before. The good news is that it should be easier than ever to use it with cross-language protocols, because the framework is now completely agnostic to the transport layer.

[yw662]

Anyway, is there any documentation of the protocol itself ?

[tikue]

Which protocol are you interested in? Tarpc services can plug in any unary request type protocol at the transport layer. By default it's only shipping with a length delimited, serde bincode transport.

Are you trying to write a non-Rust client for a tarpc service you've written or that someone else has written? If someone else wrote it, you can simply ask them for the protocol they've used. If you're writing it yourself, you'll want to start by choosing a protocol to use. Chances are, you'll have to write the transport yourself, unless you're happy using a bincode based protocol.