| YEP: | 100 |
|---|---|
| Title: | yaq-RPC |
| Authors: | Kyle Sunden |
| Status: | rejected |
| Tags: | standard |
| Post-History: | 2020-04-13, 2020-04-21, 2020-07-02 |
This YEP is now superseded by YEP-107. The yaq ecosystem no longer uses this msgpack based custom RPC and uses Apache Avro
This YEP describes an updated remote procedure call format. In brief, the format is largely based on JSON-RPC v2, however using msgpack for data serialization instead of JSON. As such, all valid msgpack data types are available to use. This is strictly a superset of what is allowed with JSON alone.
When the project started, we decided that a daemon architecture built on top of an RPC was greatly beneficial. At the time, we chose JSON as the serialization because of its ubiquity and human readability. The first implementation was an RPC interface of our own design. We then found JSON-RPC, which was largely very similar to what we had come up with, but importantly had a standard written and a community of people using it already.
As time progressed, we found the JSON to be prohibitively verbose for sending large numeric arrays, and thus looked to msgpack as an alternative that is more compact and can handle binary data. This format is less human readable, but is more easily machine readable, and easily converted into human readable format.
This YEP addresses the standardization of this format.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.
All member names exchanged between the Client and the Server that are considered for matching of any kind should be considered to be case-sensitive. The terms function, method, and procedure can be assumed to be interchangeable.
The Client is defined as the origin of Request objects and the handler of Response objects. The Server is defined as the origin of Response objects and the handler of Request objects.
Any valid msgpack type MAY be used. For maximum compatibility with other serializers (e.g. TOML used for other parts of the yaq ecosystem), Map keys MUST be String types.
In addition to the standard types, the Timestamp extension is explicitly required by yaq-RPC. Any extension types specified will be separate YEPs.
Valid types include (with various sizes/binary encodings): - Integer - Nil - Boolean - Float - Raw UTF-8 String - Raw Binary Data - Array of valid type objects (heterogeneous) - Map of key-value pairs (keys are UTF-8 Strings) - Timestamp - YEP-110 defined N-Dimensional array
A rpc call is represented by sending a Request message to a Server. The Request message has the following members:
A Notification is a Request message without an "id" member. A Request message that is a Notification signifies the Client's lack of interest in the corresponding Response message, and as such no Response message needs to be returned to the client. The Server MUST NOT reply to a Notification.
Notifications are not confirmable by definition, since they do not have a Response message to be returned. As such, the Client would not be aware of any errors (like e.g. "Invalid params","Internal error").
If present, parameters for the rpc call MUST be provided as a Structured value. Either by-position through an Array or by-name through a Map.
When a rpc call is made, the Server MUST reply with a Response, except for in the case of Notifications. The Response is expressed as a single msgpack Map, with the following members:
Either the result member or error member MUST be included, but both members MUST NOT be included.
When a rpc call encounters an error, the Response Message MUST contain the error member with a value that is a Map with the following members:
The error codes from and including -32768 to -32000 are reserved for pre-defined errors. Any code within this range, but not defined explicitly below is reserved for future use. The error codes are nearly the same as those suggested for XML-RPC at the following url (and exactly those defined by JSON-RPC): http://xmlrpc-epi.sourceforge.net/specs/rfc.fault_codes.php
| code | message | meaning |
|---|---|---|
| -32700 | Parse error | Invalid Message was received by the server. An error occurred on the server while parsing the Message. |
| -32600 | Invalid Request | The Message sent is not a valid Request message. |
| -32601 | Method not found | The method does not exist / is not available. |
| -32602 | Invalid params | Invalid method parameter(s). |
| -32603 | Internal error | Internal yaq-RPC error. |
| -32000 to -32099 | Server error | Reserved for implementation-defined server-errors. |
The remainder of the space is available for application defined errors.
Implementations in python for the daemon and client portions are provided.
See JSON-RPC v2. This was the existing implementation.
JSON-RPC is nicely quite simple. It is also relatively ubiquitous, and based on an even more ubiquitous serialization format. msgpack was preferred for its brevity and increased type availability. While not as ubiquitous, implementations exist for all of the most popular languages.
While the structure of the requests and responses was quite nice, JSON as a serialization language is relatively limited and much more verbose compared to msgpack. The lack of binary data types in particular leads to explosion of extra communication bandwidth needed, without actually transmitting additional information. This becomes especially relevant as we look to support daemons sending large arrays of (usually floating point) numbers, where JSON requires sometimes as much as 2.5 times the number of bytes required. Attempts to shoehorn in binary data are not satisfactory solutions, as they either end up using many escape sequences.
msgpack-rpc is an existing standard for doing RPC using msgpack.
In many respects, it is a very good specification. Where it falls short for our use case is that we have come to rely on the ability to use keyword based arguments for the RPC. This is not supported by msgpack-rpc.
As such, strict compliance to the standard does not suit our usage.
Additionally there is not a particularly large (or at least vocal) community using the standard.
zerorpc similarly does not provide for keyword arguments (though with a note for a suggested workaround). The zerorpc specification is listed as "incomplete", and has not seen updates in 5 years.
zerorpc is also rejected because it specifies more that we wish to specify here: - zerorpc must use ZeroMQ sockets - zerorpc implements heartbeat, streaming, and other more complicated communication modes
This was deemed a configurational and practical nightmare to organize and assure data integrity.
See a rejected implementation.
This was an idea for reducing some of the size burden incumbent on JSON for transmitting large arrays.
Using base64 introduces additional complexity, and requires copying data into new memory locations, something that at least in theory can be avoided with msgpack in many cases.
Much of the specification text itself is mirrored from that in the JSON-RPC v2 specification. As such, their copyright notice applies to sections taken verbatim, or nearly so:
Copyright (C) 2007-2010 by the JSON-RPC Working Group
This document and translations of it may be used to implement JSON-RPC, it may
be copied and furnished to others, and derivative works that comment on or
otherwise explain it or assist in its implementation may be prepared, copied,
published and distributed, in whole or in part, without restriction of any kind,
provided that the above copyright notice and this paragraph are included on all
such copies and derivative works. However, this document itself may not
bemodified in any way.
The limited permissions granted above are perpetual and will not be revoked.
This document and the information contained herein is provided "AS IS" and ALL
WARRANTIES, EXPRESS OR IMPLIED are DISCLAIMED, INCLUDING BUT NOT LIMITED TO ANY
WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR
ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
The remainder of the content is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
built 2023-10-10 23:52:42 CC0: no copyright