Custom (de)serializer support?

[cometkim]

I love that this library provides good platform compatibility and the ability to extend its transport. But the wire format is fixed to JSON.

I wonder if it can provide the ability to customize (de)serializer, or even the entire (De)evaluator implementation as well, for example:

declare interface Serializer;

newWebSocketRpcSession("wss://example.com/api", {
  serializer: myCustomSerializer,
});

To allow compacting or compressing wire messages?

[smartboyathome]

This would especially be great for those of us who need to handle binary data as part of our events. JSON can't handle this efficiently, as it requires base64 which increases the size of data by 33%, but MesaagePack can handle binary data without encoding.

[kentonv]

Yeah... I think it would make sense to adjust the RpcTransport interface so that you can capture the messages before the last JSON stringification step. This would also be useful for the MessagePort transport, which doesn't actually need the values to be stringified (since MessagePort can transmit arbitrary stuctured clonable objects).

That said, I think there's another way we might solve the base64 problem. WebSocket supports both textual and binary message types. What I'd like to do is, when we find a sufficiently large Uint8Array in a message, pull it out and send it as a separate binary message. Then send the JSON representing the rest of the structure separately, with some sort of an embedded reference to the binary.

[smartboyathome]

That seems overly complex, as now you are introducing more state into the server library in order to track binary data and associate it with JSON. Allowing the use of a custom serializer makes this trivial, and the hard work is done for us. I really don't like the idea of defining yet another format just for this use case.

[EqualMa]

For now you can implement a custom RpcTransport whose send method accepts a json string, parses it and then sends whatever you want. Its receive method should receive data of that format and return the serialized string.

[cometkim]

You can keep the JSON serializer as the default behavior to keep usage simple.

[qwtel]

Are you accepting PRs for this? I have a version of this working qwtel/capnweb and could clean it up.

The design is roughly: High level functions accept an optional Codec interface that implements encoding and decoding. The typeForRpc function has moved into the Codec interface so that Uint8Array, etc. can be declared as "primitive" to avoid tagging. Includes implementations for object encoding (to avoid stringification for postMessage) and node:v8 serialization.

[kettanaito]

This would be very much appreciated!

I'm not sure what folks meant by implementing this in your transport, given the serialization happens in RpcSessionImpl, and you are always getting JSON serialized messages in your transport. If the serialization fails, the transport receives the error message straight away. That leads me to believe the transport is a bad place to introduce custom serialization/deserialization.

See:

capnweb/src/rpc.ts

Line 516 in 7c4207a

msgText = JSON.stringify(msg);

Bad news is that implementing a custom session won't work either as the RpcSessionImp is hidden behind a private #session field that you cannot access:

capnweb/src/rpc.ts

Line 767 in 7c4207a

this.#session = new RpcSessionImpl(transport, mainHook, options);

I'd argue that serialization sounds like a concept that a transport should handle, but I'm quite new to RPC overall so I might be wrong here. My assumption is that you use custom transports to do things like RPC via MessageChannel or something, and that implies you might need to adjust the serialization logic to suit your transport means, too.

@kentonv, what do you think if the RpcTransport interface had an optional serialize/deserialize methods that custom transports could implement? The idea is:

1. RpcSessionImpl.send taps into transport.serialize() if defined, otherwise JSON.stringify() (current behavior).
2. RpcSessionImpl.readLoop taps into transport.deserialize(), if defined, otherwise JSON.parse() (current behavior).

[burdiyan]

@kettanaito the RpcTransport would need to change, right? Its send and receive methods would need to work with unknown or something like this, instead of string in order to support different serialization formats.

[burdiyan]

Hm... I see @qwtel's approach of defining a WireMessage type which is a union of different types supported on the wire. It should work.

https://github.com/qwtel/capnweb/blob/dfa317322d930797164e1fa62990c1b6e7c2213d/src/codec.ts#L7

[bdmackie]

Much higher level but if you stick with JSON and want to support encoding of particular types without significant tailoring, Svelte Kit's approach is pretty simple:

import type { Transport } from '@sveltejs/kit';

export const transport: Transport = {
	BigD4: {
		encode: (value) => {
			if (!(value instanceof BigD4)) return false;
			return [value.toString()];
		},
		decode: ([value]) => new BigD4(value),
	},
};

[burdiyan]

@qwtel will you submit a PR for this?

[qwtel]

If there's interest from a maintainer and approval of the high level design I could prepare one.

The design is roughly: High level functions accept an optional Codec interface that implements encoding and decoding. The typeForRpc function has moved into the Codec interface so that Uint8Array, etc. can be declared as "primitive" to avoid tagging. Includes implementations for object encoding (to avoid stringification for postMessage) and node:v8 serialization.

fwiw, my fork has since grown more experimental and has a README that explains what's going on: qwtel/capnweb-x.

[terion-name]

I was really surprised that capnweb doesn't support binary serialization out of the box. Using capnproto looks like natural thing here...
Definitely needed IMO