Implement legacy encoders, not just decoders?

[domenic]

jsdom/whatwg-url would use this, to get proper support for the encoding parameter, if it were available! Then anchorElement.search would work correctly in non-UTF-8 documents in jsdom.

[ChALkeR]

@domenic Which error handling mode does that need? html or replacement?

More generically: does anything need a mode that handles encoding errors in any other way than either throw or return a replacement char?

Ref: https://encoding.spec.whatwg.org/#encoders-and-decoders:~:text=to%20output.-,%22html%22,-Push%200x26%20(%26)%2C%200x23

[domenic]

Specifically we need https://url.spec.whatwg.org/#string-percent-encode-after-encoding, which uses https://encoding.spec.whatwg.org/#encode-or-fail, which needs "fatal". (There is no "replacement" for encoders, just for decoders.)

Interestingly, from my reading it looks like throwing is not quite the right behavior; instead it kind of bails out at the first error and returns what it has gotten so far? The note about ISO-2022-JP below #encode-or-fail discusses this a bit.

It's a pretty terrible legacy setup!

[ChALkeR]

There is no "replacement" for encoders

UTF-8 TextEncoder.encode, TextEncoder.encodeInto and TextEncoderStream replace lone surrogates with replacement codepoint while processing the input. While that's not a mode of the inner encoder instance, that's the observable behavior of the exposed high-level interface anyway

As of v1.7.0:

1. single-bytes (and x-user-defined) can encode in fatal (throwing) mode
2. utf8/utf16 can encoded in fatal (throwing) or loose (aka replacement) mode that replaces lone surrogates

There is no unified API yet, and no multi-byte encoders yet
Perhaps I'll add them later

[ChALkeR]

Also, like Unicode, this will also be a difference for gb18030 / gbk multi-byte encoding, which can represent the replacement codepoint validly:

> escape(new TextDecoder('gb18030', { fatal: true }).decode(Uint8Array.of(0x84, 0x31, 0xa4, 0x37)))
'%uFFFD'

And given that the replacement for lone surrogates is not performed by the encoder mode but by input processing, encoding lone surrogates is not expected to throw for that but will encode replacement codepoint (like with Unicode)

Will need to check what happens for other encodings and how to represent that as modes (which don't have to follow the spec naming on non-/encoding.js entry points)

That said, §6 states:

Standards are to ensure that the input I/O queues they pass to UTF-8 encode (as well as the legacy encode) are effectively I/O queues of scalar values, i.e., they contain no surrogates.

Implementing e.g. legacyHookEncode that just throws on non-wellformed strings should be ok I guess?

[domenic]

I'll try to give this a try in whatwg-url and jsdom soon!

That said, §6 states:

Standards are to ensure that the input I/O queues they pass to UTF-8 encode (as well as the legacy encode) are effectively I/O queues of scalar values, i.e., they contain no surrogates.

Implementing e.g. legacyHookEncode that just throws on non-wellformed strings should be ok I guess?

That sounds correct.

To be clear, it seems like errors come from other more idiosyncratic conditions. E.g., https://encoding.spec.whatwg.org/#gb18030-encoder step 3 just fails on U+E5E5. Tracing the algorithms back, I guess that means it encodes to [0xE5, 0xE5].

[ChALkeR]

It doesn't seem to need truncation by itself, it needs all chunks + errors with their codepoints

And the iterative way of calling that in the spec requires exposing streaming-like encoders which maintain state even after encountering an error.

The difference between using encode or fail in https://url.spec.whatwg.org/#string-percent-encode-after-encoding + step 5.4 there and just encode from https://encoding.spec.whatwg.org/#encode which uses the html mode (that does the same) seems to be when when percentEncodeSet includes digits (never) or doesn't include html escapes (0x26, 0x23, or 0x3B) - it escapes those regardless of percentEncodeSet if they are part of an error codepoint.

Inside the URL spec, non-UTF8 percent-encode after encoding is called with special-query percent-encode set, query percent-encode set, application/x-www-form-urlencoded percent-encode set

1. special-query percent-encode set and query percent-encode set escape 0x23 but not 0x26 or 0x3B (unless they are a part of the error!)
2. application/x-www-form-urlencoded percent-encode set includes 0x26, 0x23, 0x3B and shouldn't be different from just running encode with html mode

(1) is what the url parser needs, it won't work with just html mode as it needs to differentiate between & in data and & in html escapes...

Perhaps the best way to cover this would be to provide a single-call result vector e.g. in a form of [<Uint8Array>, codepoint, Uint8Array, codepoint, codepoint]. But I'm not sure yet, this doesn't look clean
Streaming-like methods preserving encoder state after errors (which the spec describes) don't look nice too though

... or a special html-query escape mode that would percent-encode the bytes that are not in the percent-encode set (but that differs by the set)
... or a callback for handling errors

---

Perhaps a callback for error handling would be cleaner

[ChALkeR]

step 3 just fails on U+E5E5. Tracing the algorithms back, I guess that means it encodes to [0xE5, 0xE5].

I think that means that for url decoding it should treat it as %26%2358853%3B (even in modes that don't escape 0x26 / 0x3B

[ChALkeR]

@domenic Are there any good tests for multibyte encoders?

[domenic]

I don't know if I'd count them as "good", but the ones jsdom is failing are:

"**/*-encode-form*": [fail-slow, depends on form submission]
"**/*-encode-href*": [fail-slow, whatwg-url does not support non-UTF-8 query strings]
big5-encoder.html: [fail, whatwg-url does not support non-UTF-8 query strings]
iso-2022-jp-encoder.html: [fail, whatwg-url does not support non-UTF-8 query strings]
legacy-mb-schinese/gb18030/gb18030-encoder.html: [fail, whatwg-url does not support non-UTF-8 query strings]
legacy-mb-schinese/gbk/gbk-encoder.html: [fail, whatwg-url does not support non-UTF-8 query strings]

They all depend on web platform features (form submission or URL query string encoding) that indirectly use the encoders, so they might be difficult to run in a Node.js environment.

I would be happy to review and approve any PRs that refactor them to be more platform-agnostic (e.g., some core test or expectations files plus a some browser-API-specific "runner" files). Or you could probably hack together some scripts and/or vm environments that make them work as-is.

[ChALkeR]

I think the best bet for now would be to cross-test against reference implementation from WPT.
Assuming that works

Will test after landing everything except iso-2022-jp

[domenic]

On prototyping with the current API it seems a bit awkward that I have to hard-code a list of which encodings are single- vs. multi-byte into my calling library.

[ChALkeR]

@domenic That should not be needed, once all are ready a unified API will be exposed (and once we figure out what it should look like for error handling)