Tighten up language about characters in Unicode context

[mbtaylor]

Following James Tocknell's comments at issue #89.

[msdemlei]

For the benefit of other reviewers, here's the glossary (cf. https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-2/#G14527):

• code point is roughly what you'd call a unicode character
• the code unit is the "word" in the serialisation stream, i.e., bytes for our char type and 16-bit ints for our unicodeChar
• a code unit sequence is 1 .. n code units encoding a single code point. I'm not 100% sure if the "single" actually strictly holds, but I think so.

If all of this is right, I'd say let's merge with that one comment I have.

[msdemlei]

This needs to be "midway through a code unit sequence" (or "byte sequence", if you prefer).

[mbtaylor]

I don't see the term "code unit sequence" defined at the reference you give (though I can see it used in that text). Is the definition there?

[msdemlei]

On Mon, May 04, 2026 at 08:47:04AM -0700, Mark Taylor wrote: mbtaylor left a comment (ivoa-std/VOTable#90) I don't see the term "code unit sequence" defined at the reference you give (though I can see it used in that text). Is the definition there?

I don't think so. I've looked for something like a glossary in the document but failed to find one.

[mbtaylor]

In absence of a definition, it seems like a potentially confusing term to use ... "code unit sequence" (even worse "byte sequence") just sounds to me like a sequence of code units [not necessarily aligned on a code point boundary]. However, in this context I suppose there's not much else it can mean.

In section 6 I see I've used the form of words

MUST NOT be truncated midway through the multi-byte representation of a code point

do you think that would be better?

[msdemlei]

On Tue, May 05, 2026 at 04:50:18AM -0700, Mark Taylor wrote: In section 6 I see I've used the form of words > MUST NOT be truncated midway through the multi-byte representation of a code point do you think that would be better?

"Better" I don't know about. Less jargony in any case, and I'd say that makes in preferable.

[mbtaylor]

@aragilar can you comment on whether this addresses your concerns at #89?

[aragilar]

@mbtaylor Yes, I think these changes remove the ambiguity that was there before.

I don't think this is worth sticking in the VOTable document (I've never seen a standard which puts it in), but I presume there's no requirements around normalization forms? If you haven't come across it, here's some example Python code showing the different forms for the Angstrom character:

>>> import unicodedata
>>> unicode_str = "Å"
>>> unicodedata.normalize("NFC", unicode_str)
'Å'
>>> unicodedata.normalize("NFC", unicode_str).encode("utf-8")
b'\xc3\x85'
>>> unicodedata.normalize("NFD", unicode_str).encode("utf-8")
b'A\xcc\x8a'
>>> unicodedata.normalize("NFKD", unicode_str).encode("utf-8")
b'A\xcc\x8a'
>>> unicodedata.normalize("NFKC", unicode_str).encode("utf-8")
b'\xc3\x85'

I suspect this is something clients defer to their language/library/framework to handle.

[mbtaylor]

Gosh, there's always more to know about Unicode, isn't there? But yes, I agree that normalization is out of scope for VOTable, I don't think it needs to go beyond explaining how to encode a sequence of code points.

Thanks for the approval.