Fix console getting confused by unicode characters

[ydirson]

The OS should have set a proper environment for the console to draw
line characters. Disabling UTF-8 just breaks this when the console
has been properly configured for UTF8.

[freddy77]

I didn't find that ISO 8859-1 setting anywhere (ESC %@). But it looks wrong anyway. First, ISO 8859-1 has not the semi graphics characters we need (like borders). Second today the world is more around using utf-8 not iso8859-1 (although I doubt vt100 supports utf-8 in all versions).
When I test with Qemu it looks like the borders are fine, when connected using Putty they looks wrong.
Which environment are you finding this issue?

[ydirson]

Which environment are you finding this issue?

When running in a Almalinux 10 base install

[freddy77]

Which environment are you finding this issue?

When running in a Almalinux 10 base install

Local, remote? Bare metal? BIOS or UEFI?

[bernhardkaindl]

I didn't find that ISO 8859-1 setting anywhere (ESC %@).

https://linux.die.net/man/7/utf-8 (From the Linux manpages package):

The official ESC sequence to switch from an ISO 2022 encoding scheme (as used for instance by VT100 terminals) to UTF-8 is ESC % G ("\x1b%G"). The corresponding return sequence from UTF-8 to ISO 2022 is ESC % @ ("\x1b%@"). Other ISO 2022 sequences (such as for switching the G0 and G1 sets) are not applicable in UTF-8 mode.

When I test with Qemu it looks like the borders are fine, when connected using Putty they looks wrong.

Putty has a config:
Window → Translation → "Enable VT100 line drawing even in UTF-8 mode".
and it's not enabled by default.

More details on how to configure Putty versions behave on Linux/Windows: https://stromasys.atlassian.net/wiki/spaces/KBP/pages/15138858/Cannot+get+correct+line+drawing+on+Linux+UNIX+with+Putty
and also https://serverfault.com/questions/475925/how-to-fix-putty-showing-garbled-characters

So sending "\x1b%@" to putty should disable UTF-8 mode and allow the line drawing to work in Putty. That sounds messed up.

@ydirson I'm guessing that this is for putty:
To keep the sequence for putty, the code could check if the tty is a local Linux console tty or serial / pseudo tty and not emit the sequence for real Linux console ttys? (or switch all terminals to UTF-8 mode, see my next comment)

[ydirson]

Which environment are you finding this issue?

When running in a Almalinux 10 base install

Local, remote? Bare metal? BIOS or UEFI?

UEFI guest in XCP-ng, standard console in XO

@ydirson I'm guessing that this is for putty

no, no putty here :)

[bernhardkaindl]

I want to send this idea:

But as this image shows, the default radiobutton selector for "Handling of line drawing characters" is: "Use Unicode line drawing code points":

As https://linux.die.net/man/7/utf-8 says:

The official ESC sequence to switch from an ISO 2022 encoding scheme (as used for instance by VT100 terminals) to UTF-8 is ESC % G ("\x1b%G").

Thus, if we use

print("\033%G") 

to instead switch all terminals (including putty) to UTF-8 mode then it should work for new Linux consoles and Putty, I'd hope.

[ydirson]

This tty mode change naturally does not impact just host-installer, so for diag purposes pstree is a great way to quickly check the effects of such changes.
What I get is

• out of the box, pstree shows semigraphics as expected
• after printf "\033%@" the pstree output (like that of the installer TUI) uses 1 character per byte in the UTF8-encoded semigraphics, completely messing up the display

[bernhardkaindl]

after printf "\033%@" the pstree output (like that of the installer TUI) uses 1 character per byte in the UTF8-encoded semigraphics, completely messing up the display

Of course, the terminal's mode (vt100 drawing -> "\033%@") and the Unix terminal charse (locale) must match.

• If using "\033%@", then the effective LC_CTYPE (also affected by LANG and LC_ALL must not end in .UTF-8)
• If locale|grep LC_CTYPE ends in .UTF-8, "\033%@" would cause a mismatch, and sending "\033%G" should switch the terminal (back) to UTF-8 mode, I hope also Putty should honor it and have working line drawing (using UTF-8 code points) then.

[ydirson]

@ydirson I'm guessing that this is for putty
no, no putty here :)

oh sry I misunderstood :)

if we use

print("\033%G") 

to instead switch all terminals (including putty) to UTF-8 mode then it should work for new Linux consoles and Putty, I'd hope.

It will likely work on all UTF-8 enabled envs, which unfortunately is not yet universal, as this putty example seems to show.
It seems to make sense to setup putty for UTF-8 when connecting to a host using an UTF-8 locale.

Though, in my tests things are not 100% clear, as using LC_ALL=C (or even LANG=C) does not prevent the TUI from issuing UTF-8 sequences, whereas pstree as expected does switch to ASCII output, so it's really not just the locale, it seems to be the TUI toolkit itself making inconsiderate assumptions, and for this forcing UTF-8 as you suggest might be the right solution, until/if the TUI starts to behave more correctly, in which case we'll be able to just leave the tty alone as my patch was suggesting.

I'll make the change as you suggest.

[bernhardkaindl]

Hi, approving.

Of course, with the reservation that a 2nd approval is needed, which would be @freddy77 regarding not breaking the cases XenServer cares for.

[freddy77]

I surely will want to check

• Qemu machine (well, more as a quick test, not 100% needed if next one works)
• a real bare metal machine
• remote connection with Linux (no regressions)
• remote connection with Putty (no regressions)