Recent Readline changes

[dawidi]

A recent update to the readline library seems to have made "bracketed paste mode" enabled by default. Whatever that is, it obviously doesn't work on the Tandy TELCOM and introduces extraneous control sequences that mess up the screen. I'm seeing ?2004h when I hit Return in bash, and?2004l just before the prompt after a program ends.

After some research, I found that adding a line of set enable-bracketed-paste Off to ~/.inputrc turns this new feature off, maybe you want to add that to your readme.

[hackerb9]

Yeesh. That sounds awful. I'm not experiencing it yet. What version of readline are you using?

[dawidi]

I'm on Arch Linux and updated recently, pacman reports readline 8.1.0-2 which is about 3 weeks old.

The Readline changes document looks like they did mean to consider the terminal type before enabling the feature, but apparently it doesn't react correctly to the Tandy terminal. Not sure if that is a bug in readline, but the Tandy definitely doesn't need whatever bracketed paste does, so turning it off explicitly makes sense anyway.

Btw, even with this feature turned off in .inputrc, I still see those escape codes pop up in some places, e.g. when starting and quitting nano. But at least they're not constantly mixed in with my command lines anymore.

Coming from a DOS perspective, it's kinda mind-boggling how many layers of abstraction, complexity, potential problems (as well as flexibility, admittedly) Unices manage to squeeze into text terminal access.
Also I wouldn't have expected something as "archaic" as readline to be introducing new and problematic features these days...

I'm beginning to think it'd be more future-proof, at least for the purpose of simple extended file storage, to hook my Model 100 up to a little custom BASIC program running in DOSBox (or one of my vintage PCs) instead of solving all of the mysteries of half a century of Unix TTY support and keeping up with the changes that are still going on there :-)

[ThomasDickey]

You're referring to
y. Terminals that are named "dumb" or unknown do not enable bracketed paste by default.
which is (in context) saying that it's applied to everything. terminfo/termcap/terminals have never been a strong point of bash/readline.

[hackerb9]

I'm on Arch Linux and updated recently, pacman reports readline 8.1.0-2 which is about 3 weeks old.

Wow, you're living life on the edge.

The Readline changes document looks like they did mean to consider the terminal type before enabling the feature, but apparently it doesn't react correctly to the Tandy terminal.

Yuck. Sounds like Bash 5.1 / Readline 8.1 need to have this bug reported. I'm still on Bash 4, so that's why I didn't experience the same problems.

Btw, even with this feature turned off in .inputrc, I still see those escape codes pop up in some places, e.g. when starting and quitting nano. But at least they're not constantly mixed in with my command lines anymore.

Are you using a conditional in the .inputrc? Does something like this work for you?

$if term=td200
    Control-^: previous-history
    Control-_: next-history
    Control-]: backward-char
    # Use 'stty quit undef' to make right arrow work.
    Control-\: forward-char
    # Shift + BkSp to delete next char.
    "\d": delete-char
    # Disable ANSI colors for completion
    set colored-completion-prefix off
    # Disable bracketed paste escape sequence in bash-5.1
     set enable-bracketed-paste off
$endif

Coming from a DOS perspective, it's kinda mind-boggling how many layers of abstraction, complexity, potential problems (as well as flexibility, admittedly) Unices manage to squeeze into text terminal access.
Also I wouldn't have expected something as "archaic" as readline to be introducing new and problematic features these days...

Perhaps it's a security measure. I haven't read enough about bracketed pâte mode yet, but the changelog mentions differentiating between keys that are actually typed and characters that are pasted in.

As @ThomasDickey pointed out, the problem is likely that readline is ignoring terminfo. Many programmers seem to have forgotten one of the "archaic", hard-learned lessons of long ago: “Never hard code terminal escape sequences”. The database already exists (terminfo), the library already exists (ncurses), so we have very little reason to keep making that same mistake.

I'm beginning to think it'd be more future-proof, at least for the purpose of simple extended file storage, to hook my Model 100 up to a little custom BASIC program running in DOSBox (or one of my vintage PCs) instead of solving all of the mysteries of half a century of Unix TTY support and keeping up with the changes that are still going on there :-)

Or, you can update your UNIX box only as often as you update DOS on your PCs. ;-)

[hackerb9]

You're referring to
y. Terminals that are named "dumb" or unknown do not enable bracketed paste by default.
which is (in context) saying that it's applied to everything. terminfo/termcap/terminals have never been a strong point of bash/readline.

@ThomasDickey, is "bracketed paste mode" supported by terminfo? I couldn't find it in the man page and when I did a web search for it, I only came up with people saying that terminfo ought to have support for it.

I'd like to file a bug with readline/bash that they should use ncurses, but I want to have my facts straight.

Thanks!

[ThomasDickey]

That could fall into this existing catch-all:
# XT (bool) Terminal understands special xterm sequences (OSC, mouse
# tracking).
But we're talking about bash/readline, which haven't incorporated anything from ncurses within recent memory.

The actual control sequence isn't there (nor are the OSC, mouse-tracking), essentially for two reasons:

• they require some special handling (mouse is supported as a special case)
• there's only one existing flavor

There's comparable stuff (such as tmux's developer suggesting the capability corresponding to DECSCUSR), which some applications use. No one's proposed doing that with the bracketed paste stuff, and my to-do list is long.

[hackerb9]

That could fall into this existing catch-all:
# XT (bool) Terminal understands special xterm sequences (OSC, mouse
# tracking).

Hmm... sounds non-optimal to overload the meaning of a capability, especially one that is only "documented" in the source code of some third-party software. This is too reminiscent of when terminals went around claiming to be "XTerm compatible" when what they meant, at best, was that they were compatible with a certain version at a certain time.

But we're talking about bash/readline, which haven't incorporated anything from ncurses within recent memory.

I mentioned ncurses because it would solve several problems for me if Bash used terminfo properly. I was thinking that, with this recent major release of readline, they might be open to suggestions. It's been silly enough that the project has been hard-coding "ANSI color" escape sequences, but hard coding "bracketed paste" is so ludicrous, I hoped this could be the point where they see the value of terminfo.

I don't expect Bash or Readline will ever be linked with ncurses proper, but perhaps it could use a subset, like libtinfo or, if they preferred, a re-entrant alternative like unibilium.

The actual control sequence isn't there (nor are the OSC, mouse-tracking), essentially for two reasons:

Oh, oops. How hard is it to fix that for bracketed pastes?

* they require some special handling (mouse is supported as a special case)

Is that the case for bracketed paste mode? We could still have a boolean signifying support, right?

* there's only one existing flavor

Part of what I love about terminfo is that, even if there is only one flavor, it's still easier for developers to do things the right way instead of hard-coding obscure sequences in their programs. This allows for future growth and flexibility, improves clarity, leverages code reuse, and reduces bugs. Terminfo is a win all around.

There's comparable stuff (such as tmux's developer suggesting the capability corresponding to DECSCUSR), which some applications use. No one's proposed doing that with the bracketed paste stuff, and my to-do list is long.

What are the steps required to add a capability to terminfo? I mean, to do it right, not by fiat, as screen apparently did with XT. Is it something someone like myself could do and contribute to you?

I apologize for my ignorance, but it seems like the hardest part is deciding on an appropriate name and semantic meaning. (E.g., should there be both boolean and string capabilities.) Then it would just be a matter of documenting it in terminfo(5) and adding it to libtic, libtinfo. Or, am I missing something major?

[dawidi]

Wow, you're living life on the edge.

Yeah, bit of a split personality. I still run Windows versions that are long out of support (and DOS of course), but all my Linux machines are rolling distros.

Are you using a conditional in the .inputrc?

Yes, like in your example, but of course for term=td100 in my case. As I said, disabling bracketed paste there does turn the feature off for readline - but apparently not for nano (and possibly other programs?).

And yes, it does seem to be meant as a security feature, but imho it should be implemented in the client - just prompt the user about filtering out any risky control characters before actually sending the pasted data to the host. The host doesn't need to have a special "safety squints" input mode for this.

terminals have never been a strong point of bash/readline

Without the context of this discussion, this is like saying "cars were never meant to be used on roads", but yeah...

The rest of your discussion is quickly escalating beyond my paygrade, I'll leave you to it ;-)

[ThomasDickey]

That could fall into this existing catch-all:
# XT (bool) Terminal understands special xterm sequences (OSC, mouse
# tracking).

Hmm... sounds non-optimal to overload the meaning of a capability, especially one that is only "documented" in the source code of some third-party software. This is too reminiscent of when terminals went around claiming to be "XTerm compatible" when what they meant, at best, was that they were compatible with a certain version at a certain time.

well now they're often caught up to the subset containing features that they want.
None cover more than about a quarter of xterm's control sequences.

The actual control sequence isn't there (nor are the OSC, mouse-tracking), essentially for two reasons:

Oh, oops. How hard is it to fix that for bracketed pastes?

The usual (preferred) way is for some developer who wants to use a feature, proposing to add the
extended capability and then using it when it's available. The tmux developer's done that a few times.
I reused some existing extensions from screen.

Just adding extensions that someone "might" use doesn't work for me.

* they require some special handling (mouse is supported as a special case)

Is that the case for bracketed paste mode? We could still have a boolean signifying support, right?

sure - someone has to start off expecting to use it in their program ;-)

* there's only one existing flavor

Part of what I love about terminfo is that, even if there is only one flavor, it's still easier for developers to do things the right way instead of hard-coding obscure sequences in their programs. This allows for future growth and flexibility, improves clarity, leverages code reuse, and reduces bugs. Terminfo is a win all around.

yes... but to date, most of the bracket-paste scripting has been done by people who prefer to hard-code things.

There's comparable stuff (such as tmux's developer suggesting the capability corresponding to DECSCUSR), which some applications use. No one's proposed doing that with the bracketed paste stuff, and my to-do list is long.

What are the steps required to add a capability to terminfo? I mean, to do it right, not by fiat, as screen apparently did with XT. Is it something someone like myself could do and contribute to you?

Using it in an application is the way to go
(which makes it demonstrable, so that someone can see what problems it solves or doesn't solve).

I apologize for my ignorance, but it seems like the hardest part is deciding on an appropriate name and semantic meaning. (E.g., should there be both boolean and string capabilities.) Then it would just be a matter of documenting it in terminfo(5) and adding it to libtic, libtinfo. Or, am I missing something major?

It's data-only. I don't see that the ncurses library would have to know about it (barring someone inventing a set of functions for manipulating it, integrating that with the input code, etc). More likely, it would be something listed
in the terminfo database, to explain what it is.

[hackerb9]

The actual control sequence isn't there

Oh, oops. How hard is it to fix that for bracketed pastes?

The usual (preferred) way is for some developer who wants to use a feature, proposing to add the extended capability and then using it when it's available. The tmux developer's done that a few times. I reused some existing extensions from screen.

Just adding extensions that someone "might" use doesn't work for me.

People would use terminfo for bracketed paste mode, as seen in the bug report for tcell. Unfortunately, it appears the developers in that case implemented special case logic.

What are the steps required to add a capability to terminfo?

Using it in an application is the way to go (which makes it demonstrable, so that someone can see what problems it solves or doesn't solve).

Got it.

So, my next step at this point would be to talk to a project, like tcell, and ask them to start using a bracketed paste capability as if it existed in terminfo. After that, I'd get back to you and see if we can make it official. Right?

[ThomasDickey]

yes - that would work.