
Zsh and Fish’s simple but clever trick for highlighting missing linefeeds - ingve
https://www.vidarholen.net/contents/blog/?p=878
======
ComputerGuru
One of the fish devs here. I definitely didn’t come up with this hack but I’ve
worked on improving some edge cases in the past. Our implementation differs
from zsh some although I think we’d originally adopted their approach.

This feature is still surprisingly complicated with many caveats. For example,
some terminals will wrap to the next line if you are at $COLUMNS, some at
$COLUMNS+1, and some not at all. Some advertise they do one thing and do
another. If you miscalculate, you’ll end up skipping an extra line or none at
all, maybe overwriting existing output or emitting the missing line break
glyph when it should be hidden. We’ve ironed out most of the cases but every
once in a while someone will be running an esoteric Pseudo tty (usually not
under X) and run into issues.

There are also issues with selection of content. Some terminal emulators will
bug out when you try to copy-and-paste content because they get lost with the
output past the current location in cases where a line break was already
emitted. It’s always the terminal emulator that’s broken but you’d be
surprised at how broken some of the most popular emulators are. The entire
codebase is littered with workarounds for tty emulators; if someone comes to
you and says “fish doesn’t work in $terminal but bash does” (either because
it’s minimalistic and is missing a feature that would cause the breakage or
because it has a workaround), we almost always end up treating it as a fish
bug unless we’re able to get it fixed in the upstream terminal
package/OS/software/whatever, whom we file reports with all the time.

It’s insane the configurations and platforms people use shells under and I’m
personally deeply appreciative of everyone that takes the time to file these
bugs - in addition to making the software better it really does shed light on
some extremely archaic bits of posix standards and Unix behavior and I’ve
learned so much from the many deep-dives that start off with “this is kind of
quirky behavior observed under certain circumstances” and ends up requiring
architectural changes deep within because some underlying assumption doesn’t
hold universally true as exemplified by a *nix that hasn’t seen a new user in
fifteen years or something. It’s why I love hacking on fish; I must have one
of the craziest collection of virtual machines to test edge cases, as do many
of the other devs (especially @zanchey and @faho, although no one does a deep
dive like @ridiculousfish).

~~~
dahfizz
Out of curiosity, I installed fish and set my prompt to be blank[1]. Based on
this article, I would expect to see the missing line feed character as the
first character on the line after every command, but I didn't. Do you know
why?

[1] My `fish_prompt` function just does `echo -n ''`. I also tried `true` and
an empty function body, I don't know much about fish scripting.

~~~
ComputerGuru
Nice catch and great show of initiative there! It's actually _not_ overwritten
by the prompt but rather by the "hack"/post-execution handler itself,
depending on where the glyph was emitted (which in turn depends on if the
command output included a trailing \n or not) when the line is cleared after
guaranteeing we're at col 0 of the next line. If the glyph wrapped to this
line, then it would be cleared. If it didn't, it'll show on the line above.

I think the missing clue is that the hack is followed by `^[[K` (clear line
from the cursor to the right), before control is passed to `fish_prompt` to
paint the prompt or whatever else.

Here's some sample output (stripped of the hack spaces and any ansii escapes):

    
    
         mqudsi@ZBOOK /m/c/U/mqudsi> echo -n hello
         hello¶
         mqudsi@ZBOOK /m/c/U/mqudsi>
    

`fish_prompt` isn't called until the cursor is already at column 0 and the
line is clear; the "missing line ending glyph" (here ¶ rather than ⏎ because
fish correctly detected that the latter is likely not supported by the font
used by my terminal emulator in yet another hack) is emitted at the end of the
_previous_ line.

You can see all this and more by piping the output of a `fish --interactive`
session to `xxd -C` or `hexyl` or whatever (that'll disable tty feature
detection and force ansii escapes to be emitted that you can then inspect).`

(You can also verify by simply using `sleep 1` as your fish_prompt - you'd
expect to see the glyph appear at least while the `sleep` was running if fish
were waiting on/expecting the prompt to overwrite the glyph.)

~~~
koala_man
I did indeed skip some details in the article. In the case of Zsh, the output
actually ends with `\r \r` which is probably meant to handle this exact case
by having a space overwrite any marker in case the prompt doesn't.

~~~
ComputerGuru
Ah, I couldn't remember what it was that we did differently from zsh, but I
knew there was something.

I would imagine - depending on the terminal emulator - the zsh behavior would
cause copy-paste-cruft in the form of trailing whitespace while _appearing_
correct, and would guess this was why we changed our implementation.

------
ridiculous_fish
_waves_ Great article about shell hackery. It's good to clear up the
misconception that shells know what's on the screen. They don't even know
where the cursor is!

One important reason for this hack is right prompts. If rprompt has width 7,
the shell moves right by $COLUMNS-7, outputs the rprompt, and then moves left
to return the cursor. What happens if during the move-right phase, the cursor
wraps to the next line? move-left doesn't "wrap back", it just pins against
the left side! So your prompts get split across lines, your right prompt
floats somewhere in space, and your input may even overlap it. It's quite
confusing to the user.

So if you have a right prompt, the shell has to be very sure it's on a new
line when it starts!

~~~
JdeBP
CUB does not wrap at left margin, but BS _sometimes_ does.

* [https://unix.stackexchange.com/a/198445/5132](https://unix.stackexchange.com/a/198445/5132)

------
Smaug123
That is _distressingly_ clever, and quite beautiful. It's one of those
solutions that makes me wonder whether I could ever have come up with it.

~~~
GuB-42
It is almost like a magic trick.

You probably know some of them along the lines of : think of a number, then do
a series of operations, and then I can guess the result. It can be done with
cards too. In reality no matter what your initial choice is, the end result is
always the same. The trick is to combine relatively complex functions (ex:
"take the sum all digits of the number", "add 1 if odd") in a way that produce
something simple (ex: "the result is 5").

Here the trick is to express the function "pos(x,y)=(0, y+1)" in term of the
functions "pos(x,y)=(0,y)" and "pos(x,y)=if x<$column then (x+1,y) else
(0,y+1).

It is also the basis of the "abstraction inversion" anti-pattern.

~~~
messe
This is almost certainly going to be somebody's horrific programming interview
question at some point in the next six months.

------
gnachman
I just discovered this yesterday when a user filed a bug against iTerm2. It
caused a problem for reflow because it looks like one long line. Took a minute
to makes sense of what was going on, and then I thought it was a neat trick
but it wouldn’t have occurred to me to post on hn about it. Glad others
appreciate it!

~~~
macintux
Since it can’t be uttered often enough: thank you for iTerm2. I’m not a power
user, but it and Emacs are generally my first two software installs on any new
Mac.

~~~
spurgu
+1

I've thought about switching back to Linux but I wouldn't want to live without
iTerm2.

Edit: This prompted me to make a modest donation.

------
zokier
This made me think how trailing newline is actually bit weird. Wouldn't it
make more sense to just have shell prompts always start with newline and
commands not outputting trailing newlines as gratuitously as they do now?

~~~
patrec
In unix every text file is basically a (possibly degenerate, Nx1) matrix and
every row is terminated (and demarcated) by a newline. This makes it much
easier to filter, concatenate and rearrange rows. Try rewriting this:

    
    
        cat <(cmd1...) <(cmd2...) >> some_output
    

for a world without final terminating newlines.

~~~
zokier

        cat foo <(printf "\\n") bar
    
    
    ?

~~~
patrec
Nope :)

(Hint: what's the neutral element?)

~~~
patrec
You'd need something like this:

    
    
        cat <(cmd1... | awk -vRS='\0' '""$0') <(cmd2... | awk -vRS='\0' '""$0') >>output

------
qwerty456127
> The shell could use pipes to intercept all output, and relay it onto the
> terminal. While it works in trivial cases like whoami, some programs check
> whether stdout is a terminal and change their behavior, others go over your
> head and talk to the TTY directly (e.g. ssh‘s password prompt), and some use
> TTY specific ioctls that fail if the output is not a TTY, such as querying
> window size or disabling local echo for password input.

Where do I read more about this? I'd like to understand terminals, pipes and
the differences. And how does the mouse (in case of terminal apps which
support mouse, e.g. Midnight Commander) fit in this model.

~~~
Dylan16807
A pipe is like a special file that takes text from one program and puts it
into another.

A terminal is either a physical object with a screen and a keyboard, or
software that pretends to be such an object.

The terminal can display text, but also certain sequences of text do special
things. For example, if the terminal receives the character for the escape key
followed by [?1000h then it doesn't display it. Instead it turns on mouse
click support. So a program connected to your terminal that wants mouse
support can send that sequence. Then if you click, your terminal _types_ an
escape key, followed by [M, followed by three characters to say what buttons
you're holding and where the mouse is located. Now the program will know you
clicked.

~~~
qwerty456127
Thank you very much, now I have an idea of how does mouse support work in a
Linux terminal.

However, I still don't understand how does a program know it's connected to a
terminal rather than some pipes.

~~~
surajrmal
Both the terminal or pipe show up as a preopened file descriptor to your
process. You can pass that file descriptor to the isatty posix function call
which will inform you whether the file is a tty (in most cases a terminal) or
not.

------
sime2009
OMG, as a developer of a terminal application I've encountered this little
symbol but just couldn't figure out where it was coming from. I assumed it was
something I was doing wrong on the emulation side.

------
enriquto
I don't know if I'm disgusted or delighted by this solution, but it doesn't
leave me indifferent!

~~~
messe
As I said in my other comment: Both.

It's vomit-inducingly beautiful.

------
kazinator
Interrogating the terminal emulator to get the current column is in fact a way
smarter, more robust solution that will work regardless of the terminal's
behavior when printing at the rightmost column. Also, fewer characters are
exchanged with the TTY in the happy case.

Proof of concept, using Bash on Ubuntu 18.04:

Define this function:

    
    
      getcol()
      {
        local savetty=$(stty -g < /dev/tty)
        local ttyesp
        stty raw min 16 time 100 < /dev/tty
        printf '\e[6n' > /dev/tty
        read -s -r -n 16 -d R ttyresp < /dev/tty
        stty $savetty < /dev/tty
        printf "%s\n" ${ttyresp#*;}
      }
    

Then this PS1 for testing:

    
    
      PS1='$(if [ $(getcol) != 1 ]; then echo '[noeol]' > /dev/tty; fi)\$ '
    

Test:

    
    
      $ echo good output
      good output
      $ echo -n bad output
      bad output[noeol]
      $ echo -n  # no output case
      $
    

WFM

~~~
JdeBP
Until you can name at least one common current terminal emulator that does not
respond to DSR 6, you have not tested this enough. (-:

~~~
kazinator
Why would I search for a situation that's going to be a definite WONTFIX?

I made up my mind some fifteen years ago that if a terminal isn't ANSI
conforming, it can be casually disregarded as something unsupported.

It's like testing that web pages work with Mosaic from 1993.

~~~
JdeBP
Because you are claiming to be providing something that is "smarter" and "more
robust". Not actually testing it enough to find the common case where it
doesn't work gives the lie to that claim. It indicates if anything a _lack_ of
smartness, and _is not_ robust. (Even Thomas Dickey's Stack Exchange answer on
the subject, which is better than what you've given here, isn't robust, as M.
Dickey didn't describe how to robustly parse a CPR sequence.)

When people talk about "ANSI sequences" and "ANSI conformant", they also
indicate that likely their knowledge comes from old wives' tales and samizdat.
The actual existing standards are ECMA-35 and ECMA-48, and the "E" in "ECMA"
does not stand for "ANSI". The world of terminals is alas full of people
who've worked off informal "escape code lists", or the Appendix for ANSI.SYS
in the back of the Microsoft manual for MS-DOS, and don't actually understand
in the first place what it is to be standards conformant.

For starters, it is ECMA-48 _in both directions_ , and _your code_ has to be
standards conformant, which that absurdly fragile string matching is not.
Would you have _your own_ "it's not conformant, it can be disregarded" applied
to _you_? Because it very much does.

In truth, (partly as a result of people working from samizdat and old wives'
tales) most terminals and terminal emulators are like your code, not
conformant, and by your metric _everything_ should be "casually disregarded",
leaving nothing at all; which is hardly a sensible position.

~~~
kazinator
> _When people talk about "ANSI sequences" and "ANSI conformant", they also
> indicate that likely their knowledge comes from old wives' tales and
> samizdat._

Or maybe some of them are actually referring to ANSI X3.64.

> _Not actually testing it enough to find the common case where it doesn 't
> work gives the lie to that claim._

I happen to maintain code that outputs VT100 sequences to control the screen;
it's tested on GNU/Linuxes (various emulators), Cygwin, Solaris, MacOS,
FreeBSD. Plus remote terminals like PuTTY, various Android SSH clients
(ConnectBot, JuiceSSH, ...). So from that and other past work I'm confident
that the escape sequence (which can be found in the DEC VT100 manual) works
fine in more places than I care to support. (If the only piece of code in this
area I had ever written were the above script, then of course I wouldn't be.)

The script itself isn't portable; it uses nonportable features of stty, and
Bash extensions of _read_.

The parsing of the terminal's response is probably adequate for the intended
use, and was labeled clearly as "proof of concept". It won't handle serial
line noise. (Actually, nothing will do that under all conditions anyway. Even
if parity is enforced, the response can be damaged in such a way that it has
valid syntax, but wrong digits giving incorrect coordinates.) Virtual
terminals transmit the response reliably.

If you can think of a way that a _malicious_ terminal emulator could target
and exploit the flimsy parsing, do share.

~~~
kazinator
addendum: real version of the code should handle decimal integers with leading
zeros.

------
twic
What happens if you resize the window after this trick has been used?

~~~
scarlac
I just tested in Terminal.app (macOS) with zsh and several things happen: If
you do nothing but run `echo -n 'hello world'` and then resize the window, the
next line prompt will be jittery (it'll jump left and right as you resize but
ultimately return to the correct position). My prompt is "14:41:44 ~ ️
<cursor>" and if I resize fast enough, the previous line (the one with the
output `hello world%` starts displaying characters on the right side.
Characters from the next line. If I do it fast, it becomes obvious what it's
doing: Fragments of the next line is copied to the previous line, so the right
part of 'hello world%' is filled with: `111141414:314:3914:3914:39:` after a
few resizes.

If you press return again (executing no command) then the buggy output is now
fixed and no magical behavior occurs.

It may be easier to just see, so here's an example:
[https://www.loom.com/share/5eab7cfb4590475a8881fe3e91d92738](https://www.loom.com/share/5eab7cfb4590475a8881fe3e91d92738)

------
simias
I love this article, as a long time ZSH user it's one of these things I've
always wondered how they were implemented but never cared to investigate
myself. It's really a clever hack too, if I had been asked to implement this
feature my default approach would definitely have been to find a way to
intercept the command's output somehow.

------
nneonneo
A slight improvement: adding "\\\033[K" to the end, i.e.:

    
    
      PROMPT_COMMAND='printf "⏎%$((COLUMNS-1))s\\r\\033[K"'
    

kills the extra spaces at the end, which can help alleviate some of the
weirder line wrapping that occurs when resizing a terminal window.

(I believe zsh uses something like \033[J${PS1}\033[K for similar effect).

~~~
JetSpiegel
This is a great addition to the post, thanks for this.

------
chungy
This is such a wonderful little hack, I've placed it into my .bashrc. Thank
you!

I did actually modify it a little bit, got rid of the two beginning %s in
favor of ⏎. It's not a character I expect to really happen to end legitimate
output of a program, so it seems more than good enough to keep it from being
accidentally intended.

~~~
opk
Apply some attributes to it if you want it to be clearer that it isn't part of
the legitimate output of a program. This is why the zsh default is a % in
standout.

And for zsh user's if you have a setup where unicode fonts can be replied
upon, `PROMPT_EOL_MARK` can be set to change the mark. It allows all the usual
zsh prompt expansions for stuff like coloring.

------
gumby
This is indeed a clever approach -- and I've been using printing terminals for
over 40 years.

------
Ericson2314
I want to swap the layering shell and the terminal emulator, so each shell
command (that needs it) gets it's own pty. I think this will enable a much
nicer UI and also simplify things.

~~~
opk
This is what tmux is for.

~~~
Ericson2314
No this is _not_ what tmux is fore. IMO tmux and screens are giant hacks:

\- multiplexing: better to use ssh multiplexing, which is just nice in general

\- persistence: Yes, you want to open a pty on a the host, but you should
actually emulate the terminal on the guest: one should just forward all the
pty messages between the guest and host (graphical emlator). In other words, a
lot more like regular ssh

\- sharing: graphical emulators should just understand that multiple can be
hooked up, have some support for this, we can relay the input from one
emulator to the others as needed.

For my terminal-inside-shell, I would use a customer server + protocol for
managing all the ptys (remember because backgrounded commands there can be
multiple).

------
stared
I am waiting for GPT-2-based models for ZSH. Something in the line of TabNine,
but for the terminal.

------
enriquto
I do not like this feature. How can you distinguish the output of a program
that outputs a newline from one that doesn't? This is intentional obfuscation.
If you are bothered from where your prompt starts, add a newline at the
beginning of your prompt.

~~~
Rogach
There is a "missing linefeed indicator" symbol (usually %) that is output if
there is no trailing newline.

So the output of the program that outputs a newline looks like:

    
    
      $ program
      output
      $
    

And output of the program without trailing newline:

    
    
      $ program
      output%
      $
    

There is an issue if the program is expected to output "%" at the end, of
course.

------
amelius
> Contrary to popular belief, the shell does not sit between programs and the
> terminal.

Perhaps it should?

~~~
soraminazuki
For better or for worse, unix simply wasn't designed that way. Changing this
now would mean breaking so many assumptions hard-coded into a vast majority of
unix applications. Considering how complex unix TTYs are, there is practically
zero chance this can be done transparently to existing programs. It's going to
need a redesign of the I/O system, at which point, we might as well call this
new system Plan10.

I also think there are problems with the approach itself. Making the shell sit
between programs would complicate the interaction between various programs and
introduce new bottlenecks in the shell. Additionally, it likely won't be an
improvement over the current situation. Rather, it would just introduce
another layer of incompatibility in the shell that programs would have to deal
with.

~~~
hnlmorg
I did this originally with my shell and it's actually not as painful as you'd
think. Most CLI tools don't care where fd 0, 1 and 2 are coming from / going
to (after all, if you're piping those programs then they're no longer writing
to the TTY anyway).

The only programs that cause an issue are tools that send ANSI escape
sequences (eg for ncurses) and check if STDOUT is a TTY before sending them.
In those cases you'll just get a message written saying something along the
lines of "STDOUT is not a TTY".

In theory you could write another workaround to fix that workaround but it was
a horrible kludge that I didn't like in the first place so ended up finding
another solution (which isn't as elegant as the zsh / fish fix and I'll soon
be adapting that into my own shell too).

~~~
soraminazuki
> The only programs that cause an issue are tools that send ANSI escape
> sequences (eg for ncurses) and check if STDOUT is a TTY before sending them.

Sure, but aren't those programs the whole point of having sophisticated
terminal emulators? If plain streams of text was all we cared about for CLI
applications, it would indeed make things a lot easier. So much easier, in
fact, that we can dump our current terminal emulators in favor of a more
simple alternative without ever having to complicate the shell. But since
people do care about terminal UIs, this isn't a realistic solution.

~~~
hnlmorg
> _Sure, but aren 't those programs the whole point of having sophisticated
> terminal emulators?_

What we currently have isn't sophisticated terminal emulators. It's a buggy
superset of hundreds of kludges due to 60s years of legacy. I mean there's a
lot I do love about the design but there's a lot to legitimately dislike as
well.

> _If plain streams of text was all we cared about for CLI applications, it
> would indeed make things a lot easier. So much easier, in fact, that we can
> dump our current terminal emulators in favour of a more simple alternative
> without ever having to complicate the shell._

The shell isn't the complication here. The shell is just a program launcher.
The real issues with terminals is:

\- Formatting is inlined with the data stream.

\- There isn't any type information in pipes (it is just a raw byte stream) so
applications can't easily do context sensitive processing.

\- ASCII isn't just a text format but also a data format (there's ASCII
characters for tables and records) and flow control (EOF) and job control (^c,
^z).

\- It's that responsibility for all of this is divvied up between the TTY
driver (in Linux's case, the kernel), the terminal emulator, and any user
space applications reading and writing to their respective file descriptors.

\- Terminals were never built to be API driven and while there are some
interactive ANSI escape sequences they're not widely supported by terminal
emulators. So from a shell or tty perspective, the terminal is a black box

\- Plus since file descriptors are literally files and TTYs are literally
files, it means actually any other process can write to the TTY even if
they're not part of the shell's process group (hence why tools like `wall` can
exist). So even if the shell could keep track of what was in STDOUT of the
processes it spawned, it can't possibly know if anyone else has written to
that same TTY.

There some ingenious design in Linux terminals but there are soooo many rough
patches thrown in too.

> _But since people do care about terminal UIs, this isn 't a realistic
> solution._

Right, but I was never suggesting people don't care about terminal UIs nor
that shells don't offer a valuable function. In fact the opposite is true:
I've written my own shell because I thought I could create a better UI/UX than
Bash.

------
beaker52
This explains why I've seen % when the prompt has been missing for whatever
reason.

------
ChristianBundy
I learned a new thing today, thanks for posting this!

