
Introducing the Windows Pseudo Console (ConPty) - mnkypete
https://blogs.msdn.microsoft.com/commandline/2018/08/02/windows-command-line-introducing-the-windows-pseudo-console-conpty/
======
ChuckMcM
This is pretty huge. For as long as I can remember the response to command
line applications talking to command line applications was "Why would you want
to do that? Use (RPC | shared memory | some other IPC mechanism)." And nobody
at Microsoft seemed to understand how much simpler it was to use ptys. They
seem to have completely capitulated to the notion ptys and are dropping them
into the next release of W10. I wish this had happened 10 years ago but hey,
I'll take it.

~~~
cryptonector
It's the reality of the market, which is why Windows is adding Linux
compatibility (as is every *BSD, Illumos, ...).

But also it's the fact that three decades of not even life support has left
the Windows console in pretty sad shape -- the folks tasked with getting it
into better shape were bound to see the value of ptys.

Lastly, don't forget that Windows NT was meant to be a console OS, like VMS.
There must still be people, even if very few, at MSFT who appreciate text-
oriented apps.

For me, the tty/pty, shells, screen/tmux/..., ssh, and so on, are the things
that make Unix so powerful. The fact is that Win32 is far superior in a number
of areas (SIDs >> UIDs/GIDs, security descriptors >> {owner, group, mode,
[ACL]}, access tokens >> struct cred), but far inferior in the things that
really matter to a power user trying to get things done.

~~~
SideburnsOfDoom
> Lastly, don't forget that Windows NT was meant to be a console OS, like VMS.
> There must still be people, even if very few, at MSFT who appreciate text-
> oriented apps.

I expect that, like Linux compatibility, most of it is not about "apps" but
about being better at running in the cloud, where a (virtual) machine or
container needs to be as light as possible, and to be configured and a service
launched in it as unattended/automated manner as possible. Stripping out the
GUI and making command lines work better works towards these goals.

~~~
cryptonector
That's a really good point.

------
caf
Will there be a terminfo database entry for ConPty? What TERM string should we
expect to see?

To elaborate: although an ordinary POSIX pty doesn't inherently have a
terminal type - that's entirely down to whatever emulator is connected to the
master side - the way the ConPty system translates Console API calls into
terminal control codes means that it necessarily needs to pick a terminal
emulation, which all actors in the ConPty system are expected to use.

A terminfo database entry would be useful both for applications running on
*NIX hosts but displaying on a remote ConPty master somewhere, as well as for
porting existing terminal applications to Windows where they will run on a
ConPty slave.

As a follow-up question, presumably this means that the SSHD running on
Windows as a ConPty master needs to translate between whatever terminal
emulation the ssh client is connected to and the one expected by ConPty /
ConPty apps (in the same way it must translate between the native ConPty UTF-8
and the remote charset)?

~~~
zadjii
So this is a confusing situation on Windows.

Commandline applications on linux rely on a TERM setting (with termcaps) to be
able to know what VT sequences the terminal is able to support. On Windows, we
only really have one terminal, conhost.exe, and our goal there is to be
compatible with TERM=`xterm-256color`. That's why you'll see that WSL has that
set as the default term setting.

Now even with ConPTY, when a client writes VT sequences, they still need to be
interpreted by conhost. This is because technically, a console application
could use both VT and the console API, and we need to make sure the buffer is
consistent. So clients should still assume that they should write out
`xterm-256color` compatible sequences.

Now on the other side of thngs, the "master"/terminal side of conpty, we're
going to "render" the buffer changes to VT. Fortunately, we'd dont really need
a deep VT vocabulary to make this possible, so the VT that's coming out of a
conpty is actually pretty straightforward, probably even vt100 level (or I
guess vt100-256colors, as insane a termcap that would be).

It's definitely a future feature that we'd like to add to make conpty support
multiple different TERM settings, and change the sequences we emit based on
what the terminal on the other side is going to expect.

We haven't really gotten into the nitty gritty of all of this quite yet, so if
you find bugs or have feature requests, we're happy to take a look at them.
You can file issues on [our
github]([https://github.com/microsoft/console](https://github.com/microsoft/console))
and we'll add them to our backlog

~~~
caf
Thanks. It makes sense to pick an existing terminal like xterm-256color to
target - that way you don't have to worry about a new terminfo database entry
getting distributed out.

The nitty-gritty can get quite nitty - things like bracketed paste and set
window title.

------
cryptonector
What next? Job control signals?? :) (EDIT: How about tmux?)

Anyways, this is fantastic. Finally, proper ssh functionality!

This will encourage development of console (text-oriented) apps for Windows,
which I hope will be much simpler. Interfacing with the console can be really
difficult if you're coming from *nix. Ideally all the WIN32-specific code in,
e.g., jq[0], could be ripped out.

[0] [https://github.com/stedolan/jq](https://github.com/stedolan/jq) (look in
src/main.c)

~~~
tom_
The lack of signals in Windows is the very opposite of a flaw! - Windows has
just never pretended you can get away without a message loop.

~~~
quotemstr
Windows does have signals! It just splits them into a few facilities. POSIX
"synchronous" signals correspond to SEH exceptions and can be handled roughly
the same way --- except that signals have process global handlers and Windows
has thread-local ones, because the glibc people are sticks in the mud and are
hostile to any attempt to make signals suck less.

For asynchronous signals, like SIGINT, Windows create a new thread out of thin
air to deliver your app a notification. That's not really all that much better
than a signal from a concurrency perspective.

Windows even has APCs, which are like regular signals that are delivered only
at explicit system call boundaries.

Every operating system needs some mechanism to tell a process to do something.
Windows has evolved an approach that isn't all that different from Unix signal
handling.

~~~
cryptonector
Spawning a new thread to handle a signal _is_ much better than preemption: you
then have no concerns about async-signal-safety that aren't plain old thread-
safety concerns. I'd much rather have thread-safety constraints than async-
signal-safety constraints.

~~~
Vogtinator
Linux can do that as well, see SIGEV_THREAD.

~~~
quotemstr
Only for AIO, which few use.

~~~
andreyv
You can spawn a thread manually and use the POSIX sigwait() function.

~~~
quotemstr
That won't stop the process-wide registered signal handler working and it
won't do anything about being able to reliably handle synchronous signals.

------
amluto
For me, the most surprising thing is that the new PTY devices use UTF-8. Not
UTF-16 or UCS-2 or weird little endian variants thereof, and not even wchar_t.

This is so un-Windows-like.

~~~
cryptonector
UTF-16 is garbage. Windows is stuck with it because it was too early an
adopter of Unicode. Oh the irony. This may set Windows on a path to
deprecating UTF-16 -- godspeed!

~~~
swozey
Could you elaborate? I've been under the guise for most of my career that
doubling a digit leads to huge benefits that I'm too comp-sci ignorant to
understand.

~~~
JonathonW
UTF-16 has a bit of a funky design (using four byte/two code unit surrogate
pairs to encode code points outside the basic multilingual plane) that
ultimately restricts Unicode (if compatibility is to be maintained with
UTF-16, at least) to 17 planes, or 2^20 code points (about 1 million).

UTF-8 uses a variable length encoding that allows for more characters-- if
restricted to four bytes, it allows for 2^21 total code points; it's designed
to eventually allow for 2^31 code points, which works out to about 2 billion
code points that can be expressed.

(Granted, this is all hypothetical-- Unicode isn't even close to filling all
of the space that UTF-16 allows; there aren't enough known writing systems yet
to be encoded to fill all of the remaining Unicode planes (3-13 of 17 are all
still unassigned). But UTF-16's still nonstandard (most of the world's
standardized on UTF-8) and kind of ugly, so the sooner it goes away, the
better.)

~~~
swozey
Thank you, this was an incredibly understandable explanation.

------
JdeBP
I've been waiting for two decades to revise this particular Frequently Given
Answer.

* [http://jdebp.info./FGA/capture-console-win32.html](http://jdebp.info./FGA/capture-console-win32.html)

~~~
red75prime
I suppose it's mostly TUI programs, which use low level console API. So did
you try to capture output of something like Far Manager[0]? If so, will it be
much simpler to parse escape sequences of VT100?

[0] [https://farmanager.com/](https://farmanager.com/)

------
zadjii
Hey I'm one of the Console devs who's been working on this feature for a while
now. I'll be hanging around in the comments for a little while to try and
answer any questions that people might have.

TL;DR of this announcement: We've added a new pseudoconsole feature to the
Windows Console that will the people create "Terminal" applications on Windows
very similarly to how they work on *nix. Terminals will be able to interact
with the conpty using only a stream of characters, while commandline
applications will be able to keep using the entire console API surface as they
always have.

~~~
zokier
While it is nice that MS is focusing on console and command line now, it seems
to me that you are mostly working on improving compatibility with legacy UNIXy
stuff.

Do you have some vision or plans to go well beyond the classic UNIXy style of
console and command line? I'm thinking in the lines of projects like DomTerm
[http://domterm.org/](http://domterm.org/) which could have nice interactions
with e.g. PowerShell.

~~~
joeyaiello
PM for PowerShell here!

I haven't seen DomTerm before, but it looks pretty awesome. At a glance, it's
basically a GUI-fied tmux hosted in Electron? It would be awesome to have in
Windows, but wouldn't that just require that DomTerm add support for these
ConPty APIs?

In any case, I'm more interested in your proposed interactions. Did you have
anything cool in mind? Given that we ship PowerShell on Linux, we could
theoretically do some stuff there (including within PowerShell on WSL) before
it's hooked up to ConPty

~~~
sime2009
I'm not the person you were asking, but this should interest you.

I've been working on a terminal emulator ( Extraterm
[http://extraterm.org/](http://extraterm.org/) ) with some novel features
which would dovetail nicely with how PowerShell works. The first is the
ability to send files to the terminal where they can be displayed as text,
images, sound, etc or as an opaque download. Extraterm also adds a command
`from` which lets you use previous terminal output or files, as input in a
command pipeline. See
[http://extraterm.org/features.html](http://extraterm.org/features.html)
"Reusing Command Output" for a demo. This opens up other, more interactive and
iterative workflows. For example, you could show some initial data and then in
later commands filter and refine it while visually checking the intermediate
results.

What I would like to do sometime is integrate this idea with PowerShell and
its approach of processing objects instead of "streams of bytes". It should
then be possible to display a PowerShell list of objects directly in the
terminal, and then reuse that list in a different command while preserving the
"objectness" of the data. For example, you could show a list of user objects
in one tab and then in another tab (possibly a different machine) grab that
list and filter it the same way as any normal list of objects in PowerShell.
You could also directly show tabular data in the terminal, let the user edit
it "in place" in the terminal, and then use that editted data in a new
command. It allows for more hybrid and interactive workflows in the terminal
while still remaining centered around the command line.

Extraterm does these features using extra (custom) vt escape codes. ConPty
should allow me to extend these features to Windows too.

~~~
joeyaiello
Ooooh yeah, that sounds awesome! Going to share this with some people on our
team (lots of folks love and use Hyper already, but we're always looking for
new stuff to play with).

I would highly recommend you check out the excellent HistoryPx module[1].
Among (many) other things, it supports automatically saving the most recently
emitted output to a magic `$__` variable. Theoretically, you could save a lot
further back, but you may start to run into memory constraints (turns out .NET
objects are a little heavier than text... ;) )

[1]:
[https://github.com/KirkMunro/HistoryPx](https://github.com/KirkMunro/HistoryPx)

------
asveikau
While we're talking Unixisms, Windows needs a dup2(2). That is, given a
HANDLE, you should be able to swap out its backing kernel data structure with
that of another HANDLE.

Without this, I/O redirection is slightly broken. Last I checked you can't
change where stderr goes after the process starts, for example. [SetStdHandle
doesn't do it at the right layer.]

------
hoppelhase
I always liked the Console API where you can set the color of the text without
actually changing the text that is written to Stdout. No issues when piping
the output somewhere else. No need to check whether the output is getting
piped.

~~~
zadjii
That'll work just the same as it always has :) Existing commandline
applications won't be affected by this feature, but it will open the doors for
an entirely new class of applications.

~~~
hoppelhase
The existing Console API won't be extended with features of VT codes, or will
it?

~~~
zadjii
Oh we added support for VT sequences in commandline apps years ago - case in
point, WSL.

see [this docs page]([https://docs.microsoft.com/en-
us/windows/console/console-vir...](https://docs.microsoft.com/en-
us/windows/console/console-virtual-terminal-sequences)) for a (surprisingly
incomplete) list of VT sequences we support, and how to use them.

------
quotemstr
Finally! I've been waiting ten years or so for this API. It's about time that
alternative terminal emulation becomes possible on Windows.

------
hoppelhase
If I use the System.Process API in .NET and redirect the Stdin/Stdout to a
stream inside my application, does the framework spawn an invisible console
and scrape the output? Or does this work differently? I always did it that way
and thought the 3rd party terminal emulators also do that. Why do these
emulators have to do it differently?

~~~
quotemstr
That API is using pipes.

------
exikyut
Wow. I remember the photo miniksa posted to GitHub when this was in process:

[https://github.com/Microsoft/WSL/issues/111#issuecomment-238...](https://github.com/Microsoft/WSL/issues/111#issuecomment-238592841)

 _Awesome_ to see it's finally up and running! \o/

------
borekb
I currently use ConEmu + zsh via MSYS2 as my preferred shell on Windows. I
need to run many interactive programs like `python`, `node` etc. via winpty,
e.g.:

``` alias node='winpty node.cmd' ```

With the new ConPTY, will I be able to run native Windows programs directly?
If so, that would be huge, winpty (while I'm really thankful it exists) is a
PITA in practice, see e.g.
[https://github.com/Microsoft/vscode/issues/45693](https://github.com/Microsoft/vscode/issues/45693).

------
linuxlizard
This is very exciting. I'm looking forward to seeing where it goes.

------
mschuster91
Is there any way to get this backported to Windows 7 - or run a W7 userland on
top of a W10 kernel? I'm actually serious about this one, I can't stand this
semi-"mobile-first", flat UI of newer Windows generations, and the privacy
invasions and ads are other hard blockers for me - but that WSL layer or the
new console subsystem seem to be pretty nice features.

------
red75prime
I hope control-S (XOFF) is disabled by default.

------
voltagex_
Wow, there might be able to be a proper ncurses port now!

~~~
zadjii
Functionally, we support all of the VT sequences you'd need to make ncurses
work resonably well on windows for a few releases now (ever since WSL was
introduced). If you could build an ncurses that assumed TERM=xterm-256color,
then you might be able to get it to work on windows.

------
docode
Where can we try a .NET solution with this ConPty?

------
mobilehnuser
Thanks to WSL and this, I'm very hopeful that my next development laptop can
be a windows device

------
21
Does this mean that it will now be easy to port terminator to windows?

~~~
zadjii
I sure hope so! I can run cmd.exe inside gnome-terminal running on WSL right
now - granted, WSL is doing some magic to make that all work, but it should
still work for terminator to do it to.

------
nasoieu
Something looks very eerie in that Admiral Grace Hopper picture. Is it
shopped?

------
kpil
\-- Those who don't understand Unix are condemned to reinvent it, poorly.
Henry Spencer

~~~
oblio
This is such a cliché.

Do you think that the people who implemented the Windows Console, especially
the people working on Windows NT, did not know about Unix? People try
different approaches, sometimes they don't work out.

And it's not like Unix is the Word of God, anyway, it has plenty of flaws.

(Yeah, after a long time on internet forums I get kind of touchy after someone
copy-pastes the same old and tired line.)

~~~
kpil
Since it took them 20 years to make a half-decent shell and 30 years how to
figure out how stdout should work: no they had no clue.

Maybe they knew how a kernel should work though, but weren't the NT guys old
VMS guys? That's a totally un-unixy OS actually.

~~~
oblio
1\. They didn't care about command line tools, in many ways (usability)
they're a huge regression. After all, it was in the name of the product:
Windows.

2\. Who says that in-band communication like Unix is doing is necessarily
better? See pastejacking and other shenanigans.

------
partycoder

        HRESULT WINAPI ResizePseudoConsole(_In_ HPCON hPC, _In_ COORD size);
    

If Microsoft is in the mood to fix old problems, right ^there you've got
another old problem: its bizarre API that is different to everything else.
Designed that way to lock everyone into their OS.

In 2018 nobody has the time to learn this. Just use a cross-platform API and
if it doesn't run on Windows then just don't run Windows.

As a developer, using Windows for development is against your own best
interest. If you like to be treated as a dog that is not allowed inside the
house, use Windows.

~~~
Jasper_
What cross-platform API would that be? ioctl(tty_fd, TIOCSWINSZ, &size); ? How
does the user get the TTY FD? open("/dev/tty0")? Or should they implement SYSV
compatibility and use "/dev/vt0"? Or perhaps follow FreeBSD, which has
"/dev/ttyv0"?

