Regarding a public API, this GitHub comment suggests that people at Microsoft are working on it.[1] Of course, Microsoft can't commit to anything. As the winpty maintainer, I hope some kind of public API emerges to reduce my maintenance burden and make winpty more robust and efficient.
Aside: About a year ago, I studied how the console handles worked (especially in relation to console attach/detach/alloc and CreateProcess), and I discovered a bunch of interesting things. For example, the "fake" handles were inherited regardless of the bInheritHandles parameter to CreateProcess, and an entire "ConsoleHandleSet" was inherited if a process attached to another process' console. Real handles in Win8 seemed to be either "bound" to a particular input/screen buffer or "unbound" so they worked with any attached console. It was possible to have a console open with no attached processes.
In the course of testing, I found some neat bugs like:
- A small program that blue-screens Vista
- The inheritable flag on a console handle can be changed on any Windows version, except for Windows 7.
- CreateProcess has an undocumented(?) behavior where it duplicates stdio handles into the new process rather than inherit them. (The handle values will generally change.) Up until Windows 8.1, a stdio handle of INVALID_HANDLE_VALUE was duplicated into a true handle to the parent process. (Because INVALID_HANDLE_VALUE is actually the GetCurrentProcess() pseudo-handle, you see.)
- Assorted inconsistencies in Windows XP and WOW64 (probably fixed in newer OSs).
I'm not sure how much confidence I have in the spec. It is backed by a big test suite, but there are so many cases to consider.
Cool. There's a third option that you might want to consider for winpty. (I was experimenting with it before I left Microsoft.)
The basic idea is to use a DLL shim to provide your own console API implementation for a process and its children --- you can replace the system console API with your own by shimming DLL import and export tables. You'd use your own type of pseudohandle for your fake console handles; programs written to work with the old console pseudohandles should work correctly with your pseudohandles, provided you've hooked the relevant HANDLE-using APIs.
Yeah, I considered that route, but I was concerned that it'd be unreliable -- maybe with other programs doing clever tricks with import/export tables, or maybe with antivirus. I can't think of a specific reason it wouldn't work, though. I think it'd have to override CreateProcess to propagate the API hook. IIRC, ConEmu's CreateProcess starts a child in a suspended state so it can install its hook before resuming the child. Maybe it'd have to hook GetProcAddress, too.
I was assuming I'd use genuine console handles, but recognize that they're associated with the special console, so divert their API calls.
Maybe the technique would make programs slower. I know that's a complaint people have about ConEmu.
There's a similar (4th?) technique I've considered -- instead of hooking every process attached to the console, hook APIs in the conhost process and reimplement the internal protocol. It should avoid the performance problems and confine the hackiness to one address space. The trouble is that the protocol is undocumented. A small change could break everything without warning, but MS could also redesign the whole thing, making a fix impossible.
I also tried using the console accessibility APIs [1] to at least synchronize scraping and forwarding to the pty. The problem with this approach is that the console code issues EVENT_CONSOLE_UPDATE_REGION and such with its internal locks held, so attempting to read from the console from inside the accessibility handler deadlocks.
To do that, you have to attach the conhost.exe process to its own console with AttachConsole, right? I tried that and also noticed the deadlock. An out-of-context handler avoids the deadlock, but then it's not synchronized.
The other interesting problems I found with that API were: (1) when editing an input line, EVENT_CONSOLE_CARET notifications were delayed, (2) if I hit ESC when editing an input line, the line would clear, but there'd be no update-region notification, and (3) it's impossible to distinguish between a call to ScrollConsoleScreenBuffer and the whole buffer scrolling because we've reached the end. ScrollConsoleScreenBuffer might scroll only part of the buffer.
winpty doesn't use the WinEvents API (yet), but it might be good enough to reduce polling when the console is idle.
You mean like on current up-to-date versions of Vista and Server 2008? With a PoC on the github page? You realise that you have found a security vulnerability and are disclosing it publicly, right? Userspace must never crash the kernel, even if not further exploitable, especially so if it's possible for an unprivileged user. Be responsible, send a mail to secure@microsoft.com
Yes, it's a tiny program that (1) detaches the console, (2) attaches a new console, (3) closes all screen buffer handles, then (4) creates a new screen buffer. The fourth step caused a BSOD on Vista and Server 2008. AFAIK, they were up-to-date.
I did report it to Microsoft before making it public. The reply was:
> Thank you for contacting the Microsoft Security Response Center (MSRC). I would suggest trying on a local VM to confirm BSOD. However, this currently is just a local DOS, which would would not be something we would investigate further. If you have any additional information on how this could be further used to exploit another user or a remote DOS, please let us know and we will look into it.
> For an in-depth discussion of what constitutes a product vulnerability please see the following:
In fact, the same bug exists in XP, but the NULL page is mapped in XP which is why it does not BSoD. So far I have not seen writes, only reads.
Update: it looks like the NULL pointer is accessed several times in the code. They mostly are 16-bit accesses, so it is probably not addresses. They seems to be in font-related code.
If you haven't seen it, I'd also recommend looking at the excel1040[1] spreadsheet. It's been developed for about 20 years by a single author, AFAICT. It has numerous forms, schedules, and worksheets, and it makes aggressive use of formulas and locked cells to ensure that calculations are correct. The spreadsheets closely resemble the actual forms, though I don't think people are actually mailing them in as-is.
This is intended behavior, actually! By default, mypy doesn't type check the interior of functions that don't have any type annotations. This lets you slowly introduce types to a large codebase without having to convert everything at once (or even file by file). Adding a type annotation to "bar" will cause mypy to check it:
def foo(n: int):
pass
def bar() -> None:
foo("123")
>> main: note: In function "bar":
>> main:4: error: Argument 1 to "foo" has incompatible type "str"; expected "int"
Qualifiers might also be an issue. My impression so far is that every combination of differently qualified type is supposed to require a different _Generic association. (GCC 5.2 and Clang 3.7 differ here.) That would require a huge number of _Generic associations, but it might be possible to strip the qualifiers using a comma operator, _Generic( (0,(x)), ... ). A comma expression is not an lvalue, so perhaps it cannot be qualified. DR423[1] askes that question.
The example does not actually use compound literals. (i.e. _arg's initializer would work in C89.) It does use two other C99 features, though -- variadic macros and a for-loop-scoped variable declaration. The biggest compatibility issue, though, are its use of GCC extensions:
- Statement expressions: e.g.: int x = ({ int y = 0; y; });
- If zero-args are allowed (e.g. bar()), then _args is 0-sized, which is also non-standard.
These GCC extensions are somewhat common. Clang has them, and (I think) EDG's frontend does too. MSVC does not have either of them, even in MSVC 2015.
That said, a compound literal might be a good way to remove the extension use, but as long as bar() returns void, the do-while(0) trick is also sufficient for replacing the statement expression.
bar() results in trailing commas. My impression is that C99 allows them in array initializers. (I saw an example in the n1256 C draft.) I don't recall whether C89 also had them. __VA_ARGS__ is expanded twice, but I think that's OK, because sizeof() only evaluates its operand when it has VLA type, and it never will. This code will work in MSVC 2013 (if not earlier).
Using the online MSVC compiler at http://webcompiler.cloudapp.net/ (which runs Visual C++ 19.00.23720.0), it failed with these error message:
Compiled with /EHsc /nologo /W4 /c
main.cpp
main.cpp(20): error C4576: a parenthesized type followed by an
initializer list is a non-standard explicit type conversion syntax
main.cpp(21): error C4576: a parenthesized type followed by an initializer
list is a non-standard explicit type conversion syntax
main.cpp(22): error C4576: a parenthesized type followed by an initializer list
is a non-standard explicit type conversion syntax
main.cpp(23): error C4576: a parenthesized type followed by an initializer list
is a non-standard explicit type conversion syntax
If for some absurd reason you really do need to emulate a "compound statement expression" in MSVC (because you need to declare some temp variables in a macro that can be used for assignment), it turns out that it can be done with a "lambda expression":
#include <stdio.h>
int bar(int n, const char *p[])
{
int total = 0;
for (int i = 0; i < n; i++) {
total += printf("%s\t", p[i]);
}
putchar('\n');
return total;
}
#define bar(...) [](){ \
const char *_args[] = {NULL, __VA_ARGS__}; \
return bar(sizeof(_args)/sizeof(*_args) - 1, _args + 1); \
}()
int main(/* int argc, char **argv */) {
int total = 0;
total += bar();
total += bar("a");
total += bar("a", "b");
total += bar("a", "b", "c");
printf("total: %d\n", total);
return 0;
}
Compound literals were never formally added to C++.
Still, that hasn't stopped various front-ends (including GCC and Clang) from adding them to their C++ dialects. My code compiles with Clang, but not GCC. GCC complains that it's not OK to take the address of the array compound literal.
FWIW, C99 makes it clear that the object has automatic storage duration lasting until the end of the enclosing block. i.e. There shouldn't be an issue with dangling pointers.
My impression is that it's impossible in MSVC's C dialect today. (Obviously there's std::min in C++.)
AFAICT, it could be done using _Generic and inline functions, if/when MSVC adds _Generic. I'd expect to see _Generic before seeing any of the GCC extensions, but I'm not aware of any commitment from MS to add it. There are some issues with _Generic and qualifiers that might slow its adoption. e.g. There's an open defect report regarding qualified rvalues (DR423), but even lvalues seem to behave differently between GCC 5.2.0 and Clang 3.7.1. GCC ignores an lvalue's qualifiers but still allows a "const int" branch. AFAICT, a const-qualified branch is never selected with GCC?
I'm wondering whether it's possible to use _Generic to require that the left and right operands have the same type (either before or after integral promotions). Qualifiers are an obvious nuisance, but the bigger problem is that even the unselected branches must still compile, and C does not have tuples.
Someone just sent a patch this morning upgrading it to Clang 3.7, which is available in the Ubuntu 15.10 repository, but not earlier. There are also official x64 llvm.org binaries for 14.04 that work.
I don't actually use the program on a daily basis, but I do pull it out on special occasions. Most recently I used it while tracking down https://github.com/atom/electron/issues/3380. It's helpful because Electron/Chromium is big and I'm unfamiliar with the code.
SourceWeb needs a list of C++ invocations (compile_commands.json), but it ignores compiler errors, so I'm not sure how precise the invocations need to be. If all the C++ source files in a directory were naively listed, the index might still be useful. I've never actually done that, though -- I normally use the sw-btrace tool to build the JSON file.
The indexer comes up with a globally-unique name for each "symbol" it indexes. This turned out to be a hard problem for things like templates and macros. The indexer names static functions by prepending a filename, e.g. "bio_ssl.c/ssl_write". For local variables, it also appends a file offset. With templates/macros, it can determine that a single identifier corresponds to arbitrarily many different symbols, and when you right-click that identifier in the GUI, it attempts to show them all in a popup menu, which fills the entire screen.
Will this OpenSSH server be able to run interactive console programs (like cmd.exe or python.exe), or will it be limited to (say) PowerShell?
Windows doesn't have a good API for hosting a console--it's not like Unix, where a pty has a master end and a slave end. Trying to run a console program in mintty.exe (https://github.com/mintty/mintty/issues/56) or Cygwin SSHD fails for this reason. I wrote a tool, winpty, that makes a best-effort attempt to emulate a Unix pty master by scraping the console buffer, but it has some limitations, so I'm not sure Microsoft would want to use it. Maybe they would expand the console API?
I too was very skeptical. Something about posts from "The Powershell Team" about porting sshd makes me deeply cynical that they would get the integration right from a layering perspective, i.e. as if they would make it support powershell and nothing else.
Fortunately this example shows it going straight into cmd and then they invoke powershell as a next step:
C:\Master>ssh.exe -l user@127.0.0.1
user@127.0.0.1's password: **********
Microsoft Windows [Version 10.0.10566]
(c) 2016 Microsoft Corporation. All rights reserved.
user@DEV-10566-829 C:\Users\user>powershell -File -
Cool. Looks like that is essentially listing DEVICE=ANSI.SYS in config.nt? Google results imply some things about 16-bit emulation here -- was that post about running actual DOS ANSI.SYS in a VDM running actual DOS COMMAND.COM, within a native Windows terminal?
This seems like a colossal mess waiting to happen. I'm glad that I read their plan, so that I can completely avoid it.
The allure of Ubuntu LTS releases (currently 12.04 and 14.04) is the extended support cycle -- practically speaking the kernel is the biggest component of this for a lot of people.
If you use the 3.16 kernel (from Ubuntu 14.10) on 14.04, you are now on a 6 month support cycle [1]. One which doesn't even have defined dates (they're all still TBD). Now you have to keep running on the kernel upgrade treadmill to maintain support, moving to backported kernels from progressively newer Ubuntu releases every 6 months.
> practically speaking the kernel is the biggest component of this for a lot of people.
Really? I mean, step back from your initial gut reaction and ask yourself how many software developers could even name which version of the kernel their code runs on without checking. Most developers never make a syscall directly and, increasingly, aren't even calling something like libc directly because they use higher-level libraries.
Sure, some people really care cause they recently hit an issue with specific drivers and a few people are using a really new feature, but that's a much smaller group than the number of people running code which doesn't even depend on Linux, much less a specific point release.
Looking at the release notes, I'd say there are a LOT more people affected by real bugs which are fixed in 14.04.2 than will be affected by hypothetical bugs nobody seems to have noticed yet:
Overreacting much? The shortened support is limited to the packages you upgrade. It's a 9 month cycle with a new version available every in 6 months. Most importantly, these are just packages, they can be uninstalled of they don't work and you fall back to LTS versions. And this is all the same for 12.04.
For some reason, Chrome seems to work fine with extensions on my Ubuntu 14.04, kernel 3.13.
I agree with those saying this is completely stupid. In corporate environments, there are a lot of Ubuntu LTS, RHEL, and CentOS desktops. They won't be upgraded to >= 3.16 for years.
Aside: About a year ago, I studied how the console handles worked (especially in relation to console attach/detach/alloc and CreateProcess), and I discovered a bunch of interesting things. For example, the "fake" handles were inherited regardless of the bInheritHandles parameter to CreateProcess, and an entire "ConsoleHandleSet" was inherited if a process attached to another process' console. Real handles in Win8 seemed to be either "bound" to a particular input/screen buffer or "unbound" so they worked with any attached console. It was possible to have a console open with no attached processes.
I tried to formalize the behavior I observed here: https://github.com/rprichard/win32-console-docs
In the course of testing, I found some neat bugs like:
- A small program that blue-screens Vista
- The inheritable flag on a console handle can be changed on any Windows version, except for Windows 7.
- CreateProcess has an undocumented(?) behavior where it duplicates stdio handles into the new process rather than inherit them. (The handle values will generally change.) Up until Windows 8.1, a stdio handle of INVALID_HANDLE_VALUE was duplicated into a true handle to the parent process. (Because INVALID_HANDLE_VALUE is actually the GetCurrentProcess() pseudo-handle, you see.)
- Assorted inconsistencies in Windows XP and WOW64 (probably fixed in newer OSs).
I'm not sure how much confidence I have in the spec. It is backed by a big test suite, but there are so many cases to consider.
[1] https://github.com/Microsoft/BashOnWindows/issues/111#issuec...