Separate error output was around for at least a decade before this. I know MTS had it in the 1960s, and I don't think it was their original idea. I used a CDC for a while and they had it too. So while this is the story of how standard error was introduced into Unix, it is not the origin story of the concept of standard error.
Unix's standard error is definitely not the first invention of a sink for errors. According to Doug McIlroy, Unix got standard error in its 6th Edition, released in May 1975 (http://www.cs.dartmouth.edu/~doug/reader.pdf). 5th Edition was released in June, 1974, so it's reasonable to suppose Unix's standard error was developed during that 11 month interval. By that time, Multics already had a dedicated error stream, called error_output (see https://multicians.org/mtbs/mtb763.html, dated October 1973).
All the same, I'd be willing to believe that Unix's standard error could have been an "independent rediscovery" of one feature made highly desirable by other features (redirection and pipes). It's not clear how much communication there was among distinct OS researcher groups back then, so even if other systems had an analogue, Bell Labs people might not have been aware of it.
The story that I recall about the origins of stderr is that without it, pipes are a mess. Keeping stdout to just the text that you want to pipe between tools and diverting all “noise” elsewhere is what makes pipes useable.
Right, I was alluding to the original Mac's filesystem, with separate data + resource forks, requiring all sorts of hacks to transfer files to and from them. Due to all the trouble of working with that across other platforms, Mac eventually gave that up at a filesystem level, and sprinkled ".DS_Store" files everywhere.
TCP/IP streams are bidirectional, but there is a limited way of sending "out of band" data, though it is not used as much. It would have been nice if the stdout/stderr multiple streams extended to TCP/IP networking and even HTTP messages too.
> TCP/IP streams are bidirectional, but there is a limited way of sending "out of band" data, though it is not used as much.
It's not real "out of band" data: that's something wholly invented by the Unix socket API. TCP itself just has an "urgent pointer", which addresses some byte further in the data stream that the receiver doesn't have yet, with the intent that higher-level protocols could use it as a signal to flush any data up to that pointer to observe whatever the urgent message is. There's nothing in the protocol itself to actually send a message separately from the rest of the stream.
This is interesting: MIMO channels to a process. Single stdin/stderr/stdout is effective for a single OS process, but with so much pulled up to user land (e.g. workers via green threads) maybe it makes sense to introduce multichannel i/e/o.
Sure. Make metadata out-of-band rather than in-band, so that the ungovernable mess of Unix-standard plain ol' text streams is replaced by structured data.
So, well then: allowing programs to consume and emit JSON - is this progress ?
I think having a plain byte stream is actually better, in a worse is better sort of scenario.
A plain byte stream can be easily aligned to work with any future or past encoding fashion. Consider the situation if them that designed unix had not been so aggressively minimal. We would probably be complaining how streams had to be ASN1 encoded and how much a pain it is to define the schema for what should be a simple ad-hoc data transfer.
As it stands, you can put whatever object format you want on top of the stream. I think it is the same with the files. I am sort of pleased we are not stuck with some obsolete no longer relevant, screwball structured format from the 70's that all our file have to conform to. instead our file are a simple range of bytes and we can impose whatever structure on them that we want.
To me the advantage of XML in this would be CDATA segments, that can represent (almost) any text (that isn't the CDATA terminator string) without requiring text escape hell.
And if some sort of JSON supported said CDATA, well, it is no longer JSON.
Sure, it is arguably cleaner to explicitly isolate error condition from general process meta-data. So in my OP diagram you can add an errout coming down from the proc-box. I've used this actual pattern in-process to hook up pipelines of active/passive components. However, there is no sense (imo) to propagate the errout of P(n) to P(n+1), so 2-in, 3-out.
p.s. That is pL(n).stderr -> pE.stdin, where pL is the 'business logic' and pE is the system's error processing aspects. I.e. the error processing component's stdin is the stderr of the logical processes (Lp), so there is a uniform process model applicable to both logical and error processing elements of the pipeline.
The issue is how to do this within the limits of line terminal interface (CLI). In code (as in in-process chaining) that aspect is a non-issue.
Similarly, the SAS System, originally written on an IBM mainframe in 1971 (probably OS-360 / MVS), and featured an input file (the SAS program itself) and two outputs, a LIST (the desired analytic output) and LOG, which contained status, warning, and error messages. It's not quite stderr, but clearly reflects similar thinking and was probably based on extant practices at the time.
> "One afternoon several of us had the same experience -- typesetting something, feeding the paper through the developer, only to find a single, beautifully typeset line: "cannot open file foobar"
reminds me of those t-shirts or digital billboards displaying some system error that we've all seen as memes
What I hate about stderr is that it's character-based, not line-based.
I often get output from multiple threads or multiple processes garbled together on the same line. I know how to fix this, but I feel my OS should do it for me.
that may help, but if a write writes more than PIPE_BUF bytes, it isn't guaranteed atomic by the kernel. similarly, stdioing a line of more than BUFSIZ may result in multiple write calls. i don't think posix makes any guarantees there (this is just an empirically based speculation) and i'm fairly sure the c standard doesn't
Don't confuse the C stderr (which is of type FILE) from posix STDERR_FILENO file descriptor (i.e. 2). FILE (in POSIX, and in C since C11) guarantees that each I/O operation is thread safe (and flockfile in POSIX can be used to make larger operations atomic). A low level POSIX file descriptor is not thread safe (although of course the kernel will protect its own integrity). BUFSIZ only matter when writing to a pipe from distinct file descriptors.
you can see that the single fprintf call resulted in three separate calls to write(2), even though it is only a single line of desperate screaming. those three calls happen at three separate times, typically on the order of tens of microseconds apart. if that file descriptor is open to, for example, a terminal or pipe or logfile that some other process is also writing to, that other process can write other data during those tens of microseconds, resulting in the intercalation of that other data in the middle of the screaming
threads are completely irrelevant here, except that i guess in an exotic scenario the 'other process' that is writing to the file could conceivably be a different thread in the same process? that would make your remarks about 'distinct file descriptors' and thread safety make sense. but we were talking about entirely separate processes writing to the file, since that's the usual case on unix, and in that case no form of thread-safety is worth squat; what matters is the semantics of the system calls
i don't think posix makes any guarantees about how many calls to write(2)† a call to fprintf(3) will result in, though i haven't actually looked, and i don't think wg14 concerns itself with environment-dependent questions like this at all
What he's saying is that as long as you don't mix usage of stdio and raw write(2), you won't have any interleaving problem; because there's a lock, which is why _unlocked variants exist.
that is correct in its own sphere of applicability, but incorrect in the scenario i was discussing, with entirely separate processes writing to the file, since that's the usual case on unix, because each process has a separate lock
(amelius, however, did mention the possibility of multiple threads!)
it also wasn't what they were saying
this thread is starting to remind me of the 'i'm not your buddy, pal' cascades from reddit
stderr having line buffering turned off by default is intentional. You want to see the output immediately and not have it stuck in a buffer that might be lost if the program crashes or freezes.
In my experience, the biggest offender is programs trying to do syscalls directly (possibly for async-signal-safety), but not being aware of `writev`. Especially programs that do colored output can be really stupid here. Sometimes there are stupid programs that use multiple processes to do colored output even (IIRC CMake is a big offender here, but CMake is infamous for refusing to fix bugs)!
The pipe buffer is big enough that sane programs aren't likely to run into problems. The math:
PIPE_BUF is 512 per POSIX but in practice 4096 on Linux (probably others too?). If we assume a horrible-and-unlikely 12 formatting characters per real character (and assume a real character is non-BMP and thus 4 bytes, but still single-column), Linux has enough for 64 characters. With more reasonable assumptions (mostly ascii, no more than 4 formatting changes per line) we get more like 6 lines of output being atomic on Linux, and even POSIX being likely to get at least one whole line.
Lack of a standard way to control standard stream buffering is a big pain point for me sometimes. I'm still salty the libc+environment based approach was rejected by maintainers. And it also cannot be fixed on the kernel side since buffering is purely userspace feature.
This is not exactly the most earth-shattering revelation, but man: handling and communicating errors always seems to be a source of a vast amount of inelegance in software.
I'd argue we haven't really "solved" the optimal way to do error handling in programming: Using union types remains one of the best options, but even that has its downsides. Consider the ergonomics of forwarding an error type multiple layers in a Rust program: you can remove some of the boilerplate by strapping macros on top, but I'd argue that's more of a bandage than a fix. Most other programming languages are either using exceptions, which I don't like as they complicate control flow behavior significantly, or simply ignore error handling entirely (like C and Go; Both of them provide some standard facilities for dealing with error values, but handling it is completely manual. I do like this, since it's very straightforward, but it nonetheless is just sidestepping the problem.) And even trying to keep it simple can create new problems, like of course the way pthreads has to contort errno into a thread-local, for reasons obvious.
And while stderr has created a somewhat unified channel for dumping errors into, once they've bubbled up to the point where the program needs to output it, there's an almost unlimited amount of opinions on exactly how error logging should work. Some software won't use stderr by default, others only uses stderr for specific types of errors. Some software dumps everything that isn't data output into stderr, including e.g. `--help` text, whereas some software uses stdout for anything that isn't explicitly an error (Which often leads to me needing to pipe --help to less twice: once without, and once with 2>&1.) Categorization of error logging is also somewhat contentious: should there be a "warning" severity? should you split errors into modules? Formatting, too: what should be in a log line? Should logs be structured into a machine-readable format such as JSON?
It was probably a bad omen that even very old versions of UNIX ran into problems dealing with error logging and wound up needing to bifurcate things. Few programs feel as 'lazy' as UNIX; if UNIX couldn't ignore the problem, god knows the rest of the software was doomed.
It does have the union type Result<normal, error>, but most people throw/catch Error.
In Swift, error is a simple value (without a stack frame) and thus is as cheap as a return value, but can be handled/caught anywhere in the call chain like an exception.
Error is a protocol that tags any type, so it can carry any details you like, and your catch can switch on the type.
But it's only now (10 years on) that they're declaring error types in the function signature. In this world, it turns out that not throwing is the same as throws(Never). It took this long because it's unclear (but possible) that per-type error handling helps, mainly with libraries.
Serializing/tracking the originating (thread) context and avoiding merge conflicts in error streams seems like the unsolved problem. Both Java and Swift have structured concurrency with parent/child relations with derived cancellation/termination. Perhaps later that can include errors.
Here I was thinking I might have another fun little bit of trivia for my Statistics class, like the T-test one from the other day (https://news.ycombinator.com/item?id=40485313). Different standard error!
Timestamping or sync points, so that if I pipe multiple streams (say stdout and stderr) I can keep them in sync further along when various buffers may have been involved.
Metadata, such as magic file types.
Structured data (this may link with meta data, and maybe there is even a way programs could negotiate what to send to each other).
Have there been systems which support other additional streams?
When using PowerShell, I find it useful that it handled progress separately, so it doesn't interfere with piping (putting aside that cmdlets are .NET-based objects anyway).
Is there something like stdprogress?
i think contiki needs less than 16k of ram to run a webserver, most of which is the tcp stack. my own httpdito is about 2k of code and uses a 1024-byte data buffer, but that's sweeping tcp under the linux kernel rug
this is just to say that you could probably have run a webserver on a pdp-8/s, which was about the size of an atx case and would be a reasonable controller to build into a phototypesetter at the time
