One thing to keep in mind when looking at GNU programs is that they're often intentionally written in an odd style to remove all questions of Unix copyright infringement at the time that they were written.
The long-standing advice when writing GNU utilities used to be that if the program you were replacing was optimized for minimizing CPU use, write yours to minimize memory use, or vice-versa. Or in this case, if the program was optimized for simplicity, optimize for throughput.
It would have been very easy for the nascent GNU project to unintentionally produce a line-by-line equivalent of BSD yes.c, which would have potentially landed them in the 80/90s equivalent of the Google v.s. Oracle case.
Remember that "yes" is the bottom end of complexity here, not the top. As the utility grows larger that stops looking like "asking for trouble" and starts looking like "often the only sensible solution". Expecting people to extend programs in C is often "asking for trouble", after all!
One point in favor of this simple version is that it's immediately obvious that it doesn't do the same thing as the OpenBSD version. In OpenBSD `yes a b c` will only print "a" while in GNU it prints "a b c". I did not catch that when I was reading the more complicated modern version.
At the risk of sounding like a copyright newbie, shouldn't that be covered by just doing a 'clean room' implementation? As long as you can verifiably prove that you didn't copy the source, it should fall under general use (as there's really only one way of doing such a thing), right? Much like Apple can't patent rectangles, although they tried.
Normally I think readability is more important than speed. But in this particular case, I think GNU is doing the right thing optimizing the code to the limit.
This is the beautiful part of Unix: small tools that do only one thing well. Programs following this philosophy are very good abstractions. They do one very well defined thing so you can use them without having to understand how they work. I have used Unix for years and I've never felt the need to read the source code for `yes`. And because they do a very small thing, even if you need to read them, the overhead of optimization is not that much, for example, the optimized GNU yes is just under 100 LOC if you remove comments and help boilerplate. Yes, it's longer than the BSD version, but it's just a matter of minutes to understands what it does.
I totally disagree. Nobody will ever want to use `yes` at 10 GB/s. They will want it to be reliable, and this sort of over-optimisation increases the risk of bugs.
I've used 'yes' many times to generate huge amounts of data quickly. Back then, it never had the small string optimisation, but you could always run 'yes InsertReallyLongStringHere' to spew out data much faster than /dev/urandom or even /dev/zero
I'm glad it runs fast, and I hope that all OS utilities are optimised (and tested, of course!) instead of making their source code pretty. The fact is, most people want to use programs, not read them.
I agree, all I remember is that when I tried it, /dev/zero sometimes sucked performance-wise. I can't recall the exact circumstances as it was some time ago, and could have been on any of Linux/FreeBSD/SunOS/HP-UX/IRIX - perhaps it was the fastest common way at the time?
On a recent x64 Linux, /dev/zero seems plenty fast enough now:
$ dd bs=8k count=819200 if=/dev/zero of=/dev/null
819200+0 records in
819200+0 records out
6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.331137 s, 20.3 GB/s
$ yes | dd bs=8k count=819200 of=/dev/null
819200+0 records in
819200+0 records out
6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.959551 s, 7.0 GB/s
So how did that redirect even work; should we be doing a `mknod` to make a "yes" device to make the comparison work (can we, does it help other than in my naive imagination).
I know this works, but how come we can see the output of pv when it is redirected to /dev/null? Maybe I just don't understand how pipes and redirection works since I rarely use Linux :(
> I know this works, but how come we can see the output of pv when it is redirected to /dev/null?
From pv's man page:
> Its standard input will be passed through to its standard output and progress will be shown on standard error.
> Maybe I just don't understand how pipes and redirection works since I rarely use Linux :(
The Windows/DOS command line has the same concepts[0], though it's probably used less often: by default a process has 3 FDs for STDIN (0), STDOUT (1) and STDERR (2).
At a shell
* you can feed a file to STDIN via <$FILE (so `</dev/zero pv` will feed the data of /dev/zero to pv), the 0 is optional
* you can pipe an output to a command (other | command) or the output of a command to an other (command | other)
* you can redirect the STDOUT or STDERR to files via 1> and 2> (the "1" is optional so 1> and > are the same thing) (you can redirect both independently)
* you can "merge" either output stream to the other by redirecting them to &FD so 1>&2 will redirect STDOUT (1) to STDERR (2) and 2>&1 will redirect STDERR (2) to STDOUT (1), you can combine that with a regular redirection so e.g. `command >foo 2>&1 ` or with a pipe (`command 2>&1 | other`)
And you can actually create more FDs to use in your pipelines[1] though I don't remember ever seeing
You're redirecting stdout to /dev/null and pv is writing to stderr. If you use &> instead, stderr and stdout will both be redirected to /dev/null and you will see no output at all.
While 'strace yes > test2' is just a constant stream of write() calls.
The difference matters if you're benchmarking e.g. some new SSD compared to a tmpfs on a machine with 100+ GB of RAM. It's always better if the tools have less overhead, because the comparison is more meaningful.
Also consider that it can be faster to write to a local network than to disk. I've never done it, but I imagine that the kernel's not going to want to deal with your /dev/zero calls if it's spending all of its time writing to a 10GB switch. I can imagine some very specialized storage servers that could spend most of their time writing from memory buffers to a network switch, or if you're troubleshooting a slowdown in the networking itself.
When I started this comment, I didn't think you were measuring what you thought you were measuring with those straces. 'strace yes > test2' only watches 'yes', not '> test2' (which is handled by the shell). Here's what your command outputs:
How does this possibly work? File descriptor 1 is supposed to be the terminal, not a file! Of course, the magic of file redirection:
open("/tmp/test2", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 # Open the file, get FD 3
fcntl(1, F_DUPFD, 10) = 10 # Copy FD 1 (the terminal, STDOUT) to FD 10 temporarily
close(1) = 0 # Close original FD 1
fcntl(10, F_SETFD, FD_CLOEXEC) = 0 # Close FD 10 (the terminal, copy of STDOUT) when exec() is called
dup2(3, 1) = 1 # Copy FD 3 (the file) to FD 1 (STDOUT)
close(3) = 0 # Close FD 3 (the file's original descrptor)
I had the impression that you thought "yes" pipes the output to shell, and then the shell writes to disk. That's incorrect. " > file" means redirecting to a file, and therefore all write system calls actually write to disk.
That's exactly what I thought. Now, had you asked me how redirection worked I would have said "the redirection operators cause the shell to attach a file to the descriptor," but I'd never actually thought through the implications of that in terms of what syscalls get made, and the output of strace presented a rather visceral demonstration of the implications of this clever bit of design.
The ability to get 10 GB/s dummy date into a pipe from the command line could come in handy at some point, for stress testing or something. I am not sure if it is over-optimized. (And even with the optimizations the risk of bugs should be very small.)
which is a common way of doing it (for any command with any inputs and outputs, not just the above ones), i.e.:
command < input_source > output_dest
All three pv command invocation variants, the one here and the two above, work. And it becomes more clear why they are the same, when you know that the redirections are done by the shell + kernel, and the command (such as pv) does not even know about it. So in all three cases, it does not see the redirections, because they are already done by the time the command starts running (with its stdin and stdout redirected to / from those respective sources). And that is precisely why the command works the same whether it is reading from the keyboard or a file or pipe, and whether it is writing to the screen or a file or pipe.
/dev/zero doesn't "write" anything in the sense that yes writes, since it's a character device and not a program. The Linux kernel's implementation of /dev/zero does not write one byte at a time.
You're right of course; and actually I believe the kernel will simply provide as many bytes as the read() requested; so the speed should mostly depend on how you access /dev/zero. IE, the user above was using cat and I think with dd and a proper block size it'd be much faster.
I was under the impression that cat automatically used a sane size for reading. Now that I think of it I cannot think of a source, other that to point out my own anecdotal experience.
When I was writing raspbian images to SD cards to use on a raspabery Cat and DD took within a few seconds of each other on an operation longer then a minute. Since then I have been using cat where I could, but I didn't think to right down the numbers though.
Note that cat+gnu awk was faster than just gnu awk - but mawk was faster still (reading a not entirely small file).
And in a similar vein of gp comparing Gnu and Openbsd, note that openbsd cat is a little more convoluted than the simplest possible implementation (at least to my eyes):
yes will give repeated data though, not just zeros, seems like it's more useful here - as others have pointed out, also uses less syscalls than /dev/zero.
If you want to maximize readability and simplicity then writing `yes` in C is a bad choice. It is much easier, cleaner and shorter to write it in python or just use the shell which would normally be using the `yes` in the first place. Since `yes` is used in shells and builtins can be considered very reliable, here is a implementation as a shell function:
yes(){ while :; do echo "${1:-y}"; done; }
Python:
import sys
while True:
if len(sys.argv) > 1:
print(sys.argv[1])
else:
print("y")
And if you don't need the features of `yes` and only need a loop that prints 'y', then there really is hard to beat the simplicity of:
The C example has three includes, two conditionals, two loops, and one function definition. The python example has a single include, conditional, and loop.
For readability purpose it is easier to go through each lines of the python program than the OpenBSD C code. Its not massively different, but its distinguishable enough that I would choose the python version if I wanted to maximize readability, minimize syntax requirement and did not want to use shell script.
The Shell function is in my view the superior choice if the audience is a programmer than know the shell script syntax. It is just a single loop and is written in the environment that the program is intended to be used in. The only drawback there is the speed.
Most of what makes the C program bigger comes from the fact that the C program does more. Your python example doesn't call pledge(). Remove that from the C program and it drops to one include, one conditional, and two loops. Further, counting the two loops against C doesn't make any sense: it's entirely up to the programmer whether to have a conditional containing two loops, or a loop containing a conditional. Both languages could naturally do it either way.
Exactly. That's the reason I write stuff in C instead of my favorite interpreted language, Ruby. When you write something in C, that's it. No large interpreter plus runtime needed.
You've been on a tear of uncivil and unsubstantive commenting, and it has to stop. Often a good strategy for this is to slow down. High-quality reflective posts are what we're after instead of dismissive, snarky reflexive ones, and the former come more slowly.
If the speed of yes is bounded by memory speed, doing anything useful will almost certainly consume that data at a far lower rate. Putting it on a disk, pushing it over a network, etc. will almost always be slower than yes is able to generate data.
The typical use case is piping it into another running program. Maybe someone wants to do that really quickly rather than putting it on disk or pushing it over a network.
It's not about running 'yes' at 10GB/s, it's about less overhead to do a simple job. If this version of yes is 100x faster, that implies it using 1% of a cpu to do the same work that would otherwise occupy 100% of a cpu. This leaves more of the machine to do what is likely to be the intended task.
Unix userland tools have evolved over decades to be as efficient as possible because they have historically underpinned everything the operating system does. The faster each tool works, the faster the other tools that depend on them work. If increased efficiency results in a bug, that bug can then be fixed, making it a net gain for system stability.
Easy! Write a program that does a brute force check of Goldbach's conjecture on all integers. For each integer that passes the check, print a line. If you can prove this correct (or incorrect) you'll probably win a Fields medal.
I agree with the general point that you could prove such a simple program correct relatively easily but that does still have a cost, which is always a concern in an open-source project. You still need someone to step up and do that work and continue to verify it periodically in the future – if that code is doing complicated things with buffering, that opens up possible odd bugs due to stdlib, gcc, maybe even kernel behaviour changes which might not affect simpler programs.
Not a huge bit of work to be sure but for a non-commercial project you might have trouble finding a volunteer who cares about that tedium.
Absolutely, I once trivially proved a 1000,000 line implementation of return 0 to be correct. I don't know why all comments are bothered by how much overkill this yes implementation is. Maybe they don't hold 100,000 PhDs like we do, am I right?
It's not really optimized to the limit – or perhaps it is, but then the limit is fairly easy to reach.
When I saw this item here I reached for the terminal and wrote a simple (and non-compliant) alternative that simply initialises 8150 bytes of "y\n" and then loops a write forever. I understand that it is not a fully standard-compliant yes, and that maybe GNU yes is indeed fast, but that awfully simple program that takes all of 10 lines (everything included) and took me all of a minute to write performs just as well as far as pv is concerned.
(I eventually completed a feature complete yes but I still think that simply not using `puts` is hardly optimising to the limit.)
A nitpick, but I notice that the BSD implementation do not catch errors when printing. In theory it could get EINTR and only write partial amounts of argv[1], especially if the argument string is really long and the program runs for a extended amount of time. The GNU version do catch EINTR.
Naturally it could be that the C function puts used by most people in OpenBSD is implemented with built in EINTR catching loop, or that OpenBSD do not interrupts writes.
You'd have to check the various POSIX standards to be sure - and have to then verify that the OS/libc actually follows them, but you can pretty much rely on every libc's I/O wrapper functions to handle interrupted system calls or incomplete writes. I've never seen any code check the return value of a printf() to verify that all the characters were printed.
As you say, the GNU versions definitely handle EINTR - the linux man page for puts() just says it returns a non-negative value on success, it's not even specified whether it returns the number of bytes written or not.
`puts` is an "stdio" function, not a system call. It won't EINTR, it correctly resumes if the underlying write() EINTRs. If it does get an error, it will return EOF, then you'll have to call `ferror()` to find out which error.
But the point stands; there's no error handling in the OpenBSD version. But that could be considered a design decision; the OpenBSD version never gives up on writing data until you kill it; the GNU version bails at the first error.
I think the GNU version is surprisingly readable for what it does. I have seen code that is a lot less readable than this and doesn't have the performance benefits described in the article.
I generally look at some BSD sources when I want to know how some unix tools are working, they are always much much more readable than the GNU equivalent.
It's to make sure the buffer is flushed. Streams are closed, but not explicitly flushed at exit. Stdout only because it's the only filehandle with a buffer in yes.
In normal use, GNU `yes` does unbuffered IO on stdout. However, it does use buffered IO for --help and --version messages; it sets atexit(close_stdout) to cover both of those cases at once, rather than handling them both separately.
According to the article/experiment you don't have to bork up the code much, just copy your stuff into a big buffer before you start printing it (and print using write(2) instead of puts)
This works easily for the default case, which prints "y\n" (two bytes), which is likely to divide BUFSIZ. To handle the general case with the same efficiency, you have to have a buffer size that is a multiple of both BUFSIZ and the length of the string to be printed. It appears that GNU yes will not do that and simply does unaligned writes in this case, which is likely to be considerably slower (possibly slower than write would have been).
> It appears that GNU yes will not do that and simply does unaligned writes in this case, which is likely to be considerably slower (possibly slower than write would have been).
Why would it be slower to do a single, say, 8190 bytes write instead of 2730 3-byte writes?
And it's also a source of many bugs (in the past and likely in the future as well). In most use cases today I'd rather have a slightly slower userland which is easily read (and audited) then one which compromises quality for speed in edge cases.
This work certainly has value but it is frustrating to keep up with everything being changed. And if changes affect me do the fashions and attitudes of those making the changes have synchronicity with the way that I use computers?
In the old days I think we would have left yes in c because the compiler will build it on any platform.
My first experience with Linux is porting land.c to one of the commercial Unix. Now I first met TCP/IP on a 3B2 and later met systems sold by SMI and DEC and SCO and we all mostly constructed packets the same way and a guy called Stevens had written some nice books about this that everyone had. I think I recall the commercial and free BSD also did things the usual way. But whoever figured out this interesting phenomenon land.c demonstrated happened to be a Linux user and this platform had some different ideas about it. I saw rewriting the relevant parts as an annoying, menial task.
The /r/programming discussion of this is interesting [1].
Someone does a Go version and gets the same speed as GNU yes. Someone else tries several languages. This person got the same speed in luajit, and faster in m4 and php. Ruby and perl about 10% slower, python2 about 10% slower still, and python3 about half that. The code is given for all of these, and subsequent comments improved python3 about 50% from his results, but still not up to python2.
I had to smile, as this thread is a microcosm of programming language stereotypes: with the python programmers tweaking code to get that extra 10% (nobody mentioned pypy..), someone trying to get javascript to work in a long running program, of course a rust implementation that isn't working well just yet (but we're all rooting for it.. set that compiler flag and rerun).. One comment about perl (turns into a one liner..). Nobody bothering to redo the Lua, Ruby and PHP. And for fun somebody throws down some fortran.
Cow means "clone on write". It's a generic wrapper for types that have both a borrowed version and an owned version. For example, `Cow<'a, str>` can hold either an `&'a str` (a borrowed string) or a `String` (an owned string), and a `Cow<'a, [u8]>` can hold a `&'a [u8]` (a slice of bytes) or a `Vec<u8>` (an owned vector of bytes).
In that Rust program, Cow is being used here so if the user provides an argument it ends up with an owned vector that contains that argument plus a newline, otherwise it ends up with a borrowed byte slice that contains "y\n". That said, there's not really a whole lot of point to using Cow here since allocation is a one-time cost and nobody cares if yes starts up a few microseconds slower, but the usage of Cow is limited to just a couple of lines so it's not really complicating anything (the actual write() call just ends up taking a &[u8] byte slice, so the usage of Cow is strictly limited to main()).
Oops, typo. I've updated my comment accordingly, thanks.
Also, I didn't realize it was Clone-on-write. Interesting, and the documentation does confirm this. I say "interesting" because the actual operation involved is `to_owned()`, not `clone()`, seeing as how `clone()` on a `&str` just gives you the same `&str` back.
One night for fun, I wrote a fizzbuzz (without actual printing) in almost all of the languages mentionned here (except fortran), and benchmarked it with 10000 executions each.
10k was just sufficient to get actual differences. Of course I went crazy and did something like 10M, but waiting 10 minutes to get the same relative differences didn't bring anything.
Remember this is without any printing: C is fast, if I remember correctly it was something like 0.015 or 0.03s while at the other end of the spectrum I had to wait 0.1s for JavaScript. C results varied from simple to double because of the OS I guess because it was so quick.
It isn't a really significant benchmark though, for any language. It was one of those nights. Ah, and I remember a large difference between go run and go install.
Actually the second python version is about on par, however it's running on a slower system. He got 6.78GiB/s from GNU yes and 6.76GiB/s with python3. Some one later compares it with python2 which in fact is slower with this codebase.
m4 is orders of magnitude slower (notice the different unit).
If you write the Python script using bytes (not Unicode), then Python 3 is faster than Python 2, at least for me. Python 3 is 20 % slower than GNU yes and Python 2 35 % slower than GNU yes.
e: I think in your last sentence you are comparing results from different computers.
I was under the impression Python was magnitudes slower than compiled native languages, even when you aren't abusing Pythonic features that are computationally expensive like list comprehensions. And this isn't even in Pypy where this loop would be JIT'd and then run natively anyway, unless cpython does some kind of JIT? Is is just that the buffer is so large that the kernel handler is taking so much of a % of the runtime that the inefficiencies of the language stop mattering?
If anyone, like me, is wondering what "yes" is used for. You can use to pipe "y" to commands that require interactivity, so if you just want to say "y" to all the inputs, you can use "yes" to do this:
Yes, but that's the only error it suppresses, and that's because "not existing" isn't really an error for a tool that's trying to delete something. In my example foo/bar existed but I didn't have permissions to delete it, and as you'll see it printed out that error.
Back when I worked at the Genius Bar at Apple Stores I saw a customer come in and talk to a 'Genius' about their MacBook being "slow". After a quick bit of troubleshooting, he just opened up 4 terminal windows an ran yes in all of them, and did some hand wavy explanation about diagnostics.
> In 2006, the yes command received publicity for being a means to test whether or not a user's MacBook is affected by the Intermittent Shutdown Syndrome. By running the yes command twice via Terminal under Mac OS X, users were able to max out their computer's CPU, and thus see if the failure was heat related.
For reference, I had that exact same issue on my MBA - went on for years. Many times while watching youtube or doing something Garageband related.
At the time I had 10.11 on it, but that OS had itself been upgraded from 10.10.
When 10.12 came out, I decided to install fresh - so I backed everything up and installed onto a new SSD drive. The issue has now gone away completely....
Hm, my old Dellbuntu laptop does the same. It's been dropped a few times in its 10 years of service, and now sometimes will shut down if I tap it too brusquely, regardless of the temperature. The BIOS will report it overheated, and i don't know what's going on.
the limit isn't the processor, it's how fast memory is. With DDR3-1600, it should be 11.97 GiB/s (12.8 GB/s)
I don't understand this reasoning. Why is it being limited to main memory speed? Surely the yes program, the fragments of the OS being used, and the program reading the data, all fit within the L2 cache?
It is NOT limited to the external RAM speed and the best proof is that it actually uses over 40GB/s of memory bandwidth.
For each byte of data passing through pv:
1. the byte is read from yes memory to CPU registers by the write syscall handler in the kernel
2. the byte is written to kernel's internal buffer associated with the pipe
3. the byte is read back in the read syscall called by pv
4. the byte is written to a buffer in pv memory
5. and thas's the end because write syscall executed by pv on /dev/null very likely doesn't actually bother reading the submitted buffer at all
edit: Actually it might only be 20GB/s because on Linux pv seems to use the splice syscall to transfer data from stdin to stdout without copying to userspace.
This is also the reason why further "optimization" of this program in assembly was a fool's errand: the bulk of CPU load is in the kernel.
All good valid points, but I'm still a bit surprised that the limit is not higher, I thought that the L2 cache was over an order of magnitude faster than main memory (plus, as someone pointed out in the reddit thread, the peak memory performance should really be double the quoted 12GB/s due to dual channel memory).
The actual throughput then, once you include OS copying, is either 2 or 4 times the quoted speed (depending on splice usage), so we're either at main memory theoretical speeds, or double main memory speeds. Intuitively, I'd still have expected that it should be a larger multiple.
(A quick search can't find me any reliable Intel L1/L2 cache speeds/multiples to quote, so I admit this comment is more speculation than it should be!)
L2 cache and copying "y" bytes have very little to do with this; I suspect if you could produce high-granularity timings it would almost all be in the syscall overhead.
A read() syscall takes longer than a getpid() syscall because read() has more work to do, it actually does a data copy of len bytes, which takes some time (and will be faster/slower if data is cache hot)
What we call the "syscall overhead" is what happens before and after the actual data copy, switching between user and kernel mode.
You make that overhead negligible by calling read() with a large size.
(Many, many years ago I was working on the Zeus web server, and we went to surprising lengths to avoid syscalls for performance.)
Snap!
IIRC, ZWS used shared memory to mirror the results of the time() syscall across processes, to save a few nanoseconds on some operating systems :) That was before Linux and other OSs used techniques like the vsyscall/VDSO mentioned in the stackoverflow discussion...
Very rough numbers - the 9.13/9.33 difference flip-flopped when I ran the commands again. Binding both processes to the same core is definitely a performance hit though. There might be some gain through a shared cache, but it's lost more through lack of parallelism.
I tried 2/4 as not sure how 'real' cores vs 'hyperthread' cores are numbered. These numbers are from a i7-7700k.
Assuming pv uses splice(), there is one only copy in the workload: copy_from_user() from fixed source buffer to some kernel allocated page, then those pages are spliced to /dev/null.
If the pages are not "recycled" (through LRU scheme for allocation), the destination changes every time and the L2 cache is constantly trashed.
I only learned of pv from this article so I can't speak much about its buffering. I would guess that the kernel would try to re-use recent freed pages to minimise cache thrashing. But anyway, on the 'yes' side, the program isn't re-allocating its 8kb buffer after every write(), so there's a lot of data being re-read from the same memory location.
Sure, but this is a pipe between two tiny processes, hopefully with very little else being run on the computer at the time (otherwise all bets are off for any benchmarking). There's no kind of 'real' I/O going on (in terms of stuff like screen, disk, NIC, and so on)
There's no reason that the L2 cache needed to be flushed at any point - the caches are all dealing with physical memory, rather than virtualised address space, so the fact that there are two processes here shouldn't stop the caching from working.
I am probably way behind the current state of the CPU judging by the downvotes I got so if you are saying there is no reason and the data can be written into a device without leaving the CPU I will just concede my ignorance.
Don't fret about the downvotes, these magic internet points aren't redeemable anywhere :)
It's completely possible for the data to not (all) leave the CPU. If the caches are large enough, then the pages full of "y\n" will be still resident in the cache when the next iteration of the program overwrites the same pages again. Then the CPU has no need to send the original page out to main memory.
If you did
for(;;) buffer[(i++)%size] = 'y';
then you'd be correct. However, you do i/o. And the 'y's appear at the i/o driver and have to be made visible for the device it's driving, which can be anything, including a process on another CPU core in another socket. If they remained in the issuing CPU cache, I fail to see how the destination device could possibly see them. There are some devices which can snoop the cache (like SoC GPUs and other CPU sockets on some archs) but the snooping is much slower than memory bus. Writing to memory is the only way which a) guarantees the data is available elsewhere and b) is the fastest.
You could make "yes" faster with the tee() syscall. Keep duplicating data from the same fdin (doesn't actually copy) and it becomes entirely zero-copy.
next step is playing tricks with memory mapping so that the buffer is large in virtual memory but only takes a single page in physical memory, fitting in L1 (at least on the read side).
I've realised straight tee() is actually wrong - it works fine for piping to something but "zcyes > /dev/null" won't work, it needs vmsplice() like mrb said.
It's come to my attention that lower numbers are not better here. I have filed a bug, we'll get to the bottom of this shortly. I want to apologize to all our users, this issue does not reflect the values and principals we at Pure JavaScript `yes` hold dear https://github.com/Sequoia/yes/issues/3
I was not going to post this because hacker news has this ethic (?) of down voting anything that seen as not positive. Perhaps we should have discussion about that, I'm not sure that's a good thing but I'm not in charge here.
The top comment is:
"It's a shame they didn't finish their kernel, but at least they got yes working at 10GiB/s."
which as an OS guy, someone who has been working on Unix for 30+ years, as a guy who was friends with one the QNX kernel guys (they had perhaps the only widely used microkernel that actually delivered), that's hugely amusing and spot on. The GNU guys never really stepped up to being kernel people. Bitch at me all you want, they didn't get there. It's a funny comment, especially coming from reddit.
> hacker news has this ethic (?) of down voting anything that seen as not positive
We must not be reading the same Hacker News...
Anyway, the comment you're quoting is just a shallow jab that belittles the GNU developers' work without contributing anything new or meaningful. It's telling that you had to spend two paragraphs to justify cross-posting it here.
You say that and then immediately become an example of what hes talking about. This "shallow jab that contributes nothing new or meaningful" is, in some circles, known as a "joke." I'm continually frustrated by people who think that misinterpreting comments as harmful is a useful activity.
Not really -- I didn't downvote that comment. And I still dispute the notion that negativity is rare or always shunned on this board: to the contrary, it's so commonplace that an actual rule [0] had to be added to try to sway things in the other direction.
Jokes have their place, but bringing up the failure of Hurd in every GNU-related post is banal. And saying they "never really stepped up" to your level as a mighty kernel developer, as if the people who brought us glibc and coreutils lack an understanding of OS internals, just seems rude and curiously out of touch.
So negativity here is not rare, when people don't like something they are fast to jump on it.
What I was trying to get at is this, if you care about your upvotes, hacker news promotes a sort of hive mind. Which is somewhat like "say only nice things unless you are clearly swatting down something that is obviously wrong".
Which is mostly fine, fantastic in fact. I'm fine with it, hacker news is really pleasant because of the (more or less ) quality of the posts and especially the quality of the comments. I'd much rather have it be this way than a free for all, those go bad pretty fast.
So I'm for the hive mind, I was just pointing out that you can't make jokes and be upvoted. The joke wasn't banal at all IMO. GNU has done a lot of good, I've been there since the beginning and paid attention along the way. They have also been pretty self serving with their choices, every project has to sign away their rights and then GNU takes full credit for the project even though they had nothing to do with it other than it being GNU troff for example. Given their tendency to take credit for stuff that they didn't do, and their claim that they can do an OS but clearly can't, that joke is funny as heck. If you don't get that, sorry, you haven't been paying attention.
() The quality of the posts in the area of programming, especially systems programming, is spotty. Some stuff is great, stuff I didn't know (there is a lot of that here and I'm very grateful for it, it's why I stick around), some stuff is meh, and then there is stuff like "wow, look at $OBVIOUS, isn't that cool?" that gets upvoted. That last one I just don't get, but whatever, the good stuff is good. The signal/noise ratio here is better than any other programmer oriented site I've found.
Years ago I read a similar experiment about max. CPU data flow. Guy was testing how much data can his CPU pass in a second. He was writing it in C, using some Linux optimization, optimizing code for CPU caches, using some magical C vectors that are optimized for such purpose. He got some help from someone working at Google. I tried to find that post but never succeeded. Does anyone here know it?
This will attempt to open a child shell process with whatever output of "yes" is, interpreted as shell script.
But before that, parent shell has to buffer until EOF. With "yes" output being unbound, this means unbound buffering / memory growth. That is, until OOM killer take notice and shut it down.
Whole thing will probably take few seconds to a minute (depending on how much free RAM there is vs. how fast it is), peg a single CPU core in the process, and will recover cleanly (except for shell in question being terminated).
...unless system in question has single CPU and large+slow swap. Then yes, bring-it-to-its-knees.
Single CPU is fine nowadays. It will be slowish if you ran it with no nice or priority.
RAM can be tweaked by preventing Linux heuristic overcommit via sysctl vm.overcommit_memory=2. (0 is only recommended if you do not run broken applications. It so happens that many JS and Java VMs are broken on memory pressure.)
I'm guessing it tries to read the output and return it.
Since yes doesn't actually terminate it's going to generate a massive temporary variable to store it's continuous output.
That kind of question doesn't make sense for open source code.
Somebody wanted to optimize 'yes', so they did. There doesn't need to be a good reason, just like there doesn't need to be a good reason for a person to read a certain book or watch a certain movie, other than they want to do it.
The question was "do we need it? (for a specific use case)" rather than "should we do it?".
And it does make sense for Open Source code because the resources are limited hence other features and/or bug fixes might be more important than pushing data at full speed
> And it does make sense for Open Source code because the resources are limited hence other features and/or bug fixes might be more important than pushing data at full speed
That's not how open source works, though. There's not a group of people obligated to work on the GNU utilities. There's not a central project manager ordering people around telling them what changes to make.
Declaring that 'yes' is generally fast enough already doesn't imply that it was fast enough for the person who spent their time optimizing it. Somebody needed (or just wanted) 'yes' to be really fast, so they did the work and submitted the changes back.
> If one just makes it faster without major benefits that patch is likely to be rejected
IF the project were overwhelmed with incoming patches, then I would agree that there's probably higher priority changes than optimizing 'yes'.
But I'm almost certain the GNU project is not being overwhelmed with changes to the core utilities. Everything else being equal (code quality, readability, test coverage, etc.) there's really no reason to reject a patch that demonstrably improves the project, even in some silly way like making 'yes' really fast.
This is software engineering. Thinking about the tradeoffs and whether they are appropriate or warranted is bread and butter. It very much makes sense to ask whether all of this is merely optimizing for a benchmark at the expense of other factors that might turn out to be more important, such as long term maintainability, portability, behaviour in some common cases which are significantly unlike the benchmark, and so forth.
Several people have made some of these points here and in other discussions; and not only do they make sense, they are an important part of the discipline.
Is the optimized implementation documented well enough that other people coming to it anew can understand it in another 10 years' time? How many of the magic constants have to be tweaked as the operating system evolves? Is it clear to maintenance programmers how they need to be tweaked? Does optimizing for the benchmark skew the implementation too far away from one common case where one is only wanting a few "y"s (because, say, the program being told "y" only asks 17 questions) resulting in every invocation of the yes program rapidly generating KiBs or even MiBs of output, that lives in pipe buffers in kernel space unused and only to be thrown away? Does it make more sense to put buffer size optimizations in the C library where all can benefit? What is the benefit of optimizing for Linux-only system calls on GNU Hurd? Will we optimize yes for GNU Hurd, too, in the same codebase? How much conditional compilation will we end up with? If the C library is improved to do better in the future, is it a problem that the optimized for benchmark program no longer automatically gains from it? How much of a problem is it that a GNU program is now not portable to systems other than Linux, and is locked into one operating system?
And what about other benchmarks? Many have noted that this benchmark pretty much relies on the fact that the output of the yes program is simply thrown away, and no real work is being done by anything else on the system. What about a benchmark where it is? What about a benchmark where it is important that yes issues a system call per line, yielding the CPU to other processes so that they can process that line before yes bothers to print the next? What about a benchmark that measures kernel memory usage and treats lower as better? What about benchmarks that measure how small the yes program is; not just in terms of code but in terms of memory, I/O, and CPU usages; because the memory, I/O, and CPU should actually be given to more important programs than yes on the system that are actually the system's main function? What about low impact?
Yes, it's fun to focus narrowly and optimize yes so that it generates as much output as it can for one specific toy usage. But in engineering one has to ask and think about so much more.
A detail you seem to be missing is that you are not limited to a shell script. The shell sets up the pipeline, but the members of the pipeline can be written in arbitrary languages and just have each stdout linked to the next processes stdin.
As a result you can process very large volumes of data and consume (not waste, consume) significant system resources to perform your processing.
I would just pre-allocate a static array of "y\n" of size BUFSIZ, write it out in a loop, and call it for the day, skipping the whole malloc and filling loop business.
Make the static array BUFSIZ * 1024 to trim the syscalls by a factor of 1000.
Have a few pre-written arrays for the common cases: "y", "Y", "n", "N", etc. Those are the fast cases (or the benchmark optimized cases, like, what Volkswagen did). Have another pre-allocated static array to fill in with other input.
Let me check that I have correctly understood the scenario you're anticipating.
1. Someone uses "volkswagen" as a verb meaning "cheat in benchmarks".
2. Volkswagen takes them to court, on the basis that it is slanderous or libellous to associate Volkswagen with cheating in benchmarks.
3. Counsel for the defence reminds the court that Volkswagen were in the news for a protracted period for a benchmark-rigging that saw them hit with a multi-billion-dollar fine, slashed a third off their stock price, saw their CEO resign, led to there being a "Volkswagen emissions scandal" page on Wikipedia, etc., etc., etc.
4. [THIS IS THE BIT I'D LIKE YOU TO FILL IN FOR ME.]
By volkswagen arguing that this is industry standard behaviour, and the entire media campaign is already libel and slander.
And they’d have a pretty good case with that, considering the original report all this media campaign was based on called out 6 carmakers, but the media (and the poster) only call out Volkswagen.
No, but you don’t wanna get sued for libel and slander. If VW can realistically show that it is industry-standard behaviour (which is pretty obvious), then they might even have a chance to win.
The VM subsystem is much slower. Also possibly a not-up-to-date GNU grep, on my (ancient) MBP I get 26MB "native", 780MB GNU and 2.8GB for TFA's C program (2.9 if I replace the malloc by a stack-allocated array, weirdly enough)
With that malloc overhead, I expect GNU yes to be slower when only a few bytes are read from it.
So, what's the distribution of #bytes read for runs of 'yes'? If we know that, is GNU 'yes' really faster than the simpler BSD versions?
Also, assuming this exercise still is somewhat worhtwhile, could startup time be decreased by creating a static buffer with a few thousand copies of "y\n"? What effect does that have on the size of the binary? I suspect it wouldn't get up much given that you can lose dynamic linking information (that may mean having to make a direct syscall, too).
Unless you are statically linking, one malloc doesn't significantly affect your startup time. When only a few y's are read, time is going to be dominated by ld.so, by a large margin.
Measurements are really noisy, but I seem to get significantly better numbers than that when I use fsplice() on a pre-generated few pages of file data instead.
I thought this was a fascinating read but it left a serious question lingering in my mind, which is a little out-of-scope for the article, but I hope someone here can address.
Why did the GNU developers go to such lengths to optimize the yes program? It's a tiny, simple shell utility that is mostly used for allowing developers to lazily "y" there way through confirm prompts thrown out by other shell scripts.
is this a case of optimization "horniness" (for lack of a better word) taken to its most absurd extreme, or is there some use case where making the yes program very fast is actually important?
The stated use case for the perf improvement was "yes(1) may be used to generate repeating patterns of text
for test inputs etc., so adjust to be more efficient."
I've personally used it for generating repeating text and filling disks in system testing, so I appreciate it being faster at those tasks. I also sometimes use it as a signal generator for a hacky load generator, like so:
But doesn't this make the typical use case (just a few "yes"s needed) slower, since first it has to fill a buffer?
I would write() the buffer each time it gets enlarged, in order to improve startup speed.
Also: The reddit program has a bug if the size of the buffer is not a multiple of the input text size.
And it's increasing the buffer by incrementing one at a time, instead of copying the buffer to itself, reducing the number of loops needed (at cost of slightly more complicated math).
If your only overhead is filling an 8K buffer, I don't think your user is going to care. Taking one microsecond instead of one nanosecond doesn't matter all that much when you're going to lose way more than that in pipes, the kernel, the program you're piping it to, etc.
Maybe filling a disk or flooding a network connection as in
yes | ssh server "cat > /dev/null"
But yes take arguments so there might be more use cases:
$ man yes
NAME
yes - output a string repeatedly until killed
SYNOPSIS
yes [STRING]...
yes OPTION
DESCRIPTION
Repeatedly output a line with all specified STRING(s), or 'y'.
> But doesn't this make the typical use case (just a few "yes"s needed) slower, since first it has to fill a buffer?
One could put a write in the memcpy loop, so that first it writes one copy of the string, then two, then 4, 8 (etc.), meaning time to first byte is short, but it quickly gets up to the eventual asymptotic speed with a full buffer.
GNU yes is fast because it is coded with the assumption that it's not answering any real question, such as "can I combine this free code with a proprietary program?" or "Would you accept the following monstrous patch to GNU Coreutils /bin/true without a copyright assignment?"
How about a /proc/bin/yes for this? Like most /proc files, it would appear to be empty. Executing it would involve a fs/binfmt_proc.c file in the kernel source, which would be a handler for this sort of executable. That would get the job done entirely in the kernel.
man, I just spent like 8 minutes today writing a python script to use up all the disk space on some servers (part of ops readiness testing) when I could have just used this trick.
`yes` will help me on the "see what happens when something uses all the CPU and memory" test case. Thanks Reddit/HN!
dd can also be useful for that kind of thing since you can use a source like /dev/urandom to generate random bytes if you're trying to avoid compression and the adjustable block sizes can be optimized for the underlying storage system.
I did actually use dd to begin with but it didn't work as I wanted it too right away (can't remember why, probably something SUSE related) and by then I'd used up 6 mins of the 15 mins I gave myself for the job.
I also had to do the same test on windows, so python won
It's almost funny that you try to refute, or rather, dismiss, criticism of PHP with "but it's fast", when none of what you cite even mentions that. I just want to say: STFU.
It doesn't in fact imply that PHP is slow. What it says about PHP (besides "php seems to fail in most categories" which doesn't imply any single specific failure because most != all) is this: "But php has both an ugly syntax, horrendous stdlib, and fame of security issues." You will notice that none of those specific complaints is about speed.
None of those comments are complaining about PHP being slow at repeatedly writing a string whose length is a multiple of the page size to standard output. That is a pretty niche case, and hardly refutes even the complaint that PHP is slow, much less all the complaints about PHP that aren't related to its speed.
FWIW, when I run the tests there's no substantial difference between Perl, LUA, or PHP.
>>>Did you notice PHP outperforms any other scripting languages?
I am not sure what was the last time I had to choose a language by a single dimension. If we are talking about performance, scripting languages rarely shine. On the top of that, performance critical things in scripting languages are often implemented in a compiled language (like Pandas for example).
while(1){write();} is about as far away as it gets from actual application code. This topic may be fun, but remains an utterly pointless micro-benchmark.
>OS X just uses an old NetBSD version similar to OpenBSD's
>NetBSD's is 139MiB/s, FreeBSD, OpenBSD, DragonFlyBSD have very similar code as NetBSD and are probably identical,
It really depends on how fast your memory and processor are.
Everything there was benchmarked with an i7-4790 with DDR3-1600, and very very little running in the background.
Two orders of magnitude difference can't easily be blamed on CPU, in this case it depends on how optimized yes is. I have some older version of coreutils on one machine here and performance is similarly abysmal.
It is common in markdown formatting to indicate a command line with backquotes. It tells the markdown compiler to apply special css and possibly syntax highlighting. So, a lot of people who write about shell commands and setup tutorials, or use markdown on something like github will automatically identify it as a command.
tl;dr someone who doesn't understand how i/o works gets a small insight into how memory and a cpu work and decides "Buffering is the secret" and "You can't out-optimize your hardware"
Can we have a new flag for posts by people who don't know what they're doing so I can skip them? I am serious.
I think this subthread is a good example of what can happen if we quote the guidelines without context. It's a counter-intuitive trap I've also fallen into, so I wanted to mention it.
In this case, this subthread could've been nipped by mentioning that it's ok to be abrasive to an extent as long as it's followed by a substantive comment. It's not diplomatic, but it's not against the rules.
The guidelines are the framework the community is built around. They're not sufficient to explain why the community is built around them. (We probably wouldn't want the guidelines to be so complete, either. It's better to explain the motivations on a case-by-case basis.)
It's a valid question but looking at his posting history doesn't convince me that there's a way to word things which would result in a more productive response. Note that I lead with the observation that there wasn't anything to his post other than the attack — I don't think there's too much more to say about that before hitting diminishing returns.
It's a valid question but looking at his posting history doesn't convince me that there's a way to word things which would result in a more productive response.
You could be right. But if you were in their shoes, you'd want to be given the benefit of the doubt.
The commenter was not being combative or abusive in any way. They did point out the other person was wrong politely. They still got downvoted to hell.
People don't come to HN for civility. They come because they can be smug polite idiots around other smug polite idiots and "the rules" protect and enforce this behavior.
If you don't believe that disproving something stupid is hard, I give you Donald Trump. Sometimes you just need to speak out against stupidity without providing a groveling mathematical proof.
A cursory glance at that commenter's recent history shows little evidence of politeness or intention to be polite.
You don't have to like this community but surely you have better things to do than making it worse by trashing it like this?
Also, invoking US politics drama to score points in this discussion topic shows quite a lack of sense of proportion. We are talking about a single person's casually-communicated recollections of their past experiences with a computer operating system. It really isn't worth getting wound up about.
This is a great example of not reading charitably. You're reading a bunch of stuff that just isn't in the text you're replying to. Who said anything about big data? Who is trying to write journal publication quality comments? You seem to have an axe to grind that has nothing to do with this thread.
My reading was just that the parent commenter was telling a story by relating an interesting anecdote. That's exactly what I come here for. People won't do that as much if the common response is weird aggression like your comments in this thread. That would be a shame.
You could have just written a little comment saying, "in my experience, you're incorrect about the speed of reading from /dev/zero". The difference in your remembered experiences and your conversation about it would have been interesting to me. Instead you picked a fight, which is not interesting.
> in my experience, you're incorrect about the speed of reading from /dev/zero
> Instead you picked a fight
So I harshly pointed out that they're mistaken. Then it was pointed out to me that I wasn't being nice. So I responded by showing why I didn't feel the need to be nice and provided my reasoning for it.
You can point out how people are wrong and people won't have a problem with your posts. Be a jerk about it and people will. Hacker News, for its (many) faults, doesn't ascribe to the "you can be a sneering asshole but if you're Right you're Right". It's very possible to be both right and a jerk. It's incumbent upon you to avoid being a jerk or to not whine when people don't like jerks. This is basic-functioning-person stuff.
I don't know who is right or wrong. I am a software developer and familiar with Unix/Linux. I actively try to be well informed about other topics related to this. When engineering I actively try to remain emotionally dispassionate. I understand what the argument is about and I see how in a technical sense either side could be "correct" here.
Despite all that, the level of vitriol and defensiveness you have demonstrated makes me want to agree with everyone else. I am curious what the opinions are of people less informed on this topic and more emotionally in touch. You damage your side of the argument, not with your content, but your portrayal of the content.
Some of the things you said have merit, but they are ignored and derided because they are attached to you and you are attached to vitriol and vitriol rightly gets ignored and derided. The best thing you can do for anything resembling your side of the argument is to just back down. If you don't say anything for a day or two it will all just blow over and you can try again later when people are more willing to listen. You can try approaching the conversation like an engineering problem and perhaps see that manners and politeness are to conversation as lubricant is to an engine (or perhaps high clocks speeds to software). You can have a conversation without manners, run an engine without oil or run software on low clocking parts, but it won't turn out as well as it could have.
I think "you don't know what you're talking about" is substantively different than "in my experience, you are wrong about this". Firstly, lots of people who do know what they're talking about are wrong about lots of things. Making it about the person instead of about the thing was (for me) the problem with your comment. Also, "in my experience" is not just polite fluff. I have come across diminishingly few things in computing that are not dependent on circumstances.
The interesting version of this conversation would have been you saying, "no that sounds wrong to me", followed by a discussion of (to the best of your memories) what kind of hardware, drivers, kernel and compiler versions, etc. you were each using. Maybe you would have convinced me that there is no combination of those things for which the original commenter's experience could possibly have been right. But your throwaway ad hominem comment didn't, and couldn't ever, accomplish that.
> But notice how you still post and the downvote brigade still obliterates your score.
And even posts with actual data are being downvoted :)
This only goes to show just how subjective and emotional this community has become, where everyone just wants to share "but in my opinion" or "but in my experience" without having to back up any of their arguments, or even have any of their arguments or claims challenged and questioned and called out.
This essentially turns concrete, testable claims into happy dreaming about technology. And that's really what this has become: everyone sharing their idea of what something should be and presenting it as fact.
It has literally nothing to do with people saying "in my experience" or even you choosing to challenge an argument. It has to do with you being a jerk about it. The people you interact with through your keyboard are real and how you treat them matters.
The only point you've made today is that self-righteousness exists and exhibits itself as rudeness. And you know it's rude; you're actively trying to downplay it by calling how you've chosen to approach other people as "harsh". You should use this awareness to be better, not to continue to be unkind. Because, ultimately, this is a you problem.
> And even posts with actual data are being downvoted
Of course they are, its not your data that people don't like. They aren't downvoting the post, they are downvoting you. People don't like you, and anything you say will be downvoted at present. Remove yourself or mod might.
You have made no points today, unless you are trolling then you have angered many and that is your measure of success. I will not respond to you further.
> Seriously, is HN fostering a culture where it's OK to spew pseudo-intellectual garbage as long as you're being nice?
The comment I called out had no technical content, argument, etc. and was only an attack on someone's credibility. That contributes nothing of value and in my book is worse than “pseudo-intellectual” because it's not even trying to contribute anything other than noise to the conversation.
You would not be getting such intense negative reaction if your comment had been “Do you recall which operating system that was?” or “How long ago was that?”. That's how an actual engineering conversation should go, especially in a field such a long history of people uncovering unexpected performance issues caused by unintended consequences.
As a simple example: in the early 2000s, a colleague reported that tools like grep and wc were running slower than our storage. That seemed odd since those are usually considered fast and certainly hadn't had that problem years earlier but a little testing confirmed that the switch to assuming Unicode by default had made a previously unoptimized code path the default and I'm sure the person who did that had assumed that someone would have noticed if code which had been shipped for years was that slow. The fact that for many years before and after you could mentally bucket those Unix shell tools into the “fast” bucket didn't change the fact that there was a multi-year period where you had to check.
I have no reason to doubt that joosters' reported experience actually happened, especially given how rocky the 90s Unix world was. That was back when you couldn't even rely on a vendor's own C compiler not producing pathologically bad code. Someone having a /dev/zero implementation which triggered extra or slow kernel transitions, unnecessary access checks or audit logging, etc. would be disappointing but far from shocking[1].
More to the point, what do you hope to gain from such an unnecessarily combative style? You're not preventing any bad outcome, helping yourself or others improve, or otherwise doing anything other than lowering the level of discourse here.
Oh dear. So sorry that my memory is crap enough to not provide you with the exact details you require.
...every comment is trying to look like a paper published in a highly acclaimed journal
It was an anecdote about my previous experience with 'yes' and /dev/zero, it seemed somewhat on-topic. I'm not trying to justify anything or win magic internet points.
If you're expecting journal-quality posts, perhaps you should go read a journal and not an internet forum chat.
You're free to spout your vitriol just like the people downvoting you into oblivion are free to use their downvote ability to express their dislike with your attitude. Free speech means you're free to express your ideas, it does not mean everyone else is forced to listen to them.
It's hard for me to view your comment as anything other than flamebait. You're calling out someone in an aggressive tone, "You don't even know what you're talking about! Bah-humbug!" and then crying crocodile tears that other people are "being mean" to you.
> You're free to spout your vitriol just like the people downvoting you into oblivion are free to use their downvote ability to express their dislike with your attitude. Free speech means you're free to express your ideas, it does not mean everyone else is forced to listen to them.
Did I disagree with that anywhere? Do you have a reference comment by me to point to?
> It's hard for me to view your comment as anything other than flamebait.
Ok.
> You're calling out someone in an aggressive tone, "You don't even know what you're talking about! Bah-humbug!"
I said I don't think they know what they're talking about. Without any of the shenanigans you portray here.
> and then crying crocodile tears that other people are "being mean" to you.
What? Where did I claim anyone is being mean to me? I made my argument to everyone who asked why I was being combative.
So again, do you have a reference comment by me to point to?
You admit that you don't have enough details to know exactly what they're referring to, yet you're confident enough to call their personal experience "obviously incorrect"?
It's not an argument, it's an explanation based on personal past experience. You can't find any evidence, and you don't need to, because the claim being made won't influence how you do things today anyway.
People's personal experiences can be interesting, and that adds to the conversation. Your overwrought criticism adds nothing. Kindly stop.
> You admit that you don't have enough details to know exactly what they're referring to, yet you're confident enough to call their personal experience "obviously incorrect"?
I didn't admit to not having enough details. You have two tools:
1. Reasoning about the abstractions and implementations of how yes and /dev/zero work: the claims become obviously incorrect, even their own tests later in the thread show that on a "modern system".
2. Actually running the tests they claim (which would prove them correct): no freaking way to do that. Even they do not offer any numbers to back their own claims because everything in the argument is a maybe or a remember or a might or a may have.
> People's personal experiences can be interesting, and that adds to the conversation.
People claiming their personal experiences are fact without proving anything is hugely problematic. Remember that woman claiming her child got autism due to vaccines? Remember that US president claiming a child got autism due to vaccines?
I remember seeing the earth as flat when I was working as an astronaut on the ISS. Go check my "personal experiences".
I really don't see how it's "obviously incorrect" when yes only needs to make system calls for writing, whereas /dev/zero needs to make system calls for both reading and writing. Furthermore you're assuming that both are highly optimized, whereas it's entirely conceivable that /dev/zero is less optimized.
If what you're trying to convey is that abstract reasoning based on fundamental principles without any consideration of real-world complications is a superior approach to determining what's true, well, I reject that.
> I really don't see how it's "obviously incorrect" when yes only needs to make system calls for writing, whereas /dev/zero needs to make system calls for both reading and writing. Furthermore you're assuming that both are highly optimized, whereas it's entirely conceivable that /dev/zero is less optimized.
When you're "only writing", there is a counterpart that needs to read, and you have to wait for that, same as /dev/zero. It's more conceivable /dev/zero is more optimized than yes: you know, because of how pipes and streams are an essential part of a UNIX or UNIX-like system.
> If what you're trying to convey is that abstract reasoning based on fundamental principles without any consideration of real-world complications is a superior approach to determining what's true, well, I reject that.
I agree with that. But the original comment only gives us few options to check for its validity, and some of those are not feasible.
1. Actually finding all those systems, version-specific, combinations of userspace tools, setting them up and running the tests on hardware that was at OP's hand "back then": Not feasible.
2. Reasoning abstractly would tell us quickly that the original comment is plain wrong.
3. Running the tests on a modern system (some default linux install or whatever) shows that (2) is even more probably correct.
Is it "more conceivable" or is it "obviously incorrect"? You switch freely between arguing that your conclusion is more probable and arguing that it's obviously correct.
I agree with you that your conclusion is more probable. But that's an entirely different thing. It is more probable that a Tesla P100D is faster than a Pinto. It is not obviously incorrect that one time a Pinto beat a Tesla, maybe because the Tesla was dragging a bunch of rocks and the Pinto was dropped off a cliff.
Make up your mind what your argument is: is the statement in question merely unlikely, or is it "obviously incorrect"? Once it is, make your argument without being a jackass.
I responded in the terms you were using, hoping that it would become clearer to you.
> It is not obviously incorrect that one time a Pinto beat a Tesla
Yes it is, because OP forgot to mention the Tesla was dragging a bunch of rocks and the Pinto was dropped off a cliff. So was I supposed to have the power to read OP's memories and go back in time to check whether they provided us with all the information we needed to check their claims? No, I worked with exactly what's in the comment.
So without catering to your language/terms you use: it is obviously incorrect. Even OP's own comment later in the thread (on a modern system though, mind you) shows that: https://news.ycombinator.com/item?id=14544002
You appear to be incapable of understanding any middle ground between "obviously incorrect" and "obviously correct." Is the concept of uncertainty that foreign to you?
Really? Because when it comes to something testable, like in this case either running a test of yes vs /dev/zero or looking at the source code of everything involved, there isn't any uncertainty that OP's statement is obviously incorrect.
The only uncertainty is OP's use of "back then", "from memory", "maybe", "could have", etc... without providing any evidence to support their statement.
They were talking about things in the distant past. (Distant in computer terms.) The only way there isn't uncertainty is if you tried all the systems that were available at the time in question.
The claim is testable, in theory. You didn't test it, nor did anyone else here, so the fact that it's testable doesn't remove any uncertainty.
> If what you're trying to convey is that abstract reasoning based on fundamental principles without any consideration of real-world complications is a superior approach to determining what's true, well, I reject that.
People are suggesting that it's because of the use of cat. So, if someone writes a program, and the only difference is between /dev/zero and yes, and one is slower than the other, it's fairly reasonable to make the assumption that /dev/zero is slower.
https://github.com/coreutils/coreutils/blob/master/src/yes.c
https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c