
A Conversation with Arthur Whitney (2009) - radicalbyte
http://queue.acm.org/detail.cfm?id=1531242
======
rubyn00bie
This little section just blew my mind...

=================================================

BC: Do you ever look at your own code and think, "What the hell was I doing
here?"

AW: No, I guess I don't.

BC: Wow! I confess that I tend to write comments for my future self. I know
that when I come back to code I've written, I often don't recall instantly
what the problem at hand was or how I solved it. Now you've got me thinking
that maybe I'm just in the wrong language. When you're at this higher level of
abstraction, maybe it's easier to see your intent.

In terms of debugging your code, obviously the power of a terse language such
as K or Q is that, presumably, it's easier to find bugs by inspection. How do
you debug them?

AW: In C I never learned to use the debugger so I used to never make mistakes,
but now I make mistakes and I just put in a print statement. K is interpreted,
so it's a lot easier. If I'm surprised at the value of some local at some
point, I can put in a print, and that's really all I do.

BC: That works well when you have deterministic inputs. What if the nature of
the problem is just less reproducible—for example, if you were in an event-
driven system where you had a confluence of events that led to a problem?

AW: It has been 20 years now that I've had Wall Street customers—they're doing
2 billion transactions a day and they have trillion-row databases—and in those
20 years, there was one time where we couldn't reproduce the bug. That was
nasty. I knew the kinds of operations that they were doing and I finally found
it by just reading my code.

=================================================

I can't believe, well I guess I can, and have to, that people out there can be
this good. Is there something missing from the reality painted here?

In general, Mr. Whitney seems like a freakin' genius, and I'm gonna take a
stab at learning some K just to see what it provides. For those interested,
from a previous thread today about Mr. Whitney's work, is Kona an open source
implementation of K:

[https://github.com/kevinlawler/kona](https://github.com/kevinlawler/kona)

~~~
fidotron
I had dealings with him about 15 years ago, and he really is that good.

However, you need to appreciate he's that good because he's able to see
solutions that are easy to implement, and not that he has any special
implementation talent. K itself is tiny. The programs he's written in it are
an order of magnitude less, yet do a frightening amount for their size.

~~~
stuntprogrammer
Yes, he is that good. I worked directly with him for a few years and it deeply
changed my long term approach. Side effect is that it became harder to deal
with the "normals" ;-) seriously though, it made me very impatient with the
sorry state of the "state of the art" in the valley.

~~~
beagle3
Just grokking K had that effect on me ... I can't even estimate how much
farther that will go if I had a chance to work with Arthur.

And I keep thinking "ignorance is bliss" might have been right in this respect
- perhaps I would have been better off had I not lost my appreciation for the
(now obviously sad) state of software construction. (Engineering it is clearly
not, by and large).

But I wouldn't go back. Would you?

~~~
stuntprogrammer
No, I wouldn't go back. That said, it has made it rather difficult to fit
comfortably in standard settings. I've been successful so far but taking
Arthur's lessons and applying from various domains, to software, and into
cluster/system/soc arch in my case, has been viewed as rather unorthodox.

I find that especially in the valley, adherence to buzzwords and fashion of
the day is a little too common for my taste now.

------
radicalbyte
I really like Arthur's philosophy of rewriting code, it's something we don't
often have the chance to do.

------
juliangamble
I didn't 'get' K until I read an interview with Arthur Whitney in the ACM
Queue. Apart from performance, you're also getting craftsmanship.

[http://queue.acm.org/detail.cfm?id=1531242](http://queue.acm.org/detail.cfm?id=1531242)

It turns out he rewrites the whole compiler from scratch in C every four
years.

This is a snapshot from the source code of 'J' also by Arthur Whitney - the
precursor to 'K'.

[http://keiapl.org/rhui/remember.htm#incunabulum](http://keiapl.org/rhui/remember.htm#incunabulum)

This is it described by Ken Iverson: "The final impetus that got J started was
the one-page interpreter fragment that Arthur wrote, recorded in Appendix A of
An Implementation of J [29] and also reproduced in Appendix A below. My
immediate reaction on seeing the page was recoil and puzzlement: it looked
nothing like any C code I had ever seen. (“Is it even C?”) However, Ken
counselled that I should reserve judgment. "

[http://keiapl.org/rhui/remember.htm](http://keiapl.org/rhui/remember.htm)

~~~
theoh
There's a thread on Reddit from a few years ago that gives a more mixed
picture. Sure, the C code is terse, but that comes partly from doing things
like deferencing pointers without checking they're not null. One of the Reddit
commenters describes it as "optimistic" coding.
[http://www.reddit.com/r/programming/comments/8cckg/arthur_wh...](http://www.reddit.com/r/programming/comments/8cckg/arthur_whitney_on_apl_k_q_and_elegant_code/)

Without wanting to psychologise this too much, when I put the reckless coding
practice together with a paragraph like "Whitney is no respecter of rules. One
of the scariest things I ever did as a young man was following him through
central Toronto on a bicycle." I get the impression Whitney thinks he doesn't
have to follow other people's rules because he is an infallible genius.
Seriously unbecoming and offputting.

~~~
lobster_johnson
Well, clearly Whitney has been hugely successful with his odd little language.

There's the case for a kind of extreme pragmatism — writing code that does the
one thing it needs to do, without particular concerns for things like
abstraction.

We all write one-off scripts that are short and precise, but for a myriad of
reasons won't scale to anything bigger or more general: we cheat and use
global variables; we target one specific use case but don't allow
parametization/configuration to support others; we target one OS and a certain
constellation of dependencies; and so on.

What Whitney has accomplished, I suspect, is that he has found a way to reduce
the problem surface by going for extreme simplicity in absolutely all areas of
his system, which is by making certain assumptions. Everything is lists or
simple data types, there are no abstractions (by which I mean things like
interfaces, generics, abstract data types, even custom types), almost
everything is in-memory.

You can do this with a Lisp or a Scheme, too. But the temptation is always to
reach out for abstractions and patterns and build complicated systems out of
subsystems. A database would have a cache manager, a page manager, a planner,
an index manager, an SQL parser, a transaction log manager, etc. etc.

K shows that you don't _need_ go for this clean, compartmentalized, layered
subsystem approach if the whole program is small enough to make it easy to see
the whole system on the screen. You can use global variables because it
doesn't violate any layering. You don't abstract things like indexes and
operators into abstract internal APIs because you know at any point what they
are.

An example of this kind of extremist simplification is how kdb+ stores tables:
It memory-maps the entire table file and treats it as an in-memory list.
Instead of having to invent a paging mechanism and create functions that work
on tuples in pages, they can simply reuse all the vector-oriented K operators,
which work on the tuples exactly the way they work on other in-memory lists.

I don't think I could write K, but it is certainly a source of inspiration
when designing systems. We developers have a tendency to pattern-match and
generalize, when often "ad hoc" techniques would be better. The old adage is
"make everything as simple as possible, but not simpler", but it's actually
possible to make things simpler.

~~~
theoh
I don't understand your last sentence. Things can be made simpler than
possible?

If you focus just on inner loops, this kind of simplicity is widespread and
essential for performance. APL and its derivates are great specification
languages for array manipulation. If you are writing a networked window
system, on the other hand, a "high ceremony" language is more appropriate
because you need to cope with protocols and abstractions. Right?

~~~
lobster_johnson
My point is that it's wise to challenge assumptions about what "simple" means.
Self-imposed constraints often yield more novel, more optimized solutions than
when you have all the freedom in the world. Maybe flat files interacted with
via fopen() works better; maybe we just don't need our layers to accrete until
we have AbstractSingletonProxyFactoryBean.

A networked window system — perhaps not the best example. With that one I
think the assumptions are flawed (ie., that you can make one without
introducing painful leaky abstractions) to begin with. ;-)

~~~
theoh
Well, in terms of whether the example is a well-posed question, a networked
window system isn't too different from the WWW, which as we know by now works
pretty flawlessly.

Perhaps the key contrast here is between the tendency toward static
mathematical perfection of the APL family and the notion of dynamic languages
for building systems that Alan Kay et al would propose.

"APL is like a beautiful diamond - flawless, beautifully symmetrical. But you
can’t add anything to it. If you try to glue on another diamond, you don’t get
a bigger diamond." \--Joel Moses

There's a talk by Ian Piumarta that tries to get to the core of what the VPRI
guys feel is important. I'm not sure about it but it seems worth mentioning in
this context: Ian Piumarta "Building Your Own Dynamic Language"
[http://www.youtube.com/watch?v=cn7kTPbW6QQ](http://www.youtube.com/watch?v=cn7kTPbW6QQ)

Also, this: [https://www.cs.cmu.edu/~aldrich/papers/objects-
essay.pdf](https://www.cs.cmu.edu/~aldrich/papers/objects-essay.pdf)

