
Ruins of forgotten empires: APL languages - erehweb
http://scottlocklin.wordpress.com/2013/07/28/ruins-of-forgotten-empires-apl-languages/
======
jacques_chester
The most common "perform operation on bulk data simultaneously" language in
wide use today is SQL.

Joe Celko coined a phrase for this: _think in sets_. Not in loops, not in
linear instructions -- but in sets, being transformed into other sets. It's a
very powerful way of thinking. Most of the time, _you don 't need ordered
evaluation_ and so loops are imposing unnecessary guarantees on you.

On the more general problem of our collective historical ignorance ... yes.
That is us.

Well not entirely. That decades of the relevant literature is locked up in
Fort ACM and IEEE Castle has a _lot_ to do with it.

~~~
jamii
The BOOM guys have also shown that thinking in terms of unordered collections
translates much better to distributed systems, where guaranteeing order is
expensive:

[http://boom.cs.berkeley.edu/](http://boom.cs.berkeley.edu/)

~~~
beagle3
I'm going to read about BOOM later, but can you give an executive summary
about why that is?

People who subscribe to the Relational Database dogma often claim that sets
are superior to vectors, and yet despite asking a lot of them I never received
an example -- whereas I have numerous real life examples in which vectors are
superior.

~~~
jacques_chester
The main thing is that order isn't always necessary, but maintaining order
isn't free.

So if it isn't necessary, why pay the cost?

Moreover, the order you want will change from case to case. Today I care about
the alphabetical order of names, tomorrow about the order of dates, next week
I am most interested in the top 100 transactions by value.

Imposing a single order imposes an access pattern which may or may not hold
true. With sets I delay that binding until the last possible moment, when I
understand it best of all.

~~~
beagle3
> The main thing is that order isn't always necessary, but maintaining order
> isn't free.

> So if it isn't necessary, why pay the cost?

But if you use the APL/K approach, you don't pay the cost. There _is_ in order
to every result, but you can only rely on it if you know it (or force it, by
some sort).

Specifically, my question is this:

The order of results of "select ... where" without an order-by clause in SQL
is never guaranteed. The order of a ksql/q result is sometimes guaranteed, and
can be used. Where's the cost of just giving every result row a running index,
which some operations can rely on / change, and most ignore?

~~~
jacques_chester
That the order isn't guaranteed is a _feature_ , not a bug. It frees the
implementation to collect records in the most efficient fashion and from
having to maintain ordering.

> _Where 's the cost of just giving every result row a running index, which
> some operations can rely on / change, and most ignore?_

The cost is that you need to keep an index updated. A trite observation, but
there it is. Sometimes that cost is worth paying, sometimes it isn't. SQL
doesn't forcefully extract that cost, it lets the database programmer or
administrator decide, based on local conditions, what is best.

~~~
beagle3
> That the order isn't guaranteed is a feature, not a bug. It frees the
> implementation to collect records in the most efficient fashion and from
> having to maintain ordering.

Yes, I understand the theoretical argument. I'm looking for real world
examples where that actually helps anything, or theoretical examples where it
is superior (in the sense that "there is no way to assign order which would
not move the problem to a higher complexity class").

> The cost is that you need to keep an index updated. A trite observation, but
> there it is. Sometimes that cost is worth paying, sometimes it isn't. SQL
> doesn't forcefully extract that cost, it lets the database programmer or
> administrator decide, based on local conditions, what is best.

My point is that ksql does the same -- except that in most cases, the cost is
0, so you get the order "for free"; when the cost is not zero, ksql doesn't
make guarantees, and you'll need to ask for ordering, just like in sql. Every
sql engine already assigns order in one way or another (ROWID, etc.), except
it is unpredictable, non-standardized, and doesn't provide most of the
benefits even though it is there.

And the "no order" default often does have a cost; e.g., if you are trying to
store e.g. tax brackets in your database and find out which tax bracket a
salary falls into:

Either you store both lower and upper bound of bracket in a record, which
means you can't normalize (and it is hard to express the integrity requirement
of "no gaps" as a constraint) - or you do normalize, and then every query is a
self join or correlated query that many optimizers won't handle properly.
Whereas in ksql, you store it normalized (just start-of-bracket), order by
start-of-bracket, and then get the bin that corresponds to your question.

~~~
jacques_chester
I'm wondering if we're talking about different things here. You're talking
about vectors, which to me suggests an ordered collection of primitives. I'm
talking about sets of relations.

The main difference is that a relation is not a primitive. It is a tuple of
primitives.

The point being that sometimes you care about those relations being ordered by
date, sometimes by amount, sometimes by ID and so on and so forth. The
database imposes no order on you and so you can impose one at will.

The database equivalent of maintaining order for every single-column selection
is to have per-column indices.

Try that in production on a busy OLTP system and let me know how you go.

Where array/vector languages and SQL see eye-to-eye is in basically abolishing
the loop and having a logical model where operations are applied
simultaneously to whole collections.

From talking to you the difference is when order is imposed. In cases of time
series analysis, where there are single numbers and a natural order, I see the
value of array/vector languages (or one of the various proprietary temporal-
SQL databases). But most problems of interest aren't vectors with a single
natural order.

Incidentally, I don't see why you think the tax-bracket thing is such a killer
case. Oh noes, a small join!

~~~
beagle3
> The database equivalent of maintaining order for every single-column
> selection is to have per-column indices.

I guess that's where the misunderstanding comes. In K/APL, every column has an
implicit "natural index" \- the first item has index 0, the second item has
index 1, etc. The thing that is always guaranteed is that, in a result set,
the natural indices of the different columns of the same tuple correspond.

 _Some_ operations guarantee a particular natural index order, but many do
not. An additional index on the values that allows you to find an exact value
(e.g. hash) or range of values (e.g. a b-tree) may or may not exist, just like
in a relational database.

This "natural implicit index" does not take memory or anything to maintain,
but it is useful.

> Try that in production on a busy OLTP system and let me know how you go.

ksql runs my queries approximately 100 times faster than optimized MySQL
(using MyISAM - faster, but no acid guarantees; ksql is acid).

> Incidentally, I don't see why you think the tax-bracket thing is such a
> killer case. Oh noes, a small join!

That was just for known terminology, not knowing where you come from. I have
tables with thousands of such brackets. And while I haven't tried Oracle or
MS-SQL for them since 2006 or so, it was incredibly easy to confuse their
optimizer into doing a complete cross product on those tables and filter it -
which makes it takes a second per query instead of milliseconds when the query
is optimized properly (or microseconds using K).

This is known as an "as-of" join in time-series queries, and any database
without explicit support for it usually takes thousands of times more
resources than needed for such a join.

------
cabacon
I have had a huge amount of fun lately going through the J labs. Start at
jsoftware.com to download the interpreter, then the labs are all built into
the interpreter. You can also run it on iOS. The terse code actually means you
can type in real programs even on an iPod keyboard.

When I taught people to play frisbee, I used to throw with my off-hand when
teaching, because it made me pay attention to the fundamentals I was trying to
teach. Learning J had some of that same feel. I felt like a newbie, but when I
saw the kinds of audacious programs you could write, I was hooked. Highly
recommended for a bit of fun in your spare time.

~~~
saryant
I went to a small university with only 8 CS professors. One of them was the
author of J's array parallelization and he taught our Intro to Functional
Languages course.

That class was fourth in the undergrad sequence, coming after C, C++ and Java
so it was quite a shock to, well, everyone. He took us from finding the
average of a list all the way through to implementing a CMS in J (our
department web server _still_ runs on J because he's since retired and no one
else knows how to maintain it).

This was easily the hardest class in the entire department. Students who
sailed through everything else would struggle to pass. It's a fascinating
language but not an easy one to grasp.

------
lmm
This happens with proprietary languages. How many people can afford $100k/CPU,
however good it is? How many can play around with a language like that at
home? Who's going to write infrastructure around such a thing?

Hadoop is pretty crap - I know, I used to use it professionally. You know what
else is crap? Unix. People are going on about cgroups as an exciting new
feature, when better implementations were around what, thirty years ago in
Real OSes.

It doesn't matter. Commodity wins, and sooner or later if there are good ideas
to be had we'll find them again.

~~~
stass
How APL is proprietary? Anybody can implement it, and there are opensource
implementations available. The 100k/CPU price the author referred to was about
the high performance database integrated with a vector processing language
engine.

It is not about commodity. cgroups-like functionality was available in many
opensource OSes for years (e.g. FreeBSD jails), but was easily dismissed by
majority as worthless. This is exactly what the author was talking about
referring to Lady Gaga -- the modern programming community is driven by
fashion regardless of how good or bad the current fashionable technology is. I
do not find it surprising however, as it is a direct result of the order of
magnitude growth of the programmer's population, which is behaviorally much
more close to the general population currently that it used to be.

~~~
lokedhs
What open source implementations are those? I spent some time recently
learning APL, and the only one I could find was NARS2000 which is Windows-
only, very buggy and has practically no documentation available.

If you know of any others, please let me know. I certainly haven't found
anything.

~~~
derekp7
There's A+, from Morgan Stanley. It is actually a variant of APL, but it looks
mostly like it (unlike J, it retains APL's hieroglyph characters, etc.)
Licensed under GPL.

~~~
No1
The A+ website is [http://www.aplusdev.org/](http://www.aplusdev.org/) for
those interested.

~~~
scottlocklin
FWIIW, I couldn't get this to build.

~~~
derekp7
It is also in the Fedora repositories (yum install aplus-fsf). I would imagine
that it would be in Debian/Ubuntu too, but I don't have a current install to
check. If all else fails, you can probably grab the source RPM to see how
Fedora/Redhat got it to build.

------
perlgeek
I am a bit disappointed by the low ratio of technical content to rambling. And
much of the technical content seems to set apart APL from C, but not from many
modern languages.

For examples languages like Fortress contain primitives for parellelized
loops, and most modern languages also contain enough functional programming
that you rarely have to write for-loops.

Function composition and interactivity aren't a unique feature of APL
languages either.

There are some other things that are important for programming today:
Maintainability, richness of available libraries/modules, packaging,
versioning and so on.

Where would I even look for existing APL libraries? My searches for a HTTP
library for APL (for example) have turned up nothing, while searching for a
HTTP library for several other languages that came to my mind brought up
useful results.

~~~
klibertp
There is HTTP library in J (APL "spiritual successor") standard library. There
is one for APL for sure - it's just less popular and so the google results are
less helpful.

Unique features of J (I'm writing about J because I know it, while I don't
know APL) are that all the basic verbs (functions) can operate on arrays as
easily as on atoms. There is no special syntax or feature for adding two
matrices, for example, standard `+` verb does it naturally.

The second unique feature is a tacit - point-free - style by default. This
means that function composition is the default, not function application. You
can define new words by just slapping old ones together, without worrying
about threading params through them.

The third unique feature (and probably unique to J, previous are found in APL
too) are complex rules of composition (hooks and forks) as well as words that
modify functions. You can express very complex things in very little space
thanks to this.

J is very interesting, unique language which are very well suited to one thing
- very fast and efficient array transformations - and completely useless for
other things. I hear it's used in a financial and banking sectors, for rapidly
prototyping and sometimes deploying statistical and mathematical algorithms.
If you have a good enough use case, then learning J will pay off. For anything
less than full time commitment, I think that R od Python+Pandas is better.
Still, I'm very happy I tried to learn J, as it's completely unlike anything
else and it helped me greatly to understand numpy/pandas for my main job.

------
mattdw
He'd rather write in Beethoven languages than Lady Gaga languages. Which is
fine, but there's something to be said for "takes 3 minutes to perform with
one voice and one computer" as against "takes an hour to perform with a team
of 30 highly specialised professionals."

To stretch his metaphor.

------
lcedp
I had some fun with J. But it's (you know what I'll say) too cryptic..

Let's take an example from RosettaCode. For instance, merge sort:

    
    
        merge     =: ,`(({.@] , ($: }.))~` ({.@] , ($: }.)) @.(>&{.))@.(*@*&#)
        split     =: </.~ 0 1$~#
        mergeSort =: merge & $: &>/ @ split ` ] @. (1>:#)
    

I could probably write it, but read what it does in a few weeks?

~~~
gd1
I find it much easier to read some Q/KDB I've written earlier than the
equivalent C++ or C#/Java. For three reasons:

1) It is extremely terse. A small block of code will do what pages of
boilerplate class definitions, interfaces, setters/getters, declarations,
typedefs etc. will achieve. There is usually no need to navigate through a
spaghetti chain of virtual function calls.

2) It is interpreted. I can quickly (without compiling) cut out chunks of the
code and run them to understand what they are doing. I can create variables on
the fly, fill them with test data, and run the function. If it calls another
function, I execute the name of the function and it returns the function code,
since functions themselves are first class variables.

3) It actually becomes easy to read because it is easy to parse, for both
machine and human. K/Q is read right to left, and so breaks some traditional
conventions. For example, 5*6+3 will evaluate to 45, not 33. At first this can
be disconcerting, but eventually you will realise that parsing a line of code
no matter how complex it looks is quite simple - you mentally start at the
rightmost token and start gobbling. This is probably why the interpretor is so
fast as well.

As someone mentioned below, I'm not sure why you'd consider it "too cryptic".
Anything is cryptic when you first encounter it. I suppose the difference is
that Q/KDB provides an (extremely valuable) solution to the financial world -
the handling of big data long before it was even called that (a tick database
with billions or trillions of rows) so there is a valid motivation to learn
the language and it doesn't feel cryptic at all once you've immersed yourself
in it.

~~~
lmm
Python wasn't cryptic when I first encountered it. There are many things wrong
with python, but using ordinary words for operators ("or", "in") was one thing
it got very very right, and I wish there was a more functional/typed language
that took the same approach.

~~~
beagle3
APL and K draw their inspiration from math.

Python wasn't cryptic, but it is confusing unless you have a programming
background.

the fact that "and" and "or" short circuit.

The fact that "1/2 is 0" (at least before Python 3), and that "1000+1000 is
not 2000", even though "1000+1000 == 2000"

Or that "0.3*3 <> 0.9" (if you knew that <> means not equal)

------
ginko
> APL is a mistake, carried through to perfection. It is the language of the
> future for the programming techniques of the past: it creates a new
> generation of coding bums. \- Edsger W. Dijkstra

~~~
jacques_chester
You can find an acidic Dijkstra quote for every language of his time.

Dijkstra, I think, viewed mere _languages_ as an unhappy by-product of
computer science and hankered to do it all with mathematics, actual utility be
damned.

~~~
gngeal
Alan Kay seems to think along similar (though not identical) lines these days.
The sheer inertia of ordinary programming language implementations means that
by the time you get to use it, you may realize that something "slightly
different" would have been better, but you seldom have any other option than
to "suck it up".

~~~
jacques_chester
Economists have a term for it -- Path Dependence. Some decisions have a high
enough reversal cost that they are in practice irreversible. This means that
one's future decisions are dependent on historical decisions over which one
might have had no control. In our case: once a technology is deeply rooted,
it's hard to displace.

More closely, the Rogers model of diffusion[2] has 5 factors of diffusion
(relative advantage, compatibility, complexity/simplicity, trialability and
observability).

Generally even the most advantageous language transitions score well only in
relative advantage. The other diffusion predictors are missing or weak.

[1]
[http://en.wikipedia.org/wiki/Path_dependence](http://en.wikipedia.org/wiki/Path_dependence)

[2]
[http://en.wikipedia.org/wiki/Diffusion_of_innovations](http://en.wikipedia.org/wiki/Diffusion_of_innovations)

------
jdn
The authors response in the comments to an overzealous attempt at political
correctness was wonderful.

~~~
ddellacosta
Not only was that comment
([http://scottlocklin.wordpress.com/2013/07/28/ruins-of-
forgot...](http://scottlocklin.wordpress.com/2013/07/28/ruins-of-forgotten-
empires-apl-languages/#comment-6301)) juvenile, it displays exactly the kind
of privilege and ignorance which allows sexism to continue in the world of
computer science and software engineering, and which dissuades women (and men
who dislike this kind of behavior) from participating in the larger world of
compsci/software engineering. In many cases, once you realize that these kinds
of assholes rule the roost, it's easier just to go away and do your own thing,
rather than disagreeing or speaking up. Especially when we get further support
like your comment. "Oh, what a card he is!"

It's okay to disagree--I didn't find the "men use Apply(), not for" thing to
be particularly offensive myself, honestly--but there are many ways to
disagree without resorting to the kind of hateful, caustic language the author
used. This is not how well-adjusted adults disagree when having a discussion.

~~~
jdn
I very much doubt the response reflects his attitude to women in computer
science, and your comment regarding "this is not how well-adjusted adults
disagree when having a discussion" is somewhat moot, as it wasn't a
discussion. This is where I believe his anger came from, and where my support
of his response also stems. The cause of women in computer science (which as a
student in a very male dominated CS department, I wholeheartedly support) is
not helped by people derailing everything in the manner in which the
commentator did. Like many political and ethical discussions, there is a time
and a place. If he had said something outrightly and obviously sexist, yes,
call him on it. But as it is, I think he had every right to get annoyed and
want to nip that derailing in the bud.

~~~
ddellacosta
_as it wasn 't a discussion_

It never got the chance to be a discussion, as the author responded to the
opening volley with a shotgun blast.

That's his prerogative, of course, as it is his blog, and he can take his ball
and go home if he wants to.

 _The cause of women in computer science (which as student of a CS department,
I wholeheartedly support) is not helped by people derailing everything in the
manner in which the commentator did._

It's not a derail. It is a topic which came up because of language the author
used, and which is an ongoing problem in the community of software developers.
While you are free to believe in a platonic ideal of "pure ideas and
discussion," it doesn't exist in this messy world of relationships and human
language and politics. If you are still a student, I suggest you take a
gender/sexual/racial politics course and try befriending some people with very
much opposite viewpoints to your own. You may be surprised at your shift in
thinking.

But leaving that aside for the moment, let's say the author's main intent was
to "nip that derailing in the bud." What do you think the most effective,
least hurtful way to approach that would have been?

1) Ignore the comment, 2) assuming he is the administrator of the WordPress
blog he writes on, delete/hide the comment, or 3) lash out in a juvenile,
sexist, hateful fashion.

I'm going to go with 1 or 2 being better than option 3 any day of the week.

~~~
verroq
How is

> Apply() is the right way for grown-assed men do things.

sexist?

Am I not reading enough into it or something? As far as I can tell somebody
was making a fuss about some non issue and was shot down for it.

~~~
king_jester
If you are someone who is not a man, and you read this, how exactly are you
supposed to take it? This article is written by a man and features exclusively
discussion and pictures of men or languages invented by men. The whole article
paints men == software developers/programmers. The only time it talks about a
woman is when it tries to demean Lady Gaga for making money while extolling
the virtues of male composers.

~~~
verroq
>This article is written by a man

So?

>features exclusively discussion and pictures of men or languages invented by
men

You've got to be kidding me right? The majority of programming languages are
made by men, it is statistically likely any discussion of programming
languages involve those made by men. You might as well attribute any
discussion of programming languages as sexist.

I'm not sure if this response should have a /sarcasm tag on the end because
it's more sexist then the original comment.

------
stiff
There is no shortage of excellent work being done today, too. With the large
systems that are being created today I would take Python over APL or J any
day, can you imagine building a company like Google or Facebook on top of APL,
even assuming you would manage to find a team of good APL programmers? It is
not a language fit for cooperation nor for large systems, it is a problem-
solving language, back from the day where people used computers for relatively
small tasks.

Plenty of the good things in APL are now available in other forms anyway.
NumPy with it's powerful set of linear algebra primitives and the huge set of
implemented algorithms is just as great at data crunching. Want mmapped files?
No problem:

[http://docs.scipy.org/doc/numpy/reference/generated/numpy.me...](http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html)

That's not to say APL is not worth studying, it is built on powerful ideas.
But to study the masters is to criticize the masters, not only blindly admire
them. And history is not all romantic.

~~~
Joeri
Maybe we've got it backwards. The only true problem in software development is
miscommunication, and the bigger teams get the worse they perform per-
programmer, with the most productivity loss experienced by the most senior
team members. On large teams, the most experienced team members are often
handicapped by having to operate in a context that remains comprehensible for
junior team members, investing so much time in communication that they spend
not luch of it on building things. that in turn produces a sort of lowest
common denominator effect, where designs are deliberatey kept overly
simplistic so they will be easy to communicate.

The more expertise is applied to a solution, the less likely it is that people
can cooperate on that codebase. The languages and frameworks of today try to
minimize the need for expertise to get things done, because expertise doesn't
blend well with communicating about code. I think part of the ideas in the
article is that by running away from expertise we have often arrived at
inefficient solutions. We scale out, with huge codebases that are like cities,
sprawling and vast yet with each component isolated and easy to comprehend.
Learning from the masters means understanding complex ideas which are based on
scaling up, on building a project-specific grammar that encapsulates the
problem domain so elegantly that it allows for one line features, at the
expense of being incomprehensible to someone new to the codebase or the field.

~~~
jacques_chester
This actually sorta rhymes with the design goals of Java -- a broadly
featuresome language, with lots of safety nets, that could be used relatively
effectively by the median programmer.

~~~
jdbernard
Having worked in Java for several years this is both a great strength and
weakness in my opinion. On the one hand it keeps everyone speaking the same
language as opposed to some languages where it seems each team is speaking
it's own dialect. On the other hand it can shackle the truly proficient, makes
it easier for complacent programmers to remain so, and masks (temporarily) the
incompetence of people who should not be writing production systems.

~~~
jacques_chester
Yes, but Java acknowledges that the bulk of programmers lie near a median.
It's a language for an industrial age, full of automation and great steel
buttresses. Those who hanker for surgery, or lightsabres, or precision
archery, or any number of analogical departures -- they naturally drift to
other languages.

------
hat_matrix
How is lush an old language? Looks like a language that was released last
decade, but just never caught on.

~~~
scottlocklin
Lush as a language was developed in 1987, just like it says on the website.
Older versions of it are reading the numbers off of your checks when you
deposit it in the ATM machine. The ideas it embodies are significantly older
than common lisp: it is a very pure hackers lisp: sorta has the eau de
zetalisp about it (not that I really know anything about zetalisp, I ain't
that old, but I've read about it). Studying it will make your brain grow.
[http://lush.sourceforge.net/credits.html](http://lush.sourceforge.net/credits.html)

~~~
hat_matrix
Actually it says that Lush's predecessors were developed in 1987. Version 1 of
Lush was released last decade.

But even if it was 1987, that's not older than Perl or Python.

~~~
scottlocklin
Well, if you download, say, TL3 from Leon's page (a subset of SN), you'll
immediately see that the "predecessors" are essentially the same language.
Same Ogre, same UI. You can even call pieces of it in modern Lush. The name
change is irrelevant.
[http://leon.bottou.org/projects/tl3](http://leon.bottou.org/projects/tl3)

------
robomartin
I used APL extensively for about ten years. I was introduced to the language
by a physics professor in college who was quite vocal about having his
students learn the language. I ended-up using it professionally for a wide
range of applications, from inventory management and core business operating
systems to robotics and DNA sequencing.

While the language is wonderful I would not use it for production systems
today. It is my opinion that the language came to be too early and, as a
result, has not seen innovation for decades. It also does not help that the
leading APL implementations cost thousands of dollars to license. APL is still
being used extensively in financial institutions in Wall Street.

J has been mentioned here a few times. Even though J is the work of APL's
creator, Ken Iverson, whom I had the pleasure of meeting at various APL
conferences, I consider it an abomination and a huge step backwards. I have a
feeling Ken might have sought more commercial appeal by using ASCII characters
rather than specialized notation. Ironic because of [1]. However, in making
this switch, he created a horrible mutation that anyone can rightly recoil at.
The language looks like a random vomiting of ASCII characters on the screen.
Sad.

Many have commented on APL on this thread. In most cases it is obvious that
these comments come from lack of or limited experience with the language. The
power of APL comes from two things: Abstraction and Notation.

Abstraction is very much like objects in OO languages or specialized symbols
in mathematics.

Notation --through specialized symbols-- is huge. Ken Iverson's own "Notation
as a tool for thought" [1] covers this in far more depth than I could attempt
here.

I'll just mention one example: music notation. A page in an orchestral score
looks like gibberish to someone unskilled in the art. However, to a skilled
professional it reads, well, like music. While I am not at that level, I can
read guitar or piano sheet music and imagine, for lack of a better word, the
music with reasonable accuracy.

That's what happens with APL if you really know it: You read it like music.
Much of the criticism and misguided opinion comes from people who simply don't
have this level of proficiency with the language. All other languages allow
you to be a pretty decent "hobby" programmer in the sense that you don't have
to be able to read it like music and you can still produce and maintain
software. APL, for non-trivial work, requires true expertise, not superficial
exposure. It rewards you with unparalleled productivity and expressive power.
It really is a cool language, but you have to invest the time to learn it and
read it like music.

Another way to put it: It's like vim. It's a shit code editor until you invest
the time and effort to use it instinctively, to play it like a reasonably
skilled musician would play a piano: without having to think about which
finger to use to press a certain key to produce the desired tone.

That's APL. It's very different than using C or Java. It requires expertise,
time and dedication. And perhaps that's the source of its demise. Today most
programmers don't care about diving to those depths. Most want to learn the
basics and poke around API and code libraries to get shit done and consider
super-simple stuff like pointers confusing and complicated.

BTW, to me Python has a number of APL-like features that make it fun and
productive. I really enjoy it.

[1]
[http://www.jdl.ac.cn/turing/pdf/p444-iverson.pdf](http://www.jdl.ac.cn/turing/pdf/p444-iverson.pdf)

~~~
znowi
> BTW, to me Python has a number of APL-like features that make it fun and
> productive

Can you please elaborate on this and give a few examples?

~~~
robomartin
Here's one example from playing around on Project Euler:

    
    
      def problem_1a():
          return sum(set(range(3,1000,3) +  range(5,1000,5)))
    

While verbose, this is the kind of structure you would see if this was written
in APL. In fact, you could translate this almost directly into APL.

------
dangoldin
I've been doing the Project Euler problems and always love seeing the APL-like
solutions. Amazingly expressive stuff that would be fun to get into some day.

------
eliasmacpherson
My only knowledge of APL is second hand via Scott Meyer's "More Effective C++"
where he gives it as an example of the benefits offered by lazy-evaluation,
saying that rather than compute the results of each statement in series it
makes a compound statement where it computes only the result needed.

In my own situation APL is a language looking for a problem - I don't
currently have a use case for it.

[http://books.google.ie/books?id=azvE8V0c-mYC&pg=PT121&lpg=PT...](http://books.google.ie/books?id=azvE8V0c-mYC&pg=PT121&lpg=PT121&dq=more+effective+c%2B%2B+APL&source=bl&ots=46hf09Ni3j&sig=rqutwxurvUWv5uQcL9KpQ1IbKI0&hl=en&sa=X&ei=BBz1UYOOGvGP7Ab8oYHICA&redir_esc=y)

------
useful
1/2 our codebase is in APL. It is a good calculator.

------
prezjordan
If anyone here is proficient at J, please (pretty please) contribute a guide
to Learn X in Y [0]!! I don't run the site, I just really want a J guide in
this format.

[0]: [http://learnxinyminutes.com/](http://learnxinyminutes.com/)

~~~
scottlocklin
Try this:
[http://www.jsoftware.com/help/primer/contents.htm](http://www.jsoftware.com/help/primer/contents.htm)

Or, download it and run one of the intro labs; that's even better.

------
dmpk2k
As a random aside, why do APL programmers try to import their idioms into C?
Take a look at the VM code in J or Kona and prepare for a shock.

It's too bad. I find VMs interesting, but I wouldn't touch that C with a ten-
foot pole. It's IOCCC quality.

~~~
gruseom
It's a distinct style of C that evolved over decades for implementing APLs.
It's more akin to an embedded DSL than it is to normal systems code, and the
reason people use it is not that it's obscure but that it fits the problem
space well.

The "shock" of looking at it is exactly the same as the shock of looking at
APL or K programs to begin with. They don't look anything like code that most
people are used to, but since they tend to be highly regular, the impression
of noise is misleading.

~~~
dmpk2k
I'm not sure a general lack of useful commenting and a subsistence on single-
variable names is defensible. It sure seems like a case of "You can write
Fortran in any language".

------
znowi
There's a great link in the original comments to a video from 1974, where
Iverson, Falkoff, and others discuss the origins of APL. It was a pleasure to
watch. A peculiar contrast in language and attitude to today's rockstars and
ninjas :)

[https://www.youtube.com/watch?v=8kUQWuK1L4w](https://www.youtube.com/watch?v=8kUQWuK1L4w)

------
jingo
"Big data problems are almost all inherently columnar..."

Can you name some that are not?

------
mathattack
Great article. It seems like once you get beyond the hype of Big Data, you get
to some old technologies whose primary use cases are finally coming to light.
The more things change...

------
michaelochurch
q is a neat language, and I spent some time studying it when I was in finance,
because it's useful to know q/kdb if you're a quant.

Here's an offensive but accurate taxonomy. There are 6-month
languages/technologies and 20-year languages. The 6-month set are optimized
toward being quick to learn _fast_. Java, for example, inherited C++ syntax
while cleaning up the OOP, so the C++ programmers of the time could pick it up
in quickly.

Now, q and kdb are 20-year languages. It doesn't take 20 years to be
productive with them, of course; it's that they're designed to optimize a
person's productivity over 20 years, and not in the first 6 months. Unlike the
various distributed tools out there (Hadoop, Hive) it will take you a long
time to learn them, because they aren't like anything else. Superficially, q
is Lisp-like, but the performance implications are very different and it
doesn't look like a Lisp at all. (It parses right-to-left, for one example;
also, it's written as a stream rather than in nested s-expressions.)

Many of these modern tools being bashed have grown up in a world where 20-year
tools can't really be built; they'll never catch on commercially in a world
where people change jobs every 2 or 3 years, when MVPs are more important than
solid decades-proof engineering, and where employers are pretty averse to
their people taking time to just learn new technologies.

There's too much volatility in the world for it to make sense for most people
to learn q and kdb right away. You can get half-decent in a couple months, but
it'll take much longer to learn _why_ those tools are (for some use cases) so
damn powerful.

This 6-month vs. 20-year tool concern is part of why I've gravitated toward
Clojure recently. I haven't seen anyone else do quite as good a job of
balancing those two need sets as Rich Hickey and the Clojure community have.
Clojure is (becoming) a 20-year language, but it's no harder to become
basically productive in than Python or Ruby.

------
dschiptsov
The beauty of a programming language comes from being _small and consistent_.
Such as Smalltalk, which has only 6 reserved words and implements message-
passing as if it is a function call. This is the brilliant idea of Smalltalk -
message-passing + blocks + booleans and out of this everything else is
implemented. Classes are of second importance, but delegating (via message
passing) to parent or another object was invented here.

Scheme, on the other hand, was also minimalist and consistent - s-expressions
and about dozen special forms, and out of this everything else could be
implemented, including closures, continuations, and classes and message
passing, if needed (CS 61A 2008 does this for you).

Haskell is.. well, everyone know what Haskell is.

As for APL, in my opinion, the power of a language comes not from using
esoteric math notation, but, like in CL or Scheme, from possibility to easy
add new control structures (call/cc) and special forms (with macros) or even
create a new languages, such as Arc.

~~~
rossjudson
Minimalism is oversold. K specifically eschews minimalism, providing an
_essential_ list of operators and combiners (verbs and adverbs). It's no
"6-word" language. Every character _is a word_ in other languages.

We don't have six characters in the English language.

~~~
kansface
Right, and the English was designed to perform the function that it does.
Perhaps a sane implementation of the language would use far fewer characters.

~~~
lttlrck
"designed" is the last word I would use to describe English :)

~~~
jdbernard
He was being sarcastic. I think his point is that because English was _not_
designed it is far less efficient than it could be, and far more complicated
than it need be. Therefore it is not a good language to imitate when we go to
design programming languages.

------
cmccabe
This paints a really misleading picture of what is available in Hadoop right
now. Columnar data formats? Yep, we've got them-- see Parquet
[[http://parquet.io/](http://parquet.io/)]. Yes, Hive spends a lot of its time
reading and writing data. This is because it decomposes SQL queries into sets
of MapReduce jobs, all of which must take input from the filesystem and write
it to the filesystem. That's one of the reasons Cloudera Impala was developed
[[http://blog.cloudera.com/blog/2012/10/cloudera-impala-
real-t...](http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-
queries-in-apache-hadoop-for-real/)].

If you don't like for loops, yep, we've got that too-- use Scala or another
functional programming language to write your MapReduce jobs. Or use SQL,
which is a lot more powerful than APL, and a lot more accessible to businessey
types than functional programming. SQL is a declarative programming language,
by the way-- I don't see any for loops there.

The efficiency argument makes no sense either. Does the author understand that
a just-in-time dynamically recompiling virtual machine is faster than an
interpreter? If so, he doesn't mention it anywhere. You know, sometimes old
technologies are just... old. You could at least compare Java and the JVM to
something like Lisp machines.

People overestimate the gains that are to be had from mmap. I am currently in
the process of adding mmap support to HDFS and I know what I am talking about.
mmap gives gains, but only when the data is coming from memory, and only when
you can reuse the mmap. Otherwise, you're better off reading even a 512 MB
file via read() and write(). The reason is that syscall overhead is not that
high on modern UNIXes (like Linux), and page faults involve a transition into
kernel space anyway.

~~~
beagle3
I have used Hadoop; it is miserably slow (as in 10 to 1000 times slower AND a
lot harder to work with) then the comparable K program.

If mmap doesn't bring you significant gains, you are doing stuff wrong. E.g.
You serialize objects. Don't.

Hadoop is horrible. I've heard Spark manages reasonable performance (subject
to the shackles that Hadoop compatibility entails), but haven't used it.

If you think SQL is "more powerful" than APL... Qualify that. Because e.g. APL
is Turing complete, whereas SQL isn't.

~~~
technolem
As a sidenote, SQL is: [http://stackoverflow.com/questions/900055/is-sql-or-
even-tsq...](http://stackoverflow.com/questions/900055/is-sql-or-even-tsql-
turing-complete). Not that you'd even want to use it as such.

~~~
beagle3
:) a corollary of Greenspun's tenth rule: every language spec will be revised
at least until it is possible to implement a slow, incomplete version of
Common Lisp inside it.

