
The K Language - vonnik
http://kparc.com/k.txt
======
ktamura
While I really like K/Q/J/APL (some of this is Stockholm Syndrome: I spent the
first 3 years of my career writing K/Q at a trading firm, eventually becoming
the human debugger for my team's K/Q "investment". It got so annoying that I
worked on a little parser library to help me debug:
[https://github.com/kiyoto/ungod](https://github.com/kiyoto/ungod)), they are
doomed from a adoption/future viability point of view.

1\. Steep learning curve without benefits: APLs like K/Q/J optimize for
concision to the point of unreadability. Sure, it might be fun to decipher
clever K/Q one-liners, but if you have to do it everyday for work in
production, it's just exhausting. This lack of readability has limited
adoption and the growth of its user community. For any technology is to have a
viable community, it needs to be usable by a critical mass. K/Q in specific
but APLs in general never had that, especially with more user-friendly
alternatives like R/Matlab for scientific computing.

2\. No open source community. There's Kona
([https://github.com/kevinlawler/kona](https://github.com/kevinlawler/kona))
by Kevin Lawler, but again, its community has been small because the
underlying language is not design to be usable by enough people.

3\. Domain specificity of array languages: These days, array languages are
just too limited in its scope. Scientific computing, the bread and butter of
any array programming language, has been made possible in other languages (ex:
Python), making array languages not powerful nor compelling enough to use
because most of its strengths can be found in other languages and its
weaknesses are too crippling.

I heard that Arthur Whitney found his successors (a pair of Russian
programmers from St. Petersburg, iirc), so K/Q will likely to be around beyond
his retirement. That said, I think its user base will continue to shrink for
above reasons.

~~~
jsmthrowaway
I got exposed to kdb+ while interviewing in quant and would kill to use it in
a non-financial metrics setting, but the language and ecosystem just can't let
me get there. kdb+ is so fast and powerful you'd think it impossible.

They're not directly comparable, but kdb+ would crush InfluxDB head to head in
an apple/orange comparison if used for the same thing. I actually can't think
of any time series-like store that remotely comes close to the feats I have
seen kdb+ with Q accomplish. Too bad the productivity and knowledge overhead
is _so_ high.

Seriously, I'm working in realtime metrics right now and I spent a week
playing with kdb+ as the central store. I can tell there's a gold mine there,
but I'm just not swinging a big enough pickaxe. If there was a dumb
Python/Go/etc typical ops hacker frontend to that whole system I would be
throwing every dollar I have to my name at it; my kingdom for burying that
system under a few layers of abstraction and paying some quant Q hacker a
Ferrari or two a year to hide the sausagemaking from everyone else.

(Related aside: Like building furniture from stacks of cash? Learn K/Q/kdb+
and head for NYC.)

~~~
gd1
It is very sad that they won't open up, and very short-sighted as well. I'm
sure Whitney and KX are perfectly happy with their millions, but they are
sitting on one of the most impressive software achievements around and
refusing to share it. It's not even like it would cost them. First Derivatives
could grow a consulting empire. I mean, look at open-source MongoDB. It is a
piece of shit, and it is valued at $1.4B. KDB is finest thing I have ever
used, and First Derivatives purchased a controlling stake in KX for £36.0M. It
makes no sense.

Maybe kOS will be open-source. Maybe it will be the big one, Arthur's chance
to change the course of software development history.

~~~
shitgoose
"they are sitting on one of the most impressive software achievements around
and refusing to share it"

they tried. About a year ago they released 32 bit version for free with
shockingly liberal license - anything goes, including commercial use,
unlimited. About couple of months ago they pulled it back and replaced with
free non-commercial use only. well, they are the owners, it is their call, but
still k will remain as "one of the most impressive software achievements
around", orthogonal to most of the [fucked up] IT trends of the past 15 years.
It is a shame if it fades into a quantitative elitist oblivion.

~~~
gricardo99
This is the classic "innovators dilemma". My 2-cents: Kx has a good business
selling kdb+ licenses to the big banks and trading firms. They've been
exploring how to expand beyond this niche, but many big data/analytics
startups go straight for the FOSS solutions. In addition to the cost/licensing
advantages of FOSS, a large user-base and ecosystem has developed around
things like mongoDB, giving them compelling business advantages over kdb+
(even with kdb+ performance advantages).

Although mongoDB is a demonstration of how to build a successful business
around open-source, it's very different to start out that way rather than move
that way after you're already established as closed/licensed product. Kx would
have to essentially kill-off their existing profit stream with the goal that
they could build a larger business around FOSS kdb+. That's a risky
proposition, fraught with many pitfalls.

It is a shame they backed-off of the fully-open 32-bit version. At least that
had some potential to spur some user-base/ecosystem growth (at a minimum, this
could have encouraged the development of novel
clients/editors/REPLs/debuggers/charting/etc...), without threatening their
core profit stream.

~~~
stuntprogrammer
It's not obvious to me that an open core business can sustain the necessary
margins to be interesting as an engineering company rather than a glorified
services business. There are been very few examples (friend of mine argues
it's just RedHat).

You are correct though that their model over the years has been to extract a
large amount, up front, from a small number of users. They (mostly FD) have
failed to make the leap to users outside finance due to what I'd call cultural
reasons. They also lack strong technical leadership imnsho (they're, at heart,
not an engineering company).

I'd certainly take a swing at doing an open source version for them but not
clear to me that they'd know how to play it.

------
wyc
Why use such a thing? Array languages are very powerful for dealing with high-
dimensional data. R, matlab, octave, and Julia can all do this and are
satisfactory for most people. However, proponents claim that languages such as
K are much more expressive.

The APL family gets a bad rap for being "read-only", but I think it's mostly
because they've fallen out of favor and have become unpopular. People who have
experience working with these languages can encode/decode incredible amounts
of information to/from a few lines once they see "phrases" and "sentences"
instead of "my cat chewed the 56k modem line". We don't write Integral() in
calculus when we want to take an integral, so why not have similar shorthands
in computation?

Here's the vocabulary of the sister language J for comparison:

[http://www.jsoftware.com/help/dictionary/vocabul.htm](http://www.jsoftware.com/help/dictionary/vocabul.htm)

"The one-page thing":

[http://code.jsoftware.com/wiki/Essays/Incunabulum](http://code.jsoftware.com/wiki/Essays/Incunabulum)

Naive K-Nearest Neighbors implementation:

[https://github.com/wyc/snippets/blob/master/j/knn.ijs](https://github.com/wyc/snippets/blob/master/j/knn.ijs)

Community: [https://www.reddit.com/r/apljk/](https://www.reddit.com/r/apljk/)

~~~
minikomi
A nicer to look at J Vocab has been published recently:

[http://code.jsoftware.com/wiki/NuVoc](http://code.jsoftware.com/wiki/NuVoc)

It's a fun language to use to do /r/dailyprogrammer puzzles.

------
payne92
That looks like more of a grammar.

A specification would describe, among many, many, many other things, how big
an int is and what happens for overflow, etc.

~~~
baddox
To be fair, those things could all be deliberately unspecified. Granted, the
shortness of a language's spec is less impressive when it is simply the result
of the language being woefully underspecified.

------
RodgerTheGreat
Maybe this would be an appropriate place to drop a plug for iKe, a sandbox for
writing interactive, graphical sketches in your browser using K:

[https://github.com/JohnEarnest/ok/tree/gh-
pages/ike](https://github.com/JohnEarnest/ok/tree/gh-pages/ike)

------
BWStearns
Does anyone have a good resource on this family of languages or otherwise on
any of them individually? Every time I've run across references to K/Q I've
poked around a bit and it feels like they kind of exist as linguistic
curiosities in some vacuum with a vaguely unexplained creation myth about a
man named Whitney.

Edit: found this paper which appears to be around the birth of array languages
by the developer of APL.
[http://www.jsoftware.com/papers/tot.htm](http://www.jsoftware.com/papers/tot.htm)

------
nickpeterson
How would one actually go about getting a username and password, I see these
languages often, and know this is the true successor since Whitney is working
on it, but i think all of these are neat. Or should I just learn
Kona/kerf/xxl?

~~~
olympus
tl;dr: I recommend learning J.

If you just want to play around (no commercial use) you can download kdb+ from
here: [https://kx.com/software-download.php](https://kx.com/software-
download.php) but you only get the 32-bit version for free, so keep your DB
under 4 GB. With kdb+ you get K, the language, Q, which is a superset of K,
and a column (time-series) oriented database engine. I recommend against using
it, because if you want to do something meaningful (like make money somehow)
then you need to buy a commercial license. I hear that they cost around $2k
per machine (or possibly per core).

Kona is nice but it is a full version behind K (Kona is a copy of K3, and kdb+
is on K4) and it isn't quite as fast/optimized. It's great if you are set on
using K but don't have the cash and don't need database integration.

Kerf (by the same guy who made Kona) has a friendlier syntax but seems to be
commercially oriented similar to K (as in email us for free demo software but
pay up when you want to do it for real).

J is open source, free for commercial use, has some decent tutorials written
for it, and even has a free database engine (JDB requires a license for
commercial use, but Jd seems to be free for everything. It probably isn't as
fast as K, but it has a nice feel to it. The runtime is under 3 MB and even
with a Qt IDE and a bunch of extras it's under 30 MB. You can make Qt base
GUIs with J and do quite a few things.

I started learning J just last week and have been doing some problems on
Project Euler to get familiar with it. I like it quite a bit. It even has
built in functions for prime factorization, finding the prime number sieve,
and and extended precision mode which makes solving some problems very easy.

------
aplorbust
I think [http://kparc.com/lisp.txt](http://kparc.com/lisp.txt) would be easier
for the HN crowd to digest. Better marketing.

K is like Scheme but with vectors instead of lists.

If you cannot handle the one character APL verbs, you can always sacrafice
some resources and wrap them in verbosely named functions.

Look at q/q.k

~~~
nickpeterson
That is honestly quite helpful. I can't seem to find an actual link from the
main kparc to that page. Any other hidden gem pages on that site?

------
boulos
I enjoy the sudo of a non-https curl that copies this to /bin/k...

------
wrp
K/Q is famous for extremely short code, but that is partly due to a
misunderstanding of K/Q's nature. Sure, some things are expressed more
concisely because you are operating on arrays rather than doing loops, but
most of the conciseness comes from two other factors. First, there are
syntacitc tricks to use fewer characters, e.g. single-character names and
$[c;t;f] instead of IF-THEN-ELSE. But the big thing that most people ignore is
that K/Q is a DSL. It is designed connect to a server, acquire tick data, and
do time series analysis on clean numeric arrays. Stick to those operations and
your code is amazingly short. If you expand the notation to something more
readable and try doing more general programming tasks, the verbosity becomes
about like Lua.

~~~
icsa
IMHO, I would respectfully disagree. K/Q canbe used for all manner of software
systems.

PageRank can be done in a few lines of code - with clarity approaching the
mathematical description.

The solutions to the Advent of Code
([http://adventofcode.com/](http://adventofcode.com/) and
[http://kparc.com/advent/ad.k](http://kparc.com/advent/ad.k)) are readily
expressed/solved in k/Q.

That said, it excels above other languages/systems at systems like real-time
trading.

------
anonu
Has anyone used kdb+ in a real-time, mission critical setup - where kdb+ is in
the critical path? I am referring mainly to trading applications where kdb+ is
in the critical path. Can it be used as a low-latency / high-throughput
system, reliably?

~~~
dintech
I know at least Bank of America and Marshall have/had this.

------
mathattack
Interesting. 1010data was built on a version of K as well. Very fast
computation, and a spreadsheet GUI to hide the language complexities.

------
jsgrahamusxxx
Another K derivative is klong. I've communicated with the author and he is
helpful and willing to fix things. Enjoy

------
tlack
So is the if syntax shown at the bottom coming to K or is that only for
illustration? If not, why not use $[c;t;f] then?

~~~
RodgerTheGreat
A good question. Older versions of K had if and while as syntactic sugar. You
could always do without them by using cond and adverbs. K5 did away with these
syntactic forms, and for whatever reason Arthur decided to put them back in
k6.

------
phaefele
Has anyone looked at SciDB
([http://www.paradigm4.com/](http://www.paradigm4.com/)) as a replacement for
KDB. Seems like it would fit the bill and has a community edition. I do see
some usage in the financial markets.

------
treerex
The symbolism implicit in the characters used in APL is lost with J/K/Q and
any other ASCII variant. I highly recommend reading Ken Iverson's Turing Award
Lecture too.

------
RodgerTheGreat
Specifically, the reference card here is describing "k6".

------
cousin_it
For anyone wondering what K is, I'll repost this old email exchange between an
ML person and a K person (it's already online, so no privacy violation):

    
    
        On Wed, 12 Oct 2005, Fermin Reig wrote:
    
        > Dear Rico,
        > 
        > I have read Mark Joshi's book on C++ design patterns (as well as your
        > review) and I'd like to share my opinion of the book with you. (By the
        > way, I find the information in your website very useful.)
        > 
        > My interest in the book was to try to learn how theory is put into
        > practise. The best way to do that is by actually implementing things, so
        > I did: I reimplemented his C++ code in OCaml, a functional, OO language
        > that I like using. However, instead of a direct translation of C++
        > classes to OCaml classes, I chose a non-OO design. The result is quite
        > interesting from the point of view of code complexity and programming
        > productivity (and hence cost). Here's a summary:
        > 
        > Monte Carlo (datatypes only)
        > C++: 264 lines of code (LOC), (7 classes, 20 methods), 12 files
        > ML : 33 LOC, (3 datatypes, 1 function), 2 files
        > LOC ML/C++: 33/264 = 13%
        > Binomial trees (datatypes only)
        > C++: 264 + 138 = 402 LOC, (10 classes, 30 methods), 18 files
        > ML : 33 + 12 = 45 LOC, (3 datatypes, 3 functions), 2 files
        > LOC ML/C++: 45/402 = 11%
        > Binomial trees (datatypes + main algorithm)
        > LOC ML/C++: 165/522 = 32%
        > 
        > I'm not claiming that Mark's C++ code is bad (actually, it's quite
        > good). However, using a better tool results in obtaining a better final
        > product. (Performance of Ocaml's code is competitive with C++ as well.)
        > 
        > I have written slides with more details about this, which I have shown
        > to a few friends. If you would like to see them, I can email them to
        > you. (I have also told Mark Joshi about my evaluation.)
        > 
        > Regards,
        > Fermin
    
        Hi Fermin,
    
        Not bad. But my K implementation only uses 2 lines of code though...
    
        EuroOpt:{[S;K;r;v;T;n;po] / spot, strike, rate, vol, expir, steps, payoff
    
        / initialize constants
        p:(%2*m)*(1%D)-d:a-m:_sqrt -1+a*a:.5*(_exp dt*r+v*v)+D:_exp -r*dt:T%n;
    
        / apply n binomial steps
        :*n{-1_ D*(p*1!x)+x*1-p}/K po' S*(1%d)^(2*!n+1)-n
        }
    
        ...and here are some test cases...
    
        po:{0|x-y}; / arbitrary payoff function: max(0, K-S) for put
    
        / compare results for n = 16, 32, 64, 128, and 256
        \t EuroOpt[5;10;.06;.3;.5;;po]' _16*2^!5
    
        / pass in payoff functions for put, call, and digital \t 
        EuroOpt[8;10;.06;.3;.5;500;]' {0|x-y},{0|y-x},{y>x}
    
        / investigating convergence for a digital option with 1-300 steps...
        EuroOpt[8;10;.06;.3;.5;;{y>x}]' 1+!300
    
        So yes, I agree with you. Functional languages are cool and C++ is
        verbose & restrictive. Unfortunately, in the real world people don't care
        about any of this... :-)
    
        Cheers,
        - Rico
    

And that was the previous version of K, before Q/kdb+. Things are probably
even more concise now, but I've stopped following this about 10 years ago.
Good times.

~~~
kd0amg
I feel like I'm missing some context here. Which of Mark's/Fermin's examples
did Rico rewrite in K?

~~~
cousin_it
I guess it's TreeEuropean from Joshi's code:
[http://www.markjoshi.com/design/DesignCPP.zip](http://www.markjoshi.com/design/DesignCPP.zip)

------
riffic
This is not an acceptable method of packaging, distributing, or installing
software:

sudo curl -u USR:PWD
[http://kparc.com/download/k](http://kparc.com/download/k) -o /bin/k;sudo
chmod +x /bin/k

------
Chris2048
What is K/KDB/Q/J like wrt concurrency?

~~~
aet
Two processes can't write simultaneously to the same table partition.

------
alphapapa

        sudo curl -u USR:PWD http://kparc.com/download/k -o /bin/k;sudo chmod +x /bin/k
    

No. No, no, no! Bad developer, bad!

~~~
riffic
Not sure why a downvote was left without a comment but I'll upvote you and
leave a comment that I wholeheartedly agree. This is not an acceptable method
of packaging or distributing software.

~~~
alphapapa
Sometimes I feel like you are expected to check your sense of humor at the
door here. :|

