
BQN: An APL Variant from Marshall Lochbaum - chrispsn
https://mlochbaum.github.io/BQN/
======
mlochbaum
That's me! It's a pretty exciting time for BQN because I've just put the self-
hosted version online where you can try it out. In addition to exploring a lot
of new APL ideas[0], BQN is also a test of the Co-dfns[1] approach to
compilation, meaning that with an appropriate runtime the entire bytecode
compiler could be run in parallel on a GPU. It's still in proof-of-concept
stage, so that the online compiler is very slow and one running on dzaima/BQN,
another implementation, compiles only about as fast as Go despite having a lot
less features. However, having written several thousand lines of BQN I'm very
happy with the language's fundamentals, and I feel it's a big advance over APL
or J.

[0]
[https://mlochbaum.github.io/BQN/doc/index.html](https://mlochbaum.github.io/BQN/doc/index.html)

[1]
[https://mlochbaum.github.io/BQN/implementation/codfns.html](https://mlochbaum.github.io/BQN/implementation/codfns.html)

~~~
zerker2000
I've always been curious one decision pervading various APL: what are the
relative merits of right to left evaluation order, vs more conventional left-
to-right method chains? The core principles are incredibly elegant, but I must
admit hitting "expressions are executed right to left, except if you hit a {}
lambda, wherein separate statements are ordered left to right, but are
themselves expressions going right to left again" is always kind of jarring.

~~~
mlochbaum
For left-to-right execution, the dissonance between execution and statement
separators (including functions and list notation) is very bad as you note.[0]
For right-to-left execution, assignment has a similar problem: the name really
needs to be on the left to allow easily scanning definitions, but this
conflicts with the evaluation order. Particularly statements with multiple
assignments in them don't work at all. Right-to-left is usually framed as a
more "declarative" or "functional" style (and note that application and
composition of mathematical functions follows this order) while left-to-right
is considered "imperative".

In my first major language, I,[1] I did change to a left-to-right order. For
BQN, which is intended to stick much closer to traditional APL, I didn't want
to make such a large break from the methods that have worked in the past. I do
think I'll introduce some mechanism like a "pipe" that goes in statement order
so that longer chains of functions and operators can be built up without such
a discontinuous reading order.

[0] [https://mlochbaum.github.io/BQN/problems.html#right-to-
left-...](https://mlochbaum.github.io/BQN/problems.html#right-to-left-multi-
line-functions-go-upwards)

[1]
[https://github.com/mlochbaum/ILanguage](https://github.com/mlochbaum/ILanguage)

~~~
zerker2000
Ah, I see ILang deals with the issue by… not having a name binding primitive,
making you SKI-calculus your arguments into place. (Compare the piles of roll-
swap-dup you get in varless forth dialects, though not as bad due to there
being two directions args are passed in from.)

I suspect there is room here for an unprincipled "preceding word/pattern is
defined by the following line/s" loose-binding operator, to restore
skimability, and a separate inline destructuring let form that cannot leak out
of lexical scope. Which I suppose would warrant lambdas - perhaps with ⍺ ⍺⍺
⍺⍺⍺ to denote enclosing function input, rather than argument/operand/hyperand,
though nonconcrete functions would stress the type system as is

~~~
mlochbaum
It does have : for assignment, although there's no scoping, and you have to
assign to symbols that are surrounded in single quotes. You can see some
examples at
[https://github.com/mlochbaum/ILanguage/blob/master/examples/...](https://github.com/mlochbaum/ILanguage/blob/master/examples/primes.md)
. They need extra parens, since : is an ordinary function and has the symbol
on the left. Your overall point is right; I didn't really try to address any
issues with assignment. And of course there are lots of things to try, and I
hope some of them make it into some other language I design!

------
mncharity
> The Javascript-based compiler[1] is also slow [...] this is purely due to
> the slow runtime

Top-level functions are defined with 'let'. I wonder if 'const' is more likely
to be inlined?

Array's are given a flag property 'sh'. I wonder if adding a property to an
Array drops it off Array operation fast paths? Perhaps interferes with its
optimization on element kinds?

Functions are given a flag property 'm1' xor 'm2'. So shape polymorphism,
different hidden classes, and associated burden on inline caches at call
sites, property accesses, etc. Better to always set both. I wonder if adding
any property to a function drops it off function call fast paths? Or
interferes with inlining? Perhaps these flags might be avoided using
'Function.length'?

I wonder if there's some low-hanging performance fruit here.

[1]
[https://github.com/mlochbaum/BQN/blob/master/docs/bqn.js](https://github.com/mlochbaum/BQN/blob/master/docs/bqn.js)

~~~
mlochbaum
Thanks for the tips; this looks like some good info. Overall I would say
Javascript's performance when evaluating the runtime is outstanding: not to
say it couldn't be improved, but it beat a Go version that I wrote even before
dzaima built me a bytecode->JS converter. That the runtime is a snail in spite
of this is no mystery: the VM provides very few functions, and the rest are
all blocks in the VM, with the full lexical scoping machinery. The compiler
needs to be optimizing away some of that. I'm more interested in these sorts
of optimizations because I do think I'll get the implementation on Wasm
eventually, although it will probably have to JIT things to keep up with JS.

JS should be able to figure out that all those let variables aren't modified,
and the timings don't show any difference (they did in an earlier version
based on eval() instead of Function()). The scheme with m1 and m2 can almost
certainly be improved; I guess I'll look at having a single type property
that's always set.

------
7thaccount
Hey Marshall,

Do you still work at Dyalog? Is this just a personal hobby project?

Why choose JS or Go as implementation languages if you're already quite
familiar with C? I guess web browser support is neat, but what I've always
really wanted was a single executable (no install) that can run as a REPL or
build self-contained executables that bundle the code and interpreter. I think
that could be pretty useful. I know J exists, but it is a huge install.

~~~
mlochbaum
I left Dyalog in June and have since been taking a break from employment to
work on BQN roughly full time. Eventually I'll get another job, but I hope to
keep working on BQN and won't seek to commercialize it.

I hope to have BQN running directly on at least Wasm and x86 eventually.
Currently I'm focusing on the self-hosted parts of the implementation, so the
VM is written in whatever's convenient, portable, and fast. Because the VM is
very small it's easy to port between languages—the Go port I did in about
three days without knowing any Go beforehand. But manual memory management
adds a lot of difficulty, particularly because closures can form reference
loops and have to be garbage collected. I'd rather stay in a memory-managed
environment for as long as I can to work on the unknown aspects of BQN before
attacking the known but hard memory management problems, and when I make that
jump I might skip directly to assembly/machine code because I have more
control that way.

~~~
7thaccount
Since I can't edit my current comment, I'll add that I wish your current
project the best of luck and hope it morphs into something that people want to
use and can use for development.

------
mncharity
A python or julia implementation, integrated with normal numpy, might enable
low-barrier exploration and incremental adoption?

~~~
mlochbaum
This is a nice idea. I'd thought about trying to run BQN on top of an APL
(difficult because existing APLs' boxed or nested models are at odds with
BQN's based "like a normal programming language" array model), but trying the
more mainstream APL-adjacent langauges hadn't occurred to me. numpy at least
looks promising.

~~~
mncharity
A story of "It's a python package, a DSL for numpy" could reduce several
barriers to exploration and incremental adoption. DSL's let one use the
familiar host language, but for that one little bit where you'd be willing to
learn a new and better way of doing things.

A further step in that direction, might be to have a second, more familiarly
pythonic interface. Sort of a Q for K. But with minimized cognitive distance
from a pythonic ideal, numpy api, and BQN. Attempting to create a low-barrier
space encompassing them, for learning and trade offs. Though I'd have to think
about what that might look like. Semantics and vocabulary might be learned
separately from syntax. Perhaps someone might use BQN-native syntax for
prototyping, and then dump it for release as company-acceptable "just using a
normal python library" verbose pythonic code. A counter argument is the APL
and pythonic ideals for code are rather different, so it might be worth using
python and numpy simply as infrastructure, without worrying about chasing
hearts and minds.

------
smabie
Awesome work! Hopefully array languages will undergo a renaissance!

One thing is that even after reading running.md, I'm still kind of confused
about how to actually run BQN. Maybe provide an easy way to get started with
the self-hosted bytecode compiler? Or maybe there is an easy way, but I'm just
stupid.

~~~
mlochbaum
You can use the self-hosted version at
[https://mlochbaum.github.io/BQN/try.html](https://mlochbaum.github.io/BQN/try.html)
. I haven't built any direct way to run it locally yet; you have to require
docs/bqn.js from Node.js, and then call the result on strings to run them. You
can also call test/tj.js with any number of string arguments to evaluate them
and print the results (as formatted Javascript objects, not with nice
formatting).

This stuff is all shifting underfoot now, as I only put the online version up
yesterday. I'll probably make a real executable/REPL to use locally pretty
soon, and I also intend to make some or all of the code blocks shown in
documentation dynamic so you can change the code and reevaluate.

~~~
smabie
Thanks! Gonna give it a try now.

I've actually developed my own array language inspired by kdb+/q and forth
called xs: [https://cryptm.org/xs/](https://cryptm.org/xs/)

It's a little more pedestrian than what you're doing: APL/J have this ethereal
magical quality that k/q seem to lack for whatever reason. Probably because of
the lack of tacit programming (which xs supports btw).

It's a Java world, we array language folks need to stick together. Maybe we
can even break the glass ceiling and get array languages used for front-end
work in a couple decades!

~~~
mlochbaum
I'd seen xs before, actually. I spent a while building concatenative array
languages (nothing this complex though—file functions! The luxury!) before
deciding that I really need some more structure to make sense of the code.

A good amount of implementation discussion goes on at
[https://chat.stackexchange.com/rooms/52405/the-apl-
orchard](https://chat.stackexchange.com/rooms/52405/the-apl-orchard) . I'm
sure we'd enjoy your contributions.

~~~
smabie
I think there's a conflict between the essence of array languages (building a
DAG of monadic (in the FP sense) operations and letting arrays flow through
it) and the one dimensionality of a stream of symbols in a text file. Ben Lynn
[[https://crypto.stanford.edu/~blynn/c/apl.html](https://crypto.stanford.edu/~blynn/c/apl.html)]
puts it best:

"Worse still, tacit hooks, forks, and ranks obscures the beautiful category
theory connections: hooks and forks are function composition and application
within the Reader monad, while adjusting the rank of a function corresponds to
applying functors."

Hooks are a structured approach to this problem, while stacks are an
unstructured one. And there will always be tension between the two for as long
as the current textual model of programming is used (or maybe no matter what
representation we use). How obscure are we willing to get in order to get
closer to the beautiful DAG we have in our head?

I of course don't have a good answer to that, and I think the dominant
paradigm of array languages has set to be discovered. APL gets us close, but
we clearly agree that there's more work yet to be done.

Looking at your bytecode design, I was reminded of this old paper about
building APL hardware:
[http://www.softwarepreservation.org/projects/apl/Papers/1970...](http://www.softwarepreservation.org/projects/apl/Papers/197002_An%20APL%20Machine-
Abrams.pdf) . Not sure how relavent that is, but maybe it'll be the
inspiration of a beautiful VM architecture!

------
meigetsu
If you could get this to run fast on GPUs you might get some converts from
other APL family languages.

