
The BEAM needs an APL-y language - mpweiher
https://medium.com/@gordonguthrie/the-beam-needs-an-apl-y-language-6c5c998ba6d
======
chrisaycock
Performant execution for any vector language (APL/J/K) requires amortizing the
array actions. For example, _summing an array_ should invoke an action in the
VM that adds together all of the elements of the array with only a single type
look-up.

I am skeptical of targeting an array language to a VM that requires a type
check on every element of the array. Numpy handles this by performing all its
actions in a C library independently of the CPython VM.

~~~
geocar
> I am skeptical of targeting an array language to a VM that requires a type
> check on every element of the array. Numpy handles this by performing all
> its actions in a C library independently of the CPython VM.

If arrays are implemented as binaries then you should be fine.

> For example, summing an array should invoke an action in the VM that adds
> together all of the elements of the array with only a single type look-up.

Erlang makes it easy to add NIF when you want sum to be a little bit faster. I
think this is the last problem anyone needs to solve though, the elegance of
array-language solutions is far more important than their performance. That
they're _also_ performant is how we know all other languages are wrong :)

~~~
equalunique
Would be interesting to hear what kinds of BEAM-specific concepts would be
important for a well-designed implementation.

I was curious to learn more about what a "NIF" is and this was helpful towards
understanding:
[http://erlang.org/doc/tutorial/nif.html](http://erlang.org/doc/tutorial/nif.html)

Also I found this which shows there are many examples where NIFs extend what
Erlang can do: [http://beam-
wisdoms.clau.se/en/latest/interfacing.html](http://beam-
wisdoms.clau.se/en/latest/interfacing.html)

------
girvo
Agreed. BEAM is absolutely fantastic, and having other languages to be able to
leverage its power can only be a good thing.

I have high hopes for Alpaca personally, a super modern ML that targets BEAM.
I think that in a similar way to how an APL-like would allow leveraging BEAMs
building blocks in unique ways, an ML type system that’s aware of BEAM would
allow for some seriously powerful, seriously correct programs. There’s a lot
of research into algebras for concurrent and parallel processing, which slot
nicely into ML-likes.

[https://github.com/alpaca-
lang/alpaca/blob/master/README.md](https://github.com/alpaca-
lang/alpaca/blob/master/README.md)

~~~
equalunique
That's neat. Thanks for sharing.

------
dnautics
I feel like a smooth interface between BEAM and Julia would be particularly
useful. Elixir would be a nice platform since both have lisp-y macros and 1st
class AST support, so each language could pass ASTs to each other for
execution. Julia could be used for heavy computational loads (which it excels
at), and BEAM could handle IO, reliability, supervision, and process
management (which it excels at).

An APL on BEAM would probably be terrible; since BEAM intentionally does not
support arrays. Having lists all be linked lists is a deliberate choice
because you have O(1) immutable copy, which is absolutely critical for passing
data between procs without "sharing memory", and creating an absolutely safe
concurrent system. Given the relative heavy weight of BEAM, and the featureset
it provides, it's quite impressive that for I/O and routing heavy tasks its
performance is within an order of magnitude of, say, C.

You could maybe bootstrap arrays as binaries, but as you start making
extensive modifications to the array, you will get crazy sharding of the
binary information, and performance will die.

~~~
gordonguthrie
I am writing another post which will describe better the niche I think it
should fill - an auxiliary language used to write library modules which would
be called from applications written in Elixir or Erlang.

The functionality that would be replaced is currently written by traversing
lists - this really is a business-rules language add-on to existing
infrastructure rather than a high-performance array processing language.

~~~
dnautics
julia is rolling out an all-julia database solution, and I just wish they used
BEAM (probably elixir) as supervisors on top of julia worker processes instead
of relying on julia, principally because of the lack of reliability guarantees
in julia (I love julia, but it's not really a programming language designed
for that)

------
tunesmith
BEAM is the virtual machine for Erlang, for those who were completely lost
reading this like I was.

~~~
gordonguthrie
I only tweeted it at my Erlang chums is why, didn't expect anyone else to care

------
art-w
As much as I love APL for its notation, Erlang does provides equivalent
highlevel operators for processing sequences of elements. It's not fair to
compare your highlevel implementation in APL to your lowlevel optimized (and
quite unreadable) solution in Erlang. Here's the equivalent algorithm:

    
    
        is_descendant(A, B) ->
            lists:all(fun ({Actor, Event}) ->
                            Event =< proplists:get_value(Actor, B, -1)
                      end, A).
        
        compare(A, B) ->
            A_in_B = is_descendant(A, B),
            B_in_A = is_descendant(B, A),
            {A_in_B,
             B_in_A,
             A_in_B and B_in_A,
             not (A_in_B or B_in_A}}.
    

The function `is_descendant` is nearly a translation of your APL code. I'm not
convinced that we need APL on the Beam, for the quality of this VM certainly
isn't in its array processing capabality... but I was expecting the argument
"such an array-oriented language would motivate improvements in this
direction", which is definitively true!

APL-enthousiasts often forget that APL forces you to use its builtin
operators, while modern languages provide means to implement them (and aren't
restricted to arrays). lists:all isn't a builtin in Erlang, but does it really
matters that you can write it as `^/` in APL?

Note for functional programmers: `^` is the boolean AND operator, `/` is
`fold`, and somehow there's an APL builtin hack so that `fold` knows that 1
(true) is the monoid identity of `AND`... but really `^/` is a compact
notation for `fold (&&) True`

For completness:

    
    
        all(_, []) -> true.
        all(Pred, [X | Xs]) -> Pred(X) andalso all(Pred, Xs).
    
        get_value(_, [], Default) -> Default;
        get_value(K, [{K, V} | _], _) -> V;
        get_value(K, [_ | Rest], Default) -> get_value(K, Rest, Default).

~~~
gordonguthrie
Well I'm an Erlang enthusiast not an APL one, my APL experience is reading a
manual for 4 hours on a plane and 4 hours writing APL

There are a squad of specific set related problems in Riak that the Basho
engineers skirted round or were afraid of - and they were all about Set
operations - so APL's focus on tiny notational programmes that are graspable
in the mind is appealing for that basis.

Is this the most considered blog post? No, hell no ;-) but APL struck me as
fascinating...

------
segmondy
I'm not big on the BEAM environment, but I do like to see APL in Prolog. (J in
SWI-Prolog) If anyone is interested in making this happen, ping me.

~~~
fusiongyro
Saw your comment on Reddit about this, am still confused by why you think this
is a good idea. The integration of Prolog variables and backtracking is going
to make a real mess of the array operations, don't you think, or is it going
to be a total embedding? If it's a strict embedding, why not just have a J
process and call it from SWI-Prolog?

~~~
segmondy
Well, think of the IS operator Z IS X + Y. That exists fine and works well. X
and Y can't be a variable only Z really can be. It can be annoying but that's
standard Prolog. So for the sake of discussion let's say we have

apl(Expression, Result).

Where Expression can't be a variable. If it fails to evaluate or has an error
it will return false. If APL evaluates without error it will true true or the
Result.

Prolog can be fast with list if using difference lists. But can be slow for
some vector calculations. Using CHR/CLF one can do some mathematical
calculations fast, but it runs out of memory fast for large dataset.

I might be wrong, but I figure it will be a nice experiment. My idea is to
leave all things Logical and one of off type of code to Prolog and things that
happen in groups to APL.

~~~
fusiongyro
I don't think clpfd or chr are especially large hogs of memory. Have you run
into issues there?

If you can write a smallish program that uses a subset of J (actually APL
might be easier, since it is less likely to interfere with SWI's parser) I
would be willing to try and figure out how to run it. Aim for 5-10 lines.
Email me. I like the idea I'm just not totally convinced.

I feel like if I got better at J, my problems with it probably would be
resolved. For instance, it feels to me like J is really bad at parsing (beyond
regular languages anyway) and tree structures. But there are built-in verbs
for these things, so I could be mistaken. I only threw a few months at it last
year. And I kept being indecisive about APL and J. It didn't help that Aaron
Hsu was such a passionate advocate for APL, and Roger Hui apparently now works
for Dyalog and has tried to close the loop on a lot of the ways J improved on
APL there.

My back-of-the-envelope sense, from writing a small amount of J and a fairly
significant amount of Prolog, is that J is way better at solving the core of a
certain class of problem, but you spend a lot more of your life converting the
input into an internal representation you can use and back to a result a human
might want to see. Prolog really likes a certain class of problem which is
somewhat less general than J, but it makes up for that limitation somewhat by
being amazing at parsing and having great internal representations of things.
In most of the cases where I would want non-determinism in J, I would probably
try to rework my problem so that I'm doing all the possibilities at once and
then detecting which ones succeeded. If I had non-trivial calculations to do,
I probably wouldn't use Prolog, but if I had to, I would certainly consider
calling out to C or something. My day-to-day interests seem to coincide more
with what Prolog does well than what J and APL do well.

Anyway, I think it might be a profitable area to explore. I was curious about
Brachylog for similar reasons.
[https://github.com/JCumin/Brachylog](https://github.com/JCumin/Brachylog)

Anyway, I'm curious if you could show me what you are thinking. There must
have been a smallish program you thought would work well with this
combination. Can you share it?

------
FractalLP
I think APL is for "a programming language", not "another programming
language".

~~~
kencausey
I could be wrong but I believe the word 'joke' applies here.

~~~
FractalLP
Lol I thought the author might be joking, but couldn't tell. Overall I love
APL (hobby language) and think this would be neat if feasible.

------
moomin
BEAM is great, but it’s a bad match for high performance calculation.

On the other hand, purescript for BEAM does exist...

~~~
gordonguthrie
But none of the examples I cite need 'high performance' (well CRDTs do) an
expressive language for sets would remove accidental complexity and make more
complex things simpler and less buggy

------
bobsh
Gnuapl has an interface to Erlang/Elixir: [https://elixirforum.com/t/calling-
apl-from-erlang-or-elixir/...](https://elixirforum.com/t/calling-apl-from-
erlang-or-elixir/4190)

------
galfarragem
What I would like to see is an APL-y language targeting nodeJS.

~~~
geocar
[https://github.com/JohnEarnest/ok](https://github.com/JohnEarnest/ok)

[https://github.com/ngn/apl](https://github.com/ngn/apl)

~~~
etatoby
ngn/apl is great, it's one of the most modern and complete implementations of
the language

------
madmulita
Basho's bankrupt?! When did this happen?

~~~
rrdharan
About a year ago, although they were so quiet about it no one figured it out
for another couple months:

[https://www.theregister.co.uk/2017/07/13/will_the_last_perso...](https://www.theregister.co.uk/2017/07/13/will_the_last_person_at_basho_get_the_lights_oh_too_late/)

[https://www.theregister.co.uk/2017/07/31/end_of_the_road_for...](https://www.theregister.co.uk/2017/07/31/end_of_the_road_for_basho_as_court_puts_biz_into_receivership/)

~~~
gordonguthrie
Riak the database so indestructable that is survives its parent companies
demise is going into code freeze for its first post-bankruptcy release even as
we speak

