
The Lobster Programming Language - openbasic
http://strlen.com/lobster/
======
Aardappel
Hi, author of the language here.

Not a great time for HN exposure, as I just added a lot of new functionality,
but none of that is documented yet :) But hey, all publicity is good
publicity?

For those interested, some recent things that have been happening:

\- Compile time reference counting. This does an analysis similar to the Rust
borrow checker to infer lifetimes, but unlike Rust doesn't make the programmer
jump through hoops.

\- "Inline" structs: structs are allocated in their parent, and come at zero
overhead compared to individual variables, which is particularly useful for
Lobster's heavy use of 2d & 3d vector types.

\- Multi-threading that uses separate memory spaces, meaning individual
threads can run at max speed with no GIL overhead etc.

\- A WebAssembly backend is being worked on.

\- A lot of optimization work to make the language run faster.

For reference, it has been on HN before, as early as 2013
([https://news.ycombinator.com/item?id=5904430](https://news.ycombinator.com/item?id=5904430)),
but it has changed a TON since then (e.g. it was a dynamically typed language,
now it is statically typed, with flow sensitive type inference).

~~~
calebwin
Would you mind elaborating on the "compile time reference counting"? Is this
new? How does it handle cycles?

~~~
Aardappel
It is so new I haven't even documented it yet.. hence why I said the timing
for this HN post isn't great.

So imagine the current implementation does a lifetime analysis somewhat
similar to Rust, but then where Rust would error out, this implementation
simply inserts a runtime refc increase. This gets rid of most runtime refc
overhead without the programmer needing to be clever about it.

Then, I intend to add an optional type annotation that gets you the Rust
behavior, for those cases where you want maximum control. I don't think its
the right default though.

The algorithm is more lenient than Rust, since I don't have any shared memory
concurrency for example.

I intend to write up more thorough details about the algorithm, but again,
haven't gotten to it.

It does not handle cycles. If you create cycles that you do not manually
break, you'll get a cycle report at program exit that prints the kind of
values that weren't reclaimed. You're then expected to fix your code
accordingly :)

~~~
alexisread
This at first glance actually looks similar to [http://liu.diva-
portal.org/smash/get/diva2:20899/FULLTEXT01....](http://liu.diva-
portal.org/smash/get/diva2:20899/FULLTEXT01.pdf)

And this style of static analysis has been taken even further by Luc Blaeser:
[http://concurrency.ch/Content/publications/Blaeser_Component...](http://concurrency.ch/Content/publications/Blaeser_Component_Operating_System_PLOS_2007.pdf)

The cycle checker I'll have to have a look at - I'd like to crib parts of it
for an OS I'm trying to cobble together. Your project looks very interesting,
thanks!

~~~
Aardappel
Thanks for those, I hadn't seen them. The Ritzau paper indeed appears to be
doing something very similar, where in 5.2 he describes "inner references"
whose lifetime is bounded by the outer one, I call that "borrowing", where the
outer definition (the owner) gets locked from mutation during the borrow.

What I do is a bit more involved though, as for every language construct I
indicate if it wants to own or borrow its children (and wether it wants the
parent to own or borrow), to remove maximum refcounts, also for cases which
are not simply variables. Again, I will describe this in detail.

The second paper I don't see how it relates to what I am doing.

------
dfox
Major reason why many game engines used to have native core and then some kind
of "scripting" language for game logic (think Unreal Engine before 3, but many
engines are similar) is that then you can trivially serialize the simulation
state (ie. save game) and even run the whole thing as either lock-step,
forward predicted or fully distributed network game without much effort on the
part of whoever is writing the "scripting language" part of the game code.

~~~
Aardappel
Agreed, those are nice benefits. But there are also huge downsides, in the
sense that you're writing all the glue code in the wrong language :) The
amount of speed of iteration you get from having your entire engine + game's
structure in a friendlier language is impressive (in my experience).

Also doing savegames by just serializing the entire scripting VM may work in
some cases, but can also be problematic, since you're mixing ephemeral data
with gameplay state. Last U3 engine game I worked on had explicit savegame
serialization :)

~~~
dfox
My point is that having scripting VM that is explicitly designed for game
logic allows you to serialize only the relevant state. And if the whole system
is designed correctly you can also do checkpointing and streaming replication
essentially of the game state for free. This buys you things like rendering
thread that is decoupled from the simulation (just draw the snapshot of state
that was current on the start of rendering iteration), non-synchronized joins
of multiplayer players (send serialized checkpoint and do stream the changes
until the client catches up) and so on.

~~~
Aardappel
I agree, for that level of integration you want a "gameplay scripting system",
whereas Lobster is more of a general purpose programming language that happens
to be a nice starting point for simple games & engines. I don't think in its
current form it is a great language for a large AAA team anyway, because of
its heavy reliance on type inference and lack of separate compilation.

The language could of course be adopted to these use cases, it simply hasn't,
so far.

Doing simulation on a separate thread from main/rendering is something it is
already set up to do (multi-threading uses separate memory spaces so it is
easy to ensure there's no data races between the two).

------
baldfat
I use Haxe which is a

Haxe is an open source toolkit based on a modern, high level, strictly typed
programming language, a cross-compiler, a complete cross-platform standard
library and ways to access each platform's native capabilities.

what would be the benefit for using Lobster since I can work directly with
each OS with a great scripting language built in?

[https://haxe.org](https://haxe.org)

~~~
Aardappel
Haxe can translate to an impressive set of languages and platforms, so if that
is your main need, I'd say, stay with Haxe.

Haxe and Lobster are very different programming languages however, so you may
prefer one over the other purely for the language design.

If you like portability, Lobster already runs on 3 desktop OSes, 2 mobile
ones, and the web, using either VM, C++ or soon WebAssembly modes. While not
as extensive as Haxe it is pretty good :)

------
aitchnyu
Did you create a VM that is several times faster than Python's, for a static
typed language thats as productive as a dynamic language, with true
multithreading? How did you outdo entire language communities?

~~~
tomp
I'm a huge proponent of typed languages, but there's no way that a statically
typed language could be as productive as Python (with current type systems
technology). For example, just the Pandas library offers incredible
flexibility (often dynamically based on the arguments that different functions
get) that I've yet to see replicated in a static language.

~~~
brianpgordon
My experience is that you start out very productive not being weighed down by
having to satisfy the type checker... but programs tend to grow over time and
you end up regretting it. When you have a medium to large codebase in a
dynamically typed language it gets to be more of a hindrance than a help.

~~~
tomp
I agree, but my point was rather that you _can 't_ typecheck Pandas (or a
similarly complex library). Gradual typing for the win!

------
c3534l
I don't really see the point in using Python syntax and then throwing in a
handful of random differences that don't have anything to do with the domain
you're working in. Syntax level support for vector operations or for shader
interopability I would get. Making for loops follow some randomly different
syntax, I don't see the point of that.

~~~
Aardappel
It is not intended to be Python compatible, it merely has indentation based
syntax that is similar.

The cases where it differs from Python are actually for good reason: unlike
Python that only has built-in control structures, here any function can look
like if/for/while if it takes a lambda body.

~~~
andrelaszlo
This seems like a cool way to make an ad hoc dsl. I'm not a game programmer
but I imagine it could be useful.

~~~
stcredzero
_This seems like a cool way to make an ad hoc dsl. I 'm not a game programmer
but I imagine it could be useful_

Smalltalker here. Being able to pass in lambdas very casually is a _great_ way
of producing an ad hoc DSL. Imagine a custom control structure which does a
database transaction. Just implement a function!

~~~
Aardappel
Yup, this style of programming I use all the time in Lobster.

------
throwaway190102
This reminds me of processing.
[https://processing.org/](https://processing.org/)

~~~
bluetwo
Exactly.

------
jkcxn
This looks really fun:
[https://github.com/aardappel/lobster/blob/master/lobster/sam...](https://github.com/aardappel/lobster/blob/master/lobster/samples/lazy_py_triples.lobster)

Are the generators implemented entirely in user space using continuation
passing style? And the coroutine function turns it into something pausable?

Edit: I can't wrap my head around how return returns out of all the generators

~~~
Aardappel
So the non-coroutine case of that code are all regular functions, without
using CPS or anything. The only special feature Lobster has is that when you
write "return" it will return out of the lexically enclosing named function,
not the function whose body it sits in (which can be an anonymous function).
This mean return is implemented by unwinding stack frames until it gets to the
named function, almost like a lightweight version of exception handling. It
also can't resume these functions, like you typically can in languages that
support CPS.

Now the co-routine functionality takes an existing higher order function, and
compiles it such that the call to the function argument instead does a yield.
These are really regular co-routines that are implemented by copying stack
frames off and onto the stack upon yield and resume. I just thought being able
to share code between HOFs and co-routines was cute, since they often express
the same patterns. Why would you write an iteration twice?

------
eggy
I am an old Amiga user Vic-20, A1000, A500, etc., so I am a fan of the author.

I tried running the sierpinski example with a pre-compiled Windows .exe that I
placed in the lobster root directory, and it gives me this error: "can't open
file: stdtype.lobster".

I placed lobster and lobster/modules in my path, and it still throws the same
error. I am assuming I need to build the .exe myself due to some
incompatibility with the master branch and this older .exe?

I like a self-contained game PL. I use Raylib for that type of experience
right now, but I like the concepts here. Thanks!

EDIT: copied modules to root of lobster, and now I get:

"stdtype.lobster(8): error: end of file expected, found: <"

I tried an included test file and a copy/paste of the sierpinski example, and
both throw this error.

~~~
Aardappel
Sorry, I need to do better at "getting started" documentation. The exe needs
to be placed in the "lobster" dir under the root. It searches for "includes"
relative to itself.

Not sure about the error involving "<".. are the executable and the data from
the same version of Lobster? Open an issue on github or email me for more help
:)

~~~
eggy
I'll build it as the site suggests, since I am using a binary from the release
with the master branch modules. I'll follow up on github. Thanks!

------
hyperpallium
Is the first example's last line:

    
    
      var i = find [ 1, 2, 3 ]: _ > r
    

the same as?:

    
    
      var i = find ([ 1, 2, 3 ]) a: a > r
    

i.e. argument parantheses are optional for one argument; and _ is an implicit
argument for a block.

Would multiple functions arguments be done just with parentheses, like this?:

    
    
      var i = find [ 1, 2, 3 ] (: _ > r), (: _ < r)
    

Seems like the syntactic generality of lisp or haskell, with the
predictability of performance of C. EDIT I see it's as fast as non-JIT Lua...
are there any features that prevent it from theoretically approaching the
speed of C?

PS: small detail, but that's got to be the best for-loop I've ever seen.

~~~
Aardappel
Yes, those are equivalent.

Multiple function arguments require a keyword in the caller, much like "else"
introduces the second function argument in an if-then-else. So if "find" would
take two functions, one for even and one for odd list elements, it would look
like: "var i = find_even_odd [ 1, 2, 3 ]: _ > r odd: _ > r"

There are no features that prevent it from "approaching" C's speed, merely a
question of implementation work. The language started out as very high level,
so code generation needs to be improved to not do as many things at runtime as
it currently does. The language semantics are totally able to be within 1.5x
of C.

~~~
hyperpallium
Fantastic, thanks.

The many uses of : syntax are intriguing - lambdas, functions, control,
python-style structure, data, globals. I reckon you've established it works in
all cases, but I'm not yet familiar enough to see that for myself.

For initial adoption, I wonder if more regular sample code, without the
special-case abbreviations, might be more effective? (Followed by the short
version.) OTOH, maybe at this early stage it's best to select for developers
interested/capable enough to handle it!

EDIT if : is used for both blocks and code structure, could everything be one-
line (or is \n significant?) Not great style, but helps in understanding the
syntax. Maybe returns and vars need their own lines?

    
    
      def find(xs, fun): for(xs) x, i: if fun(x): return i return -1
    
      var r = 2 var i = find [ 1, 2, 3 ]: _ > r

~~~
Aardappel
Yes, ":" appears a bit overloaded, though many of those cases are actually one
and the same.

What would be more regular sample code? Do you mean just writing "find([ 1, 2,
3 ]) x: x > r" with explicit () and explicit variables? I agree that is easier
to read, though the extreme terseness is also a feature..

Yes, everything can be one line, but in this case it would look a bit ugly:

def find(xs, fun): (for(xs) x, i: if fun(x): return i); return -1

~~~
hyperpallium
> Do you mean just writing "find([ 1, 2, 3 ]) x: x > r" with explicit () and
> explicit variables? I agree that is easier to read, though the extreme
> terseness is also a feature..

Yes, amd yes it depends on the purpose. For learning, it can be impossible to
parse; redundancy helps distinguish parts and provides a check you got it
right.

OTOH for showcasing features, IMHO, the generality of : is a more impressive
feature to demo.

Anyway, it's certainly intriguing! And maybe that's the true purpose of
showcase code...

------
Gravityloss
Very personal opinion and matter of taste: Many lambda syntaxes are a bit
weird.

    
    
        map(my_array)element:element*2 //lobster
        my_array.map{|element|element*2} //ruby
        Arrays.stream(my_array).map(element -> element*2).collect(something list something)  //java?
    

Lobster feels a bit unbalanced. Ruby has "pipes" which are a bit confusing
substitute parentheses. Java has the cool arrow syntax but otherwise it feels
very glued on. (edited and fixed, thanks for the comment)

~~~
Aardappel
Your Lobster version has a stray comma, it is: map(my_array) element: element
* 2

Note how if you write this example with multiple statements instead of
"elements * 2", suddenly the Lobster syntax looks a lot more consistent, and
the Ruby example doesn't look that great as it looks completely different from
the built-in for loop.

~~~
yxhuvud
No-one uses the built-in for loop in Ruby though. Instead, everyone uses the
each method which looks the same as map.

That said, the Ruby syntax can be a bit bulky. Your syntax feels pretty clean
TBH.

------
estomagordo
Really stupid question here, but how do I start compiling Lobster programs on
my machine (Windows or Linux)?

~~~
Aardappel
There are some releases which come with pre-built exes for e.g Windows:
[https://github.com/aardappel/lobster/releases](https://github.com/aardappel/lobster/releases)

I'd highly recommend you build it yourself though, which can be done rather
easily with VS (2019) on Windows or CMake on Linux. On Windows just press
build in release mode and you're done. More notes here:
[https://htmlpreview.github.io/?https://raw.githubusercontent...](https://htmlpreview.github.io/?https://raw.githubusercontent.com/aardappel/lobster/master/lobster/docs/implementation.html)

Then cd lobster && lobster samples/pythtree.lobster for example should run
something. Or see here how to use from command line or from your fav editor:
[http://htmlpreview.github.io/?https://github.com/aardappel/l...](http://htmlpreview.github.io/?https://github.com/aardappel/lobster/blob/master/lobster/docs/README_FIRST.html)

~~~
estomagordo
Cheers!

------
sansnomme
This reminds me of the language used in Godot Engine, maybe talk to them and
use this in the engine?

~~~
Aardappel
Hah, that would be a pretty good fit, yes.

------
tjpnz
I really like the inclusion of basic graphics routines. Not a lot of languages
include these out of the box and it's something I've really missed since I
first started toying with programming in QBasic.

~~~
brianpgordon
It's kind of surprising to me that the language is directly tied into OpenGL
given that it's a cutting-edge experimental new language and that seems more
the domain of Vulkan.

~~~
Aardappel
It's not tied to, it is implemented in it. I could swap out the rendering code
for Vulkan, an no Lobster users would need to change anything.

Also, have you actually programmed in Vulkan? Even a simple primitive can be
hundreds of lines of setup code. In Lobster they're often a single line. A
direct mapping would not make sense.

------
qarw
000000128039 000000128034 000000128039 000000128034 000000128034 000000128034
000000128039 000000128008 000000128039 000000128008 000000128013 000000128039
000000128013 000000128034 000000128039 000000128034

------
z3phyr
I enjoyed playing Cube based games, editing maps live and whatnot. Will check
this out too!

~~~
Aardappel
Thanks, let me know what you think :)

------
jbb67
Oh this looks rather nice. I'll have to take a look at this.

------
sscarduzio
Why choosing 'value' as a keyword for types? Super odd

~~~
Aardappel
I've actually since changed that to be more conventional. Now the "inline" by-
value objects are called "structs" and the regular by reference objects are
called "classes" much like C#

------
mshockwave
> Discuss things in the google+ community

google+?

------
stesch
Yeah, cool Google+ community.

------
Eire_Banshee
Portable python for games...? Does it compile and run natively?

~~~
Aardappel
It's not Python.. it just has a similar syntax.

You currently run it either as a bytecode VM (which is rather fast and runs on
any platform), or you can translate it to C++ if you want an additional speed
boost.

