
Carp, a compiled Lisp with type inference and a borrow checker - eriksvedang
http://blog.veitheller.de/Carp.html
======
sitkack
Others might be interested in Henry Baker's Lively Linear Lisp

[0] HN Discussion.
[https://news.ycombinator.com/item?id=14248419](https://news.ycombinator.com/item?id=14248419)

[1]
[http://home.pipeline.com/~hbaker1/LinearLisp.html](http://home.pipeline.com/~hbaker1/LinearLisp.html)

~~~
agumonkey
Loved that article, made me realize a lot of things about ... lots of things.

~~~
sitkack
That bibliography is amazing. The best papers also have great bibliographies
where I can dive in and learn mind altering ideas. Boring papers have boring
bibliographies that just echo a list of the big papers in a field.

------
arximboldi
I really want to like Carp but I somehow feel that the Rusty memory model does
not fit the Lisp philosophy... all the borrowing story comes from a basis
mutability.

I wish for something more Clojurish based on immutable data. Then one can
exploit the power of inferred type linearity/affinity to transparently build
safe "transients" and other cool stuff. Maybe some day I should write such
Lisp myself on top of my C++ immutable data structures [1] :-)

[1] [https://github.com/arximboldi/immer](https://github.com/arximboldi/immer)

~~~
nickpsecurity
One of the good LISP's back in the day was PreScheme that let it finally be a
C alternative efficiency-wise. It was also used in first, verified LISP. Such
a style might also be good for bootstrapping or just making more flexible ways
to code C.

[https://en.m.wikipedia.org/wiki/PreScheme](https://en.m.wikipedia.org/wiki/PreScheme)

Looking at PreScheme, Carp doing a C alternative safe without a GC is a nice
evolution of these LISP's. Next cool possibility: use a Rust-like LISP with an
OS project like Mezzano on lowest-level stuff. Might even start with Redox OS
just LISPifying its parts for a kernel and shell to start with.

~~~
progman
> Next cool possibility: use a Rust-like LISP

Why do you want a Rust with Lisp syntax? I don't see the benefits of such a
"Lisp" if you take the core benefits of metaprogramming at runtime away which
depends on the equality of code and data.

Safe software can also be written in other languages than Rust. Ada is still
industry standard of safe programming today. Even Lisp can be used to write
safe software since Lisp's memory management takes care of possible pointer
problems.

Rust shines in the field where Mozilla developed it for -- safe Internet
browsers. However, safe Internet browsers could also be written in Ada and
Lisp. The Lisp version would just not be as performant as Ada's and Rust's.

~~~
nickpsecurity
"Rust-like LISP"

The phrase was ambiguous. I meant a LISP w/ safe, memory model like Carp.
There's already LISP's in the past and present for OS's. There's also projects
that restrict the power of languages to get better efficiency in low-level
work. So, more of the latter with Rust's memory model might be advantageous.
The former with their flexibility could be built on top of that same language
with a low-latency GC for anything done dynamically or which the analyzer just
couldn't handle enough to know safety. Might lead to a great performance boost
on top of safety benefits.

"Safe software can also be written in other languages than Rust. Ada is still
industry standard of safe programming today."

You're right that Ada was a language systematically designed for safety that
people could be using right now. I gave them some resources on that here:

[https://news.ycombinator.com/item?id=15771552](https://news.ycombinator.com/item?id=15771552)

The problem: Ada does _not_ have temporal safety or flexible support for race-
free concurrency. The first part in Ada doesn't exist at all: they have to do
unsafe deallocation when their tricks like memory pools aren't enough. The
second, Ravenscar, is like a tome in its restrictions compared to the basic
rules of the borrow-checker in Rust. Rust smashes Ada so hard on "safety
without GC concept" that I told Yannick McCoy at AdaCore to get someone on
achieving parity pronto. For SPARK if nothing else since it would have
provably-safe sequential procedures whose composition was also safe w/ Rust's
method. That would be _awesome_ for projects such as IRONSIDES DNS.

"Rust shines in the field where Mozilla developed it for -- safe Internet
browsers."

Rust shines anywhere you want safe, systems code without a GC and with
concurrency. That's it's main selling point, not language design. There's tons
of applications for that as the huge ecosystem is showing us replacing all
sorts of components in other languages with safe, fast alternatives. Redox OS
did an OS in it. In high-assurance sector, Robigalia was wrapping seL4 so one
can have safe apps in Rust on a formally-verified, separation kernel.
bluejekyll upthread wrote a DNS server (TrustDNS) in it. Rust can do about
anything an analyzable subset of C can do.

"The Lisp version would just not be as performant as Ada's and Rust's."

This is correct. My recommendation to use one close to the metal like
PreScheme with Rust-style safety that outputs C for LLVM attempts to address
that. All the macros apply until the resulting code is straight-forward. The
translation should be straight-forward to C where LLVM does the heavy lifting.
One might also use techniques from projects such as Chicken Scheme, a whole-
program optimizer, and/or a super-optimizer. The speed differences might be so
minimal that nobody even cares. They already are for several LISP's on most
applications but I'm talking system code.

~~~
progman
> My recommendation to use one close to the metal like PreScheme with Rust-
> style safety that outputs C for LLVM attempts to address that.

Wouldn't it be easier to develop a very basic OS core in Rust, and a simple
Lisp 1.5 interpreter written in Rust as foundation for a full blown Lisp?

~~~
pjmlp
No, because PreScheme is still a Lisp like language, whereas Rust would be
bringing something else into the table as well.

------
partycoder
From their github ([https://github.com/carp-
lang/Carp](https://github.com/carp-lang/Carp))

The key features of Carp are the following:

\- Automatic and deterministic memory management (no garbage collector or VM)

\- Inferred static types for great speed and reliability

\- Ownership tracking enables a functional programming style while still using
mutation of cache friendly data structures under the hood

\- No hidden performance penalties – allocation and copying is explicit

\- Straight-forward integration with existing C code

~~~
dkersten
This actually sounds like exactly what I’ve been wanting! I use Clojure and
it’s great, but I’ve been wanting to use something to interop with some C++
code but while I like C++ more than most, I’d rather write in Clojure (or
something like it). Awesome!

------
michaelmior
> Carp compiles to C...It also likely doesn’t matter much, because chances are
> your machine has a C compiler.

Whether or not I can compile the code is only one of the many concerns to
consider when picking between C and LLVM.

~~~
hellerve
That’s quite true, but from the users perspective, it mostly doesn’t matter,
at least in my mind. For people interested in the internals—which is an
increasing amount of the user base—, and the people working on the compiler,
it most definitely is a concern.

~~~
michaelmior
One place I can see this being an issue for users is debugging. I haven't
tried Carp so I don't know how much of a practical issue this is. But if what
is actually being compiled is a C program, how do I tie issues in the compiled
C program all the way back to my source in Carp?

------
catnaroek
The borrow checker is _the_ reason to use Rust, in spite of its annoyances
(clunky syntax, limited type inference, non-interactive programming, long
compilation times, etc.), so it's always nice to see someone trying to provide
the upsides of Rust without the downsides.

That being said, your website is disappointingly terse regarding how Carp
recovers the advantages of Rust in the context of a Lisp derivative. What
makes this particularly suspicious is the fact that, in the past, Lisp
programmers have claimed to recover the advantages of other statically typed
languages in a Lisp setting, but there have always been huge caveats, like
“the system can be very easily circumvented, reducing the benefits of static
typing to nothing”.

The main reason why I feel confident that Rust is a safe language isn't just
the fact that rustc seems to catch many bugs in practice. That alone wouldn't
distinguish it from the countless static analysis tools for C and C++ that
have existed for decades. The main reason why, at least in my opinion, Rust is
such a trustworthy language in spite of its ginormous complexity, is the fact
that its core developers, in particular Niko Matsakis, do a terrific job of
communicating the process by which Rust's features are conceived and designed.
When you read Matsakis' blog [0], it becomes patently clear that Rust's
developers take into account the kind of corner cases that in many other
languages are only discovered months or years after the feature has been
implemented [1][2].

Other languages that inspire similar confidence in me are:

(0) Standard ML. It has a formal proof of type safety.

(1) Pony. It has a formal proof of type safety.

(2) Haskell, _as specified in the Haskell Report_ (i.e., without the crazy
GHC-specific language extensions), because its type system is similar to
Standard ML's, and there are no reasons to believe the differences introduce
any type safety holes.

(3) OCaml, _again without the crazy extensions_ (GADTs, first-class modules,
etc.), for more or less the same reasons as Haskell.

Links:

[0]
[http://smallcultfollowing.com/babysteps/](http://smallcultfollowing.com/babysteps/)

[1]
[http://joyoftypes.blogspot.pe/2012/08/generalizednewtypederi...](http://joyoftypes.blogspot.pe/2012/08/generalizednewtypederiving-
is.html)

[2]
[http://io.livecode.ch/learn/namin/unsound](http://io.livecode.ch/learn/namin/unsound)

~~~
eadmund
> in the past, Lisp programmers have claimed to recover the advantages of
> other statically typed languages in a Lisp setting, but there have always
> been huge caveats, like “the system can be very easily circumvented,
> reducing the benefits of static typing to nothing”.

It seems to me that the benefits of static typing only apply to accidental
mistakes (e.g. using a pointer to a character as though it were an integer, or
a pointer to a struct, or whatever), and thus that a system with deliberately-
circumventable static typing is just fine.

~~~
catnaroek
> It seems to me that the benefits of static typing only apply to accidental
> mistakes (e.g. using a pointer to a character as though it were an integer,
> or a pointer to a struct, or whatever), and thus that a system with
> deliberately-circumventable static typing is just fine.

Static typing is useful to enforce the integrity of abstractions across large
systems. For instance, using (language-enforced) abstract data types, you can
confidently say that a 50 KLOC program won't destroy the invariants of a
complicated data structure, because the only place where this could
hypothetically happen is a 1500 LOC module, and these 1500 LOC have been
verified to the death. Elsewhere, the internal representation of this data
structure isn't accessible. Before anyone claims this can be done using
object-orientation: No. Object-orientation allows the creation of ill-behaved
impostors that are indistinguishable from well-behaved objects unless you use
expensive dynamic checks or even more expensive whole-program (and hence non-
modular) analyses.

Static typing is also useful to enforce the exhaustiveness of case analyses.
Disregarding memory safety issues, which are largely a nonproblem in high-
level languages, the vast majority of bugs in computer programs arises from
failing to identify corner cases or fully comprehend their complexity.
Algebraic data types allow you to substitute ad hoc case analyses with
induction on datatypes, for which mechanical exhaustiveness checks are
possible, and, in fact, actually performed in practice.

Static typing is also useful as an aid to program verification. Program
properties of interest can be classified in two groups: safety properties and
liveness properties. Safety properties assert that “bad states are never
reached”, and are largely covered by type soundness (for non-abstract types)
and type abstraction a.k.a. parametricity (for abstract types). Liveness
properties assert that “good states are eventually reached”, and, while types
provide less support for verifying liveness properties than safety ones, at
least induction on datatypes provides a easy-to-use tool to verify that non-
concurrent (but possibly parallel) algorithms will terminate.

All of these benefits fly out of the window if static typing can be
circumvented.

------
igt0
I wonder why nobody said one word about Shen Language[1].

[1] [http://www.shenlanguage.org/](http://www.shenlanguage.org/)

------
jstewartmobile
So, Chicken Scheme[0] with a borrow checker?

[0] [https://www.call-cc.org](https://www.call-cc.org)

~~~
bjoli
But without dynamic typing and Cheney on the MTA and with a completely
different way of managing memory.

And not a scheme.

~~~
jstewartmobile
Scheme _is_ a lisp, and I'm pretty confident that two of the biggest selling
points of lisp are dynamic typing and a strong distaste for (if not an
outright prohibition of) mutation.

Put it as a question because if you take those two things away, wouldn't it
just be Rust with parenthesis?

~~~
steinuil
The selling point of lisp is macros, and macros alone. Different dialects will
have different characteristics: Common Lisp is dynamically typed and usually
makes heavy use of mutation, others like Lux are purely functional and
statically typed.

> _wouldn 't it just be Rust with parenthesis?_

That's pretty much what it is.

~~~
jstewartmobile
Yes, but will macros retain their power in a statically typed environment?

~~~
masukomi
They work pretty well in Crystal[1]. Some things are certainly a bit harder
than they are in lisp/scheme but you can get a lot done with them. In crystal
this is (in my experience) due to the lack of reflection support, and thus the
ability to call dynamical looked up methods. Add that in and i'm pretty sure
you'd remove >=90% of the current limitations, but even without that, you can
get a lot done.

[1]: [https://crystal-lang.org/](https://crystal-lang.org/)

------
greatNespresso
I find it really interesting, and I would definitely like to try it out (the
article is great by the way) I am not that good at C or Haskell, so that would
be a great way to improve on this too, and that would be my main objective
actually. So noob question : how would I go adding some C networking libs to
Carp ? Should I directly call the code from Carp (and is that possible at the
time) or should I embed it in the language core and go the haskell route ? I
am new to this but would really love to embark into the adventure, for
learning purpose.

------
Veedrac
It seemed they don't distinguish between mutable and immutable references (?).
Does Carp not handle the issue of shared mutability, eg. iterator
invalidation, like Rust?

~~~
eriksvedang
Correct, not at the moment. Overall the type system is less expressive but
also less dependent on annotations. Differentiating between immutable and
mutable refs is probably coming though, it’s a useful distinction for sure.

~~~
chombier
Is there a documentation about the type system somewhere?

------
zitterbewegung
Add an interface to Caffe or Torch and this could be even cooler. I wanna try
this out. Looks like a better prescheme from s48.org

------
yawaramin
Very excited about Carp. In general I like the simplicity of Lisp but
ironically I'm not a fan of dynamic typing. Carp may be a real innovation. A
bit surprised though that the Carp C runtime functions are not namespaced.
With names like `IO_println` there's a risk of clashes. Unless the compiler is
doing some tricks to hide names?

~~~
eriksvedang
The ”IO_” part is the namespace, so no clashes!

~~~
yawaramin
It's not enough :-) Ask the OCaml folks, they're going through this pain right
now because their stdlib modules are called `Array`, `List`, and so on.
They're planning to put them all under `Stdlib`, so e.g. `Stdlib.Array` and so
on, but it's going to be a big effort with a lot of pain.

The main problem will arise when users create their own libraries; suppose
some people create a `Option` libraries and then later you want to add a
standard option type to Carp, it will be painful. Better to namespace your
stuff under `Carp` from the beginning, so e.g. `Carp_IO_println`,
`Carp_Option_map`, and so on.

~~~
eriksvedang
OK, I see what you mean now. Thanks for the feedback, I'll make sure to think
about this more before stabilising the modules.

~~~
vog
I wonder how this compares to the Python ecosystem, whose standard library is
also quite large. (Anyone remembers their famous "batteries-included" mantra?)

Python 2 didn't have a stdlib namespace, or "package" in Python speak. But
when they created their clean-slate approach with Python 3, they also decided
not to introduce a stdlib package. External Python packages just avoid the
stdlib package names, and everything seems fine.

So, is there something in the Python language that makes the missing?

Or is it just the Python community which doesn't care about (or plays down)
this issue?

~~~
mwatts15
As a Python programmer, I would consider code which shadows the names of
built-ins to be poorly written although it's permissible in small scopes
(e.g., a single block).

OTOH, for extending / wrapping there is a builtins module, which I think is
what you suggest: [https://docs.python.org/3/library/builtins.html#module-
built...](https://docs.python.org/3/library/builtins.html#module-builtins)

------
slowmovintarget
Clojure + Rust => Carp?

~~~
chubot
It sounds like arrays are mutable in Carp, so it doesn't feel like Clojure to
me.

As far as I understand, Clojure is also not big on macros, while Carp appears
to be.

~~~
slowmovintarget
Aye, that's the one big disappointment. Maybe the first pet project is an
implementation of Clojure's immutable collections.

~~~
Rusky
Immutable collections don't play well with non-GC memory management.

~~~
dgreensp
Exactly. Someone who doesn’t see the value in a GC-less language is not going
to see the value here.

I personally think “functional persistent” data structures, as a language
default, trade a lot of runtime performance in order to achieve some
guarantees like thread-safety and functions not having side effects. Every
mutation to every array or dictionary is treated like a transaction in a MVCC
database, with the garbage collector in charge of cleaning up the records of
every past version that wasn’t actually necessary to keep around, because no
multiversion concurrency was actually necessary in that case. I encourage
languages to experiment with other methods of achieving similar benefits.

I think limiting side effects and shared mutable state is very important, but
at a local level, imperative code that mutates a data structure is highly
readable and performant. Certain algorithms practically require it. Certain
hardware practically requires it.

Functional languages let you think in terms of abstract “compound values,” but
in practice these are backed by tons of little heap-allocated objects, partly
so that lots of sharing can be done when the data structure is copied on every
change.

------
mhd
As a Perl programmer, that name confuses me slightly.

~~~
senorsmile
## confusing indeed ... || croak "Wrong language!";

------
swlkr
This project looks really good. I would love to have a clojure inspired lisp
that's fast and doesn't require the JVM.

~~~
sitkack
You might try Avian or Multi-OS Engine for embedding a JVM within your
program.

[0] [https://github.com/ReadyTalk/avian](https://github.com/ReadyTalk/avian)

[1] [https://software.intel.com/en-us/multi-os-
engine](https://software.intel.com/en-us/multi-os-engine)

~~~
pjmlp
Or even one of the commercial JDKs with AOT native code compiler.

------
mratzloff
I realize that almost every language could be described as a subset of C++,
but I read articles like this and think...

Deterministic language that has type inference, C interop, and uses ownership
to govern object lifetimes? We have that. It's C++11. auto with
std::unique_ptr and std::move().

Only slightly serious. ;-)

~~~
sitkack
So Carp could compile to that subset of C++. But C++11 isn't _just_ that. It
is all of C++11.

C++ would even more successful if it was easy to make a proper subset of it.

~~~
geokon
That's what the GSL was supposed to be

[https://github.com/Microsoft/GSL](https://github.com/Microsoft/GSL)

It seemed really cool. All compile time checks. I have no idea why it didn't
take off

~~~
pjmlp
Sure it did.

Some of the new C++17 library updates come from there, and both clang and VC++
implement those static checks.

~~~
geokon
I've only have a pretty basic understanding of how it works, but I thought the
checks didn't rely on the compiler. They're special templates that fail when
their conditions aren't met

And when I say it didn't take off - I mean that I haven't come across a single
large project that has take up using the GSL. Hope I'm wrong - maybe I just
haven't looked in the right place?

~~~
pjmlp
No, GSL is a kind of stop-gap solution, until standard catches on.

For example _std::string_view_ and ongoing design on _std::array_view_ are
based on _gsl::span_. The _gsl::byte_ is also no longer needed on C++17 thanks
to _std::byte_ as yet another example.

The GSL asserts are there until code contracts[0] get into the standard.

The magical types like _gsl::owner_ allow clang-tidy and VC++ checkers to
apply a Rust-like memory tracking usage.

Kate Gregory did a presentation at CppCon 2017.

"10 Core Guidelines You Need to Start Using Now"

[https://www.youtube.com/watch?v=XkDEzfpdcSg](https://www.youtube.com/watch?v=XkDEzfpdcSg)

As for not taking off, being initially a Microsoft proposal, it is surely used
by Office and Windows teams, specially given they already use SDL (Security
Definition Language) macros as well.

[0] [http://www.open-
std.org/jtc1/sc22/wg21/docs/papers/2017/p054...](http://www.open-
std.org/jtc1/sc22/wg21/docs/papers/2017/p0542r0.html)

------
weberc2
Does Carp support ADTs or pattern matching? I couldn't see anything in the
documentation.

------
bogomipz
I'm curious about the borrow checker, is this Rust invention or did a similar
concept exist elsewhere in some other language(s)?

Can anyone recommend any resources specifically on the idea of the borrow
checker?

~~~
steveklabnik
[https://forge.rust-lang.org/bibliography.html](https://forge.rust-
lang.org/bibliography.html) has links to previous work we built upon.

------
ameliaquining
How does dynamic memory work in this language?

------
rad_gruchalski
Looks really good. Might give it a shot. What's the concurrency story?

~~~
hellerve
None at the moment, to my knowledge. High on the list of priorities, but it’s
not far enough along, I fear.

------
sillysaurus3
I've been waiting for a non-Clojure lisp to make some headway. Immutability is
mostly a fad: look at how incredibly complicated Clojure's implementation is.
It's not worth sacrificing elegance just to attract the true believers.

~~~
macintux
Honest question: is immutability complicated in Clojure because it's hard, or
because it's running in a virtual machine with no meaningful support for it?

The Erlang VM has immutability deeply baked in, and it just works(™).

~~~
nostrademons
Erlang is typically used for problem domains (eg. fault-tolerant servers,
network brokers & routers, message queues) where immutability fits the problem
domain pretty well. If you want fault tolerance, you typically need to store
all your state externally anyway so you can replicate it, and make all your
logic stateless so you can restart & re-run it if a node fails.

There are some problem domains - eg. GUI software, web browsers, simulations,
many of the more complicated parsing, data extraction, or graph-traversal
algorithms - that are inherently mutable, and Erlang is not used very much in
these domains. C++ still rules the roost there, even though many of its
practitioners hate its shortcomings.

~~~
pjmlp
> C++ still rules the roost there, even though many of its practitioners hate
> its shortcomings.

The solution has been to move it down the stack, as visible on all Apple,
Google and Microsoft OSes, even on Qt.

It is there, making full use of the hardware in all performance critical code,
but then the actual GUI code gets written in something else.

------
thomastjeffery
Since this is about a pre-alpha language, it would be helpful for future web
searchers to have a date in the title.

~~~
mark_l_watson
Sorry to be a little off topic but I think search engines should penalize
material that is not clearly marked with date of creation.

~~~
macintux
I've seen academic papers with no date and it _drives me crazy_. I'm sure
there's some reason for it related to the publication process but can be hard
to vet a paper without knowing more about the context in which it was written.

------
nimmer
Nim [https://nim-lang.org](https://nim-lang.org) is programmable as well, more
mature, and multiple memory management models are coming out.

~~~
CyberDildonics
When you say 'mature' has the compiler stopped crashing all the time?

