This language claims to be "capabilities-secure", which I assumed meant that it uses capability-based security and is an object-capability language. This got me excited -- my work on Sandstorm.io and Cap'n Proto uses capability-based security heavily, and a high-performance object-capability language would be awesome to have (the existing ones are mostly dynamically-typed and slow, though you can use capability-based design without using a capability-based language, as many programs and most OS's actually do to some extent).
Unfortunately, it turns out they have simply coopted the term to mean something completely different. According to the documentation: "Pony capabilities are completely new, no other language has them."[0]
Please think of a new word for this; don't redefine one (or a pair) that is widely-used already.
Yup. Cap'n Proto's RPC layer is based on E's network protocol, CapTP, except sitting on top of Cap'n Proto's serialization layer instead of Java serialization. E's designer, Mark Miller, is a friend and assisted me with the design.
That's great news that you know Mark! We have talked extensively with Mark about the design of Pony, and he agrees that it is indeed a capabilities type system, and that it is capabilities secure.
Hmm well if Mark Miller says Pony is capability-based then it must be. I apologize if I've misunderstood. I'm confused, though, by two things:
1. Your documentation introducing capabilities specifically says: "Pony capabilities are completely new, no other language has them." If you really mean "capability" in Mark Miller's sense of the word, then this statement is incorrect. Many object-capability languages exist.
2. What your documentation describes as "capabilities" looks to me like type qualifiers. The documentation says "A capability is a form of type qualifier and provides a lot more guarantees than const does!" But this doesn't sound anything like capabilities.
Is it possible that Pony is an object-capability language but you are also using the word to refer to a different concept? If Pony is actually an object-cap language then that's excellent, and I even more so encourage you to rename the type qualifier concept to avoid the confusion!
FWIW, Sandstorm is very interested in adopting a high-performance ocap language, but would probably shy away from one which uses the word "capbaility" to mean something else, for fear of confusing developers.
The term "capability” is much older than Pony, and the concept was proposed in the 60s. We would have loved to use a different word, but there are not enough different recognisable words in the English language.
We use capabilities in Pony to statically avoid data races - not for security. But we believe that Pony's capabilities can also be developed further to support security - would be great to try and program Sandstorm’s security with them.
> The term "capability” is much older than Pony, and the concept was proposed in the 60s.
Right, capability-based security (my definition) originated in the 60s and has been in continual use ever since. Pony's definition is new, AFAIK (and according to the documentation).
> We would have loved to use a different word, but there are not enough different recognisable words in the English language.
I think "type qualifiers" is the correct term. The Pony docs already describe capabilities as being similar to C++'s "const" and "volatile" type qualifiers, and this term will help people understand that the concept is similar (but more advanced).
The Pony docs spend a whole paragraph explaining that its capabilities are not the same thing as any other "capabilities" I may have heard of, e.g. "You may have heard of capabilities somewhere else, but these are not the same thing." The fact that you need this paragraph suggests that you've heard this objection a number of times, which should be demonstration enough that you really ought to choose a different term. People don't usually need disclaimers like this. English is a language that has entirely too many words -- there are plenty available. Or, consider making one up. :)
> But we believe that Pony's capabilities can also be developed further to support security
Yes, it's very likely that your "capabilities" would compose well with object capabilities in order to effect security. But that's just all the more reason they shouldn't use the same word, since it will be incredibly confusing to anyone who is trying to use both.
The "type qualifiers" you refer to are an extension of the capabilities-secure nature of Pony. They are themselves also a form of capabilities, specifically because the underlying language is itself capabilities-secure.
You may be interested in the type system paper that covers this:
The term "capability" is perfectly appropriate here. A capability is something that defines how you can use what it points to. It's often used in the security context, but is also used the way Pony uses them (to define how the holder of a reference can mutate the target object).
A capability is traditionally a noun, or sometimes the combination of a noun and a set of allowable verbs, but not simply a verb.
If Pony were using "capability" to describe the combination of a variable and its qualifiers, then I'd say, yes, that's a capability -- it designates a thing, and the permission to use it. But they are using "capability" to describe specifically the _qualifiers_ on the variable, which is not correct, and is confusing.
Before you ask: Yes, Linux/POSIX capabilities (as described on the capabilities(7) man page) are similarly misnamed. This is covered in the paper, and has been a source of a lot of confusion.
FWIW, this is a touchy subject for a lot of fans of capability-based security because the story of capabilities goes something like this:
1960s-1970s: Capability-based security invented, popularized, widely used, almost makes it into hardware (Intel 432).
1980s-1990s: People incorrectly redefine capabilities in various ways (e.g. as verbs rather than nouns), then declare "capability-based security doesn't work" based on these misunderstandings (see the paper I linked). Capability systems find they have trouble making headway against these misconceptions, as the difference between the misconceived version and what capabilities really are are just subtle enough to confuse anyone who isn't immersed in it. (Meanwhile, the basic ideas of capabilities live on as object-oriented programming, but aren't enforced strictly enough to be called "security".)
2000s-2010s: People start realizing that capabilities as originally defined do in fact work ridiculously well and they become re-accepted.
I found interesting what Rust's Graydon Hoare had to say about it:
"It's very similar to earlier versions of rust. In terms of actor local GC, language provided scheduling of actors, separated function and async invocation system, mixed structural and nominal types, and an overabundance of language supported reference qualifiers. I think they did a good job at it, certainly much better than I did, but I'd be nervous about the cognitive load of the reference qualifiers (capabilities). See this page for example, or the previous few in the tutorial. We had that sort of cognitive load on variables early on and people basically rejected the language because of it, as well as associated compositionality problems. The many years of design iteration had a lot do to with factoring that space into fewer, more essential and general qualifiers. Slow, tedious, world breaking refactoring work. I hope for pony's sake this doesn't happen to them too -- it looks really nice -- but that's my biggest worry for it."
Indeed, the type system when explained in detail is not that simple. Nevertheless, we believe that it can be used without detailed knowledge.
In our experience, programmers can pick up the system fast (e.g. after a 2 hour discussion).
Similarly, programmers can easily write programs which type-check, even if they do not have a deep understanding of the type system (e.g. they need not memorise the table you mentioned). Type-checking is enough - given the guarantees by the type system, (data-race freedom and atomicity).
The use of defaults reduces the annotation burden - typically to only 10%-20% of locations which allow them.
Regarding annotations, the "Fast & Cheap" paper contains the following:
The language uses carefully chosen default capabilities
to minimise the required annotations. In addition, the compiler guides the programmer as to which annotations should be used, infers annotations locally, and performs automatic recovery in some circumstances. As a result, when implementing LINPACK GUPS (in app. F) we require just 8 capability annotations and 3 uses of recover in 249 LOC. In approximately 10k LOC in the standard library, 89.3% of types required no annotation.
This is bike shedding but I wonder if better names for the capabilities could help?
box --> view (as in view but don't touch) or ro(read-only)
trn --> once
This is incredibly cool. I read through the whole tutorial, and I love what I see, but I recommend putting some code samples on the front page.
Syntax may be less important than a language's features, but it isn't until the 4th page of the tutorial that I even have an inkling of what code in Pony looks like.
Agreed too. Here my user experience with the presentation.
Once I've understand it was a new language (cool name btw), I've look for the Hello-world example to see if the language was easy enough for me. I've found the link after reading the HN comment: http://tutorial.ponylang.org/getting-started/helloworld/
Then, I've quickly read the part on concurrent programming. But I've not understood how it is easier with Pony. It's very frustrating. So far, my understanding is there is class (called actor) that can discuss together by sending each other immutable object. I would like to see some code in the tutorial that show how concurrency is easily done in Pony.
What is your planned version 1.0 release date? What milestones are you facing in the release process? What will future support for the language look like?
"Among these, Pony is the only language that concurrently garbage collects actors."
I must not be understanding that claim correctly. Erlang does garbage collection on a per actor basis (at least that's my understanding). If the claim is that multiple actors can garbage collect simultaneously, then I guess that means Erlang only GCs one actor at a time. If so, why can't Erlang GC multiple actors at a time? It does full SMP for actor execution.
> If so, why can't Erlang GC multiple actors at a time? It does full SMP for actor execution.
It does. The article is incorrect. Erlang has a fully concurrent garbage collector among actors. One actor's GC running on one CPU scheduler will not interfere with execution of actors running in other CPU schedulers.
There's a difference between GC'ing the memory reachable from an actor and GC'ing the actors themselves. Erlang requires a "poison pill" message to kill actors.
Also, looking at the white paper I see really weird numbers.
This section in particular:
Benchmarks and preliminary comparisons with Erlang,
Scala, Akka, and libccpa Table1 & Table2
It seems that you have the exact same numbers for Erlang and Scala in table2, this is very hard to believe. You either accidentally put down the same number twice or you measured the performance of your benchmarking tool, otherwise it is extremely unlikely that two entirely different systems give the the same exact measurement. Similar story in table1. maybe I am missing something terribly obvious but this looks off to me.
>It seems that you have the exact same numbers for Erlang and Scala in table2, this is very hard to believe.
The numbers are taken from another source and it seems they didn't have the exact numbers, hence the ~9s figure which is then used to calculate 333,333. They seem to have gotten the numbers by looking up the graphs on this page:
I am not an expert in benchmarking, so maybe I'm missing something. But how is that not crazy?
If I were looking to compare two things, I would run all the benchmarks on a single machine under my control. I might look at previously published reports to make sure I was getting comparable numbers. But there is no way I would publish a comparison that I merely hoped was apples to apples. I've just had too many benchmarks depend on subtle issues, ones that I had presumed were irrelevant.
There is but what is relevant from the design of the Erlang concurrent GC is that your actor operations latency is not impacted by it. This is why Erlang is extremely suitable for HTTP routers and request dispatch because you can maintain tight SLA on the p99.99 latency as opposed something like JVM where the GC locks up all of the executions, or at least this used to be the case.
The Pony object garbage collector is fully concurrent, the reachable memory for any actor is GC'd totally independently. At the same time, Pony allows (safely, with no data races) sharing pointers across actors, for performance (ie without copying).
There's a paper on the type system that allows this:
What I am saying is that the Erlang GC is good enough from the practical point of view. I am not sure what value are you trying to add with the "fully concurrent" GC.
It's not the same thing. The post is talking about automatically figuring out that nobody knows the Pid of a process and hence that process can be reclaimed. This is a transitive notion: If a group of processes can't be "reached" from another group and the latter group is the "important" one, then you can just kill off the first group of proceses. This is why it is GC-like behaviour. Like in a GC, processes can "leak" if you forget to throw the Pid away.
Erlang's method is to form linked webs of processes and then the death of a process "poisons the web" and kills off all processes in the web. By trapping exits, you can put in stopgaps for this behaviour, which is what supervisors do, among other things.
Process handling in Erlang is more akin to "manual memory management" or ARC/RAII style memory management here.
> The post is talking about automatically figuring out that nobody knows the Pid of a process and hence that process can be reclaimed.
This seems pretty impossible in distributed Erlang. Perhaps the Pid was sent to another node (which may be alive but not currently dist connected), or was serialized and may be deserialized later.
It's like jlouis said, with Erlang you have to kill your processes off when you've finished with them and if you don't they leak. In Pony that's done for you automatically.
Thanks, I think you end up killing your Erlang processes most of the time because this is the model you follow when programming in Erlang. Using HTTP as example, while in other systems it is a really bad idea to have 1 req -> 1 process (or thread) mapping in Erlang it is encouraged. When the request is answered and the response is sent back the process dies. I think is a fairly simple model. I guess I need to look into Pony more to understand the importance of this in it.
Can you explain some more or point to the exact part of the research paper. When you say "GC'ing actors themselves vs GC'ing memory reachable from actor" what exactly do you mean? Are you talking about the process dictionary and mailbox vs the state of actor passed through its loop function?
For arguments' sake here is what a simple Erlang actor looks like:
I guess the claim was that such an erlang actor would run forever when nobody sends it a message to stop it. And with pony it would be detected that the actor is no longer reachable and it would be automatically stopped.
Erlang can "leak" processes if the process doesn't exit once it's no longer needed. Pony (I assume) will clean up a process automatically once there are no more references to it.
Exactly: it will clean an actor, once there are no messages to it, and it has no work to do, and there are no references to it (or any references come from such actors which have no work themselves -- i.e. are in a cycle).
I'm wondering about something. In big systems you'll see actors on different platforms communicating with eachother. For example, a javascript actor would communicate with a server actor. How would garbage collection work in that case? Would cycles be detected across machine boundaries?
Perhaps you're thinking of some different concept of actor?
I don't recall Javascript supporting or having any library that would enable actor model of concurrency.
The general idea of actor model is that each actor is a process with a queue (mail box) and they talk to each other via msg. Javascript does not have such a thing nor does a server actor?
From the benchmark results, this is not much faster than Erlang. Also the graphs only cover 12 cores, while it claims that "Applications can run on a virtually infinite processor count and on arbitrarily large clusters". I would love to see how it scales on thousands of cores.
It looks good, but is language alone enough these days? Reminds me of Axum C# research dialect. Microsoft decided to implement it as a framework in the end - Orleans. So did Scala. And wouldn't F# + Akka.NET be just as safe?
In Akka, message arguments have to be immutable objects - similarly for Erlang; Pony can pass isolated references which can be mutated and passed on, while still being data-race free. Scala is based on Java, and thus it is not data-race free - unless the program has been crafted very carefully.
Data races are a subset of all race conditions. We don't claim that it's impossible to wait on a network message that never arrives, of course!
However, Pony makes a messaging order guarantee that's much stronger than is typical for the actor model. It guarantees causal messaging. That is, any message that is a "cause" of another message (i.e. was sent or received by an actor prior to the message in question) is guaranteed to arrive before the "effect" if they have the same destination.
That guarantee is achieved with no runtime cost, which is pretty fun.
Pony still allows unbounded non-determinism, of course, but causal messaging gives a very comfortable guarantee to the programmer.
Your causal message ordering guarantee sounds exactly like the Erlang one. If a pair of processes (P1,P2) is given and P1 sends m1 then m2, then P2 will see them in that order, perhaps interleaved with messages from other Pids.
But your claim might be stronger, though I'm not sure in what sense it is stronger.
In Erlang, that only holds for a pair of processes, as you point out.
In Pony, it holds for the entire program. So, if A1 sends a message to A2, and then A1 sends a message to A3, and in response to the second message A3 sends a message to A2... in Pony, A2 is guaranteed to get the message from A1 before the message from A3.
Hmm, thinking some more on this, this holds locally in Erlang, but not in the distributed communication. If we have m1: A1 -> A2, m2: A1 -> A3 and then in m3: A3 -> A2, then the message ordering will always be that m1 comes before m3 in A2 (locally).
Locally, a message pass never fails and always delivers the message directly. It can't be in-flight, so m1 is placed in the mailbox before m2 happens, and thus m3 will always come later.
In the distributed setting, A1, A2 and A3 might be living on three different nodes, and then m1 might get delayed on the TCP socket while m2 and m3 goes through quickly. Thus we observe the reverse ordering, m3 followed by m1. This is probably why the causal ordering constraint is dropped in Erlang. Is pony distributed?
That sounds... interesting. So if A1 is on a different segment, but has a faster link (less congestion?) to A3, than A2, even if it takes 1000t time to send the first message from A1 to A2, and only 10t+t time = 11t time to send a message, from A1 to A3, and from A3 to A2 -- the plan is to guarantee ordering of messages m1, m3 at A2 (delaying m1 at least 989t)?
I suspected as much. It is a pretty good guarantee. Have you found any use for it yet in programs? That is, a program where the causal message ordering is preferred and removes code from the program?
That's just what I was thinking about: Where would this be useful?
On the other hand I could imagine that this could also cause problems: If you have a large program, could then this causality cause some messages to be delayed for a (too) long and not directly visible time in order to achieve the causality guarantee? But the claim was that there is no runtime overhead.
You have an actor A13 that communicates with actor A2. The guarantee allows you to break up A13 into the two actors A1 and A3 that now both communicate with A2 and be done with it.
In erlang, you might not've done so, because you know nothing about message ordering in non-tree-shaped topologies.
Especially if they're trying to include distribution (per a comment elsewhere). Pretty sure you can't have linearalizable messages between nodes without additional runtime cost compared to the non-linearalized case.
Distribution probably (almost definitely) kills the runtime cost, I can't imagine how it couldn't. I'm much more interested in the causality guarantee in the context of the distributed systems.
Oh, agreed. But it's doable, provided you don't mind losing availability (which is...an interesting choice for a programming language), and you don't mind the runtime cost.
But I agree, exactly what guarantees they're trying to provide (and is it opt in?) is an interesting question.
I found:
Clebsch, S., & Drossopoulou, S. (2013). Fully Concurrent Garbage Collection of Actors on Many-core Machines. ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, 553–570. http://doi.org/10.1145/2509136.2509557
Sorry about that, we'll get more docs up as soon as we can.
For now, check out some of the included packages, like net/ssl. That calls C extensively (libressl, or openssl if you like). Or regex, which uses PCRE2.
I couldn't find any information about the authors of this language. It seems like a fairly accomplished yet complex project, so it would be quite good to know who are the people working on it.
Looks very promising, hopefully there will be a strong community around it, imho communities make or break a language or a framework no matter how good the syntax or the features are.
An unusual little feature is that assignment returns the old value of the target, e.g. a = b returns the old value of a, so that a = b = a just swaps the contents of a and b.
No. Dynamic linking is possible, for example with C libraries. 'No loadable code' means that a programs code is not modified during runtime, nor extended with new code (for example like java applets or RMI applications). We have been looking into hot code loading though.
It's also worth noting that the language does have a None primitive, but the distinction from nil or null is that the concept of "nothing" is wrapped in it's own type entirely. This means that it would be a type violation for a method of return type String to ever return a None. Instead, a method that you expect to return a String or a None would need to have a union return type of (None | String).
Obviously, I have never written anything in Pony, but this absolutely seems like an advanced use case. It exists because 'C' has a concept of null, so it needs to be handled. The manual even admits that all bets are off when dealing with the FFI.
One class having a method called "is_null" doesn't mean that the language has null built into it or any inherant concept of what that means. Just like having a method called "is_functional" doesn't mean it's a functional language.
Looks like the `handle` in question there in the File module is an instance of a Pointer, which presumably is some sort of wrapper around a raw pointer. It has null in the sense that the pointer can be null; this isn't the same thing as how nulls are traditionally used (e.g. an "empty object" which can appear in the place of where an actual object might be). So it's not a language level concept.
I'm guessing the capabilities correspond to polarized substructural types plus additional structure for who can read/write to a given area of memory (http://www.cs.cmu.edu/~fp/papers/fossacs15.pdf).
i.e.: iso = linear, val = affine, ref = unrestricted, box = readable, trn = readable/writeable, tag = uniqueness?
Capability defaulting seems a little dangerous (not that it's not type safe, but it having such information be implicit seems like a good way to encourage people to forget about it; on the other hand, if the compiler is good enough about warning people it shouldn't be a problem).
I'm not sure subtyping is worthwhile for this language (since it complicates the type system without much of a gain wrt constrained polymorphism).
Can you put up some example applications? The tutorial only has code snippets. I don't want to look at individual features in isolation, I want to get a feel for how one solves problems in the language.
1. Because Pony is both type and exception safe, actors (the Pony equivalent of an Erlang process) don't fail in the catastrophic sense. Right now, if an actor wants to report that it can't go on to some other actor, it just sends out messages. Obviously, we are big fans of Erlang, and big fans of the Erlang supervisor pattern, so we'll be writing a Pony package that puts a nice interface around this.
2. Yes! Pony has first class pattern matching already (along with algebraic data types, i.e. tuple types, union types, and intersection types, to make pattern matching really useful). It allows matching both on type and on standard or programmer defined structural equality. As I mentioned, we're big Erlang fans, so we definitely won't ignore stuff like this.
Bonus: if this language thing doesn't work out you can just pivot to selling the unlimited-resource and failure-free hardware and operating system you've invented...
Snark aside, a system that always assumes "clean shutdown" is going to massively simpler to design. It's also going to be unusable in real-world environments.
Stopped being interested after reading "It's deadlock free!", "because Pony has no locks at all"... yeah but actor X waiting for actor Y which waits for actor X is a deadlock. It's not a matter of lock primitives, it's a matter of software. (yes, x-comment from reddit)
No. If actor X is waiting for actor Y and Y for X and neither of the two has messages to process, they are potentially subject to GC.
The question is, what does X is waiting for Y mean? 'Waiting' would imply that there are language constructs to actually wait. Pony doesn't have that. An actor that is blocked (i.e. has no messages) and that satisfies necessary RC invariants will be collected.
So I haven't seen yet in the website a code sample showing how a simple ping-pong actors would communicate, so I can't even test quickly if this is possible with the language.
If the program can implement a lock between the two actors, then your language is not deadlock free. Even a loop by checking if some variable has changed by the other actor (if possible in this language, I still don't even know by the deep lack of examples), and the other actor doing the same thing, that can be claimed as deadlock.
A recognizable syntax is a big win, for starters. Do people still cling to the idea that it doesn't matter? How many tens of thousands of "gotchas" and "good parts" do we need to hear about from languages that have poor syntactic choices, from the viewpoint of the average programmer?
I don't think so. The language is (almost) whitespace insensitive so the only way to know where a certain control statement terminates is by having the `end` token.
It is unnecessary on the top-level though, since the top-level expression will end where the next one begins. Which means you probably can't have nested class/function definitions.
This technique is not new and is used in OCaml, for example.
This is correct, classes and functions are not dependent on indentation, they just don't need an end. And yes, there are no nested classes and functions (other than in object literals).
Unfortunately, it turns out they have simply coopted the term to mean something completely different. According to the documentation: "Pony capabilities are completely new, no other language has them."[0]
Please think of a new word for this; don't redefine one (or a pair) that is widely-used already.
[0] http://tutorial.ponylang.org/capabilities/introduction/