Hi! Wow, the website just got up a couple days ago... that was fast for hitting the HN homepage...
But the work has actually been happening for a couple of years, and the foundational layer, Spritely Goblins, is already something you can use: https://docs.racket-lang.org/goblins/index.html
But that's not really the most interesting part of Spritely Goblins... the safe to run in a mutually suspicious network distributed programming environment is.
Some background: I was one of the co-authors/co-editors of the ActivityPub social network standard... you might know it because it's the protocol used by Mastodon, but it's also used by Peertube, NextCloud, Write Freely, Funkwhale, Pixelfed, etc. I actually got involved in standardization because we needed it for the previous project I worked on, MediaGoblin. After standardization of ActivityPub ended, it turned out that a number of ActivityPub-using applications such as Peertube and Pixelfed and Funkwhale were using the protocol we worked on standardizing to do the kinds of things we meant to work on in MediaGoblin. So I asked myself, given the finite amount of time I have on this planet, what's the best use of my time? At this point I thought (and with a lot of feedback and phone calls with close friends): I should work on advancing the things that the federated social network can't do. And there are quite a few of those because ActivityPub left some gaps in the spec, especially around authentication and authorization, but also because people aren't building as rich of experiences as I think are interesting and possible yet... they're mostly mimicing contemporary social networks.
So you can see in that long list of things on the Spritely website that one of them is Fantasary, which is the goal of bringing distributed virtual worlds to the fediverse. That may sound like a hyper-ambitious and also kind of strange goal, but consider that modern social networks are really degenerate versions of massively multiplayer games; you can chat in both, but most social networks don't have the sense of space and rich interaction that games do. But I also want to make more obscure peer-to-peer technologies available and in the hands of users without the usual "Oh no those are only for the bad people on the internet"... normalizing things can help. Those of us old enough remember when demanding https was similarly brought up with the "well, that's only for people who have something to hide". A use case of e-commerce changed that. Well, I want another use case... but one that involves fun. Hence the distributed game stuff. We can see how motivating building worlds together is by the success of minecraft.
But what if the Fantasary component fails? It's the most ambitious piece! That's actually okay, because it's really a driver for all the other layers. Distributed programming that's safe to run in a hostile network, portable encrypted storage, the ability to safely run untrusted code... all of these are useful things, and useful things to bring to users generally, even if the game approach fails.
But actually all of these layers are quite well researched and possible. Part of the reasons that Spritely development happened before two years before the launch of the Spritely website is because when we ran MediaGoblin, we opened up the project and within a couple of weeks had dozens of developers... but that was quite feasible to accomodate because MediaGoblin was a fairly "traditional" web application, easy enough to point to how it works by other similar web applications. Spritely is treading a lot of what appears to be "new ground" to most people, so I had to figure out how it works enough, and lay enough of the foundation, before I was comfortable making a public image for it and asking people to start looking at it. But with the last release of Goblins with the beginnings of distributed network programming and with a fairly reasonable tutorial, that's starting to change.
But I said "new ground" to most people, but few of these are actually new ideas. Most of them are old, well researched ideas, from a somewhat obscure (but becoming quickly less so) branch of computer science called "object capability security". I've been very fortunate that the object capability security community has been willing to take the time to walk me through how many of these concepts work... Spritely wouldn't be possible without their help.
Spritely Goblins looks so good I'll be pinching it at a later date!
I'd be keen to see what your thoughts are on transactional logs here- you're using snapshots so technically you could do a transaction log, which in turn can be used to help resolve distributed remote call issues eg. if you have 2 nodes and a split-brain scenario you can use the transaction log with a (vector) clock and defined types to resolve the split when connectivity is restored.
Additionally, do you have any thoughts on GC of objects? I'd have thought that Object Capability Security would factor into this.
Hi there! Yes, the transaction log stuff is of interest to me, and ties in with the "Questie" stuff (the one listed as inspired by the causeway debugger). Not quite on the front burner yet, but it'll have to be to make our lives easier.
Goblins already supports distributed (acyclic) GC! And yes ocap security does tie in nicely. In a sense ocaps make this really easy... if you think about an ACL approach the user has to point at the object they have access to and the object has to point back, but since ocaps are based on "having references to objects", in general GC locally can just be the very GC that the native runtime provides. Across runtimes (in the distributed code), distributed GC likewise falls out easier, though the algorithm doesn't know how to handle cycles when Alice on machine A points at Bob on machine B which points at Alice on machine A again. So you have to manually cycle break in those cases, but they're relatively rare.
But that requires special cooperation to traverse the object graph held by the runtime's garbage collector, which we don't have access to in Racket (or most languages) unfortunately. It would really be cool to see that tech revived somewhere though.
which details an RAII + message queue solution for managed memory without GC. The idea there is that provided your message API is well defined, you can determine the queue size in advance, and thus deterministically manage the memory (ie. object lifetime).
I'm mentioning it here, as if capabilities and RPC are added to the mix, you can treat remote objects as part of the memory DAG - RPC calls are just another message on the queue (capabilities here act as a root trace as if you have a message on the queue, you can trace back to the root) and the object can be removed when the queue is zero.
In addition, as per Rust, we should be able to lean on the compiler somewhat, to do some static cycle analysis.
I don't think you can get away from having a cycle checker at high level, but with a transaction log + managed memory + functional code it should make a cycle checker easier to write.
Lastly, I was thinking about the DAAS space - all of this distributed stuff will require authentication (authN) and authorization (authZ).
I think primarily this comes down to IBS (Id Based Signatures) for authN, and then storing encryption secrets to access resources in a distributed hash tree (https://wmcode.nl/dh3/dh3_paper_v0-5.pdf), but any thoughts on how to secure items without POW (proof of work)?
Effectively the DHT is a POS (proof of stake) as you define who owns the resources by attaching an access hash to the secret.
We're not using E, we're strongly inspired by it. It's written in Racket/Scheme because it's a very easy environment to do experimentation with language ideas in. It turns out everything we've built didn't really require language-level ideas, and could be ported to any language that has sane lexical scoping support. And with the CapTP networked layer, we might even be able to achieve interoperability.
In fact, we're in talks with the wonderful folks at Agoric, who are building very similar things in Javascript-land: https://agoric.com/
And we want our two CapTP implementations to be compatible. If we do that, you can have programs written in Scheme/Racket talking to programs written in Javascript just fine.
I'd love to read a discussion about how ActivityPub is too complex, but searches for "AcivityPub overcomplex" and "ActivityPub complex" didn't return anything in the first ~20 results or so. Do you have a pointer to one of the places it is described as overcomplex?
But the work has actually been happening for a couple of years, and the foundational layer, Spritely Goblins, is already something you can use: https://docs.racket-lang.org/goblins/index.html
I think people don't tend to get very excited until they see the time travel demo, and then they say "whoa, something cool is going on here" https://dustycloud.org/blog/goblins-time-travel-micropreview...
But that's not really the most interesting part of Spritely Goblins... the safe to run in a mutually suspicious network distributed programming environment is.
Some background: I was one of the co-authors/co-editors of the ActivityPub social network standard... you might know it because it's the protocol used by Mastodon, but it's also used by Peertube, NextCloud, Write Freely, Funkwhale, Pixelfed, etc. I actually got involved in standardization because we needed it for the previous project I worked on, MediaGoblin. After standardization of ActivityPub ended, it turned out that a number of ActivityPub-using applications such as Peertube and Pixelfed and Funkwhale were using the protocol we worked on standardizing to do the kinds of things we meant to work on in MediaGoblin. So I asked myself, given the finite amount of time I have on this planet, what's the best use of my time? At this point I thought (and with a lot of feedback and phone calls with close friends): I should work on advancing the things that the federated social network can't do. And there are quite a few of those because ActivityPub left some gaps in the spec, especially around authentication and authorization, but also because people aren't building as rich of experiences as I think are interesting and possible yet... they're mostly mimicing contemporary social networks.
So you can see in that long list of things on the Spritely website that one of them is Fantasary, which is the goal of bringing distributed virtual worlds to the fediverse. That may sound like a hyper-ambitious and also kind of strange goal, but consider that modern social networks are really degenerate versions of massively multiplayer games; you can chat in both, but most social networks don't have the sense of space and rich interaction that games do. But I also want to make more obscure peer-to-peer technologies available and in the hands of users without the usual "Oh no those are only for the bad people on the internet"... normalizing things can help. Those of us old enough remember when demanding https was similarly brought up with the "well, that's only for people who have something to hide". A use case of e-commerce changed that. Well, I want another use case... but one that involves fun. Hence the distributed game stuff. We can see how motivating building worlds together is by the success of minecraft.
But what if the Fantasary component fails? It's the most ambitious piece! That's actually okay, because it's really a driver for all the other layers. Distributed programming that's safe to run in a hostile network, portable encrypted storage, the ability to safely run untrusted code... all of these are useful things, and useful things to bring to users generally, even if the game approach fails.
But actually all of these layers are quite well researched and possible. Part of the reasons that Spritely development happened before two years before the launch of the Spritely website is because when we ran MediaGoblin, we opened up the project and within a couple of weeks had dozens of developers... but that was quite feasible to accomodate because MediaGoblin was a fairly "traditional" web application, easy enough to point to how it works by other similar web applications. Spritely is treading a lot of what appears to be "new ground" to most people, so I had to figure out how it works enough, and lay enough of the foundation, before I was comfortable making a public image for it and asking people to start looking at it. But with the last release of Goblins with the beginnings of distributed network programming and with a fairly reasonable tutorial, that's starting to change.
But I said "new ground" to most people, but few of these are actually new ideas. Most of them are old, well researched ideas, from a somewhat obscure (but becoming quickly less so) branch of computer science called "object capability security". I've been very fortunate that the object capability security community has been willing to take the time to walk me through how many of these concepts work... Spritely wouldn't be possible without their help.
If you want to hear more, the video embedded on the site is a good overview, but here's a direct link: https://conf.tube/videos/watch/18aa2f92-36cc-4424-9a4f-6f2de...
Anyway, happy to answer questions if anyone has any. And feel free to join our irc channel... we're #spritely on irc.freenode.net!