Hacker News new | past | comments | ask | show | jobs | submit login
Motoko, a programming language for building directly on the internet (stackoverflow.blog)
155 points by markdog12 on Aug 27, 2020 | hide | past | favorite | 82 comments



Looks cool. One thing though, the page says

> Actors are a great model for concurrent programming because they automatically prevent race conditions (thanks to atomicity and encapsulated state) and deadlocks (because execution never blocks)

This isn't really true afaik. You can still have logical race conditions if you don't synchronize messages (but they won't be architectural races), and you can still have deadlocks if two actors wait on messages from each other at the same time.

I feel like people often treat race conditions and deadlocks as small, computer-architecture ideas when really they're quite broad concepts. You may be able to prevent them automatically at certain scales, like say a single uint64, but when you try to treat two uint64s as a single uint128 then suddenly race conditions are possible again, despite the guarantees your language has made on the smaller scale.


I agree completely! And this shows up in just about any language because of that — however — I kinda get where the author is coming from.

When a certain amount of state is encapsulated within a single actor (or a single message for that matter) you can act on all of it sorta transactionally since no other actor has access.

This is probably more power/control than some programers have had in other languages or paradigms.

Your full program state dosn't retain this property once it spans actors but what can ya do.


Code sample:

Actor model with explicit synchronicity and access scoping:

  actor Counter {
    var value = 0;
    public func inc() : async Nat {
      value += 1;
      return value;
    };
  }
Types and specificity:

  type Tree<T> = {
    #leaf : T;
    #branch : {left : Tree<T>; right : Tree<T>};
  };

  func iterTree<T>(tree : Tree<T>, f : T -> ()) {
    switch (tree) {
      case (#leaf(x)) { f(x) };
      case (#branch{left; right}) {
        iterTree<T>(left, f);
        iterTree<T>(right, f);
      };
    }
  }
Source: https://sdk.dfinity.org/docs/language-guide/motoko.html

Not to my taste personally.


Pretty interesting. Basically your code is an eternal continuation, which is quite a nice persistance style (e.g. used by our universe, works pretty good). The code looks relatively nice and clean, except the hashsigns. Compiler seems simple and Candid will help quite a lot with interop and speed up the development.

Think I will enjoy trying this once its available


Async is difficult to get right without proper generics and type parameterization support in the syntax. Brackets help collapse expressions to single lines, or expand them as needed. I like having that flexiblity.


Looks like Scala, but uglier.


an impressive feat in itself


The last snippet looks Rust-y.


Too many curly braces, too many abbreviations, and worst of all, templates with <>-syntax. You couldn't build an uglier language if you tried.


Which alternative do you propose for templates, instead of angle brackets <>?


I would follow Scala's lead and use square brackets [] for generics.


The binary ! operator. (inspired by Dlang)


> Instead, the Internet Computer implements orthogonal persistence—yet another old idea where a program has the illusion of running “forever” and its memory staying alive (at least until it is explicitly taken down). In Motoko, this means that developers do not have to worry about explicitly saving their data or bother with files or an external database: whatever values or data structures are stored in program variables will still be there when the next message arrives, even if that is months later.

My guess is that storage persistence is the Godzilla of all leaky abstractions. There are huge tradeoffs in regards to consistency, durability, random access, performance, and cost. In addition, there are huge issues with lock-in. If I can save my data to sqlite or postgress or even S3, I know that I will be able to migrate the data to other languages and systems pretty easily. But if all my data is in a abstract "Internet Computer persistence pod", that is a huge issue.


This model also seems to ignore data deletion, which is a huge concern for any serious business of any size.

You can’t have users’ personal data lingering in a soup of opaque cloud continuations for months. There needs to be a way to strongly commit deletions and audit that the data is gone. And once you build that on top of the “Internet Computer”, my guess is that it will look disappointingly like a very slow ordinary database.


IMO, the next step for distributed programming, and one web assembly could finally enable, will be communicating with code, not messages.

Imagine you're building an app that lets users of a music streaming service see how much time they spent listening to particular genres of music.

The way you would do this with REST, SOAP or similar could roughly be:

1. Retrieve the user's whole listen history.

2. For each track, send a request to get the track info, which includes album and duration.

3. For each album, retrieve the album info, which contains the genre.

4. Tabulate the returned data.

That's thousands of requests, and a lot of network bandwidth wasted.

GraphQL is a bit better, you only receive what you need, and you send just one request, but you still get a whole lot of track objects, which need to go through the network and be processed on your side.

With my proposed approach, you would send a piece of wasm code, a bit similar to a Lambda function, which would run on the streaming service's API cloud.

Such a function could use fine-grained API calls to retrieve what it needs. It would be executing on the service's network, so round trips would be either short or nonexistent. After tabulating the data, it would send back a simple JSON with the genres and time spent.

This approach would allow developers to extend any service in ways the original developers have never imagined. We could also re-think the way backends and frontends communicate, giving frontend devs a way to do even more, without changing the back end.


On the Windows accessibility team where I work, we've done something like this for the UI Automation API, not using WebAssembly but using our own bytecode. We call it Remote Operations. The idea is that certain parts of the UI Automation API require a very large number of cross-process requests (i.e. from the assistive technology to the application process) to do anything non-trivial. So with Remote Operations, the client (assistive technology) sends a little program to the provider (application) to do the API calls.

This feature hasn't yet made it into a stable Windows release, but it's available in Insider builds, and the high-level client libraries and accompanying tests are on GitHub here:

https://github.com/microsoft/microsoft-ui-uiautomation

Disclaimer: Remote Operations is still a work in progress. There's no public documentation yet aside from the README in the GitHub repo, and the client libraries aren't yet packaged for easy consumption. Moreover, this is not an official Microsoft announcement; I'm just bringing it up on this thread because it's relevant and I'm proud of this feature that I helped develop.


As far as I remember, NVDA either does or used to do something similar. They were injecting c++ code into other processes, so that it could quickly query their APIs and return just the data NVDA needed. I never really delved into that part of the codebase, so I might be wrong here.

As an aside, how do you get to the bytecodeyou use? Do you write quazi-assembly by hand? Did you develop your own compiler? For what language?

Will other, non-microsoft assistive technologies, like NVDA for example, be able to use it?


You're right; all serious third-party screen readers for Windows currently inject code into the application processes. In particular, they all use this technique to efficiently traverse browser DOMs and Office object models. The point of Remote Operations is to provide a way to efficiently get the equivalent information through UI Automation without the risks (in security and robustness) of injecting native code in-process.

As for how the bytecode is built, the GitHub repository I linked has a library with a WinRT API for building the bytecode at run time, by calling methods that correspond to the individual opcodes. It's an object-oriented API, so there's a class for each type of operand. And for control flow blocks (e.g. if-else and while loops), the method takes a WinRT delegate (basically a lambda) that builds the body of the block. You can see how it works in the functional tests; stay tuned for actual sample code.


Do you think it could be useful to wrap this in a LINQ provider for C# usage?


You’re really on to something with this query language for structured data idea! I’d find it super-useful as a dev to be able to think of a backing service not as a set of disconnected endpoints, but a base of operations for dealing with data.

If only such structured query language data bases existed, I bet they would get a lot of use.

(No but really: how is what you’re describing meaningfully different from SQL, especially with the restrictions you’d want to put on the code in practice?)


You're onto something with SQL here. It definitely should be a big part of this, especially for the querying data aspect. Of course, we wouldn't query over raw, database data, but over sanitized, secure views.

SQL isn't everything, though. Be aware that such APIs, for security reasons, wouldn't allow you to modify some of the data in arbitrary ways, but require you to use special functions instead. Think of being able to query the number of upvotes, call an upvote function, without being able to just change the upvote count.

Maybe a mix of wasm (as an imperative platform) combined with a structured query API? Maybe a programming language with something like Elixir's Ecto or C#'s LINQ? Not plain SQL, but an imperative language with the power of relational queries?


Your characterization of sanitized secure views as something other than data in a database strikes me as vitalistic (and dubious).

The “communicate with code, not messages” in your first comment strikes me the same way: code is data, and data is code, and Wasm does nothing to change that. Nothing about making an interface more expressive guarantees one is making it better, and overspecification of user intent can make it much harder for the system to optimize execution.

Even if we really do want imperativity: your upvote example, even including the security aspects, is easily handled using stored procedure support that’s been available in SQL dbs for decades. It is not unfair to describe that support, and its variants (eg postgres lets you use python) as “not plain SQL, but an imperative language with the power of relational queries”.

Can this approach handle everything? No; it would be madness to try and build e.g. huge neural network models in SQL + some stored procedures. But it handles a lot, including your two examples, and handles them more compactly than a general purpose language would, and exists.

Mind you, I’m not saying your general concept isn’t great. It is great, it is a great step. It’s just that distributed programming already took it, back in the 70s.


I agree with your comment highly.

I also find the idea of arbitrary code execution on a remote host to build a derivative service to be a quite naive point-of-view. Even if you can validate that the code isn't going to pwn you - a mighty stretch - how can one justify being a recipient of free compute cycles on someone else's infra? It's not quite realistic to me


> I also find the idea of arbitrary code execution on a remote host to build a derivative service to be a quite naive point-of-view. Even if you can validate that the code isn't going to pwn you - a mighty stretch

Be aware that's exactly what you are doing when you go to any modern site. You let a remote server execute arbitrary code on your computer, trusting it's not going to pwn you. Sure, there were tons of loopholes in the first few years of JS, but now browsers and operating systems are so secure that it's mostly a non-issue. Web assembly is a sandboxing-first technology, so I don't think it's an impossible or even a particularly hard issue to solve.

>how can one justify being a recipient of free compute cycles on someone else's infra? It's not quite realistic to me

Well, there are API call limits, or something equivalent to them. Maybe computing credits in this instance. Calculating how much computing resources you're using is something that i.e. AWS Lambda does already, no reason why an open-source runtime couldn't do so either. The fact it wouldn't be native code, executed directly by the CPU, but Wasm executed by a VM, makes things even simpler.


>but now browsers and operating systems are so secure that it's mostly a non-issue

Right, which is the point - it took an absolutely behemoth effort backed by megacorps to get us to this point. These applications are public-facing because that's the key way for the corps to make money - content delivery.

Why would anyone put this same effort into building more complicated internal services enabling derivative services to extract value from their platform?

> Calculating how much computing resources you're using is something that i.e. AWS Lambda does already

Yes, exactly - another megacorp built a service that makes them huge amounts of money, and now they sell access to it. This code is certainly not publicly accessible

So, again, help me bridge this gap - what possible reason would Spotify (to further GP's example) build such a platform for? This means that Spotify needs to maintain massively more infrastructure in order to help others extract value from their platform.

It just doesn't check out. Is it objectively a more efficient method of computation? Yes. Is it also completely misaligned with corporate incentives in the modern day? In my opinion, yes.


I don't think OP necessarily implied that the compute cycles had to be free.


Right, so then he's proposing that this streaming service is enabling derivative services by not only managing the API for the streaming service but also by running their own public cloud which issues micro-charges for compute cycles. That seems even more pie-in-the-sky.


It's definitely pie-in-the-sky. I wouldn't argue otherwise.

But I do think this is a problem our industry will have to solve eventually. The volume of data we have to move between servers in order to ask questions is exploding. We move petabytes of image data between filesystems in order to train machine learning models. If the problems in this thread can be solved (admittedly a big if), moving the code to the data would be fundamentally better.


Mind you, most of this should be handled by some kind of runtime. Any complex enough web service runs HTTP Servers, databases, sometimes messaging queues, caches or even things like Kubernetes. There's no reason why it couldn't be running yet another program to manage all this.

Such a thing could become the de-facto API, if Rest or GraphQL were needed, they could even be implemented on top of it.


>There's no reason why it couldn't be running yet another program to manage all this

Yes there is - that program would need to be extremely hardened (even against DDOS style attacks - run a ton of memory-consuming cycle-hogs on a service's platform), integrated with tracking & billing, and open-source (or itself charging).

I just don't see the incentive for someone building this, and then I doubly don't see the incentive for a corp running this. Why take on this huge extra burden of infra to help derivative services? It's not going to make you money.


The code you’re talking about is a query. There are straightforward ways to do this now (including graphql). There are some issues with it, though.

The query (or code, if you will) is separate from your data, meaning the aspects of the data a query depends on can’t change without breaking the query. You can have versioned apis and keep the old apis around, or have a query compatibility rewrite layer, etc. but obviously these kinds of things have serious complications/limitations.

That approach is useful only in proportion to how broad, granular, and flexible the API is that you provide for the code/query to access. That sounds good, but such an API is a lot of work to create and a lot more work to maintain. An API is like a promise. If you make big promises, it will take a lot of effort to keep them.

Also, realistically you have to limit arbitrary code sent in from untrusted sources that runs on your server. Even completely non-malicious users will write inefficient code or have inadvertent infinite loops that will eat up your compute budget if you don’t have controls in place.

So...

It’s very often better to add a “listeningStatsByGenre” endpoint if that seems to be what your users want.


Uh... ever heard of SQL? Here's the "code" you would send:

    select s.genre,
        sum(s.duration) as total_duration
    from listen_history h
    inner join songs s on s.song_id = h.song_id
    where h.user_id = 12345
    group by s.genre


Plus

   ; DROP TABLE songs
Sandboxing in environments with no trust is always a challenge. GP's idea was tried and failed many times thus we preffer data passing now.


No doubt. It is shame we are all forced to create own DSL (e.g. GET /genre/12345) that dispatches to the above though. I really had hopes for the OData specification, but it hasn't seemed to get much traction.

It is worth noting, though, that many RDBMS allow for "read only" permission groups, so the risk of malicious DELETES/UPDATES isn't a huge problem in practice. Rather, it's the unbounded queries that could make your severs spin forever that truly represent the problem domain.


What about web browsers?


I'm pretty sure web browsers are sandboxed and that it has taken a significant effort to get there.


If the songs the user has listened to would be stored on the client, it would be trivial to run code on the client to compute anything over that data, rather than expecting the original service to execute arbitrary code.

You can move code to the data, or the data to the code.


Except that sometimes you have a phone, a computer and a smart TV, and syncing the data between all of those is non-trivial. It's easier to assume the server as the source of truth.

Sometimes you want to work with big data banks that you have no hope of ever storing. Your listening history is small enough to fit on your phone. Amazon's whole product directory, with reviews and indexes for quick data retrieval, is not.


Syncing data is an easier problem than what the parent is trying to solve. RSync is one such solution.

If you want to run compute over big data, you replicate the database with a server you own and run compute on a copy of the data. Realistically that is what Amazon would have to do for you either way - they'd be running replicas you could compute over. And those replicas need to say in sync via db replication.

Secure competition over anonymized datasets is useful for security reasons, not efficiency reasons IMHO.


This was implemented ages ago in Voyager application server written in Java. It supported a concept of agents: pieces of code that could be sent to particular location, execute and come back with the results. That was way before Java app server got their standards. Not sure what was the ultimate fate of this particular tech.


> and of course it has a flexible type system that is actually sound, ... No holes!

And this is where I went from "good faith reading" to actually not believing the article. Every general language that has made this claim has had holes - and those holes are usually patchable, but to claim a language with almost no adoption is leak proof is foolhardy at best. Just because you haven't discovered how your particular abstraction is broken doesn't mean it isn't.


Copy/pasted from google search:

> A sound type system means you can never get into a state where an expression evaluates to a value that doesn't match the expression's static type.

It's not actually that hard to make a provably sound type system. The claim can only strain belief if it includes some type-level feature that makes this difficult.

Of course, it should also be proved. e.g. Java's type system is not sound even though there is no good design reason while it shouldn't be (to be fair - it only happens in cases so intricately twisted that nobody will ever stumble on them by accident).


Soundness has a technical meaning. Point is, the type system at least isn't a shaky foundation.


If you had gotten a bit further, you would have seen the type system is so amazing it stops you "misinterpreting data". Truely the future has arrived.


The programming model seems nice (orthogonal persistence especially), but the fact that it's running decentralized is a big cost.

Maybe someone will provide the same developer niceness in a traditional cloud way. The only people who would prefer DFINITY at that point are people who really care about decentralization (which is fewer than the marketing hype suggests imo.)


Disclosure: I work at DFINITY, but not on this team. I can probably answer most questions folks may have...


* What are the usecases where this [the Internet Computer and/or Motoko] shines?

* Can you expand on Orthogonal Persistence -- is this a "per actor" persistence, or from the references to blockchain is it some sort of shared state between actors?

* How does the internet look/feel/work different once this exists?


Hi there! Diego from DFINITY here...

1. Compared to other forms of DECENTRALIZED compute, IC + Motoko shines in making web apps with User experience comparable to centralized providers (AWS, GCP, Azure). Example: You can build simple react web social network and expect quick reads (in milliseconds) and writes (1-3 seconds). Note I did not say "blocks" or "finalization." I deliberately trying not to have a leaky abstraction from the POV of the app developer.

Compared to CENTRALIZED compute, you can create "open internet services" like an open version of TikTok where the control of the app and its features can be be done by (in simple terms) voting. Example: https://www.youtube.com/watch?v=_MkRszZw7hU

More broadly, one of the things that is surprising (it was to me when I joined, and it still is), how much of the cloud stack that developers or compute providers create is not necessary when you have protocol-based compute. As a developer, some things like firewalls, databases, seem less important in enforcing the security and scalability of my apps. I realize as I type this can sound a bit naive AND esoteric... and I think the only way to really show this is by action and folks just playing with it. Few words will really convey this as much as people playing with it directly, I get that.

2. Our compute model are "canisters" which (if you are familiar with actor model) are actors that contain both their code and their state. Actors communicate by sending messages to each other (as actor model implies). Example: I could create a twitter canister. Orthogonal persistance means its state can grow without me spinning up DBs or worrying about replication and consistency.

https://sdk.dfinity.org/docs/language-guide/motoko.html#_ort...

3. Oh boy thats a big question... I don't think I can do better than our founder Dom: https://www.youtube.com/watch?v=dALucsAgAwE

Was that a helpful set of answers?


Yes, it was, thank you.


Where can we find the code that implements this language and the wasm runtime with persistence? I couldn't find it in DFINITY's github account


Hi there! Diego from DFINITY here...

Good question. It is not public yet. We have started opening more in summer 2020 (this post is an example) as things start to line up. Some of the folks in the Languages do give talks Motoko and Wasm... and expect us to open up further.

I do not expect that answer will truly sate your curiosity, and I am very humbled you are even interested to read the code and go deep. That means a lot, thank you.


If I understand the persistence stuff correctly, programs don't do explicit I/O but instead just run forever and thus use normal variables and data structures to store state.

This sounds very convenient until one fine day you decide add a new feature, so you need to change your code, but the new code has different variables and there may not be any sensible way to "port" the data.


” Motoko provides numerous features to help you leverage this environment, including language features that allow your heap to self-migrate when you upgrade the software of a canister.” https://sdk.dfinity.org/docs/language-guide/motoko.html

So in this sense it could be a bit like creating migrations to your database. Maybe on ”cloud scale” you just would like to migrate the data when it is used.

Edit: Few more details in this document: https://sdk.dfinity.org/docs/language-guide/actors-async.htm...


Okay, so the magic ingredient is that variables that are marked "stable" are treated specially. Nice!


Not sure what the problem is supposed to be. Is the idea here that the programs don't do I/O (e.g. to disk/db/etc) ever?

Because that's not probably what it happens...


Sounds more like you don't have to worry about mapping your variables to and from an underlying persistent store - it happens automagically.

It's an idea that has been tried on and off for decades and is obviously very appealing but tends to have lots of interesting problems when it has been used for a while - often when you want to access your data outside the context of your application.


Distributed consensus on all your memory sounds potentially (very) slow compared to typical server side virtual machines. True? If so, what sorts of applications is the language designed to be ideal for?


Interesting language. What is dfinity's business model?


Actually, I would love to see an example that does a call to a REST API. I see some examples in the docs to talk to React, but not this.


As this is WebAssembly couldn't you just share some in browser demo where users form a huge compute clusters over WebRTC on their own web browsers? Add in browser REPL and you rock.


No code examples in the entire article?


I was thinking it might be AI-generated, and while it could string together these common concepts very impressively, it could not generate code samples.

But I suppose it’s probably actually the kind of article the AI engines digest in order to be able to generate similar ones.


I keep seeing that wasm is safe and secure. But to my understanding wasm is just a bytecode spec. The interpreters are what have to be judged secure. If an interpreter has a hole then the sandbox is escapable. As you add jits and various cpu optimizations and other options, the more likely there's a bug there somewhere someone can exploit. I don't understand. Is there a mathematical way to verify that a wasm interpreter is secure?


Yes, there is, but you need to define what it should be secure against. Then, a proof can be constructed, given particular assumptions. I doubt this will ever be done for wasm.


Sounds a bit like Ethereum but without a cryptocurrency?


It’s more like Solidity done right, by a team of expert language designers. It’s the scripting language for DFINITY, so it does actually have a token, fwiw.


You can't build something like Ethereum without a cryptocurrency


You mean, a "solution" in search of a problem, and unsuitable for anything touching privacy or security?


Perhaps I should learn this? ;)

Idk web assembly yet though, scanning to figure out how this could be useful for an amateur web-dev.


Don’t learn this or web assembly, learn TypeScript and then Rust. TS will teach you a lot about type systems, classes, modules, interfaces, async/await, etc. Rust will teach you about memory management, filter expressions, futures, (im)mutability, etc. They are both mature languages that are usable in your web dev context and that you’ll be able to find resources to learn. No one is writing WebAssembly manually, it’s even lower level than regular assembly. It would be super painful to write directly, it’s meant as a virtual architecture that can be efficiently executed.


Makes me think we're close to the stage where different languages can be easily combined in the same project, and therefore can be created willy-nilly as easily as DSLs in Lisps.


This already exists on the JVM or the .NET CLR. The concept of an intermediate language is also old. It’s easy to call between languages as well. The “innovation” is really the true OO nature of everything being message passing, and no real persistence mechanism.


So, this is Erlang.


That sounds more like clickbait. Just as if someone took the latest marketing buzzwords and somehow tried to make a product out of it. If there was any meat to this, they would lead with the use case and say "Hey here: This is the problem the world has. Here is how we solved it". But anything blockchain related already went more like "Uhh look, here is a blockchain. Enjoy". This is worse...

This seems to be just like Ethereum without the blockchain. What does "building directly on the internet" even mean? What do I ever need distributed, cryptographic consensus for? The use-case escapes me completely, this even seems more useless than Ethereum, where I can see at least some applications thanks to the blockchain. But that thing without the blockchain just doesn't make any sense. But hey, it got WASM, so must be cool.


> What do I ever need distributed, cryptographic consensus for? The use-case escapes me completely,

Generally the use case for decentralized systems is if you want to disintermediate an existing middle-man who has proven untrustworthy, expensive, or problematic in some way. Or, you want to enable a group of parties to coordinate their activities in ways not previously possible without a (currently non-existing) trusted intermediary. The middle-man function is automated and federated. That’s it.

Any business model dependent on a middle-man is potentially a business model for decentralized systems, if the economics, risks and feature requirements compare favorably. Bitcoin addressed arguably the most impactful one - money - but there may be others.

>this even seems more useless than Ethereum, where I can see at least some applications thanks to the blockchain.

DFINITY is a “blockchain” in the general meaning of the word, aka a public decentralized computing platform with a native currency and attendent economic incentives. It may use other data structures under the hood than most current blockchains, but the use cases are mostly the same as Ethereum.


the fact that you call Ethereum useless shows that you are way too ignorant to voice such a strong opinion.

educate yourself and try again.

i work with bchains since 2013 and ethereum is a major driver for a lot of innovation and real life applications. i've built an energy trading platform on top of it that allows local business to localize their energy management. its deployed and working, granted , in a small poc here in Adam. but still.

also what do you think powers 99% of all altcoins?


is there a hub of some sort to follow ethereum projects? last time I checked (few months I believe) cryptocats was the app with more activity. also I'm curious about your project, if there's anything you're willing to share


Just search for DeFi (decentralized finance).


Ghost in the shell reference? Motoko Kusanagi


Ah, I was wondering which anime they were referring to with this...



I think the guy who made the internet is also trying to fix the worst bits rather than reinventing it. I'm backing Mr Berners-Lee. https://en.m.wikipedia.org/wiki/Solid_(web_decentralization_...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: