I've been building with electric for several months now and it's hands down incredible! Why? I think most importantly because it allows for functional composition across the network (client/server–distributed systems more generally) while also ensuring payloads are as small as possible (only what functions in each env access) which translates to FAST.
IMO this is a paradigm shift similar to GC. No one wants to manage their garbage. No one wants to manage the network. It's grunt work. Let the compiler take care of it.
I've naively rewritten a re-frame (react/redux in js) app that started getting prohibitively slow due to a bunch of deserialization on the client in electric and the performance is outstanding! TBH I'm taking a break from additional UI work on the app to do research because I realized that with electric I can build something much more amazing than I thought possible before.
Caveat: the future will not be served on a silver platter. it's early days and there are some quirks
Small bugs overall, some of which the team already addressed. One Clojure specific that is outstanding is that dynamic variables are not rebound between electric and Clojure functions. Also, the compiler currently does not optimize the callstack so the whole app stream-renders in and out which has a cascading effect. None of these are deal breakers for me, they're being worked on and I'm glad the community can now help ship fixes and improvements.
(founder here) Thank you Dennis! All, broadly, to date we've focused on achieving correct program semantics over syntax/perf/ux/dx. We're rotating onto these issues now, since we've reached a point where, after 2 years of hardcore dogfooding and 8 months of private beta, we're comfortable that brave Clojurists like Dennis will be successful and have an acceptable development experience. (Especially when compared with the benefit of strong composition at the system level - functional programming over entire systems!)
This seems similar in approach to Lamdera for Elm. Just curious, did you take any inspiration from that project or arrive at this solution independently?
(founder here) We got here from first principles by thinking deeply about pure functional programming. A lot of what is happening in pure FP ecosystems is "http servers but with monads" and we think that misses the point.
Practicalities aside (and there might be many issues and even be show-stoppers), the approach is a breath of fresh air in what appears like a stagnating swamp. How to orchestrate overall client/server computations is obviously a vital importance if not the most important design choice in the modern tech landscape. Yet while strong opinions abound, frameworks that help us to think more concretely and rationally about the tradeoffs and break the redundancy of infinite alternative possibilities are not common.
If I understand the idea correctly effectively here you have a platform where alternative choices are simple rearrangements of code.
Not to be a downer, but while I think in theory this sounds interesting, and treating frontend and backend differences as a DAG that is compiled for the different environments seems cool, I'm skeptical that it would actually be productive in a large application.
One thing I'd be particularly concerned about is debugging. One of the things that's on the roadmap is "developer experience", and I don't see any particular debug docs or helpers, so it tells me that if there is a bug somewhere debugging it is going to be a pain, even more so because I won't easily be able to see whether the problem is my client or server code as they live together and are compiled into separate things. As a critique the fact that developer experience is on the roadmap rather than thought about from the beginning is a little off-putting, but maybe that's not what was meant here.
Another thing that gives me hesitation is that I can't see how I would extend the setup to support offline capable applications. This is increasingly becoming a big thing for web+desktop apps, does this easily support that?
Lastly, the docs seem to point to use a homegrown DOM library, rather than one of the major players. How easily would this work with a library like Reagent, Helix, or UIx?
Re. abstraction weight - You're right, it's true that the program is less inspectable than before, and that it adds more distance between programmer and metal. We're aware of the issue and have already begun – we landed async stack traces a while ago (which works about the same as in React.js).
And yet. It's also true that HLL (high level languages) like C, Java, Python all faced the same issue. The benefit is a huge gain in terms of ability to reason about ever larger programs. Java gave us garbage collection. Not everyone liked that; C++ developers want to manage their own memory. But at what cost?
Our thinking is: first get the semantics right. Then polish the experience (today we are here). Then add observability tools. Then make it fast. And if we've screwed up the semantics, then the project will fail for the reasons you describe.
The saving grace is we're seeing 10x LOC reduction (18k to 2k) in rebuilding Electric's sister project, Hyperfiddle (a spreadsheet like tool for robust UI development), as well as massive gains in performance.
Re. offline-first - we think the model can be extended to that but this is a blog post for another day
One quick sketch of offline-first (there are many other ways):
imagine a database like DataScript (already OSS and built in Clojure), move key components like the query engine into Electric Clojure, now the "DAG slice" can cut right through the query engine and it's internal data structures, such that the indexes are partially on the client, and the final stages of query filtering are on the client. A network-transparent database! The bulk of the work, and sensitive data, stays on the server; but the last mile user data querying happens on the client. Electric can transparently move portions of the underlying database indexes to the client for local offline query.
0 LOC cost to userland! No loss of relational query when offline! No performance cost of putting the entire query engine in the browser (a low compute environment)!
> As a critique the fact that developer experience is on the roadmap rather than thought about from the beginning is a little off-putting
Implementing it is on the roadmap but that's no evidence that it hasn't been thought about extensively and for something like this you really -have- to think things out up front (it's more like a math proof than a CRUD app in a lot of ways) or you're going to find yourself with holes in both feet and a codebase you have to throw away and restart from scratch.
From what I can see, they've been laser focused on getting the core to be correct by design, which will have required repeatedly gutting parts of it as conceptual mistakes are shaken out, so if you -wrote- the DX code as you were going rather than simply keeping in mind that the codebase needs to be amenable to it -being- written (and periodically thinking through how you'd do so as a sort of mental dry run) I'd expect the DX parts to have to be thrown away entirely, repeatedly, which is a motivation sink and in the best case scenario would've substantially slowed down getting the core right.
But yeah, the key thing is "not yet implemented" and "hasn't been thought about at all" are very different things and I'd strongly suspect that my read of what was meant is more likely to be the right one given the nature of the project.
(please take the above as "an explanation of why I suspect their actions are fine and their chosen trade-offs make sense" rather than "let me tell you why you're wrong" because you do have a point that if you don't keep DX in mind at all in the early stages the end result tend to be unhappy-making, and, well, let's just say I speak from repeated experience of doing that that to myself ;)
Our DOM module is only 300 LOC - it's bare metal DOM point writes + Electric (reactive language) + macros for JSX-y syntax. When the programming language itself is reactive, DOM rendering falls out for free.
Mechanically, Electric is comparable to Solid.js except the reactive engine is general purpose, not coupled to DOM rendering, which is a special case of incremental view maintenance.
Electric's reactive engine is https://github.com/leonoel/missionary ; Electric can be seen as a Clojure to Missionary compiler (if you ignore the distribution stuff - which btw is optional). Missionary is cross-platform functional effect and streaming system that today is tested on JVM, Node and Browser platforms. Missionary was created by Léo Noel, who is also the architect of Electric.
I've been building with this for 2+ months. Debugging is fine (not great but when is it great? Especially in Clojure). It prints stacktraces in each env. Haven't found debugging harder than traditional client/server dev. But complexity has decreased thanks to it so less bugs.
Agreeing on the debugging concerns. Adding another layer of abstraction on top of everything else isn't going to help anyone. Creating an abstraction that fits so well that you don't have to worry about the layer underneath is extremely hard. That might not show during initial development, but will come up in the maintenance phase for sure.
A bit unsure where the observation of more offline capabilities comes from. I have the impression that things are moving in the opposite direction: Everything needs to be online.
> Adding another layer of abstraction on top of everything else isn't going to help anyone
Ignoring the point that the frameworks we already use are major abstractions on top of abstractions, if this new "abstraction" truly eliminates the complexities of client-server-client it will greatly benefit developers and users alike.
A good many (most?) of the bugs and performance problems (and security lapses) in today's complex webapps is due to the complexities of managing this client-server divide. Even when the developer does things right according to docs and examples, it's possible to end up with a system which does not perform well without solid understanding of what's happening in at the endpoints and in the middle and clever optimizations to solve the special needs.
You're right, I was a bit harsh by disregarding the current use of frameworks. There're huge upsides to them. They still hide stuff, some do it well, some not so. Electric gives me the (unproven) vibe of hiding the lower layers a bit too well.
Regarding the causes of many bugs, your guess is as good as mine. You'll continue to have bugs and you'll have to continue looking under the covers to understand them.
You're right, over-abstracting is a primary risk and has been top of mind for us since project conception in ~2012. ORM, for example, is a nightmare of over-abstraction (remember Spring Framework's AbstractSingletonProxyFactoryBean?). In fact, studying the O/R impedance mismatch from first principles back in ~2010 was what led me to functional programming in 2011.
Electric is an attempt to find exactly the right level of abstraction. The goal is to remove and flatten layers, not add them, thus decreasing abstraction weight in the end if we succeed. Maybe we fail, but first let me share some details about how we think about this:
1. I've personally failed to build this project several times, Electric Clojure is something like the 7th attempt.
2. strong composition model as a starting point, based on category theory generalization of "function" -> "async function" -> "reactive function" -> "stream function" -> "distributed function". ORM does not have this. (This rigor is in response to the past failures.)
3. Functional effect system (monad stuff) at the bottom, which provides strong semantics guarantees about glitch-free reactive propagation, process supervision (like Erlang) (transparent propagation of cancellation and failure), strong resource cleanup guarantees (DOM nodes can never be left hanging, event handlers can never fail to be detached and disposed). Already this results in tighter operational semantics than we have ever achieved with manual resource management (and, again, we tried, see past failures). Our functional effects library: https://github.com/leonoel/missionary
4. Electric affords the programmer trapdoors to the underlying FRP/concurrency primitives. Electric is essentially a Clojure-to-FRP compiler, so if you code raw concurrency and effect management, that actually typechecks with what Electric generates, allowing seamless transition in and out of the abstraction.
5. 3k LOC + 3k test LOC is the size of Electric today (includes a rewrite of the Clojure analyzer). Spring Framework is, let me go check, 59k just for `spring-core/src/main/java`, and there are like 20 other modules I excluded. Indeed it is not a fair comparison but certainly we have complexity budget to spare.
Yes, that's what I meant. I didn't mean an app that's never online (or connected to a server), since that doesn't really make sense in the context of this kind of framework, I meant something that is resilient to being offline (aka "offline capable").
based on the todomvc example, I can say that it is resilient to being offline, in the sense that whatever you do while temporarily offline is resumed when you are back online. there's no optimistic updates so it looks like the items you add while offline are disappearing, but they end up in the list as soon as you are back online.
Everything needs to be online, but "online" those days probably means cutting off constantly between 5GHz and 2.4GHz wifi, with bad ISP DNS and IPv6 routing in the way.
Regarding offline I tested the photon-starter-app repo a bit and it looks like it already supports disconnections but without optimistic updates:
Here's what I did
- killed the network
- added a bunch of items to the todo list, which disappeared from the input field but didn't appear in the list
- switched network back on, items appeared in the list
I'm curious to see how adding optimistic updates to the mix would work
(founder here) Should be straightforward; as you discovered, everything is properly backpressured by the compiler (automatically!) so no state is lost.
You can already kill the boilerplate with code generation and metaprogramming. Once you have done that, you figure out why that is not always a good idea and how to best deal with the exceptions that your particular use case brings to the table.
I'm thinking more of all the cruft that goes to handle state on the client and keep it synced to the server in your React app. Redux and the gazillions of other packages which names sound like warthogs having breakfast.
The same argument applies. I think Redux is a pointless indirection, but you can use metaprogramming to make it more palatable, at the cost of another indirection.
I don't know much about this kind of architecture, but wouldn't it be problematic from an infosec standpoint? It seems like it would be very easy to accidentally leak information from the server to the client in a subtle manner if the developer makes a mistake somewhere. And in ways which would not be obvious from reading the code unless you carefully consider the implications of each e/server and e/client with respect to the mentioned graph analysis?
Are there any mitigations in place to prevent this, except the obvious "be careful and don't write bad code"? Or have I misunderstood things completely and it's a non-issue?
(founder here) You're right that accidents can happen, and if we need to build in safeguards to prevent accidents (i.e. tag a value as server-only) – that should be trivial. To be clear, your backend still exists and has the same security model as a REST Backend-for-Frontend. The client/server markers denote at compile time where a computation (function) must occur. A scope binding like `password` will only transfer if the programmer asked it to by using it from a client region. That would be like putting passwords in a REST payload – it's a programming bug and should be caught in review. But you're right, providing safeguards is a great idea.
I agree it would be user error, but it seems to be a particularly easy mistake to make compared to the REST example, where you explicitly create REST endpoint to send a value to the client.
If you accidentally move filtering of a list of users from server-scope to client-scope during refactoring, everything will still work just as you expect it to and you'll be none the wiser – but suddenly every user has access to all the user data in the list. There's not many other frameworks I'm aware of where moving an operation to an adjacent row or forgetting to change a word suddenly (and silently) exposes the data to the client – except perhaps PHP, which does not have a good reputation when it comes to security to say the least. Altough to be fair adlpz mentioned something similar with Next.js in another thread.
Tagging values as server-only mostly seems like another thing that could be easily forgotten. Personally I would feel more comfortable if values intended to be sent from the server to client had to be explicitly tagged as mutual, with an optional compilation flag that enforces this, such that the intent to share it with the client must always be stated in plain text during creation.
As a security guy I would say this situation happens with reasonable frequency in REST web frameworks where developers are encouraged to use ORMs[0], and the design of Hyperfiddle/Electric is no more likely to fall victim to it.
I'd actually take a punt that it's less likely because of Clojure's data-orientation. Given that the above mentioned ORM-preferring frameworks typically have to define a whole new class and map the data from a server-side representation to a client-side representation, you're more likely to see developers not bother, forget or bungle the implementation aspect.
What I'm interested in is how Electric Clojure handles mass assignment (unconstrained deserialization)[1], sort of the flip side of excessive data exposure. If an (e/server) s-exp uses the symbol `is-admin?` will the server respect or discard values for `is-admin?` sent from the client?
Code is precompiled so you could add all your safeguards during compilation. Should not be too hard to enforce fo only precompiled code to run instead of exposing eval. However, I don't believe this is implemented at this stage.
Accidental code execution vuln might not be the only issue though. How do you know that state which is being checked and verified on server is not bypassed if the application is written without client/server divide? How do you know the data isn't being passed back to the client which should've always stayed on the server?
There is a client/server divide as you can see in the code examples. But I think the issue you're describing is a permissions problem, not a question of whether the compiler manages the network for you. There are traditional client/server apps with wide open permissions that allow all kinds of queries to pass through to a prod DB. Then again there are others that are heavily permissioned. Those are implementation details. If you _can_ control which code can run then you can write that code to _only_ run with certain constraints.
I tried to get hands-on experience with this hyper fiddle stuff a few months ago but I think it was still in closed beta.
I would imagine you can inspect the network traffic. I would hope it would be readable so that you could have a good idea of what's going on and errors would be more obvious.
Hey Dustin, could you share a few details of how you are funding your projects? I know that you have been working in hyperfiddle for a long while but not sure if you have any revenue from it yet.
Your projects are very “futuristic” and the kind of thing I imagine may be hard to get financial backing for, because of the novelty and other considerations… so makes me curious how you balance innovation with monetization in your work.
Personally I think it is ok to say that I am one of the people backing Dustin financially. The last project I backed in this space was Light Table/Eve who did some interesting work in the space. I think Dustin's work shows promise too.
“The Electric compiler performs deep graph analysis of your unified frontend/backend program to automatically determine the optimal network cut, and then compile it into separate client and server target programs that cooperate and anticipate each other's needs.”
This make it sound like client and server segments are determined on the fly based on prevailing conditions, but then the sample code has forms wrapped with e/client and e/server. What do you mean when you say you determine “the optimal network cut” and how do you reconcile this with the expressly marked client and server code?
By the way I’ve been following this project for a while and am very excited about it, kudos on getting to this point.
Great question. A trivial example, expr (e/client (inc (e/server (e/client x)))) should simplify to just (p/client (inc x)) and eliminate the pointless (and perf-damaging) server round trip.
It gets interesting when you introduce lambdas, revealing that this optimization is non-local – you don't know if the transfer is superfluous until you know where the function is called from, and consider the caller and callee together.
The good news is that the DAG (Circuit really) contains everything there is to know about the data flow, so insert compilers research here and poof, insanely fast app.
Looks very awesome. Will definitely give it a try. Hopefully it doesn't end up like Om.next. Om.next felt like an amazing quasi-revolutionary thing when David announced it. But it was so poorly documented and hard to get it right, almost nobody used it in production. I don't know for a fact, but it felt like it complicated Circle CI's frontend to the point that they'd decided to abandon Clojurescript altogether. It's sad, because Om.Next really did have some cool ideas. Fulcro project fixed many problems, but it was too late, the momentum for Om.Next was already wavering.
Seems like e/client and e/server are for when you'd like to force sections of code to run on one side or the other. Otherwise the compiler decides what lives where?
Magic does happen: you are rendering a table on the client, pause to fetch data on the server, and display it. And it looks like a PHP page written in Clojure.
Could someone who understands the similar blurring of lines between client and server in the F# world, through Fable or more broadly in Blazor/Bolero, comment on how this approach compares?
I may be mistaken but I think Electric Clojure renders HTMX unnecessary. Where HTMX still works by a developer specifying the network calls (albeit in a declarative manner), Electric Clojure hands over the same responsibility to the compiler.
This looks cool. If I think one step ahead, this reduces enough incidental complexity its the kind of thing that, when combined with AI, could produce the holy grail of an AI writing most of the code for a full CRUD app.
IMO this is a paradigm shift similar to GC. No one wants to manage their garbage. No one wants to manage the network. It's grunt work. Let the compiler take care of it.
I've naively rewritten a re-frame (react/redux in js) app that started getting prohibitively slow due to a bunch of deserialization on the client in electric and the performance is outstanding! TBH I'm taking a break from additional UI work on the app to do research because I realized that with electric I can build something much more amazing than I thought possible before.
Caveat: the future will not be served on a silver platter. it's early days and there are some quirks