
Designing very large JavaScript applications - kawera
https://medium.com/@cramforce/designing-very-large-javascript-applications-6e013a3291a3
======
simonw
I love Malte's take on what makes a senior engineer with regards to API
design:

> The way I would talk about myself as a senior engineer is that I’d say “I
> know how I would solve the problem” and because I know how I would solve it
> I could also teach someone else to do it. And my theory is that the next
> level is that I can say about myself “I know how others would solve the
> problem”. Let’s make that a bit more concrete. You make that sentence: “I
> can anticipate how the API choices that I’m making, or the abstractions that
> I’m introducing into a project, how they impact how other people would solve
> a problem.”

~~~
aje403
I think "How confusing is this going to be to the one consuming this"
(assuming you're not writing the front end as well) is the more important
question to ask

~~~
abritinthebay
And the one most rarely asking in my experience too

~~~
silversmith
In my opinion, asking this question is the key benefit of test-driven
development. You get to be the consumer, write down your expectations, and
then work backwards from yours-as-consumer expectations.

~~~
sbov
I find the opposite tends to happen in Java. In the pursuit of testability,
people make libraries much harder to use and understand.

~~~
purerandomness
Writing good tests is a skill that has to be developed, just like anything
else.

------
komali2
I've always disliked the "BIG TEXT CHOPPED-OUT BUT YOU'LL READ IT BELOW AGAIN
IN JUST A SECOND" paradigm in magazines and newspapers. Like, I'm reading the
article anyway, why suddenly spit some of the words in my face super big? I'm
guessing it's for skimmers, catch their eye and bring them into an interesting
paragraph that's further down from the top.

But this article does it at the beginning... we read the sentence "Hello, I
used to build very large JavaScript applications" three times. First as an
image, then as a label for the image... and then as the _first sentence_ of
the article. Why lol

~~~
bunderbunder
> Why lol

Because, even closer to the beginning, in the real first sentence of the
article (by which I mean, the first sentence after the title), it says, "This
is a mildly edited transcript of my JSConf Australia talk."

This is not an example of the "big text chopped out" paradigm from magazines,
it's an example of the, "each slide contains a brief of the most important
part of what you'll be saying while that slide is being displayed" paradigm
from conference lectures.

~~~
et-al
The problem is those slides hinder the readability of the actual article.

For a reader, you get into the rhythm of the writer's voice, only to be
interrupted by a giant image of text every other paragraph. It's a battle
between just skimming the slides, or trying to read the content.

If the author had just swapped out the "loud" text images for subtle headers,
the article would actually flow.

------
abalone
More like designing very long Medium posts.

Is there any substance to this "we invented 'enhance', a reverse dependency
pattern" thing or is it just old wine in new, javascripty bottles?

~~~
dnomad
This is just basic dependency injection.

I agree it is mildly amusing to watch Javascript devs excitedly discover these
principles but it does show that these principles are universal. Virtually
everything in this talk applies to building very large Java and C++
applications. (And here I don't mean 10mb... rather global systems made up of
dozens of services and hundreds of components.) All of this is about
modularity -- of configuration, of state, of logic, and of concerns.

~~~
selestify
Is there a single place to learn about all of the common software engineering
principles, as opposed to rediscovering universal principles over and over
again?

~~~
ivanhoe
Martin Fowler's site is a good start, especially this collection:
[https://martinfowler.com/eaaCatalog/](https://martinfowler.com/eaaCatalog/) .
And then read his books...

------
magicmouse
It is tragic that the world standardized on Javascript. They are just now
discovering some of the module issues that Prof. Wirth solved in his great
Modula-2 langauge in 1974. His system allowed separate compilation, and a way
to detect when the interface API changed so that it could be cross-checked. I
used Modula-2 to great effect on very large projects, and am greatly relieved
that the Modula-2 module system has more or less made it into JavaScript.

~~~
tzahola
Don’t worry, they’ll discover types eventually.

~~~
Joeri
Aren't most large-scale javascript applications using typescript or flow these
days? I definitely wouldn't try building a large scale javascript codebase
without either of those anymore.

~~~
tzahola
Frontend developers at our company vetoed TypeScript for being too difficult
to comprehend for JS devs.

~~~
always_good
The downside of Typescript is the constant friction from bolting a type system
on top of a dynamic ecosystem.

It's like Babel and Webpack where you incur time spent on debugging your own
tooling and getting these systems to collaborate. By the time I put in the
effort, I decided it was simpler to go all in and use a compile-to-JS language
instead of something that tried to be Javascript with types.

~~~
TimJYoung
This is 100% spot-on. With our web development product, Elevate Web Builder,
the furthest that we went in the compiler with "giving in" to the dynamic
typing in the target JS was a variant type. The rest has to be given an
external interface declaration (classes, functions, and variables) that the
compiler uses to type-check all calls to external JS or the built-in
interfaces in the browser APIs. Other products have tried to allow for in-
lining JS or simply include-ing it via special declarations, but that really
handicaps what the compiler can do in terms of statically analyzing your
codebase, which kind of defeats the whole purpose of using types in the first
place.

------
danielvinson
This was posted multiple times in the last few days

[https://news.ycombinator.com/item?id=16880991](https://news.ycombinator.com/item?id=16880991)
[https://news.ycombinator.com/item?id=16844285](https://news.ycombinator.com/item?id=16844285)

~~~
tenaciousDaniel
There should be a feature that matches the url of your link against posts
within the last 7 days, and alerts you to possible reposts prior to
submitting.

~~~
fiatjaf
There is.

------
antjanus
I'm really appreciative of the mention of the inverted import.

Inverting the flow really lets components "register" themselves and thus be
fully contained rather than be "required" by another component. It can be
powerful but it causes the problem that you never really know what all is
registered to another component.

Also, yeah, code-splitting isn't easy!

~~~
camwest
I have a talk about these ideas in the Context of a React Application last
year at React.js Conf -
[https://youtu.be/395ou6k6C6k](https://youtu.be/395ou6k6C6k). It might be
worth a watch if you’re interested in how to build applications like this on
your own.

~~~
rcanepa
Does anyone knows if this can be done in Vue.js?

------
Silhouette
I agree with much of what is written here, but the emphasis on code splitting
seems like it risks hitting the wrong target. My first question is usually
what caused even relatively simple web apps to need multi-megabytes of
"optimised" JS code in the first place.

All too often, the answer is not the application code itself but rather all
the direct and transitive dependencies, where the developers yarn-added first
and didn't ask questions later. And while obviously that offers some
advantages in terms of reusing code, it does also have a very high price in
terms of bloating our deliverables, particularly given the still relatively
immature tools for optimising JS compared to traditional compiled languages.

Maybe we should be looking at that problem first, before we start assuming
that any large JS application will necessarily produce some huge monster
outputs that need to rely on code splitting and all the overheads that brings
just to get acceptable performance?

~~~
BigJono
I think the author still has a point when talking about "very" large
applications. His example of all the Google widgets is a good one. Even if you
were to hand-craft all of them with optimised code (which I imagine is what
they do at Google), it would still likely be too much code for a single
package to load in a reasonable amount of time.

But I agree completely with you as far as other projects go. If you drop the
"very", it should be entirely possible to build a large project without the
need for code splitting. Almost nobody is doing anything to the scale of the
example talked about in this article, and almost everybody is using code
splitting.

In my experience, there's two root causes for this.

Cause #1 is that barely any front-end developers understand or consider the
cost of abstractions. Something like code-splitting is commonly seen as "free"
because it takes a couple of lines to implement. The permanently increased
complexity of the software and all that extra brain power required to grok it
over the development lifetime is never taken into account. At least half of
the devs I know are happy just banging in code-splitting at the start of a
project with zero thought and I guarantee you they'd read this article and not
understand the "sync -> async" example given to explain the downsides of code-
splitting.

Cause #2 is that devs are too eager to add new dependencies. By definition a
popular library is going to be extremely bloated relative to your use case.
When you 'yarn add react-redux-awesome-date-picker', what you're usually
adding is an overly generic solution with a large amount of customisability
and functionality that's not relevant to your use case.

My go-to example from my own experience is with rc-collapse. rc-collapse is a
container that animates to and from 0 height. It's like 200 lines and has 4
dependencies (god knows how many KB that adds). I've been using my own version
that's 50 lines and 0 dependencies in production code for years and never run
into a problem with it. I'm sure rc-collapse works around some fancy edge
cases or supports older browsers or something, but I'm almost positive the
extra weight isn't necessary in 90% of the projects that use it.

This is kind of tangential, but another real problem with this mentality is
that by implementing my own collapsible container, I learnt some important
lessons about React, the DOM, browsers etc. Devs that play npm lego aren't
generally going to get that extra knowledge, which will cost time in the long
run.

------
ngxoogler
The framework he is talking about is trash compared to modern JavaScript UI
frameworks.

There are very few established patterns and each team seems to make up their
own rules. Some of the worst code I've ever seen was a result of this
framework.

The code-splitting is probably the best feature and worked great when this was
released, but smart lazy-loading of bundles is much easier in 2018.

The most frustrating thing is that the framework developers consistently point
to benchmarks created over 3 years ago to show how great it is.

~~~
iovrthoughtthis
This article isn't about the framework though. It's incidental.

This seems a very out of place comment.

~~~
ngxoogler
I get what you're saying and maybe I'm being harsh, but the whole backdrop of
the article is this supposedly amazing framework that was built. It undermines
a lot of his points that it is poorly used.

~~~
alexanderhouse
The apps listed in the talk (Photos, Sites, Plus, Drive, Play) all seem to
have gotten much better lately? Are you actually sure you are talking about
the same framework as the author? Google has so many it can be confusing.

If you'd be talking about the closure library and dreaded goog.ui.Component,
I'd agree but that isn't it.

------
xab9
It's interesting how in 2018 web development === react. React this, react
that.

Anyway, I think beginner empathy is when you anticipate the usage of your API,
or modules. Pro empathy is when you talk about it and share your feelings and
needs and then listen to those other users' feelings and needs.

Fractured pluggable routing may be a good idea, but you know, at a certain
point a centralized route definition may be a good idea too. There's no one
solution fits all strategy.

Major routes === modules (oh boy, enhance). We're all MVC (with islands of
goodness) again. Frontend is the new backend, but it's fronted too. Doing
frontend is hard.

Dependency injection. Malte talks about the google search result page as an
example for ultra complexity - Angular tries to sell itself for very high
complexity ("enterprise") projects, yet I would eat my hat if Google plus or
even the search result page would be better in Angular. The frontend landscape
is fractured and everyone tries to sell you his or her snake oil.

Base bundles: yes, they suck. That's why common chunk creation is delegated to
the bundler (like webpack) and then you believe that if a datepicker used more
than three times it's okay to load it all the time. Or you can be a smartass
and do require-ensure like logic all yourself (maybe it's not your fault,
maybe just a very clever manager made you do it). And then you will feel very
smart and good and after a year or so you realize how it was stupid and then
someone comes along and relegates it back to the bundler.

Large applications in javascript are like Frankenstein's monster, but it's
better to have a monster than a pile of dead flesh jolted by electricity with
each and every release. My two cents.

~~~
stratigos
> It's interesting how in 2018 web development === react. React this, react
> that.

Its quite popular for sure, but after just doing a job search for about 6
months, I came across a surprising number of firms using ng, ember, and even
vue in a couple places.

modern ruby on rails with ujs is still hugely popular

~~~
xab9
You're lucky. I've been through a jub-hunt cycle, my experiences here are:
lots of legacy stuff (mostly angular1), tons of react, some nu-angular and one
vue (which is going to be my next job, not just because I love vue, but also
because they understood the tech stuff I had been talking about, unlike many
others).

------
RoboHacker111
This was a very interesting article!

I'd currently consider myself a mid-level engineer, and I have recently been
put in a position where I must lead a team of many devs and incorporate all
their code. I did understand about half of this article, but it also indicates
that there's a lot left for me to learn.

I'm curious what kind of blogs or online courses or books I should search for
if I wanted to better develop this skill within myself, and to design and
architect these sorts of large applications with multiple contributors?

------
darkmarmot
My company also rolled our own framework to handle the case of 'very large web
applications'.

In our case, every component is a file (which is becoming more typical) -- but
it differs in that every file dependency is automatically resolved and
downloaded in a lazy, on-demand fashion. With HTTP 2.0, this means you can
update a single file without a build process -- and everyone receives the
update as a minimal change.

Also, all paths are relative or built dynamically using something like C
#define, so it's easy to refactor by moving entire directories about.

~~~
tytytytytytytyt
Don't the popular web frameworks handle lazy loading things?

~~~
darkmarmot
Most of the popular frameworks generate a single JavaScript bundle that
comprises an entire application. Some people break these bundles into multiple
parts manually in a process called 'code-splitting' in order to do a really
chunky kind of lazy load.

~~~
ngxoogler
Code splitting via webpack is not a manual process.

Does your hand rolled framework handle minimization and obfuscation? Probably
not because any changes would require a rebuild.

Are your assets cached? If they are, I'm curious how you handle the cache
busting when a single file gets updated.

~~~
ec109685
With HTTP/2, you can send files down independently, without creating a large
bundle, so each file can be cached. You couldn’t obfuscate function names
across files though unless you made the process deterministic based on module
name.

~~~
kilburn
But you need some cache-busting functionality anyways (otherwise how does the
client know that file "something.js" has changed?

~~~
ec109685
Etags could be used to cause the browser to revalidate the contents without
too much overhead since the connection is already establishd, but good point.

------
G4BB3R
I though the article would have just one line saying "Use Elm"

~~~
napsterbr
After 1 year using Elm, we've had a lot of great and not-so-great experiences.
For a large app I still think choosing Elm was overall a better choice than
javascript, but it's not without downsides.

Recently a new dev joined the front-end team. He had js background and no
previous functional programming experience. In less than one month he was
already shipping production code with confidence on our ~40kloc app. That goes
to say that 1) I don't think lack of Elm programmers is a problem if the
company is willing to train its employees and 2) I'm sure the front-end lead
and the new dev would have a much harder time if we were using javascript.

Surely this is _our_ experience. YMMV.

~~~
always_good
The killer feature of Elm is something I discovered a year after taking a
break from Elm: I revisited a production Elm application I hadn't touched in a
year, approaching it almost regretfully with "why did I have to use Elm?",
certain I'd have no idea what was going on after such a long absence.

What actually happened was that I was immediately productive.

Without having to credentialize in the code, I was able to stub out new
features and push an update. An infrequent experience for me in other
languages. I had only a figment of an idea of the code I had to write, but I
got started and the compiler helped me the rest of the way.

I still don't understand how to confidently organize an Elm app. But that
describes my relationship with every front-end Javascript framework as well. I
regularly choose poorly between component vs upstream state in React. But what
I _can_ do is refactor Elm code at a level that would be downright expensive
in Javascript.

As far as the sour graping going on in some nooks of the Elm ecosystem, I'm
reminded of this post by Rich Hickey:
[https://www.reddit.com/r/Clojure/comments/73yznc/on_whose_au...](https://www.reddit.com/r/Clojure/comments/73yznc/on_whose_authority/do1olag/)

------
arvinsim
I really dislike these frivolous images that can be easily represented by
blockquotes. It just wastes data.

------
auvi
what is the metric to consider a JS app as "VERY large" as the author puts it?
If SLOC is used as a proxy then how code from libraries (i.e. code that the
team didn't write) are considered? Recently when I generated a Ruby-on-Rails
app with webpack enabled, it downloaded 8148 JS files in the node_modules
folder. 675K SLOC. I thought it was absolute madness.

~~~
Silhouette
The size of node_modules probably isn't a great metric to use if it includes
development tools and libraries (and all their transitive dependencies) as
well as any libraries you're bundling into the files your user will download
and run.

~~~
girvo
Well, I still have to have those on my computer and in my Docker container.
And from a trust, auditing and security perspective it’s pretty horrible, even
if multi-stage builds can help

With Buble, Flow, and Roll-up, I can get a decent modern environment with as
few dependencies as possible. Upsides? Much faster, too. Less disk space.
Docker containers build faster, which is better for CI/CD. And it’s easier to
understand the codebase and dependency tree, including development tools.

~~~
Silhouette
Just to clarify, I'm not disagreeing with you that the current dependency-
heavy JS world is madness or that the bloat during development activities is
unhelpful. I'm just saying that having a large/complicated development
environment is probably a separate issue to having a large application for
most other practical purposes, but looking at the size of node_modules will
often conflate the two.

------
jordache
which modern JS framework offers component level lazy loading? Angular offers
only route based lazy loading, which as is limited as the article pointed out.

~~~
TheCoelacanth
Pretty much trivial to implement in react.

    
    
        class Loader extends React.Component {
          state = {
            Component: null
          };
        
          componentWillMount() {
            import('./Component').then(Component => {
              this.setState({ Component });
            });
          }
    
          render() {
            const { Component } = this.state;
            if (!Component) {
              return <div>Loading...</div>;
            } else {
              return <Component/>;
            };
          }
        }
    

Might want to add a few lines of error handling for production use, but that
is pretty much all you need.

~~~
xab9
Like the article (presentation) said it's not the best idea to do this with
hundreds of components (I think he mentions latency) - in fact you "somehow"
have to (should?) bundle things together and then you are back to square one
or at least have to hink about pluggability/configuration again.

~~~
TheCoelacanth
Obviously you don't want to use it for every component, you put it in key
places where you want lazy loading to happen.

------
polskibus
Does Google really use React? I got that impression from skimming the article.
That'd be interesting given they develop at least two competing tech like
polymer and angular.

~~~
Joeri
The talk mentions they use an in-house framework, but react was probably
substituted in to make it more approachable for the audience.

------
randiantech
Isnt React actually open source?
[https://github.com/facebook/react](https://github.com/facebook/react)

~~~
pnevares
You were misled by the slide image interrupting the flow of the text:

 _So, I build this JavaScript framework at Google. It is used by Photos,
Sites, Plus, Drive, Play, the search engine, all these sites. Some of them are
pretty large, you might have used a few of them._

 _[Slide]_

 _This Javascript framework is not open source._

~~~
randiantech
Oh, thanks for the clarification.

~~~
anonytrary
I found it a tad bit confusing as well, thanks for asking.

------
TAForObvReasons
> This Javascript framework is not open source. The reason it is not open
> source is that it kind of came out at the same time as React and I was like
> “Does the world really need another JS framework to choose from?”. Google
> already has a few of those–Angular and Polymer–and felt like another one
> would confuse people, so I just thought we’d just keep it to ourselves.

Given the current state of affairs, an open-sourcing of this framework would
be welcome.

~~~
leetrout
Do you have some specific reason for this (genuinely curious)?

Think we've put ourselves in a corner, so to speak, with the Angular, react,
preact, vue, ember, etc etc options and this would offer something compelling
from the authors examples?

~~~
TAForObvReasons
The article discusses one of the major problems: "React component statically
depend on their children." There are workarounds, sure, but there are all
kinds of pitfalls with the dynamic workarounds, to the extent that I find
myself breaking the react model regularly in applications (e.g. utilizing
state outside of redux and forceUpdate)

