
3factor apps: a pattern for fast iteration, resilience and high scalability - tirumaraiselvan
https://3factor.app/
======
vlucas
So this pattern forces specific technology choices on you (GraphQL), and
encourages vendor lock-in by going all-in on "Serverless" cloud functions. No
thanks.

~~~
rjbwork
We do our "serverless" using the self-hosted version of Azure Functions -
Azure Webjobs. For most of the events we can scale out to 32 beefy machines.
We also don't use graph SQL, preferring simple rest endpoints made in Web Api
2, and some reporting functions. If we need to do more than that...well, we'll
have to rethink some things.

It will be a good problem to have.

~~~
diddid
While I agree Azure web jobs are awesome, I’m not sure I would call them
serverless. They require you to provision an app service and would only scale
within whatever tier you have and ultimately share resources with any sites
you have on the same sever.

Microsoft actually has an on-prem or self-hosted version of Azure functions
that uses docker, though it’s still only in preview.

------
vikingcaffiene
Scaleability means more than just being able to accept infinite requests
without buckling. It also means writing predictable code that your teammates
(or you 6 months later) can parse and understand whats going to get touched. I
would argue thats more important for 98% of the stuff we build. You can
iterate faster when you can understand what your changes touch and see the
whole picture. Talking to what is essentially a bunch of stored procs on AWS
or similar and/or letting your front end team write the database queries to
fetch data seems like a massive footgun. Am I missing something here?

~~~
tango12
Hm...not sure why you think this goes against what you’re saying. The idea is
just to make things more asynchronous and give the front end team the ability
to consume async information. Also, giving the frontend team the power to
query what they want is not a bad thing. You should have a process to
whitelist queries before they go into prod anyway.

Incidentally serverless can also mean microservices. The idea is just to
deliberately remove state from inside the code and put it into state or
events. And then running that on something that scales 0-n. Why Is that
against your principle of scaling the codebase for the team? It’s just a dev
principle to make ops easy.

~~~
vikingcaffiene
Let me start this by saying I am a frontend developer. Lest you think I am
some curmudgeon coming at you with exclamations of the "good ol' days" of
PHP/jQuery...

I'll give you this: you can build something suuuuuper fast with this
architecture. I imagine it would be thrilling for the first, say, 6 months or
so. Hell maybe even a year if your business rules don't change too radically.
In SV that might well be enough. As long as you are building and not
maintaining/updating I doubt you'll get bit too bad. Depends on the size of
your team. In this case smaller would be better.

Its just that I've yet to see a compelling case for this type of thing for non
FAANG companies. They have very very different problems than the rest of us
and every one of them started out as a monolith (shoot Google STILL rocks
one). To me it makes way more sense to start out as one, maybe two, code bases
and then scale off things that need these performance boosts to serverless
processes etc as they are needed. The reason being that to be effective in a
given system, one needs to be able to understand the ramifications of changes
as they are made. Its much easier to do that when you can grok the whole
system vs talking to an isolated process.

Let me step through your comments/questions:

> Serverless can also mean micro services

Oh I am aware. Dear god I am I aware... I manage many of these in a given day.
These are even worse honestly because they have their own dependency trees and
keeping those up to date/compatible is a friggin' nightmare. Then of course
there is configuration management. Then there is the case of having to
maintain multiple release versions of an app and needing to apply patches to
those micro services. Oh and what if you have a multi tenant environment and
those are running different versions? Its just a mess. Thar be dragons dude.

> giving the frontend team the power to query what they want is not a bad
> thing

On this we disagree. There should be a discussion about this. A pattern to
follow and a data contract. The UI should be as dumb as possible and only know
about that contract. If that contract is not met, things should go red. I'd be
willing to concede that something like a reporting interface or a complex
filtering scenario would make sense. I just don't think it makes much sense to
let anyone pull down anything they want wherever.

> You should have a process to whitelist queries before they go into prod
> anyway.

We agree on this. However, at the risk of sounding snarky, thats what an API
contract is.

> And then running that on something that scales 0-n

Again, this is not a problem you need to solve right away. If/when it does
arise, there are a lot of steps you can take that are simpler and easier to
manage than a bazillion isolated processes that don't know about each other.

Am I saying this is a uniformly bad approach? No of course not. It might make
perfect sense for your org. I am just saying that for most of the type of work
I've done, the real scaling issues that I've encountered come from premature
optimization and unnecessary obscurity.

~~~
tango12
Thanks. This is helpful!

One question on the API contract, giving frontend teams power thing: what are
your thoughts on things like firebase on the one hand and graphql on the
other?

------
jph
Looks like a multitier architecture plan to me. I see an app tier, a logic
tier implemented in GraphQL, a state tier, and a functional services tier.

[https://en.wikipedia.org/wiki/Multitier_architecture](https://en.wikipedia.org/wiki/Multitier_architecture)

Also see the 12 factor app guide, which aims to improve adaptability and
resilience.

[https://12factor.net/](https://12factor.net/)

------
sethammons
I think this really boils down to:

    
    
      1) Have an api gateway (for them, graphql).
      2) Store state in a (most likely distributed) store.
      3) Have services interact with state via an event system.
    

All this really says is make your applications stateless. This is a trend that
has been going on for a while.

I don't think GraphQL nor serverless is required and are just specific to
their implementation.

Unless I totally missed what they were going for.

------
KaiserPro
So basically you've renamed your API layer, graphQL

one note, using serverless doesn't allow you to scale to infinity. You are
still dependent on your database/state management.

~~~
y4mi
I don't have any experience, but isn't it generally encouraged to use DynamoDB
or spanner for faas on aws or gcp because of that?

~~~
KaiserPro
Its possible, but that that scale how your model your data, and what and where
you want your data to be become very important.

If you need global state, then dynamo _might_ help, but that's still limited
to a region. Yes there are mechanism for syncing regions, but you need to
decide what's more important, consistence, or speed.

------
tango12
Hi HN!

Wrote this up based on our experiences working with early adopters of GraphQL
and serverless. There have been lots of teething pains with serverless
architectures and we've seen this as being a useful way of pushing business
logic to being "serverless" which is run asynchronously and the results are
captured asynchronously too. That way you don't have to worry about cold-
starts and a lot of error handling and resiliency concerns go into the generic
event system.

GraphQL subscriptions also fit in really well because they allow clients to
consume asynchronous results.

Feedback, ideas, experiences very welcome!

~~~
IMTDb
The issue I have with your website compared to the 12factor one is how hard it
is to grasp/implement your advices.

> 12Factor Nbr 3 : Store config in ENV variables.

Yep I can do that. In 2 hours I can convert most of my conf in env var. And my
next project will go with ENV var from the start.

> 12Factor Nbr 11 : "each running process writes its log, unbuffered, to
> stdout"

Yep, give me 5 minutes to do that.

When I read 12 factor, and decided to give it a shot, by the end of the week,
my app (2 devs full time for 2 years) got nearly everything nailed down and I
was able to benefit from it. The last remaining points were harder to
implement but I undestood what I needed to do and why.

Most of the advices of the 12factor manifesto can be applied to any
programming language/framework/architecture/buisiness and make the
deployment/management of the app at least easier.

When I read your manifest, I did not understand most of it. Even your examples
do not "talk" to me :

> In your food ordering workflow, instead of writing a payment processing
> microservice that captures different failure modes, write a payment
> processing function that processes a payment or fails. The event system
> should capture the retry or failure event which is processed thereafter.
> This ensures your business logic is simple and easy to scale.

What event system ? how does it capture the failure mode ? I have an app that
is now 4 years old, on which 6 devs are working full time, what can I do today
to make it better in a month or two ?

Why are your trying to force me to use GraphQL and make a tech choice for me ?
At no point 12factor says use "HTTP REST JSON API" even is a lot of user will
go that road, it's their choice and not forced by the maifesto itself.

~~~
tirumaraiselvan
> What event system ? how does it capture the failure mode ? I have an app
> that is now 4 years old, on which 6 devs are working full time, what can I
> do today to make it better in a month or two ?

Good question. Definitely the wording can be improved. Let me try to explain
though.

Suppose you write a simple function which processes payment. It fails. Now you
can insert a state saying failed with error code 1001. Now this insert creates
an event which invokes a serverless function specifically aimed at resolving
this error.

This way you have smaller cohesive functions which do "atomic" work as much as
possible.

PS: Co-author here.

~~~
fiatjaf
What is a "serverless function"?

~~~
bobwaycott
I’m guessing they mean something like AWS Lambda.

------
andjd
tl;dr:

>[An] architecture is analogous to the redux dataflow model on a react app,
but applied to the fullstack.

The page has no concrete recommendations as for what tools could work to
implement this pattern except GraphQL. Without a pretty strong knowledge of
the tools out there, you won't know how to implement this architecture pattern
well.

There's a lot of hand-waving past non-trivial problems and there's not much
help on what tools/patterns/best practices will help that. For example, the
architecture requires a fully async backend, yet that all state changes be
atomic.

Similarly, the architecture requires `Your event system should deliver the
events reliably to other microservices.`. Not a trivial thing to achieve.

Also, this architecture promises to `Scale your backend infinitely` because of
using serverless services, but because everything is being funneled through a
centralized state, That server/database has the potential to be a bottleneck
that keeps you from scaling, and the architecture has no advice as to how to
build your state to be scalable.

------
da02
So let's say an event is emitted, does this happen?

* "payment processing" handles it -> changes state -> another event emitted -> "restaurant approval" processes it -> changes state -> another event emitted -> "delivery assignment" processes it

Or does this happen?

* event emitted -> "payment processing" and "restaurant approval" handles event asynchronously, -> state changes -> event emitted -> "delivery assignment" handles event.

~~~
tpetry
So instead of clear code to handle the error you have a dozen logic components
watching for some events and modyfing state again triggering events and
modifying state. Yeah, i absolutely can see how this approach will NOT
increase maintainability because now you have 84 logic functions watching for
some graphql events, and they may never be down for a microsecond because they
would miss a (critical) state change: graphql subscriptions are not like
persistent queues

~~~
tirumaraiselvan
That is why events must be persisted and reliably delivered. And in the
frontend, you must subscribe to the state of the world and not necessarily
each state change. This way if frontend goes down and comes back up you can
show the new state instantly.

~~~
rjbwork
Exactly. We created our own even store on top of a SQL database. It's quite
simple to do, and it is blazing fast because it is custom built for our use
case. It's an append log of every event, their time, their type, and some
other info about them (account, user, etc.). The event itself is stored as
json, and we can just use Json functions of SQL Server to query those for
diagnosis if need be. We've also got failure reporting built in, and strict
referential integrity for various other tables to link into the main event
storage table.

It was actually a pretty fun weekend project.

~~~
da02
Right. But, what if the client (eg browser) would ask the system if the
message has been fully processed? Then the client can re-send the message if
the Server lost the original message? For example:

    
    
      client: "Hey, buddy! Did you process that message I sent you? Here is the ID."
      Server: "No. Send it again."
      client: "Ok. Here is the message again."
      [Server processes it, completes processing it, sends back response.]
    

If done this way, wouldn't you be able to reduce the need for persistence of
the original message and events from the client? (Not that persistence of the
original event on the Server would be a bad thing.)

~~~
rjbwork
We can do this via the store's aforementioned failure reporting. I guess it
could be better? I prefer to have a log of all messages since forever, as in
Event Sourcing.

Seems like that is more trouble than it's worth to try to make everything
happen purely on a message bus, though the mechanism for letting the system
know a new event arrived is a message bus, it just includes the message id,
for us.

------
gregorygoc
I’m waiting for 1factor apps.

~~~
rjbwork
"Just write teh coad"

------
mothsonasloth
I've never used GraphQL, would using it mean I do not have to define DTOs or
other API objects, I would just create models from whats been passed into
GraphQL?

Talking in a Java context here.

------
ah-
What would you use nowadays to actually implement the real time GraphQL
subscriptions and event system in practice?

~~~
tango12
We built [https://github.com/hasura/graphql-
engine](https://github.com/hasura/graphql-engine) ,
[https://hasura.io](https://hasura.io) that sets up the plumbing for this
architecture with state being Postgres.

There’s a lot of other tooling emerging in this direction too. For example,
set up a GraphQL server and subscriptions with Postgres and listen/notify and
use SQS with lambda. Or dbezium + Kafka + custom service for event delivery.

------
orev
“3factor” strikes me as a bad name. In IT, 2-factor universally refers to
login security/multi-factor login. “3factor” sounds like you’re adding another
security method to the login screen.

~~~
vkjv
It's a play on "The Twelve-Factor App"
[https://12factor.net](https://12factor.net).

[https://3factor.app/#comparison-
to-12factornet](https://3factor.app/#comparison-to-12factornet)

~~~
ChristianBundy
This may be silly, but I think copying the format and spelling out the number
might make this look less like "2-factor":

> The Three-Factor App

