
Scaling to 1M active GraphQL subscriptions on Postgres - tango12
https://github.com/hasura/graphql-engine/blob/master/architecture/live-queries.md
======
aidos
Just another little shout out for Hasura here. It’s a really lovely system to
use and the team behind it are awesome to deal with. Super friendly and
helpful.

We evaluated a lot of stuff recently before settling on Hasura. Prisma, aws
appsync, postgraphile, roll our own socketio backend etc. I had a few
hesitations about Hasura. In particular I was hoping to lean on Postgres RLS
for the authorisation, and to be able to customise the graphql endpoints a bit
more. After hearing the rational from the guys about how their system is
better than RLS and seeing the patterns they have for implementing the logic
I’m really happy we made the leap.

We’re still in early days, but it’s been really solid and a real pleasure to
use.

(No affiliation, but happily paying for their support licence)

~~~
ndarilek
Out of curiosity, what is the rationalle for abandoning RLS?

In a previous project, I used Postgraphile and enjoyed it. Sure, RLS and
PLPGSQL were a bit awkward, but I knew they were well-tested and had lots of
eyes on them, so I felt comfortable that if an RLS policy blocked access to a
row, then that particular role wasn't getting access. I also enjoyed working
with a text-based format that more clearly fit within my other workflows of,
well, working with other text-based formats. :)

Another part of it is that I'm blind, and Hasura seems to make doing things
outside of its web interface somewhat painful. Sure I can write a YAML
migration by hand, but the YAML migration format seems more machine-friendly
than user-friendly. Yesterday I needed to create a function, and the SQL web
interface wasn't showing me the line number on which my function had an error.
Ultimately I dropped to the psql command line, cut-and-pasted the function in
manually, and right away got back the line number and fixed the issue.

Please don't get me wrong. I'm not trying to hate on Hasura. But the fact that
I can't just drop to a non-YAML text-based format and throw down a few checks
to secure my tables has been an endless source of frustration for me. So if
there _is_ a non-NIH reason for abandoning RLS in favor of a separate security
system, then maybe knowing that might help me be a bit less annoyed. :)

~~~
ZitchDog
I can't speak for the author, but having used RLS for authentication I would
not recommend it. It complicates nearly every EXPLAIN plain, and it can cause
performance headaches in even the simplest of queries.

We had to mark many internal functions as LEAKPROOF in order to get them to
perform. It was not possible to move our database into RDS, so in the end we
had to abandon the RLS authentication layer.

~~~
ndarilek
Thanks for this. Lots of developers fall into the trap of discarding concepts
because they aren't Facebook/Google scale solutions. The app on which I used
Postgraphile/RLS was likely to have only a handful of users, so I thought more
about solving the problem effectively than about whether the solution would
scale, so RLS seemed like an airtight system. Knowing that it kills
performance puts things into perspective. It at least suggests which tool
might be best for which job. :)

~~~
BenjieGillam
I suspect many of these concerns were with PostgreSQL 9.5 and 9.6 where RLS
did indeed have many performance issues. PG 10 fixed a large number of these
issues, and PG11 even more. I would recommend re-evaluating RLS performance on
these latest PostgreSQL versions.

~~~
aidos
In Hasura’s use case it’s a little different. They’re working on the
assumption that you have loads of identical queries that you want to run that
differ only in the session variables (and hence the rows they see). Because
they’re outside Postgres, they can collate it all and execute a single query
for multiple users. Within Postgres you don’t have that luxury. It’s a trade
off, but not using RLS suits some use cases better.

------
Thaxll
I was curious about the code so I went to check it, it's written in Haskell,
never seen Haskell before, it's really hard to follow for someone not used to
it: [https://github.com/hasura/graphql-
engine/blob/master/server/...](https://github.com/hasura/graphql-
engine/blob/master/server/src-lib/Hasura/Server/App.hs)

Not sure how people enjoy FP.

~~~
lpage
Haskell and ML languages look like symbol soup when you're used to
mainstream/C style languages. Once you understand some of the basic syntax
they become quite readable and easy to reason about as a series of
transformations.

Step one, which applies to most functional languages, is knowing how to read a
Hindley-Milner type signature [1].

Armed with that you can often gain a surface level understanding of what some
function does - and what the program does in aggregate - just by reading type
signatures and seeing what calls what.

Hoogle [2] is a good resource for figuring out Haskell syntax, like what's up
with all of those $ signs.

Converting OCaml to ReasonML syntax with sketch or reason-tools [3] makes
OCaml less foreign.

With basic knowledge of Haskell and ML syntax you'll find many FP languages
readable.

[1] [https://drboolean.gitbooks.io/mostly-adequate-guide-
old/cont...](https://drboolean.gitbooks.io/mostly-adequate-guide-
old/content/ch7.html#tales-from-the-cryptic)

[2] [https://hoogle.haskell.org/](https://hoogle.haskell.org/)

[3] [https://reasonml.github.io/docs/en/extra-
goodies](https://reasonml.github.io/docs/en/extra-goodies)

~~~
jakear
“Quite readable” is a stretch... I know of no other language that needs a
hoogle equivalent.

That being said, my experience with Haskell is limited to working on some
meta-meta-programming research in college, where every file used at minimum 3
different GHC extensions. So I may be biased.

~~~
mlevental
>at minimum 3 different GHC extensions. So I may be biased.

are you kidding? most files i see have all of these

[http://dev.stephendiehl.com/hask/#the-
benign](http://dev.stephendiehl.com/hask/#the-benign)

~~~
jakear
Again, I don’t have a clue what I’m talking about. But I seem to remember
seeing a lot of those “dangerous” ones. If that’s just the state of Haskell,
my bad.

------
londons_explore
> we’ve currently fallen back to interval based polling to refetch queries.

This fancy "subscribe to events when the results of this query change" system
boils down to just polling the database...

~~~
josephg
Yep. I’d love to see an implementation of this in the wild built on top of
logical decoding:

[https://www.postgresql.org/docs/9.6/logicaldecoding-
explanat...](https://www.postgresql.org/docs/9.6/logicaldecoding-
explanation.html)

~~~
Liron
Me too. Here are my thoughts on the current state of "realtime databases":
[https://hackernoon.com/why-eve-will-be-perfect-for-
realtime-...](https://hackernoon.com/why-eve-will-be-perfect-for-realtime-
apps-92b965b80ad)

------
themccallister
I’ve been using Hasura for around 3 months now. It has changed the velocity
and way I approach our projects. I have a small team that works as a startup
inside of a larger organization and this was crucial to delivering some key
projects... plus very little code. The event trigger really help reduce our
costs as well, using the Serverless.com framework with Lambda Go has been a
huge success in our organization!

Thank you to the Hasura team for all you do and in turn allow me and my team
to do!

------
thegoleffect
When I first saw Hasura, it was AGPL licensed which makes it unusable in a lot
of use cases so we passed on it. Looks like they converted it to Apache
license two months ago. I suppose it is worth another look, I'm glad they made
the change.

------
dmitryminkovsky
Hasura is such a promising application platform. I tried it a few months ago
and was really pleased. In a way it kind of felt like RethinkDB, except
GraphQL and Postgres. Was very easy to turn on with Docker Compose and play
around. Looking forward to 1.0!

~~~
z3t4
I guessed it would use the pub/sub capabilities of postgres. But it seems from
the readme that it use polling, although using the same query which allows
cache optimizations. The naive implementation can often be surprisingly
effective.

~~~
tango12
Yep. We experimented around a lot with pub/sub (listen/notify, wal-lr). But it
was getting hard to scale and heavy/spiky write loads needed careful thought.
This seemed like the right way to get started esp. given that Hasura can get
the entire sql query including authz, before moving to tackling that problem
more effectively[1].

For folks that want to play around with wal and listen/notify:

[https://github.com/hasura/pgdeltastream](https://github.com/hasura/pgdeltastream)
[https://github.com/hasura/skor](https://github.com/hasura/skor)

[1] [https://github.com/hasura/graphql-
engine/blob/master/archite...](https://github.com/hasura/graphql-
engine/blob/master/architecture/live-queries.md#when-do-we-refetch)

------
rishav_sharan
As an Indian engineer, Hasura is one the few indian startups that i really am
proud of. They are innovative, distinct, product focused and super young. The
only other company which made me feel like this was Notion Ink, but those guys
seem to have faded away.

I wish Hasura the best and would be cheering for them from the sidelines

------
gunnarmorling
I'm impressed by the numbers, but I'm having some doubts about the general
idea of exposing your database via GraphQL.

It seems to expose your internal data structures, and any change to those will
immediatly impact that public GraphQL API. Also it seems that this approach
would only work well for applications without any business logic, or with
business logic solely implemented in the database. At least the applications
I've been working on wouldn't fall into this category.

But in any case, impressive work and demos!

~~~
aleksei
You aren't limited to a single database though, so you could stream your main
database to a public facing one with a more stable structure.

But you'd have to handle changes through another API, or at least route them
differently, and then worry about consistency so probably not ideal for all
business cases.

~~~
tango12
True!

We’ve added event triggers to Hasura to kind of support this pattern via
Hasura itself. So you can create “action” tables that basically have a log of
request data (a mutation inserts an action). Hasura will then call an event
handler which can run with the action data, user session information, related
data in case there are any relationships etc. This handler can then go update
the tables that can be queried from.

Ofcourse, if the eventing system is in-order / exactly once quite a few use-
cases become feasible ;)

------
agrippanux
Hasura is fantastic. I am using it for several projects atm. Coming from
Prisma it's refreshingly easy to use and well thought out. The security/ACL
model is esp. nice and easy to grok.

------
pier25
We evaluated Hasura and the only problem we found is the integration with our
own authorization system. Otherwise Hasura is quite frankly amazing.

Our auth system has many roles per user, then each role has its permissions,
pretty common setup. The problem is that Hasura expects a single default role
per request to then evaluate against its permissions. The Hasura team has been
looking into accepting an array of roles and merging permissions and whatnot,
but AFAIK this hasn't been solved.

If you start with Hasura from scratch this is not really a problem, it
happened to us because we had to figure out how to integrate Hasura with our
current permissions.

~~~
contrasti
> the only problem we found is the integration with our own authorization
> system

But integration with external authorization systems is an extremely basic
requirement for every piece of software, certainly on the top 5 on the
requirements list for every serious solution - I can not understand how did
you forget about this?

~~~
sk5t
I haven't found this ("integration with external authorization systems is an
extremely basic requirement") to be the case at all; most solutions ship with
tightly-coupled access control mechanisms, and it is an exercise for the user
to figure out how to wield them effectively.

In Hasura's case, it looks like the flow of data to the authorization bits
shows a simple/naive approach, although it also appears this is something
Hasura are working to correct. Even as it stands, moderately complex authz
logic could be implemented via x-hasura headers or in-database associations.

------
lubesGordi
I'm curious about the underlying push tech. Is Hasura using firebase to handle
that? Then firebase relies on different push services depending on the client,
like Apple's push service for iOS and http/2's push for web (which is
supported by what percentage of browsers)?

~~~
pier25
Firebase? No, not at all. It's all Hasura and Postgres.

~~~
lubesGordi
What I'm getting at is what is the underlying tech handling push to clients?
Reading further I see that the graphql libraries handle subscriptions via
websockets. But elsewhere in this github I was seeing firebase. So it's a
little confusing. Websockets are pretty ubiquitous now but but from what I
understand maintain a tcp connection to the server. That has scalability
implications.

~~~
tango12
I think you were probably looking at docs/sample-code for a firebase auth
integration.

Hasura GraphQL subscriptions are over websockets. This post is a benchmark
that should address those scalability concerns. It’s about getting to 1M
concurrent active websocket connections.

------
StreamBright
Nice scaling numbers. Can you use Hasura without Kubernetes or Docker?

~~~
tango12
You can compile the binary yourself, but otherwise docker is the way to go.

What’re you looking at?

------
jazoom
I know there's an issue about this but it doesn't really indicate Hasura's
intentions regarding it:

Is supporting CockroachDB something Hasura is actively working on?

~~~
tango12
It’s on the cards!

Don’t want to spread ourselves too thin too early. Also working with the
yugabyte folks closely to have Hasura supported with their 2.0 release.

[https://twitter.com/karthikr/status/1128003465370693636?s=21](https://twitter.com/karthikr/status/1128003465370693636?s=21)

~~~
jazoom
> Don’t want to spread ourselves too thin too early.

I certainly understand that, but I wonder since CockroachDB is essentially
Postgres, if it wouldn't be easier to ensure compatibility from the start. It
seems like it'd be easier to just avoid using special Postgres features that
aren't in CockroachDB than to go back later and re-write all your code that
relies on those features.

~~~
tango12
Definitely. That’s why cockroach & yugabyte that speak Postgres but are black
magic underneath are a great target. However, it’s not fully compatible quite
yet
[https://github.com/cockroachdb/cockroach/issues?q=is%3Aopen+...](https://github.com/cockroachdb/cockroach/issues?q=is%3Aopen+is%3Aissue+label%3AA-
sql-pgcompat)

Citus / timescale / pipelinedb are much easier and we already support
timescale and pipelinedb well. Citus with native support for their
distribution columns is going to happen soon too! These are packaged as
Postgres extensions which is definitely first priority for us before moving to
databases that speak Postgres and eventually to other databases as well. :)

~~~
jazoom
> it’s not fully compatible quite yet
> [https://github.com/cockroachdb/cockroach/issues?q=is%3Aopen+...](https://github.com/cockroachdb/cockroach/issues?q=is%3Aopen+..).

So you're waiting for CockroachDB to implement features before you suppoort
it?

------
machbio
Write Ahead Logs would have been amazing - hate that AWS RDS does not support
WAL - could have saved so much time for some of the applications we build.

~~~
gunnarmorling
WDYM by "RDS does not support WAL"? I wouldn't know how Postgres works without
them, and they are accessible using logical decoding. wal2json is supported on
RDS and used by tools such as Debezium for streaming data changes into Apache
Kafka (Disclaimer: I work on Debezium).

~~~
machbio
Thank you, had missed this announcement.. I did check Debezium a while back,
found it appealing

------
he0001
How does it handle transactions and locking? It seem to support multiple
databases, does it handle distributed transactions?

------
rj5
I can’t wait to try this out!

