
Aquameta: Web development platform built in PostgreSQL - dustingetz
https://github.com/aquametalabs/aquameta
======
endlessvoid94
I think this is an awesome idea. I really like the Smalltalk approach of not
using files and instead representing the structure of a program purely in
memory. I also love the idea of drawing inspiration from spreadsheets and
databases instead of representing programs purely in lines of code.

I applaud this effort and can't wait to try it out!

~~~
jagged-chisel
Any long-lived executable sits in memory (notwithstanding paging, which the
database server is also subject to), and such an executable is free to
maintain any files it accesses in memory as well.

~~~
gnode
I think what was being referred to is the technique of defining a program, not
as source code files which are then compiled/run in an interpreter, but as an
in-memory VM environment (think REPLs). The memory-based environment can then
be saved to a file, similar to making a core dump.

This changes the structure of programs from a file/disk based structure, to
the potentially richer structure of the programming environment (e.g. graphs
of objects). Although it comes with disadvantages like losing the tool inter-
operability of files.

~~~
breck
I have a hard time seeing advantages of ditching files. I like this concept,
but you can have it both ways (come up with a graphical notation that you can
then manipulate with graph like tools), but at the end of the day you need a
source of truth, and the file metaphor (a named region of code) is hard to
beat (impossible to beat?). Even databases store things as files.

~~~
gnode
File systems are just one approach to storage. Databases can use file systems,
block devices, or object-based storage (such as distributed storage systems,
e.g. Ceph, Amazon S3). Traditional file systems are, by themselves, inadequate
for many use case, as they don't provide things like versioning.

The strength of the file system, is that much tooling already exists for it.

------
cryptonector
I like their meta, and semantics layers.

In the past I've built a scheme similar to their semantics layer, where a
bunch of additional metadata is associated with schema metadata via
PostgreSQL's COMMENT statement (which lets you associate a free-form text
comment with all sorts of schema elements). In that scheme we had JSON
COMMENTs and a set of views that ultimately generates a nice JSON
representation of a database's schema, including the JSON COMMENTs in the
right places, and _that_ gave us UI control that we could then use to generate
Admin-on-REST UIs from. And PostgREST can be used to get an HTTP JSON API for
free.

For event pub/sub, I've written an used an alternative (all-PlPgSQL-coded)
view materialization system that supports live-updating of materialized views
as well as recording deltas, which then can be combined with NOTIFYs. Unlike
Aquameta, the fact that NOTIFY requires no authorization, and its payload is
free-form text, I feel queasy about sending out too much information in
NOTIFYs -- instead I use them to drive queries for new deltas as recorded by
the delta recorder mentioned earlier in this paragraph. The component that
LISTENs for NOTIFYs then writes deltas to a file which is served with a
special HTTP server that supports hanging GETs of files -- "tailfhttpd" \--
and any HTTP client can then be used to tail these files.

Anyways, the Aquameta scheme is pretty good.

EDIT: I've been tempted to write an authorization-for-NOTIFY patch to PG... I
really don't like the idea that if I give someone direct access to the DB they
can NOTIFY anything they like on any channel.

~~~
erichanson
Dang some cool ideas in here.

Yeah trying to figure out how to annotate the schema was a big motivator for
inventing the meta identifier system. Thought about using COMMENT for
documentation, but once you have meta-ids, I thought it was cleaner to just
put schema annotations in a different table.

Yeah I think the LISTEN/NOTIFY system in PostgreSQL is a bit primitive. Still
working on that part of the project. We got NOTIFYs to propagate up through
nginx over a WebSocket and get them into web-world that way, but that section
of the project is still fairly immature.

~~~
cryptonector
What I like about using COMMENT with JSON content is that it makes it trivial
to extend the schema in powerful ways, and then Aquameta's meta can be
stateless views (and instead of triggers). That makes it much easier to
swallow meta.

~~~
erichanson
Oh, we actually split up meta into just the views or a additional extension
for the update triggers. Yeah the updatable views are a giant foot gun.
Essential in our case but not generally.

[https://github.com/aquametalabs/meta](https://github.com/aquametalabs/meta)

------
stagas
Tangential, from the introduction: "Centralized systems are BORING. The early
days of the web were honestly more exciting, more raw, more wild-wild-west. We
need to get back to that vibe." \- I adhere to that philosophy, anyone know of
more projects towards that direction?

~~~
erichanson
The long-term idea with Aquameta is that users run their own local database
and make connections directly to their friends/peers to basically "git pull"
new content as it appears via pub/sub. Then users would do the "browsing" on
localhost. I still have some work to do figuring out all that NAT piercing p2p
stuff, but some combination of WebRTC and headless chrome is showing some
promise.

~~~
rancor
My favorite NAT-busting mechanism is Tor hidden services, The onion network
provides the basic P2P overlay as well, so it keeps life simple. STOMP over
WebSockets is cool, and I believe can be implemented without using any Chrome
code these days.

~~~
erichanson
Hadn't seen STOMP before, cool. I think making Tor a core part of the
architecture is a little over-engineered in our case. The Internet is already
a p2p network damnit! I like WebRTC because they seem to solve just the NAT-
piercing problem specifically without a whole additional network layer.

~~~
rancor
I've worked with WebRTC NAT punching pretty extensively, and it is not the
solution to your problem. STUN and TURN both require details like proxies with
fixed IP addresses, specific DNS and SSL configuration, and so on. So for the
nodes to be independent, most users will need to set those services up.

So while the added network layer of Tor (or any overlay network really)
certainly adds a level of complexity, from the application standpoint it
actually simplifies matters for the following reason: Your application doesn't
need any longer to think about the _topology_ of the underlying IP network, as
this implicit detail dragged in by direct IP connectivity use is abstracted by
the overlay network. Instead, your application is able to interface with _any_
peer via (in the case of Tor) an HTTP proxy and a set of opaque base URLs.

IOW, you're going to need one or more additional daemons to make the P2P part
go, and Tor is legitimately the simplest thing available _today_ that
accomplishes this without ruling out other styles of overlay network from an
architectural point of view.

~~~
erichanson
Interesting.

I don't mind adding additional daemons to the system if it's essential. But,
Tor also (as I understand it, given that this is a ways outside my core
competency) adds a whole encryption and anonymization layer, which would be
undesirable overhead in at least some scenarios, which is why it seems too
heavy to be part of core. Maybe WebRTC isn't going to work reliably which
would be very disappointing. I'd like to hear more about your experiences and
could really use some help making a good decision here. Ping my email if
you're available!

------
erichanson
I'm going to Twitch stream the Aquameta install and give a little demo, and
answer questions folks might have. 5pm CDT.
[https://www.twitch.tv/events/j5MGrQ91TwSzRpo28wqGdw](https://www.twitch.tv/events/j5MGrQ91TwSzRpo28wqGdw)

~~~
breck
Very cool. Does that record? Would like to watch later.

~~~
thepaulstella
VOD is available - starts around 6min into the video.

[https://www.twitch.tv/videos/495999132](https://www.twitch.tv/videos/495999132)

~~~
erichanson
I cut out the five minutes of me trying to figure out Twitch so here's a
better link:

[https://www.twitch.tv/videos/496396478](https://www.twitch.tv/videos/496396478)

------
smt88
I guess this isn't that dissimilar from Git, except that when you snapshot
your environment, you get everything (code, IDE settings, issue tracking, and
persistent data) all in the same dump.

There's definitely something to this. I'm not sure that building the IDE on
top of it is practical (partly because web IDEs are still pretty limited and
partly because the scope is just enormous).

But if we had a way to version code that combines source, data, and issues,
I'm super interested. It just needs to pipe into existing tools rather than
recreating existing tools.

------
sheeshkebab
Call me old fashioned, but I prefer my web app spread across a dozen servers,
file systems, hundreds of virtualized processes, and few thousand folders, and
employing a small army of system enginneers (sres) and developers to maintain.

~~~
erichanson
Ha :)

------
msvan
This was announced on the Future of Coding Slack [1] not long ago. I encourage
people who are into things like structured/projectional editors, visual
programming and other wild re-imaginations of programming to join.

[1]:
[https://futureofcoding.org/community](https://futureofcoding.org/community)

------
mdemare
SQL on Rails, 2006

[https://m.youtube.com/watch?v=Es9-l1up3r8](https://m.youtube.com/watch?v=Es9-l1up3r8)

~~~
MrRadar
Unfortunately that upload is only in 240p so most of the jokes are hard to
read. I have a copy of the original version which I've uploaded here:
[https://2by2.info/sql_on_rails_screencast2_lq.mov](https://2by2.info/sql_on_rails_screencast2_lq.mov)
(VLC can play it).

~~~
degenerate
I uploaded your file to Streamable:
[https://streamable.com/o5g3f](https://streamable.com/o5g3f)

------
jagged-chisel
Experienced something like this early in my career when a co-worker discovered
PL/SQL - he campaigned for putting the entire web app into an Oracle database
with all the logic in stored procedures.

Experienced a slightly different take a few years later where Some Genius
wrote an interpreter in stored procedures in an off-brand RDBMS. Web pages
were a combination of HTML and his custom language, and these pages were
stored in the database.

I have strong negative opinions about putting a web development platform
entirely inside an RDBMS.

~~~
jacques_chester
> _he campaigned for putting the entire web app into an Oracle database with
> all the logic in stored procedures._

Oracle have already done this, it's called Application Express[0]. I used it
for my last job. In practice it was pretty good for fast prototyping and
iterating on database-backed apps. It took the schema as the source of truth
and would add useful interface features based on it (eg, dropdown lists
derived from foreign keys, converting some check constraints to javascript
validation code etc).

But on the downside it was essentially untestable and version control meant
dumping a massive file of autogenerated PL/SQL and checking it into git. For
anything more complicated than basic CRUD and reporting you'd wind up cracking
the hood to directly use PL/SQL and then skin it with APEX. The tradeoff being
that you lost some of the roundtrip niceties.

I wouldn't recommend it in general, but it fit ... _OK_ in its context. It was
included in the Oracle license, the org had experience with operating Oracle
and it allowed each of us in a small programming group to build little apps to
solve genuine business problems that were either too small to select OSS/COTS
or too specialised to find anything to select.

The best part of the job was getting rid of painful processes built around
sharing Excel and Word docs. I remember occasion when a user broke down in
tears because a (to me) trivial app meant that a stressful, high-pressure,
error-prone month-long process turned into a non-event. One of the best days
of my career and I partially owe it to a platform that I think is, in most
respects, a mistake.

[0] [https://apex.oracle.com/en/](https://apex.oracle.com/en/)

~~~
jagged-chisel
This was before an official release, late 1990s. I'm sure he'd gotten his
hands on either some documentation discussing the concept, or an early alpha
release of Oracle's ApEx.

~~~
jacques_chester
Oh very possibly.

Wikipedia reckons HTML DB (the original package) was released in 2004, but the
APEX website clear claims to being 19 years old, which would put it circa
2000.

But in the meantime if you're doing late-90s CGI with Perl, why not `SELECT
my_amazing_function()`?

~~~
jagged-chisel
Hey, buddy, we were cutting edge with the brand-spankin-new ASP! (Although, we
did have a Perl guy who insisted we'd be renaming all our .asp files to .aspl
because MS had 'opened up' the scripting engine to take whatever language
you'd want to write an engine for ... and a Perl engine existed at that
moment.)

------
52-6F-62
Well, just glancing at that source code: I do not know Postgres like I
thought.

~~~
t0mbstone
Well, to be fair, you can do all sorts of things with Postgres if you build
custom extensions for it

------
Terretta
_Aquameta has been the life project of Eric Hanson for close to 20 years off-
and-on. Functional prototypes have been developed in XML, RDF and MySQL, but
PostgreSQL is the first database discovered that has the functionality
necessary to achieve something close to practical, and huge advances in web
technology like WebRTC, ES6 modules, and more have shown some light at the end
of the tunnel._

 _Aquameta is an experimental project, still in early stages of development.
It is not suitable for production development and should not be used in an
untrusted or mission-critical environment._

Not really the basis for ’reference success stories’ approach to evaluation.
More, check out the repo and hack at Postgres, see where the hypothesis can be
made pragmatic.

------
xet7
Interviews of Aquameta:

[https://twit.tv/shows/floss-weekly/episodes/527](https://twit.tv/shows/floss-
weekly/episodes/527)

[https://twit.tv/shows/floss-weekly/episodes/449](https://twit.tv/shows/floss-
weekly/episodes/449)

------
dmix
Their argument from the introduction that software can slow down a business by
making changing/evolving the systems difficult and requiring programmers at
every step, is an interesting one:

[http://blog.aquameta.com/introducing-
aquameta/](http://blog.aquameta.com/introducing-aquameta/)

Obviously this is has been the panacea goal in business software forever and
there's a long trail of failed companies or projects who tried to do this.

IMO it's always going to require specialized knowledge of a certain level of
abstraction above the machine, it will never be as simple as pushing buttons
in a GUI. The question is making the languages/frameworks simpler and lowering
the bar.

In practice these sorts of things work great for simple scenarios but are very
brittle one you start to. Just like Excel spreadsheets used like databases it
will quickly turn into a hacky maze of things forced into places where it
shouldn't be.

I'd also never think 'PLpgSQL' when it comes to simplifying things. I'm
genuinely curious to see if they can pull it off... eliminating the file
system part is also an interesting idea.

~~~
pm90
The MO of SV startups seems to be to slap together something that has product
market fit while ignoring performance and scaling concerns, at first. Once you
have enough customers and market, you then raise funding and use that to hire
more experienced system folks who can make your system performant, efficient
and scalable.

~~~
dmix
I wasn’t talking about scaling performance wise. I mean scaling the problem
set beyond the initial easy ones.

------
zubairq
Is this comparable to Oracle Apex, which is a web development tool written
entirely in oracle PL/SQL?

~~~
dahdum
Used Apex when it first came out (as HTMLDB). I’ve never found anything that
compared to its speed in rolling out web based forms for internal use. At the
time, it was the best BI tool for ad hoc queries around. Nowadays I use
metabase, but I’ve often missed Apex speed and wished PG had it.

------
gwbas1c
Cool concept!

Has anyone built anything in Aquameta? Would like to see some success stories.

~~~
micburks
I worked on Aquameta for a few years and I'll say that having a build running
locally is a joy I've never encountered elsewhere in my career.

Every time I need a little app or something, instead of reaching for some SAAS
product, I would just build a quick prototype for myself. I could make a
prototype in a couple hours and over the course of a week or two I would
polish it up when I had a few extra minutes. Rather than using something built
for the masses, I could tweak the interface to my own taste and I owned all
the data.

------
pgt
Greenspun's tenth rule may need to be updated for IDEs:

"Any sufficiently complicated C or Fortran program contains an ad-hoc,
informally-specified, bug-ridden, slow implementation of half of a JavaScript
IDE." (previously Common Lisp)

------
e12e
> Technical goals of the project include: > Allow complete management of the
> database using only INSERT, UPDATE and DELETE commands (expose the DDL as
> DML)

There might be other reasons - but note that postgres actually supports
transactions and rollback of DDL - Oracle with full enterprise lisence has
something similar (there's undo/a "trashcan" style cache for schéma
modifications).

But in general (as far i understand; I have _not_ used this in anger) pg lets
you simply BEGIN drop table (...) ALTER table (...) (who's, something wrong)
ROLLBACK.

It's very neat, and not generally supported.

~~~
cryptonector
Yes, yes, but, it's nice to be able to treat schema as relational data itself.
Indeed, the information_schema and pg_catalog already let you do that, but the
information_schema is incomplete, and the pg_catalog is hard to use and
unstable (that is, each release can change the pg_catalog backwards-
incompatibly).

Besides letting you read schema via normal SELECTs on a well-designed meta-
schema, something like Aquameta can let you run DMLs as DDls too. That means
you can now generate and execute DMLs dynamically but without needing EXECUTE
-- you can just have a bunch of normal INSERT/UPDATE/DELETE statements
(including via CTEs) and generate schema from other data the same way you'd
generate data from data. I think that's a big plus. But then I've worked with
a database before that did this schema as data-in-a-meta-schema thing, and I
find it very comfortable.

For example, without a metaschema you have to use CREATE THING IF NOT EXISTS
then ALTER THING ... in order to apply schema changes. Whereas with something
like Aquameta you just INSERT INTO ... WHERE NOT EXISTS ... (or with ON
CONFLICT DO ...) and that's that. You can have one set of DDLs that create
schema, and the same set of DDLs also updates schema. That's wonderful.

~~~
e12e
I'm still not entirely clear on why this is a good thing. Is it a preference -
sort of hygienic unhygienic macros - for data structures?

Do you end up with old data in old schema - or is there some implicit
conversion? Is a UPDATE that drops a column equivalent to dropping a column -
and can you roll it back normally?

~~~
cryptonector
It's like functions that return functions: SQL that generates SQL. Normally,
in PG land, one generates code using SQL, using the format() function, then
one EXECUTEs it. That would still be the case for generated FUNCTION bodies,
naturally, but for everything else you could just use DDLs on a small and
well-tested metaschema that does this work for you.

Just as in Lisp you might rather not use eval but instead use macros, here
you'd prefer to use DDLs on a metaschema over DMLs. Hygiene is one of the
motivators.

And yes, you could DELETE from a view that represents columns instead of using
ALTER TABLE ... DROP COLUMN ... All the usual transactional semantics apply.
You can BEGIN, delete that row, and rollback, leaving the DB unchanged.

------
fooblitzky
"Under the hood, Aquameta is a "datafied" web stack, built entirely in
PostgreSQL. The structure of a typical web framework is represented in
Aquameta as big database schema with 6 postgreSQL schemas containing ~60
tables, ~50 views and ~90 stored procedures. Apps developed in Aquameta are
represented entirely as relational data, and all development, at an atomic
level, is some form of data manipulation."

This is triggering painful flashbacks to SharePoint development.

~~~
iamwil
What were the painful parts of SharePoint development?

~~~
no_wizard
What _were not_ the painful points of SharePoint development?

~~~
iamwil
I don't know. That's why I'm asking.

------
joewrong
reminds me of couchdb apps where you'd use document attachments to store
html/public files and serve them directly from couch.

[https://docs.couchdb.org/en/2.0.0/couchapp/](https://docs.couchdb.org/en/2.0.0/couchapp/)

------
miffy900
This is actually really similar to on-premise SharePoint development; list
data, content types, workflow definitions, front-end HTML, CSS, JS code are
just stored in the back end database.

------
slifin
Data driven applications are cool, I think I'd use Datomic over Postgres but
it didn't exist 20 years ago

------
auspex
Make sure you back up that database...

~~~
adipginting
Lol you have a great advice over there

------
jwhiz22
This greatly reminds me of my experience with AEM (Adobe Experience Manager).
Web-based IDE, version control via OSGI bundles, content repository via
Jackrabbit.

I did not enjoy my time working with it but it had some neat ideas and did do
some things well.

