
Ions: Develop cloud applications by deploying to a running Datomic cluster - simonpure
https://docs.datomic.com/cloud/ions/ions.html
======
vincentdm
Can someone explain the benefits of Datomic Ions? What problems is it supposed
to solve and how does it compare to existing solutions?

Also, what is the story regarding local development? (I also have this concern
about AWS Lambda; it seems you can only realistically run code in the actual
cloud environment)

We are investing heavily in the Clojure-ecosystem (but not yet Datomic). It
seems Cognitect is strongly headed into a direction which involves some kind
of holistic vision about a new stack, but they don't seem to explicitly
communicate this vision anywhere.

Edit: I asked the above question also on Clojure Reddit and got an interesting
response:
[https://www.reddit.com/r/Clojure/comments/8p3d5s/datomic_ion...](https://www.reddit.com/r/Clojure/comments/8p3d5s/datomic_ions_your_app_on_datomic_cloud/e09dz9e/)

~~~
jeremiahwv
My take is that one of Datomic's original value props was that it ran in your
Java app's process. That plus immutability plus cache-foo allowed you to write
your Java app as-if your entire db was not only "in-memory", but "in-your-
app's-process-memory." That is, um, "pretty dope".

Then they introduced Datomic Client, which is more of the "normal" db
paradigm. Your app includes the client lib, which connects to the running db
process "somewhere else". I don't know why they did this (easier to include?),
but it meant that the "db-in-your-app-process" value prop went away.

Initially when they deployed Datamic Cloud (a sort of managed Datamic service
deployed in AWS), they only supported the client model.

That is context for your question. The problem Ions solves is how to get that
"db-in-your-app-process" value prop while using Datamic Cloud. It seems to me
that it effectively reverses the original method. Original method was add the
DB as a lib into your app, Ions method is push your app into the db (which is
running in a configured-for-you cluster on AWS).

(It also allows you to not have to provision/configure servers to actually run
your app code in addition to the db servers, but that seems secondary to me.)

Note, that I'm not affiliated with Cognitect, and I've never used Datamic, or
Clojure in a production system. So have a salt shaker.

~~~
hellofunk
What is the diff between “in memory” and “in-your-app's-process-memory”?

~~~
derefr
An "in memory" DB is just a DB that can answer queries quickly (e.g. in O(1)
time.) You still speak to it over a network protocol (even if it's going over
the loopback interface), where everything gets serialized into long binary
strings and then deserialized again. You can maybe achieve zero-copy access
into the _packets_ the DB is sending you if the DB is communicating using
something like Capn Proto, but that's still two copies of the data: one in
your app process, and one in the DB process.

A database whose data is _in your process_ (like, say, BDB, or Erlang's ETS,
and _not_ like SQLite†) allows you to hold _pointers_ to data (usually either
"tuples" or "objects") that are in the DB, from your process, and treat them
as if they were just regular data sitting on the stack/heap. Depending on the
DB's persistence architecture, this can mean that you can even _mutate_ the
data in object form, and that mutation will be automatically persisted—because
what you're mutating is, after all, the DB's copy of the data. Unlike e.g.
Firebase, there's no secondary synchronization between "your local" copy of an
object and "the database's copy" of the object; your process is _holding_ the
database's copy of the object.

Or, if you like, you can think of an "in-process-memory" database like a
server in a distributed-RPC stack, ala CORBA or Windows COM. You ask the DB
for an object, it gives you an RPC handle to the object. Except, since you and
the COM server share memory, this handle doesn't need to do any marshalling or
IPC or locking; it's an optimized-path "direct" handle.

Another potential path to enlightenment: if you create a "flat-file database"
by just mmap(2)ing a file—and then you write structs into that file-backed
memory, and keep pointers to those structs—then you've got an "in-process-
memory database."

† SQLite _the library_ resides in your process's memory, and it can even
allocate tables that reside only in-memory (which is also "your process's
memory") but SQLite still communicates with your process through the SQL
protocol, as if it was on the other side of a socket to your process, with no
way to "get at" SQLite's internal representation of its data.

------
hellofunk
Man, am I ever a dummy! I look at the diagram on that page and it might as
well be Egyptian hieroglyphs to me. I'd have to study that thing one long time
to even understand what problem Ions is solving. I'm sure it's a real problem
with an interesting solution, but geez the modern Cloud world has really
gotten quite complex.

I feel particularly silly because I've been writing Clojure professionally for
many years.

~~~
baxtr
There are always two sides to this... I think it’s badly communicated. Someone
said (I don’t find the quote), that if you can’t explain a complex topic in
simple words, you haven’t understood it thoroughly.

~~~
VaedaStrike
If you know Clojure and AWS I really don’t think it is poorly communicated.
There’s a limit to that idiom of simplicity and understanding being
correlated.

This was a walk through of the mechanics. The TL;DR is if you are comfortable
enough in AWS and Clojure implementing your system in a scalable and simple
and wholistic way just got a bunch more accessibile.

~~~
zimablue
I've written a clojure app, used aws and lambda functions and skimming this I
really wasn't sure wtf they were talking about. Datomic... Something something
lambda functions, deployment. I know that they have an existing ami that uses
a cloudformation template to let you set up a datomic server on AWS, this is
something similar maybe? I think I could get it if I read in detail but this
is definitely poorly communicated, it reads like old Microsoft enterprise
sales copy.

------
samuell
We need more open source datalog implementations with disk based storage.

I have my hopes now on Mozilla's Mentat [1], but it seems to still have quite
some way to go, and also I wouldn't mind something similar implemented in Go
too ... At least something simple, allowing to start working with the datalog
paradigm for expressing logic and views.

[1] [https://github.com/mozilla/mentat](https://github.com/mozilla/mentat)

~~~
swlkr
Here’s a really early one based on persisting datascript

[https://github.com/replikativ/datahike](https://github.com/replikativ/datahike)

------
nickik
I really love Datomic and it solves so many problems that I usually have when
doing application development special in bank and financial context.

We are already using Clojure and I would love to introduce Datomic as well but
being a commercial database makes it pretty difficult.

This new setup is sounds cool but I would love something like that on a
Kubernetes stack. In the real world Amazon is a no-go in so many situation.

I must say I often found the setup needlessly complicated. There is not much
application for making setup and development easy when outside of AWS. No
official way to set up of the whole stack with Docker or Kubernetes for
example.

~~~
alter_eco123
> _We are already using Clojure and I would love to introduce Datomic as well
> but being a commercial database makes it pretty difficult._

I don't see why paying for Datomic would be a problem, but if the license is
as crazy as it used to be, that's a bit scary.

For example, the license at least used to forbid even saying "Datomic" aloud
in public.

Publishing benchmarks was also forbidden, which is a big red flag. If Datomic
performs well, wouldn't Cognitect want lots of benchmarks out there? If it
doesn't, they wouldn't!

> _I must say I often found the setup needlessly complicated. [..] No official
> way to set up of the whole stack with Docker or Kubernetes for example._

You don't find Kubernetes needlessly complicated? Do you _actually_ need
Docker or Kubernetes?

~~~
marcinzm
>You don't find Kubernetes needlessly complicated? Do you actually need Docker
or Kubernetes?

They make life easier in my experience especially if you need something more
than a simple self-enclosed application. For what I do we need Apache Spark,
Airflow (which in turn uses RabbitMQ and Celery if you want to scale out),
possibly JupyterHub (which also needs some way to spin up child servers), and
then the application code itself.

I've wired some of it up myself in the past and it's a massive pain in the
backside with lots of edge cases (Spark used to have a shitty massive python
script for cloud deployments which always was missing something). Deploying it
on Kubernetes with Docker on the other hand is rather simple. I basically
trade the complexity of Kubernetes/Docker for the complexity of custom
deployment/scaling/management code. In my experience, the code quality of the
former is going to be much much higher than the code that an internal DevOps
team outputs.

~~~
tedmiston
I've spent timing wiring up a similar setup with Airflow, and Celery on Kube,
Kafka too, and running Spark clusters and echo the same sentiment. I also find
Kubernetes easier/simpler compared to DC/OS, perhaps because it's a bit more
structured and opinionated.

------
bjconlan
It seems Rich (yes and the team at cognitect) might have jumped the shark on
this one. (Or this was released too early and the next conf. talk will clear
up what the problem is and what it is that this is attempting to solve).

Either way it's rare for Rich not to describe to problem a little more before
throwing a solution to it. Hopefully more will come to light in time.

The brief take away I see from this as a non AWS deployment based user is the
ability to go beyond the current limitations of only having access to core
library functions in the context of datomic. (I see that deployment of
applications also plays a part in this but not sure what that solves exactly)

------
VaedaStrike
This looks like a Godsend :)

The only qualm I have is it being only on AWS.

Other than that this enables just what I could use right about now!

~~~
mark_l_watson
DynamoDB is a preferred backend, and the us AWS only.

