
Datomic Cloud - grzm
http://blog.datomic.com/2018/01/datomic-cloud.html
======
danpeddle
There's a few people asking about how to get something up quickly to play with
- this was a problem for me, and it isn't really well explained, but for those
interested, Alexander Kiel's datomic free docker image has worked well for me
- [https://hub.docker.com/r/akiel/datomic-
free/](https://hub.docker.com/r/akiel/datomic-free/) \- and an example of it
being used with docker compose is here:
[https://github.com/dazld/urlomic/blob/cc3d1218218b5f08751dfa...](https://github.com/dazld/urlomic/blob/cc3d1218218b5f08751dfad41eabe68ceabeed7d/docker-
compose.yml) (on a wip branch for some playground thing). I'm far from a pro,
but hope this helps someone interested in this tech.

~~~
stuarthalloway
Datomic Cloud makes it much easier to get started on AWS. It takes only a few
minutes to sign up, see
[https://www.datomic.com/videos.html](https://www.datomic.com/videos.html) for
a walkthrough.

~~~
danpeddle
For sure! looking forward to becoming a customer. I guess the journey I went
through was missing some intermediate step between playing at the repl with an
in memory db (which I believe you’ve mentioned before as being a good way to
get hands dirty from zero), to having something in dev which was backed by
storage, to a toy service, to production.

Jumping into AWS, at least in my work environment, is a big jump requiring
lots of supporting infrastructure etc. We’ll get there if it works out, but
the dockerized setup is really helping in learning and evaluation.

Officially supported ways of doing this is something i’m sure you’ve
considered before, it would be nice if cognitect could reconsider this.

~~~
stuarthalloway
You don't have to be very far "in" AWS to try Cloud. The bastion server will
let you connect from anywhere, so the idea is that the only infrastructure you
need is a laptop and an internet connection.

That said, we understand the value of a local development mode and that is
possible in the future.

------
gregwebs
This is a big change in pricing approach. Previously the licensing was
$5k/year and still is for on-prem. Although you could run datomic for free for
1 year and longer as long as you never upgraded it.
[https://www.datomic.com/get-datomic.html](https://www.datomic.com/get-
datomic.html)

Now they are offering an approach where one can at least start at $1/day
(Every time you scale out this goes up? What is the cost of support?).

The cheapest way to get started is probably still to run what is now called
"on-prem AWS" (for up to 1 year), which there is already cloudformation for.
[https://docs.datomic.com/on-prem/aws.html](https://docs.datomic.com/on-
prem/aws.html)

~~~
stuarthalloway
Cloud is definitely the preferred way to get started if you are on AWS. When
running the On-Prem version you also have to manage (and pay for) storage.
Cloud also puts less load on DynamoDB, which can dominate your spend.

------
bontaq
I'm not sure it's still true, but awhile ago their EULA banned the publication
of performance statistics. Be very careful using this and do your own
performance testing.

~~~
spinlock
My experience with Datomic was that it is not appropriate for a system that
has performance requirements. We developed a system using redis while the data
team was developing the data service on Datomic. When we tried to integrate,
the whole system slowed to a crawl and we needed to do a rewrite.

My advice is to develop with Datomic from the start and not separate it out
into a service. Any other database will be so much faster that you won't know
how bad your performance is until it's too late.

Another piece of advice would be to seriously consider if you need
immutability in your database. If it's not a hard requirement, I would not use
Datomic.

~~~
lgrapenthin
Having built a variety of products with Datomic and performance requirements I
can only disagree. Especially queries are insanely fast compared to other
databases.

I don't know what your data team was doing, but they must have done it wrong.

My advice is to develop without a data team from the start ;)

~~~
qaq
would you give an example?

~~~
lgrapenthin
The main advantage is that you can do time-consistent queries against any past
database state across the entire database without any read locks. This frees
resources and eliminates the necessity of timestamp tables.

Then there is Datomics caching model in combination with its local data based
query engine. The vast majority of critical queries only read from memory.
Data that is fetched doesn't block other consumers.

~~~
qaq
What RDBMS are you comparing to pretty much everything is MVCC so writers
don't block readers and readers do not block writers mature RDBMS also are
fairly highly optimized written in C with critical paths hand written in
assembly so again would be interesting to learn for what workload Datomic is
significantly faster. The biggest performance penalty in a modern RDBMS is GC
of snapshots so if you need a design that is append only it will make modern
RDBMS more performant.

~~~
lgrapenthin
I don't care so much about optimized tight loops written in assembly, rather
about the ability to scale nowadays. Datomic uses databases like you describe
e.g. Postgres as its storage. So it would be foolish to say that it beats
their performance on a bare bones level. Instead, Datomics architecture and
information model make it much easier and significantly reduce the overhead to
design and implement applications that provide insane performance. I won't
argue that you can hand rewrite every Datomic application in its underlying
storage database and get more performance out of it if you do your caching and
coordination right. It will take you much longer though (I'd guess a tenfold
at least), likely have some very difficult to find bugs, and the result won't
be as easy to extend. With Datomic, I get memory speed performance out of the
box for the heavy hitters and so much more that it take would take some very
uncommon requirements for me to choose something else nowadays.

~~~
qaq
Well I would buy the developer productivity argument but most applications
have a mix of reporting requirements that are generally extremely hard to
implement on anything "distributed". Also in a distributed system you are
either running some consensus algorithm (paxos, RAFT etc) that will def. not
be "insane performance" or you it will have issues with consistency.

~~~
lgrapenthin
Datomic uses distributed storage, writes are coordinated by a single instance
(transactor). Reads don't block writers and immutability allows to query
consistent snapshots. Does that address your concern?

------
nickik
I love Datomic, it solves so many problems that I face daily and you end up
building a sort-off datomic on top of your SQL DB anyway.

I would have prefered a Datomic for Kubernetes to get this cross-platform and
allow it to be hosted on different and private clouds because of my job I can
not use AWS. I can't use Kubernetes either but there is at future possiblity
that it would be possible.

The downside would probebly be that it would offer less integration as the
ecosystem is not so far along as AWS. Still an easly containerised version for
devlopment would be really nice.

Thanks to the Datomic team.

------
fulafel
Great milestone, congratulations to the Datomic team!

Seems the developer forums
([https://forum.datomic.com/](https://forum.datomic.com/)) are a good place to
visit for practicalities after the web site.

~~~
ekzy
The forum is fairly recent and not everyone is checking it... You could get
help on the clojurians slack in the #datomic channel

------
pmarreck
Are there any open-source immutable databases yet? Or would the hegemony of
SQL semantics (which seem quite tied to mutability) somehow disincentivize
such an undertaking?

~~~
lvh
FSVO "database": datascript is in-memory, similar to a slimmed down version of
datomic. I think it's fair to say that datascript:datomic :: sqlite:postgres
or something. It's in-memory, but there are third-party extensions using
hitchiker trees that make it persistent.

There's also Mozilla's mentat and its predecessor atomish which talk to SQLite
on the backend.

So, sorta?

~~~
pmarreck
Did not know about Datascript and Mentat. Thanks!

------
datahead
For anyone interested in background on Datomic, these are fun and informative
resources.

The talk where I realized Rich is a data head too. Introducing his design
goals and life experiences with db(s).
[https://www.infoq.com/presentations/The-Design-of-
Datomic](https://www.infoq.com/presentations/The-Design-of-Datomic)

"Clojure for the Brave and True" author, Dan Higginbotham, on key themes and
overview. [http://www.flyingmachinestudios.com/programming/datomic-
for-...](http://www.flyingmachinestudios.com/programming/datomic-for-five-
year-olds/)

Congrats to the Datomic team and Cognitect. I hope this move to cloud opens
Datomic's design principals and ideas to a wider user base.

------
dh-g
This looks great, I really like datomic but its licensing is preventative for
small projects. Hope this helps it get more love and community attention.

~~~
fulafel
Can you expand on this, what do you find wanting with the various choices?
"Small project" can mean very different things between 0 and 10M$ budgets, but
there seem to be a lot of options reaching to the low end, even the free
Starter edition sounds pretty capable.

~~~
gonewest
Suppose you have an open source or hobby project.

The Free Edition has limitations that impact how you would write code -- it
supports only the "peer" API and not the "client" API, and is restricted in
terms of backend storage choices and number of peers. So while it's very
likely true there's perfect API compatibility when you want to step up to a
paid edition, it makes you design to the artificial limitations they've used
to set apart "free" from "paid".

Indeed, it's also not the recommended way to learn. The Datomic team says: "If
you are trying Datomic for the first time, we recommend that you begin with a
client library." In other words, the Free Edition doesn't include the client
library that they recommend you try first.

If you step up to the Starter Edition you have full functionality and free
updates __but only for the first year __. Beyond that, you start paying.
Either you run in Cloud (as low as $1 /day) or you purchase the on-prem Pro
Edition ($5000/year).

Don't get me wrong. Cognitect has the right to charge whatever they like, and
bundle features however they like. It's their intellectual property and they
made the investment to create it.

But if you're building a hobby project then even $1/day may be more than you
want to spend, and/or more than you want to impose on outside contributors to
your codebase.

~~~
fulafel
Seems it's only the free updates that are limited to 1 year with Starter - you
can keep using it forever. I wonder how/if they limit you getting a new
Starter the next year? Anyone know the details?

Even if this drives you to the $1/day edition, that should cover a lot of
small projects. Especially as you don't have to keep it running 24/7 during
development.

------
reilly3000
I’ve been in love with the idea of Datomic for a few years now but haven’t had
a project to really dig into it with. The AMI makes it irresistible to try...
but I’m really not clear on how the client and peer libraries work outside of
the JVM. Do they? It seems REST support exists with some client libs for js,
Python, etc but they explicitly state in multiple places that it is defacto
deprecated and won’t receive further development. I highly doubt given many of
Rich’s statements that they would ever break the API, but I’m trying to push a
new app towards serverless, and I’d rather not have to mess around JVM keep-
warm hacks. A CLJS library that wasn’t orphaned 4 years ago and not based on
the REST api would be enough to push me over the edge.

------
kgwxd
I really wish they'd open source Datomic and just make money on hosting. I
can't bring myself to spend anymore of my life learning proprietary software
in such a foundational domain.

~~~
dj-wonk
I'd be interested to see if people would candidly unpack their rationale a bit
more in this area.

Before I get into that part, forgive me, but I must admit that I've begun to
tune out when I hear the "make Datomic open source" commentary. Still, this
particular line of commentary returns from time to time, so I'll weigh in.

This reminds me of "meta stories" that take over the original story. On Hacker
News, I'd much rather hear about technical commentary, lessons learned, or
interesting domains and applications used in Datomic projects.

I tend to be less interested in hearing armchair quarterbacking around what
business model would work better for Datomic, particularly when the arguments
seem:

1\. largely motivated by self-interest. Open source is often perceived as
"free" to software developers. It is relatively easy to say "I would rather
not pay for this software, why isn't this open source?" Of course, open source
is not necessarily really free, due to integration and maintenance costs. In
the cases where open source projects are abandoned, projects face substantial
risk and transition costs.

2\. not framed around the long-term interests of Datomic (at least the
arguments rarely seem to make suggestions from the perspective of Cognitect,
which invests in the product and makes income from sales)

It is easy to say "make it open source". It is harder for a company to find a
business model that works. It seems that Datomic's business model is working.
There is a free tier and paid tiers.

I know that I cannot properly summarize all perspectives with the ideal amount
of nuance. Perhaps some people think it is really in Datomic's interest to be
open source.

Nevertheless, it seems to me that many arguments people make are somewhat
unexamined. Let's go a level deeper.

May I ask this: How many of Amazon Web Service's offerings are based on open
source software?

I ask that question because there are four follow-up points I would like to
make:

1\. People use AWS quite extensively.

2\. AWS is based on closed source software.

3\. I don't think it is a coincidence that AWS is so successful.

4\. It seems totally reasonable (and arguably the smartest thing to do) for
Datomic to stay with a closed source model.

Also, I have no affiliation with Datomic, but I have used it on projects.

~~~
j_s
> 4\. It seems totally reasonable (and arguably the smartest thing to do) for
> Datomic to stay with a closed source model.

It is 100% understandable / practical / reasonable, yes. It is the same to
recognize using Datomic technology as a core part of a technology stack
involves some degree of additional risk, which grows as dependencies increase
on its unique capabilities.

~~~
nightski
All dependencies carry risk. Open source projects are abandoned all the time.
What do you intend to take over maintenance of your database software if it is
abandoned? Highly unlikely.

~~~
freshhawk
> What do you intend to take over maintenance of your database software if it
> is abandoned?

Yeah. Unlikely I'd be adding any additional real features or anything but why
not? It isn't magic. I'd also expect a lot of other people doing the same and
we can share the maintenance load.

There is also a lot of functionality that is obviously backed by very nice
code a lot of people would like to see and potentially use other places. But
knowing you can do necessary maintenance and tweaks, even if it is while you
rewrite your whole db layer to work with another product rather than a
permanent thing, is a serious benefit.

~~~
chickenfries
Right, if open source is unmaintained you can still use it at the very least.
And if you can't fix it yourself, you can pay someone else to.

------
auggierose
How do things like cloud databases work with respect to the GDPR?

~~~
grzm
Looks like that's addressed:

> _" The regulation applies if the data controller (an organization that
> collects data from EU residents) or processor (an organization that
> processes data on behalf of data controller e.g. cloud service providers) or
> the data subject (person) is based in the EU."_

[https://en.wikipedia.org/wiki/General_Data_Protection_Regula...](https://en.wikipedia.org/wiki/General_Data_Protection_Regulation#Scope)

(However, this is off-topic for this submission specifically.)

~~~
auggierose
> "Furthermore the regulation also applies to organizations based outside the
> European Union if they collect or process personal data of EU residents."

It's really not clear to me how a company like datomic would cope with that in
any practical way. Are they supposed to know what kind of data their clients
store??

~~~
NoNotTheDuo
I attended a GDPR session at Re:Invent this year. Basically, there's two types
of organizations: data controllers and processors. Datatomic, AWS, Azure, etc.
would all fall into the processors category. Amazon.com, however, would fall
into the data controller category.

From a processor perspective, they have their own requirements/regulations:
[https://aws.amazon.com/blogs/security/aws-and-the-general-
da...](https://aws.amazon.com/blogs/security/aws-and-the-general-data-
protection-regulation/)

See also: [https://aws.amazon.com/compliance/eu-data-
protection/](https://aws.amazon.com/compliance/eu-data-protection/)

AWS/Datomic doesn't need to know what kind of data their customers store -
it's up to the customer to be compliant with their part of GDPR.

------
dominotw
Is there a way to play with this without using a credit card?

~~~
bpicolo
[https://www.datomic.com/get-datomic.html](https://www.datomic.com/get-
datomic.html) starter version

[https://github.com/tonsky/datascript](https://github.com/tonsky/datascript)
Datascript is also neat, it's an in-browser DB with datalog query syntax.
There's some neat stuff possible with om.next + datascript:
[https://github.com/omcljs/om/wiki/DataScript-Integration-
Tut...](https://github.com/omcljs/om/wiki/DataScript-Integration-Tutorial)

~~~
dominotw
Thank you.

------
danscan
This is awesome! Whenever I'm working with MySQL I'm usually daydreaming about
using Datomic instead. I'd love to see more adoption in the community.

------
steelbird
Is $1 a day a good price?

~~~
andrewchambers
If you have a product which makes $2 a day using it then it is.

------
puredanger
This is the mobile link, better link is
[http://blog.datomic.com/2018/01/datomic-
cloud.html](http://blog.datomic.com/2018/01/datomic-cloud.html)

~~~
grzm
Thanks for pointing that out. I can no longer edit the link, so I've emailed
the mods to ask them to do so.

Edit to add: And they have!

