

Databases - a grumble after a weekend project - hoodoof

I did a project over the weekend. I wanted a database where I could throw in JSON data, do some simple queries, get data back. You'd think that would be easy right? Install it, 5 minutes, install drivers, 5 minutes, write some code to throw in some JSON - 5 minutes, write some simple queries - 10 minutes. No way - it's not that easy.<p>Man, databases have a long way to go.<p>I knew not to use MongoDB because it takes an entire server and I was running on one single AWS micro instance.<p>I tried Amazon Dynamodb - simply not possible to throw a JSON structure into it. I dropped it.<p>I went back to Postgres - and spent hours doing configuration of users, tables, schema, and security, not writing code for my application. I deleted Postgres from my system. Why do SQL databases have massive layers of configuration and security on them - seems a relic of the past - why not let the application take care of all that.<p>I tried to think what other database might allow me to simply throw JSON data in and do SIMPLE queries to get it back.  Something without installation pain, configuration pain and usability pain.<p>In the end I tried RethinkDB.  It installed fast, the install documentation was perfect for an Ubuntu system, it seemed to work well, did not take over my machine and allowed my to throw JSON into it with no pain. However it was much more painful to do queries - Rethink is still improving in this area - even basic queries are painful. BUt the guys there are incredibly helpful and friendly and helped me to get some working queries built.<p>SO..... databases just aren't there yet.  Where is the database that simply allows me to throw in JSON and has a simple and intuitive query language that just works. One day someone will get that worked out. If rethinkDB can manage to get powerful and simple JSON queries implemented then maybe it will be The One.<p>Can anyone suggest a database that allows simple installation, simple data loading and simple and powerful queries? Something that does not require layers and layers of arcane configuration just to get going with?
======
mattparlane
I'm not sure where your idea about MongoDB taking "an entire server". It will
use as much memory as it has available to it, but of course you can run other
things alongside it.

I'm currently running PHP, Ruby, Memcache, Postfix, nginx, plus probably a
bunch of other stuff I'm forgetting, all on one server and it works fine.

MongoDB can be configured entirely on the command line and is as simple as
downloading a binary, unzipping it and running it.

~~~
hoodoof
Also I seem to recall major pain writing queries to access
substructures/embedded data within JSON when using Mongo. Unless that has
changed in the last 12 months?

~~~
mattparlane
It is true that it requires a shift to go from AND/OR SQL queries to doing the
same thing programatically. I'm not sure anything there has changed in the
last 12 months, at least not huge changes.

I've found it to be not a huge problem though, and of course MongoDB is
definitely one of the most flexible NoSQL databases in terms of querying.

------
dragonwriter
> I went back to Postgres - and spent hours doing configuration of users,
> tables, schema, and security, not writing code for my application. I deleted
> Postgres from my system. Why do SQL databases have massive layers of
> configuration and security on them - seems a relic of the past - why not let
> the application take care of all that.

"SQL databases" don't. Standalone multiuser, nondedicated database servers
(many of which happen to be SQL-based RDBMSs) do, because they are designed to
support direct user access (through general purpose client applications) and
multiple applications -- both of which need to authenticate against the
database -- from the same database server. So the application _can't_
reasonably handle security and configuration. There are SQL databases that
have less security and configuration, because they are embedded databases
rather than database servers (SQLite is a particularly well-known example.)

~~~
hoodoof
In Postgres for example there is just so much arcane configuration that needs
to be done before you can start writing code. All that extra guff should be an
add on option for "enterprise users". Why not just the database functionality
on its own, stripped of all the complex surrounding bits? Have you ever
experienced the pain of configuring pg_hba.conf? Or trying to work out the
right user access control to your database entities and then keep it all ok
when the database setup needs to be changed? Nightmare.

~~~
dragonwriter
> In Postgres for example there is just so much arcane configuration that
> needs to be done before you can start writing code.

Not really.

> All that extra guff should be an add on option for "enterprise users".

It may not be right for whatever toy project you are doing, but that doesn't
mean that it isn't right for the baseline of a product that does what Postgres
is intended to do.

> Why not just the database functionality on its own, stripped of all the
> complex surrounding bits?

The things you call "complex surrounding bits" are part of the core
functionality of a RDBMS server.

> ave you ever experienced the pain of configuring pg_hba.conf?

I've made changes to pg_hba.conf, though for simple development against a
default install I usually don't have to. I never found it painful.

> Or trying to work out the right user access control to your database
> entities and then keep it all ok when the database setup needs to be
> changed?

Never had any problem with that. If you don't have really complex security
needs -- and especially for the kind of development where its sensible to
consider application-managed security rather as an alternative to DB managed
security -- its straightforward. You can just create a user for the
application, and create a database in the cluster owned by that user, and bam,
you're done.

For more complex cases, sure, its more work, but then the work is inherent in
the more complex cases, not a product of the DB features.

------
kbenson
My first thought was that maybe we don't need things to "be there" or
"arrive", especially when they are complex and integral (in some respects) as
a choice of a database, or at least the data representation or the database.

Then I thought of SQLite, and what it's allowed, which is ubiquitous instance
databases for light usage and/or rapid prototyping. Maybe what we need is the
equivalent of SQLite for NoSQL databases. Does that exist yet?

~~~
hoodoof
How practical is sqlite for anything more than single user applications?

~~~
kbenson
Probably not very.

That said, for single user applications (such as ios/android apps) or rapid
prototyping, they may be very practical. Then again, prototyping is only as
useful as it is easy to trade out for the eventual solution, and if it has a
different query syntax than what you replace it with, you were probably better
off with the pain to set that up in the first place.

I could see a SQLite-alike JSON database doing really well in the phone app
space. Especially for those using a framework like phonegap that allows you
write in javascript. In fact, I think this is obvious enough that there
probably exist solutions for this, and I'm just not knowledgeable enough in
the subject to know of them.

~~~
jaredsohn
>prototyping is only as useful as it is easy to trade out for the eventual
solution

It is also useful if you find out from prototyping that the project is not
viable. Also, in general, if you need to iterate a lot using something that
iterates faster at the prototype stage would have extra benefit. (Although in
this particular case I think using SQLite only saves you initial configuration
time.)

~~~
kbenson
> It is also useful if you find out from prototyping that the project is not
> viable

True. I considered this when writing the reply, but decided to not obfuscate
my point. It's good that you brought it up and it got addressed though. :)

> Also, in general, if you need to iterate a lot using something that iterates
> faster at the prototype stage would have extra benefit

I'm not sure it's worth it if it's significantly different than the eventual
solution will be (and you're fairly certain it won't map). That can be as
simple (and horrible) as prototyping against NoSQL when you know production
will be SQL (or vice-versa), or as subtle as using a feature present in one
RDBMS which is buggy or non functional in a different RDBMS, which is the one
that will be used in production. The longer you code using the prototyping
data store, the more likely you are to inadvertently use a feature which will
cause you problems later in production.

Now, SQL is fairly standardized, and SQLite as a prototyping tool is unlikely
to cause a problem later if the production environment is a full RDBMS, but
you may end up not using a few features only available in the full RDBMS that
could have lead to a cleaner and/or more performant solution.

------
jlengrand
There is one thing I don't understand. Why do you want to directly throw JSON
at the database? Why not write a simple and minimal JSON to SQL layer?

You can probably do that really fast with ANTLR (<http://www.antlr.org/>) or
any equivalent.

------
memracom
More people should think of SQLITE when doing a simple project over the
weekend...

For this use case, you can use the BLOB datatype and just write a simple shim
with get and set methods that interfaces with the db.

------
jaredsohn
I have found that Meteor provides a very simple/fast way to deploy/use mongodb
if you want to build something quickly. (I just did so myself this past
weekend.)

------
balazskiss1988
You could also try out redis, it's a key-value store that is easy to use and
needs little to no configuration.

