I've built a career moving data out of flat files into RDBMS in order to get systems working properly (and quickly). I couldn't believe the hoops people jumped through just because of their dislike (or ignorance) of relational databases.
This data base model is perfect for many applications, and can easily be mixed with other data storage technologies (if you know what you're doing.)
Don't like the capabilities of your RDMBS? Then add what you need to you app. Indexing, transactions, stored procedures, audit, security, whatever. The code doesn't care where it's at. It still runs.
Not pretty enough for you? Then become a mathematician or fashion designer. Otherwise, shut up and code.
No it is not stupid. RDBMS are old thinking, and have not inovated in decades. They made sense when procedural programming was all the rage, now they just don't anymore.
My ideal database would just act like a big hash table. Where I can store an object and retrieve it at will, without having to learn any other language. (sql in this case), or use complicated ORMs, (Hibernate, or Django models). etc..
So, for some reason or the other, there hasn't been much inovation in this field. Maybe b/c the big players in databases like it as it is, and there is no incentive to inovate.
Just b/c you drank your koolaid a while ago, doesn't mean everybody else has. In the programmer's poing of view, right now databases are a necessary evil, but they could be much better.
- Give me a list of good (lifetime sales > $10,000) customers on the west coast (CA, OR, WA) who bought any product on the defective list during busy season (10/06, 11/06, 12/06) and haven't placed an order since our last email blast. We'll find out why.
- Give me a list of phone numbers (using our Caller ID) of people who called in the last 4 days, whose call wasn't answered and had never called us before. We'll call them back.
- Give me a list of our slowest moving (10%) products that haven't been ordered in the last 4 months that are sitting in prime space (Location beginning with 1,2, or 3) in the warehouse. We'll move them to pallets to make room for new stuff.
- Give me a list of all products returned from Territory 7 in the last 3 months with a Problem Code related to fit or size. We want to make sure their description is correct on the website.
I could go on (and on and on)...
Do any of these with your "big hash table" in less than 1 second and earn a seat at the table with the big boys.
Just because something is old doesn't mean it's obsolete. Maybe because it works so well.
And just because someone understands this doesn't mean they "drank kool-aid".
Nothing wrong with a little debate. But please come up with more creative methaphors.
Also RDBMS' are based on some pretty solid mathematics, unlike almost all other areas of software (the other famous one is the lambda calculus). Generally where mathematics and software intersect, you end up with some powerful stable systems that stand the test of time.
Thank you! I've bookmarked this reply, so that the next 20 times I see ardit33's argument made I can just link to it rather than pounding out my own long-winded reply or just beating my head against my desk until it doesn't hurt any more.
(The reply that I would have written if you didn't respond first would probably have emphasized the amount of code required rather than performance.)
I could have made any number of technical arguments. We'd be pissing at each other all day.
Instead, I cited a few things that concern those who pay the bills. As hackers engaging in good technical debate here, it's too easy to lose sight of those whom we serve.
It's frustrating how many smart people agree with ardit33's argument, though. I've even heard the same thing from spez. When you're mostly writing CRUD queries, you can get away with just treating the DB as a hash table, but it still means you're probably doing a lot of things inefficiently. The database needs of the company I work for now are just run-of-the-mill web app stuff, but still it's just another day at the office when I write an SQL query that could easily translate to 100 LOC worth of hash table operations.
Here's a real example from our forum code that I've used as a job interview question: each user is member of some set of groups (groups in the UNIX sense), and each forum requires membership in all of some set of groups in order to be allowed to post. Given a userid, return the set forums to which that user is allowed to post. You can do it with a single, concise SQL statement containing nested existential quantifiers, and the query plan it generates is very fast.
Let's say I have the entire structure in memory, interlinked with pointers. The solution in pseudocode:
allowed_forums = [] // empty set
for each group in user.groups
allowed_forums |= group.forums // union
That's all. I can probably write a compiler that translates this to SQL if necessary. Nice thing is that when I write this piece I don't care about how exactly data is stored and accessed.
Now let's compare it with your SQL solution. You see, mine so simple, it's not even an interview question :)
1. Your solution is incorrect. I said you need to be a member of all the groups associated with a forum, not one of them. How would your code handle a public forum (one with no groups associated with it)?
2. Suppose you have one million users. Then you're sucking up an awful lot of RAM just for this tiny piece of functionality.
3. Your pseudocode makes a lot of convenient assumptions about what functionality is already provided in standard libraries and in the accessor methods for user.groups and group.forums. Let's see it written in a real language.
4. You say you don't care about how data is stored and accessed, but given the right set of reasonable assumptions, I can conclude that the running time of your algorithm is O(n^3). The beauty of relational databases is that you usually really don't have to care. The query optimizer is pretty good at figuring it out for you.
EDIT: nostrademons, I saw that peer reply before you deleted it. That was just spookily similar :-)
allowed_forums = []
for each forum in forums
if forum.groups <= user.groups // set comparison, like in Pascal
allowed_forums |= forum // include an item
2. Let' say, I have a system that will do swapping and optimization for me.
3. My pseudocode is an assumption that it is possible to build some language/environment that would replace databases, only in theory so far.
4. Ok, maybe I'm going to use the "beauty of databases" in the implementation for my language. The goal is to eliminate SQL (and useless interview questions at the same time :)
2. There's a word for such a system. It's called a database :-).
3/4. Ok, sounds good. But like many others in this discussion, you're confounding SQL with the relational model. Your pseudocode above goes slightly outside the relational model because it's written in imperative style, but can be adapted to it easily enough just by replacing your for loop with set-builder notation and maybe a unary union operator. Sure, SQL has warts. It was originally designed as a language extension for COBOL for chrissakes. But the language you're inventing is just a prettier syntax for basically the same semantics.
But the language you're inventing is just a prettier syntax for basically the same semantics.
No offense, drfranke, but this reminded me of a conversation I had long-long ago, with a group of physicists. I was saying Fortran was going to die soon and their only argument was "see, you can do it in Fortran! you are just replacing one syntax with another!". Actually damn Fortran died a bit later than I expected, but those're just technical details :)
Ok, let's leave it at this: my YC app is for a web-based distributed RDBMS. It's going to speak SQL because that's what everyone is used to, but I've long entertained the idea of designing something nicer to work with and supporting it as an alternate frontend for the query compiler. If you have good ideas about how to do this, and think you're capable of implementing them, then please do send me a resume.
The query optimizer is pretty good at figuring it out for you.
This is not true. Let's get back and compare the intellectual effort required to solve your task in SQL and in my imaginary language. The difference in complexity is exactly the amount of effort by which you help the SQL query optimizer to do its work.
Have you heard about filters? That's what they exactly do, narrow your choices according to specific circumstances. You get the list of object that fits the exact pattern of a filter (or multiple filters), and you are done (ok. maybe in java you have to cast the object).
But that's it.
Tables and SQL are old concepts which haven't evolved thru time.
If SQL was so great, why in the heck there is so many ORM frameworks out there? Is it b/c people are stupid, or maybe b/c there is actually a need to just have a Object Oriented Database.
Yeah, list comprehensions are isomorphic to query languages. There's at least one database (Mnesia) and three database abstraction libraries (HaskellDB, SchemeQL, SQLAlchemy) that exploit this.
That does not mean that databases are useless. Your [(customer.name, customer.phone) for customer in customers for sale in sales if sale.customer_id = customer.customer_id and sale.product_id in defective_product_list and sale.date.month in (10, 11, 12) and customer.lifetime_sales > 10000 and customer.state in ('CA', 'WA', 'OR')] isn't going to help much if you have thousands of customers and millions of sales.
The examples I cited are real. (When you're in the trenches, you don't need to make them up.) The database has 1,692,438 Customers and 7,823,356 Sales Orders. All 4 questions had subsecond response. But not using a single SQL statement as you suggest. Using a combination of resources: indexes, stored procedures, common functions, and other code, applied exactly as needed. And a RDBMS.
Remember, this is a discussion of relational databases, not SQL. 2 different issues.
HaskellDB would be so awesome if it actually worked. Every time I've tried to use it, though, the backend has choked on some piece of the generated SQL.
When I tried it, I didn't even get to that point. It choked on installation; I couldn't get it working with MySQL, then I tried with PostGres and it couldn't find pg_config or something silly like that, then I gave up. (In hindsight, I was probably just missing the dev headers or something.)
That was early 2005, though. It may be better now.
"If SQL was so great, why in the heck there is so many ORM frameworks out there?"
Uh, isn't that an argument FOR the popularity of relational DBs? There would be no need for ORM tools if there weren't lots of R databases that programmers need to interface with.
To turn your rhetorical question around "Why haven't OO databases become popular if that's what so many people really need?"
I don't disagree, my comment was just trying to derail the chain of useless rhetorical questions and slow down the advocates on both sides of the fence.
Maybe b/c there haven't been any good attempts at it, and the major players (oracle) don't have incentives to build such systems.
Getting OOD Right, would be a huge task. Why do invest on it, when you are already making lots of money on the current offering?
How do these filters work efficiently on a HashTable? And most ORM frameworks I know come with a SQL-like query language. How else would you formulate complicated queries?
My solution to that: store representations of trees as simple paths like "/trunk/branch/leaf1/", "/trunk/branch/leaf2/", etc. And just rely on the application layer to parse it all out into a friendly IU. When you want a specific branch, just do "select * from table where signature like '/trunk/branch/%'
not if you cache your data structures intelligently... probably not a good solution for something like PHP or Rails where you lose your application state across requests
Or do you mean that using "like" is slow? i would think that it would not be as long as that column is indexed.
Perhaps I should have said querying recursive data structures. E.g. you can easily represent a family tree in a RDB, but it's not so easy to write a query to get all ancestors of a specific person.
Good point. It's also not the end of the world in many cases to use 2 queries (or even more if each one is blazing fast). Just check your data first to make sure you don't have O(n) where n can actually be 5,10,20+.
edw519: I'm not sure you got the point of my article. I am a programmer who happens to like relational databases. I have enough experience with RDBMSs (and the same half-baked alternatives you've run into) to at least have an opinion. My article was not criticizing RDBMSs or SQL, but rather giving my opinions on why so many programmers don't like or don't understand relational databases.
My anecdotal experience is that a lot of programmers bristle when they get into a technical area that actually has a mathematical basis or a demonstrably right way to solve a problem. Programmers are used to problems that have lots of possible solutions, none of them provably right or wrong. I've seen that attitude in the realms of databases, computer graphics, compilers, and algorithms.
A few years ago I worked with a programmer who actually set out to prove to me that bubble sort was more efficient than whatever Oracle was doing to sort rows. He was retrieving a lot of rows, unsorted, then sorting them with his own code. Even pulling out Knuth and trying to explain big-O notation to him had no effect -- that was just so much academic bullshit and opinion that didn't apply, according to him. Of course Oracle was able to sort the rows orders of magnitude faster than his PL/SQL bubble sort, and the <i>coup do grace</i> was delivered by a DBA who took the inefficient cursors out of the solution.
That kind of thing is played out every day between programmers, and when they run into something like relational theory their first instinct is all too often to dismiss it as old, inapplicable, ugly (see the other comments in this forum), rigid, etc. Those are all code for "I don't understand it and I can't be bothered to learn it." When a DBA who is charged with maintaining data integrity interferes with whatever bad idea the programmer has he or she is subjected to <i>ad hominem</i> attacks. Some DBAs no doubt deserve contempt but in my experience a lot of programmers can and should be learning from DBAs rather than sneering at them.
In the comments in this forum you'll see the "Relational databases can't work with tree-structured data" canard hauled out, as if this exact problem hadn't been addressed numerous times in the database literature (Joe Celko, Fabian Pascal, Chris Date, etc.). That's ignorance, and the older I get the more I suspect it's more willful than I'd like to think.
I'm not sure why you're not sure that I got the point of your article. I agree with pretty much all you have to say here.
What I think is "stupid" is not your article, but the fact that some programmers don't like relational databases (for whatever reason). I'll try to pick my words and make my point more carefully in the future.
I understand much of the theory behind these technologies, but, as a practitioner, I'm not in a position to base my actions solely upon theory. I HAVE to deliver. I wish I had a nickel for every benchmark I've conducted over the years. You may have noticed that I tend to avoid theoretical debate here; I'm more interested in what works.
And relational databases work very well, very often. Whether you like them or not.
I've built a career moving data out of flat files into RDBMS in order to get systems working properly (and quickly). I couldn't believe the hoops people jumped through just because of their dislike (or ignorance) of relational databases.
This data base model is perfect for many applications, and can easily be mixed with other data storage technologies (if you know what you're doing.)
Don't like the capabilities of your RDMBS? Then add what you need to you app. Indexing, transactions, stored procedures, audit, security, whatever. The code doesn't care where it's at. It still runs.
Not pretty enough for you? Then become a mathematician or fashion designer. Otherwise, shut up and code.