

Big Data Needs To Think Bigger - joshbert
http://techcrunch.com/2011/03/20/big-data-needs-to-think-bigger/

======
forkandwait
I find these articles to be long on puff, short on content. The take home is:
big data is really important, and some people think their startups will make
buckets of money either storing/ analyzing it or selling software to process
it. I don't really need 800 words (and a bunch of unwanted javascript and
tracking cookies) to tell me that.

Here's what I would like: 12 technologies or techniques for handling big data,
and why, with examples. Or a tutorial on a complex analysis that merged 3
really big datasets and came up with something useful (I dont need Hadoop to
count words, thanks). Or a list of mathematical background every data
scientist must have. But I am afraid the people who read these articles don't
actually know "their asymptote from a hole in the graph", and that they don't
have examples of big successes (maybe not yet, I grant...)

My daily job is merging building permits and tax assessor parcels to get small
area population estimates. This seems like "big data" to me (though nothing
like, say, all the twitter messages or google searches or FB friend
connections); if someone could explain to me why -- and how, exactly -- I
should move away from my SQL queries on and string matches, I would be really
interested, but I all read off of this hype is BIG, BIG, BIG, AUTOMATICALLY
DISCOVER EVERYTHING, MAKE MONEY, BIG, BIG, BIG.

Just my little rant...

~~~
pjscott
Are your SQL queries problematically slow, or do you expect them to be in the
foreseeable future? If so, can this be fixed by throwing beefier hardware at
the problem?

Unless the answers to these are yes and no, respectively, then your data is
small enough. Now, there may be usability reasons for going with different
database technologies, but from the lack of panic in your post, it sounds like
you're doing fine.

(Incidentally, if you've got data problems that aren't being satisfied by SQL
queries, then you could probably get a really interesting discussion going if
you wrote about it.)

