Hacker News new | past | comments | ask | show | jobs | submit login

Not to step on anybody’s toes, but… I often suspect that there are lots of people who tried, and failed, to learn relational database techniques and gravitate toward schemaless solutions like Mongo just because they’re easier to understand. Maybe not everybody, but more than a few that I’ve interacted with.

They're superficially easy to understand, but end up moving the complications of concurrency, transactions, and statistical query planning up into the application. Those are much harder problems to solve correctly than just learning SQL and understanding the output of EXPLAIN.

It's a common ignorance pattern: hipsters reject complex and mature effective tools that require a learning investment (in this case a RDBMS and its fancy configuration options and SQL capabilities) because they don't know what they can do, and embrace inadequate familiar and/or shallow tools (in this case, misused fashionable and "schemaless" NoSQL databases) because they are confident that they can fill the gaps (in this case application code in familiar programming languages, to the extreme of "reinventing the wheel" with completely custom frameworks).

I was trapped in this mode of thinking for a few years when I was younger. I really regret falling for it, and try to help other people when I see them falling for it. I wish somebody had set me straight, so I'd not have spent those years searching for excuses to perpetuate my ignorance.

> hipsters reject complex and mature effective tools that require a learning investment

I think there is an element of that, but let me suggest another possibility: this is driven by modern (so-called “agile”) project management techniques. Every week or so, the programmers are called on the carpet to provide “estimates” for a series of a dozen or so vaguely-worded “tasks”, each of which does have clear business value but is pretty open-ended about acceptance criteria. These estimates can only be on the order of a few hours: if the estimate goes higher, the project manager insists that you “break it down” until the individual subtasks can be estimated on the order of a few hours. This “breaking down” process, likewise, is expected to only take an hour or so - if it takes longer, you’re going to be answering for it next review period.

So you have a “task” that looks like it could use a relational database, but you don’t really know relational databases that well. You’re not an arrogant narcissist, so you recognize that something that takes other people weeks or months to learn effectively is probably going to take you weeks or months to learn effectively, as well. You _do_, however, know how to stuff documents into a document repo, and you can estimate that, and the estimate is within the acceptable four-hour maximum that the project management group that is, for some reason, now running the show, permits.

Even if you’re supremely dedicated and spend your off time studying (assuming you don’t have a family and _have_ any off time), you still have to come up with something RIGHT NOW so maybe after a few weeks of evening study you might have been able to produce something better, but the idiotic “sprint” ends on Friday, so you stuff it into a key-value store, mark the task complete, and spend orders of magnitude more time working around the problems that were introduced by myopic, shallow-thinking project managers than you would have spent actually putting together a decent solution.

So there’s that.

From what I've seen and personally experienced, the sort of inexperience-masking hipsterism he's describing starts earlier. In university or highschool, years before Agile or other management styles become relevant to the individual.

Even a superficial difficulty can make a big difference for some people. If you start out with a NoSQL database because it seems simpler, and then with each problem you run into, it seems simpler to solve it with NoSQL because that's what you're familiar with.

Often the best tool for a job isn't the easiest to learn. Presumably a person who calls themself a "Data Scientist"—or even just a developer—should be able to figure out SQL or take a class. Applied Knowledge is after all what we get paid for.

If you are a professional developer and can't figure out SQL then you don't deserve that 6 figure salary.

Many data scientists are STEM PhDs with research under their belts. This shows they can figure out complex, technical challenges and then translate it into written, persuasive materials. I'd be surprised if most of these highly learned, gritty individuals (1/2 of grad students develop mental health illnesses) could not learn SQL, even if it is tricky/ugly/less user friendly.

I use SQL daily and while it's not ideal, it provides a very reliable foundation upon which R and STATA scripts build.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact