"Engineers 3-10x more productive"
If this were generally true (of any technology X) the dollars would be flowing that way in an unstoppable torrent, and anyone using anything else would be the subject of ridicule and/or pity. It would be like writing custom websites in C.
Since that isn't happening, I have to take assertions like that with a sizable chunk of salt. The alternative would mean that there's a gigantic arbitrage opportunity that's being missed by a bunch of highly motivated people. So yeah, citation(s) please.
Managing a transition like this is far from trivial for a company facing a variety of immediate demands and constraints, even if it promises to pay dividends in the long run.
It depends on who you are. I recently used Clojure on a skunkworks feature for our product for which I had three days to complete. I am absolutely convinced that it was at least 3x smaller (LOC) and took me at least 1/3 of the time it would have taken in Java. I am an expert Java programmer and not a newbie to Clojure, but I still have to look up a lot of things.
The killer productivity feature of Clojure (LISPS and other languages) is the REPL. Trying things out, analyzing data structures, etc. is where I would have spent a ton of time in Java. Even if you're not an expert Clojure programmer, just using the REPL can save you tons of time.
So not necessarily disagreeing with you, but I also think that a non-expert in Clojure can still run circles around an expert Java programmer when considering productivity.
The workflow is very natural and you practically never have to restart anything. You can build up state in the application, see how things interact, reload functions add new functions, and so on. Using a debugger in Eclipse is a very pale imitation of this.
That's standard process for any language.
The question is: why should we assume that Clojure pays more/special dividends in the long run?
As far as I am concerned, they are all being deeply deceitful. If you make some "fact" up, and pawn it off as the truth, you are being dishonest.
Support for clojure's worldview is easy to find on the web. It comes both from the clojure community itself (see Rich Hickey's talks) and from the wider communities that clojure draws its ideas from (other functional languages, lisp, etc.) Decide for yourself if you find the arguments compelling. In my experience, the clojure community is not interested in promoting clojure for reasons of tribalism or vanity, but because they can indeed use it to produce better work faster.
Example needed. Cascalog?
* M7 Overview: http://www.mapr.com/products/mapr-editions/m7-edition
* Datasheet: http://www.mapr.com/Download-document/21-MapR-M7-Datasheet
* Benchmark: http://www.mapr.com/Download-document/52-MapR-M7-Performance...
* Docs: http://doc.mapr.com/display/MapR/M7+-+Native+Storage+for+Map...
* AWS Service: http://aws.amazon.com/elasticmapreduce/mapr/
- the management UI isn't as good as Cloudera Manager, or even Ambari in HDP2.0
- 3rd party support isn't there. If you want to use anything you find on GitHub, get ready to add support for MapRFS, and revert the API version back to 0.21
- there are bugs. Weird, specific bugs in the filesystem implementation which will bite you once every few weeks. This is one area where they lag the ASF project: many eyes do make these kind of things shallow. Their smaller install base and fewer developers means these things take a long time to notice and fix.
The MapR team is sharp and responsive. It's founded by ex-Googlers from the search-infrastructure team (http://www.wired.com/wiredenterprise/2011/12/ex-google-man/).
M7 Tables are designed to be a drop in for HBase, with better performance and without the HBase complexity (it's much easier to run M7 Tables as the primary datastore for online apps).
M7 Tables support and certification for several key projects is coming along...
Spark 0.7 already works with MapR M7 Tables with the Hive images that were released with the Eco-1310 (http://doc.mapr.com/display/components/Hive+Release+Notes, http://answers.mapr.com/questions/7803/shark-spark-over-mfs).
MapR is in the process of certifying Titan (https://github.com/thinkaurelius/titan/wiki), and it's going is going through final QA right now (https://groups.google.com/d/msg/aureliusgraphs/RTeFVssIvoI/m...).
The problem with this of course is that when a company tries to switch their more mundane engineers to Clojure, the project will likely fail.
It's too bad that these top coders are so bad at mundane tasks like high-school-level statistics.
I've seen companies end up in inversion, which means that the most apparently productive engineers are the worst ones. It's a common effect in "design pattern" Java shops with all their Visitor and Factory and Vibrator and AbstractSingletonFactory patterns. It's not pretty. Good people are either taken badly (because they keep agitating for rewrites) or become disengaged and eventually leave or are fired.
The language, or its aficionado -- either way, specific technologies have made me, in my own estimation, multiple factors more productive. Yes, there's the learning curve, and understanding things that are not explicitly stated line by line throughout the code. But then, there's getting shit done.
Unfortunately, a lot of people, processes, and institutions get hung up at the sentence before last of that previous paragraph.
I also turned other people onto some useful technologies, such as pointing a consultant at some reporting software that my department didn't want to spring for but for which -- of course -- the outside consultant had budget. I heard from her a month or two later that it had greatly eased things and turned her into a sort of reporting superhero.
Throughout my career, I've seen people who, both on their own as well as through dint of better technology awareness and selection, have been multiple factors more productive than their coworkers.
And yes, often they are fighting the trend and significant inertia, rather than being embraced.
I started up with clj and I am trying kinda heavy arbitrage here :)
The line on slide 111, "Engineers 3-10x more productive, smaller teams", is probably in reference to...
Slide 16: "Code is typically 3-10x smaller when written in Clojure."
Slide 110: "3-10x less code. Limiting factor in programming speed is essential complexity of the problem, not typing."
I have to take assertions like that with a sizable chunk of salt. The alternative would mean that there's a gigantic arbitrage opportunity that's being missed by a bunch of highly motivated people.
That there is a "a gigantic arbitrage opportunity being missed" is essentially the thesis of the talk.
He states the reason why (Clojure is cloaked in 50 years of Lisp misconceptions) on slide 21: "Lisp is an old weird AI language no one uses anymore. It's so slow it requires a supercomputer. It's impossible to hire people who know how to use it. Functional programming has been proven impractical. Also, parentheses."
The body of the talk is designed to dispel these beliefs.
Oh, it's a perfectly plausible number. It increases the productivity of each developer by a factor of 3 to 10 and also increases the maintenance costs by 3^n_engineers or 10^n_engineers. This seems to be the tradeoff that startups make.
EDIT: (yes, being a little cheeky because as we know the physical time programming isn't everything in building a website)
Lisp has a bad reputation compared to what it is. People think it's impractical, that Lispers are arrogant (due to a few vocal but influential people in the Lisp community) and that it's archaic or weird. I've addressed those perceptions, but they're out there. Clojure is, in fact, a pretty damn practical language.
Now, the arbitrage argument: in larger companies, the concerns aren't engineer productivity so much as appearances. For a middle-manager to launch a meaningful Clojure/Lisp initiative would be to put himself out there in a major way, and since programmer productivity is only one piece of a much larger system, it might not have that big an impact on the company as a whole. It might not be worth it. (Would Google's market cap go up 3-10x if it replaced Java with Clojure? Almost certainly not. Maybe the fair value would improve by 1.05x on fundamentals, which would be drowned out by month-to-month volatility.) On the other hand, startups generally can't afford (or don't think they can afford) to experiment with newer, quirkier languages. VC-funded startups tend (surprisingly?) to have stodgy closed-allocation work cultures and boring tech stacks because they're expected to allocate 100% of the "risk allowance" to one department: business-model risk. That's not to say that there aren't other kinds of companies that can (and, in some cases, do) use more innovative tools; but the two dominant cultural forces (VC startups and large corporations) in technology don't favor it.
For the large companies, I'll admit that there's an Amdahl's Law sort of effect at play. Let's say that you improve productivity of engineers by 3-10x. Does that "3x" the whole company? As I alluded to for Google, probably not. It makes something else the limiting factor. For most firms, using the wrong programming languages is nowhere near their top problem. To use Google as the example, it might deserve a 5% bump on market cap if it included Clojure (a small gain compared to monthly volatility, so small that no one could prove causality and take credit) on the white-list but the fair value would go up 50-200% (again, on fundamentals, as I can't predict investor opinion) if it implemented open allocation. Programming language innovation just isn't the top priority for most companies. But for an individual developer, the advantage conferred by a powerful language can be pretty serious.
People choose what they work on. No "headcount" numbers that have to be agreed upon by people far away from the actual work. It's the one thing Google could still do at this point that would 2-3x the company.