This. This article is half-speculation about the simplest parts of the stack that Google still isn't revealing ('probably uses Java servlets').
Google is doing a pathetic job of transparency and an even worse job of open-sourcing the real parts of its infrastructure, and from everything I've read so far, it's hurting them as much or more internally than it starves the developer community.
As far as I can tell its not hurting internally at all. The tools here are miles better than the competition. Colossus, btw, has no open source equivalent (which the article seemed to suggest might exist).
For the same reason why everyone else should open-source their infrastructure and why others do - Yahoo's Hadoop, Facebook's Open Hardware, Rackspace's OpenStack. Because others are building similar technology and cross-pollination (and honest trials) of BigTable with, say HBase, or their web stack with node, would mean that they break out of NIH and start improving both the state of the industry and their own code.
That said, I'm not saying that they need to, but that nobody should credit them with transparency or open-source goodwill in terms of actual core stack when what we're actually seeing is zero transparency and lots of in-house closed code. To the point that even what they're using is a question mark, and our guesses like BigTable often meet with a smirk and an 'oh, no, we're on to InternalProjectX, which is so much better.'
Thank you. Like others, I believe Google keeps its infrastructure secret because that is a competitive advantage. Their infrastructure is their business. It makes sense for Facebook to openly publish its datacenter design because that's not what Facebook is about: they could be running from AWS for all we care -- the social graph is kept very much closed, however. Opensourcing seems to be the radical approach to commoditizing complements.
But I would love if Google opened up about that, anyway. I am an avid reader of their service failure post-mortems because they reveal a tiny bit of what goes on inside their data centers. And because the way Google stuff fails is admirably beautiful.
Here's an ex-Google engineer who thinks Google's software infrastructure is obsolete, compared to opensource offerings:
http://rethrick.com/#waving-goodbye
I wasn't one of them, but I agree. Google's internal infrastructure has its warts, but it's generally light years more stable and reliable than anything you can get open-source. I have a teammate that worked with Hadoop in previous jobs; he said that there's no comparison between it and MapReduce. I used JSON for my previous startup's core data storage; were I to do it over again, I'd use protocol buffers, which are much, much more efficient. I use both Closure and JQuery on a daily basis; I consider them as occupying different niches, and wrote a bit more about that last night:
Absolutely. I think the criticism is that Google has credibility amongst the tech community for 'openness', which is based on shrewd PR rather than them actually behaving differently to any other large corporation.
I see proponents of Google make arguments of the form:
"Google is open so we want them to win. Because we want Google to win, it's ok for them to keep things closed to give them an advantage."
Google is and remains primary a for profit company, which has to report to it's share holders. But they are better compared to others. They made large contributions to open source, for example Android and Chromium. Neither of the two are generating direct revenue for Google.
Also they choose to publish their map reduce, page rank and other algorithms, in the first place. I think if it were not for Google, there was no Hadoop. Also if Yahoo was the first to develop map reduce, I doubt they would have open sourced it either.
My point is overall Google's existence is beneficial for the opensource and tech community in general.
Why doesn't Apple get credit for their huge investment in Webkit which was crucial (by Google's own admission) for both Chromium and Android?
I'd argue that the difference in perception is down to PR. Apple does little to make itself look like a benign actor, whereas Google puts a lot of effort into associating itself with 'openness' (and other things like 'science'). I'm pretty sure that's intentional image making.
I wonder if the Closure library (and related tools) will gain more traction now that it has been used for Google+. Any reason why more companies/projects use them?
Closure has been available for a while and is being used with Maps, Gmail, Docs, Reader etc. so I don't see why the addition of another Google product would make a big difference in adoption.
It is a large, full-featured framework, comparable to sproutcore, cappuccino or EXT. It ties in a lot with Java and (update: not GWT: I must have missed this since I never used GWT). I tried it, but found that I prefer a lighter stack of loosely coupled libraries (eg. backbone + mustache + zepto)
Actually it doesn't tie in with GWT at all. It's kind of weird - there doesn't seem to be anyway at all to use the Closure components from GWT without a lot of work.
Closure has it's own Javascript compiler (that does some type checking and minimisation). It's likely that is what you are thinking of - you'd hardly be the first.
it comes down to understanding everything you are using, and being able to swap out components for something better. ie. it is very unlikely that a single framework has the best router, controller, selector, templating etc. but by assembling your own stack you can inject personal preference and what you consider 'best'.
I do the same thing on the server side as opposed to using Django
EDIT: It is definitively only for full fleshed java-script web apps, not for quickly adding some gimmickry to a static webpage.
As it happens I just started looking into it a few days ago. Here is what I found:
The closure compiler really is it's biggest advantage. It compiles your code to really optimized javascript, with not one character too much. Only the parts of the library that you can actually use are compiled into your app. Also the library is pretty much extensive. For example their i18n code looks quite good.
The downside (in my mind) is that it forces a quite verbose java-like programming style on you (classes with getters/setters etc.). You also have to annotate your code with type information (a la javadoc) to take full advantage of the compiler. Also the DOM APIs are really ugly. They basically look like the bare browser apis.
Their is not really a big community around it either. But also means that nearly every line of code is actually used in a google application.
I havn't decided yet if I should use it for my next project.
> You also have to annotate your code with type information (a la javadoc) to take full advantage of the compiler
I consider this an upside, especially on large JavaScript products with many engineers contributing (like the ones Google builds). Having your JavaScript checked at compile time helps ensure quality code, and helps to avoid errors at run time.
I guess you are right, especially if you work in a team and want to build well documented, maintainable code. On the other hand it can be a bit tedious, if your main goal is to build a rapid prototype kind of thing.
We started our current project using jQuery/jQuery UI and I found that it didn't feel up to the task. We're creating a complex single page application with many controls, etc. Things like menus, tree controls, etc, just don't exist in standard jQuery UI. And the plugins were mostly unsupported.
I got up and running with the Closure library, very quickly, and I'm still finding it nice to use.
I'd heard of it before, but hadn't really taken it seriously. I guess it's time to start. It's been used in enough of Google's main apps now that it's probably pretty stable.
the most interesting part of this was the first comment, which was a rebuttal of the ex-googler blog post talking about how new open source projects and ideas are more advanced than the 'dated' (his quote) Google stack.
The ex-googler who compares Protocol Buffers to MessagePack is either lying about his understanding of Protocol Buffers, or hasn't looked at MessagePack beyond a simple glance.
First of all, MessagePack throws up a benchmark on their front page bragging about how it can deserialize 3 integers and a string faster than Protocol Buffers. Lovely. Of course, besides being a silly microbenchmark, that's in an array - not a keyed map, and since your protocol is going to evolve, you'll need a keyed map. Since MessagePack serializes map keys with the message, you're going to take a big hit there. Let alone when you have a protocol that grows over several years and has to support a service with over a hundred keys. Eventually, your string keys will become too big of an overhead, and you'll switch to integer keys. Then you have to roll your own translation layer between your space-saving integer keys and useful names for your keys. It's roughly at this point you wish you had picked the "creaky" solution which has done the right thing™ for almost a decade (though of course not open source for that long).
There is a bug with their API that I can't replicate on demand which absolutely messes with the history in latest Chrome. It happens sometimes when I go to someone's posts page, then about page, and then try to go back. It keeps switching between posts and about pages. When I click-hold on the back button, I see the wrong titles for many of the pages I visited previously but the links work. Pretty weird and annoying.
There is no "average HN user" - and I think most people here realize how important Java is to pretty much any company that is of interest on HN.
People who spend a lot of effort hating Java (or any platform) on general principle don't know what they are talking about. The Java platform has a lot of problem - and they are worth discussing - but mindlessness biases are pointless.
Servlets are actually a decent implementation of a not too bad idea - the problem is the rest of the system (Java server faces, which have all the conciness of XML with the prettiness of PHP and Tag libraries which are essentially a remimplementation of Java in XML and, well, Java) and the bigcorps that normally use it.
Also just because you use the JVM is no reason to use Java (which is a bad language, whoes only redeeming value is that it has fantastic IDEs for it) -- there is Scala and Clojure.
Meh, they're OK. It's pretty easy to layer something more interesting on top of them. Pop in a scripting language for the business logic and you're looking good, with lots of possible deployment options and a wealth of tuning information.
Right.