Hacker News new | past | comments | ask | show | jobs | submit login
Using Python helped YouTube ship features faster than Google Video (books.google.com)
50 points by swyx on March 25, 2018 | hide | past | favorite | 42 comments

As much as I want to credit python, I doubt that was the cause. It's typically far easier to ship a new feature at a small company, because there's no bureaucracy, no wait for QA, no need to translate text into 20 languages, etc.

No QA and not even a lengthy release process. The question always is - Are you done with the programming? Go release it.

Additionally, people are more tolerant of mistakes made by small companies. Features stopped working? Maybe couple of people will complain.

Google's features stopped working? Media, bloggers.. everyone will have an opinion on why this happened and how to resolve it.

On the other hand, Youtube messes up, people will notice within seconds. Google Video stuffs up, it might takes weeks before anyone notices.

Startups are generally more tolerant of failure and half baked code. Googlers, who have lived there whole lives achieving perfection in school, not so much. I see this all the time at big companies that hire 4.0 GPA from ivy leagues.

Let me guess. You're not a 4.0 GPA from an Ivy League, are you?

A book named "Python Interviews: Discussions with Python Experts" might indicate some bias...

Anyway, I agree with your conclusion: big companies move really slowly in comparison with smaller ones.

I think there's certainly something to be said for the agility of small teams in shipping new features but I also think it's not particularly contentious to suggest that developers are more productive in higher level languages esp. when your focus is on user experience etc.

Yes. Perhaps also brilliant engineers don't want to hack together something half-assed that barely works, but does provide the feature.

I've always liked that youtube uses PHP - much derided, but its declarative "templating" is the easiest, simplest, most direct way to generate html. Maybe that was a factor too.

But I think the youtube engineers might have a more accurate opinion on why they were quicker, than the google video engineers.

> I've always liked that youtube uses PHP

Perhaps you are thinking of Facebook?

> but its declarative "templating"

Do you have any insight onto how much of the templating aspect of PHP is used at Youtube? I know of some large PHP installations where PHP is used as a general programming language that does produce web pages, but the templating aspect of PHP is pretty much minimal, and other engines actually produce the final output.

For a so-called templating language, PHP sure falls short, and thus has a plethora of ad-hoc, informally-specified, bug-ridden, slow implementations of half of many other templating languages implemented in it.




Smarty is something from 15 years ago. Nowadays people use Twig which is inspired by Python's Jinja template engine and has similar syntax.

Yet Python has never claimed to be a templating language itself, let alone "the easiest, simplest, most direct way to generate html".

My point is that PHP pretends to be a templating language, then falls flat on its face, so people end up implementing much worse (Smarty, MediaWiki) or sometimes even better (Twig) templating languages on top of it.

In other words, it's simply not true that PHP's "declarative "templating" is the easiest, simplest, most direct way to generate html", or there wouldn't be a need for Smarty, the MediaWiki abomination (which is 17 years old but still widely used on 29 million pages), or Twig.

I'd say "to hack together something half-assed that barely works, but does provide the feature" perfectly describes the PHP development process itself.

YouTube doesn't use php.

I shouldn't have said "uses", but "used". I can't find a reference, but IIRC they used it around the time of the acquisition.

Fun fact: online code editing & hot reload is a thing for python folks.

Remember, YouTube born 13 years ago. That time, online code editing and hot reloading were big things only archived with dynamic languages like Python and Ruby.

As someone who was creating a reasonably popular 'video podcast' back then, I have a different perspective...

Uploading to Google video was incredibly slow relative to Youtube. During the period of the 'battle' between the two services, Youtube allowed only very short videos - excluding most useful content, but discouraging piracy (except for very laborious multi part uploads). G'Video on the other hand allowed uploading of essentially any length, although at much lower resolution

But it was incredibly laborious, as uploads took hours - and were liable to fail without warning (at least uploading from Europe). I recall there was also an emphasis on adding lots of metadata etc, another barrier to uploading.

The social elements of youtube - perhaps in hindsight addicting rather than useful, are rightfully acknowledged. Whats less well remembered is the sheer clunkiness of using Google Video. The service looked and responded much more like a corporate intranet than a website. Still, I - and lots of other 'vidcasters' at the time were sad to see it go.

Here's a perspective from the time - from a blogpost I wrote about different video services in 2006 (https://garethstack.com/2006/07/24/the-unavoidable-future-of...)...

"While sites like Youtube and Guba, may or may not have a future primarily as redistributors of broadcast content, they’ve done little to foster the creation of original work. In fact, by restricting the length and size of files which can be uploaded (ostensibly to reduce copyright infringement), YouTube have diminished their chances of becoming a hotbed of original content. Google video, although bravely eschewing any restrictions on the length of uploaded content (whilst foolishly restricting video quality to an extremely low bit rate), does little to foster the community creation or pooling of talent needed to inspire the development of original shows and films. Note, it’s far from clear that it was ever Google’s intention to become a generator of new IP, so Google Video shouldn’t necessarily be seen as a failure"

Yeah, absolutely. I remember that being a big thing as to what Google Video said when they bought youtube. The immediacy of the upload was a huge factor in YouTube's success. It's not hard to imagine that YouTube won via better PM. That often happens.

YouTube was also burning money for bandwidth like nobody's business. I suspect that Google Video tracked and limited bandwidth.

If Google hadn't bought them, YouTube were headed straight for bankruptcy. People forget that the YouTube deal was a big time boardroom back scratch between Google and the VC's.

The worst part is that Google subsidizing YouTube has set back a real monetization of video on the web.

I disagree with that. I think the web is monetizing video at an appropriate rate, considering most video itself is essentially worthless.

That video content is costly to produce, and kills bandwidth indiscriminately, does not mean video is valuable. Just because there’s a high bar for video to exist, doesn’t mean that cost should get passed on to consumers.

Most video is non-essential, and consumers get to be picky about where their disposable income goes. Video needs to be redeeming to the viewer, whatever their tastes may be, and whose fault is it that the surplus of junk content exists?

> I think the web is monetizing video at an appropriate rate, considering most video itself is essentially worthless.

Um, I think you are confusing cause and effect.

Isn't worthless video exactly the problem that subsidizing video delivery caused?

If you have to pay even a penny to put your video on the web, most video disappears and suddenly the quality skyrockets.

I'm not sure how YouTubes current existence given the past decade or so isn't a huge antitrust violation. They've run the platform at a massive loss, subsidized by their other business ventures to make it impossible for anyone to create a competitor. I can't imagine that if you propose to a VC that you need to spend hundreds of millions of dollars on building infrastructure that you will then need to run at a loss potentially for decades before hoping to see any profit it would go well.

"boardroom backscratch" that's the phrase I've always wanted to use but never knew it. Thx!

That's the power of scripting languages. You don't focus on the language but on solving the problem.

unrelated: the ui is impossible to navigate on mobile.

And UI has been unusable on Google Books forever!

Must be coded in C++.

I remember when Google announced video.google.com was closing, and then they backtracked and said it wasn't. And then they closed it. I have video.google.com posts that I would like to migrate, but it's gone.


No affiliation at all to the author but if you want to read more he's @driscollis on twitter and the book is at https://www.packtpub.com/web-development/python-interviews

also I first heard this story on the Talk Python podcast https://talkpython.fm/episodes/show/156/python-history-and-p...

Of course its not the full story but it is an interesting comment and from a firsthand source. That's all I know, I dont have a dog in this fight :)

Yeah, no. That is obviously not why, and it is quite telling that the guy being interviewed still, after the purchase, hasn't got the slightest clue as to why (or doesn't want to admit it).

It is disheartening to see this on the front page with so little substance.

Interviewee closely involved with the YouTube acquisition: “competitive advantage was python.”

Commenters I’m this thread: “nah it wasn’t python.”

I worked at Google during that period. It's not quite as simple as "Python > C++" although yes, for sure, you're going to produce new UI features at a far faster rate working in a high level language like Python than working with C++. Absolutely that was a factor. I don't think Google Video had hundreds of developers vs 20 though. That doesn't sound right to me.

The real reasons YouTube won were strategic, not purely a function of developer speed.

In particular YouTube focused a lot on social / discovery features and prioritised them, at a time when the importance of social wasn't so obvious and Google in particular was very bad at anything with a social dimension. So Google Video had super scalable backends, great search etc, but it wasn't very good at user profiles, commenting, channels, subscribers, discovery, content surfacing and so on. Whereas YouTube excelled at these things and was good at encouraging people to upload whatever random vids they created even if apparently worthless.

Google in contrast was less certain that free user generated content would ever be a big deal, and right from day one saw Google Video primarily as a marketplace. It focused much more on acquiring content rights from professional producers as a result. Even YouTube wasn't really sure UGC+ads was going to turn into a real business which is why they had a lax approach to video piracy for so long. Early YouTube traffic was to some extent driven by piracy of professional content, a problem that only got wiped out once Google acquired them and build content id. Even when comments were eventually added, it was a tiny box to the side of the video which took up the bulk of the web page, and there was an admonition "Please make sure your comments are useful and informative" - very Google. YouTube on the other hand made the video less prominent and what users were saying about it much more so.

Social features are a great use case for Python because you need to churn them out and iterate very fast, the cost of correctness bugs is very low, and their performance complexity is not very high, so that's the sweet spot for a scripting language. On the other hand around the time YouTube were acquired they struggling tremendously with the scaling aspects of their operation and large chunks of the site had to be quickly switched to C++. YouTube when I left was still a mix of Python and C++, with the C++ code handling things like search, transcoding, thumbnail serving, anti-spam, content id, video serving and so on. All the heavy lifting. Python was left handling the main site UI, admin tooling and not much else.

There were also some technical mis-steps by the Google Video team. YouTube relied entirely on Flash from day one. For whatever reason the GV team were far more reluctant to hitch themselves to the Flash plugin and for example they only enabled video upload via a web form about a year after YouTube entered beta, before that you had to use a desktop app to do it. They also wrote their own video playing plugin but it wasn't as good as Flash and they eventually scrapped it. Their paid for pro content also required a dedicated video player app. YouTube being a small startup couldn't get content deals as it was too tiny for the producers to deal with, so they just focused 100% on UGC and features like embedding, which acted as a giant advert for their service.

More insights here:


You say:

> Social features are a great use case for Python

And then say:

> On the other hand around the time YouTube were acquired they struggling tremendously with the scaling aspects of their operation and large chunks of the site had to be quickly switched to C++.

I fail to see how python failed youtube here.

You use the right tools for the job. And the tools that got you 10e1 views per day may not be the tools that get you to 10e10 views per day.

I once worked at a place that supposedly "engineered" their backend to handle 1,000,000 connections at once. Only everything fell over at 10,000 connections in production, and they struggled for two years to get to 100,000 connections.

The point is, you don't know what you don't know, and you won't know the pain points are until you hit them. And the more groundbreaking your service (in this case internet video distribution), the chance of your architecture being 100% correct from the start is zero.

If I recall correctly youtube also focused on embedding early on. Embedding google video videos was impossible. This lead a lot of people to youtube and google videos just missed out on it.

> search, transcoding, thumbnail serving, anti-spam, content id, video serving and so on

Python has some of the capabilities, but surely those libraries are also written in C/C++

I read it somewhere that YouTube picked up pace because it was used as the default video sharing in MySpace. I am not sure if this is true or not.

thank you for sharing your insider view. I'm just a casual observer, so I do feel a little guilty posting this story here but it is a firsthand source.

I heard an anecdote where the actual first movers on a new video feature was porn sites. They were smaller still, and probably had a more tolerant audience.

I feel like entertainment (gaming, porn, even say tourism for spaceflight) has always punched above its weight in leading technology. Really makes you think about whether we are more motivated to use our intelligence/resources to solve hard problems or to please ourselves.

This is still true, especially for building out networks and CDN's.

And Google still writes backends in C++ in 2018.

I'd imagine that any company building its own hardware will continue to develop at least some code in C++ for another 25 years.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact