Hacker News new | comments | show | ask | jobs | submit login
The story behind football's innovative yellow first down line (cnn.com)
180 points by nickgrosvenor on May 30, 2014 | hide | past | web | favorite | 89 comments

In the Eagles vs. Lions game earlier this year there was about 6" of snow on the field and tons more coming down. The field barely showed any hints of being green underneath. The photographers couldn't even use autofocus because of the size and frequency of the snowflakes.

Due to the yellow line technology, or some other augmentation they were doing live on the feed, all of the players would appear translucent for brief periods. It was really weird, but it made me stop and think about how many impressive live-processing algorithms get their hands on the video before it gets to me. It was the first time I appreciated how tough it is to do the "yellow line". This article is a very interesting insight to how it came about.

I remember that game. They were using the overlay to project all the yard markers (numbers, lines were dug out). Detroit was wearing their whites, and were sometimes mistaken as snow by the computer, causing them to appear as apparitions beneath the markings.

That was my favorite game of the year.

Does anyone have a video of that?

I never watched the this sport, but I think it could be this one? [1]

When it fails, it makes people look life frosted glass [2], but actually it's extraordinary that it's still working almost perfectly.

[1]: https://www.youtube.com/watch?v=E9qYZCl_F8I [2]: http://i.imgur.com/FExuLmY.png

Oh hum, that looks interesting, thanks!

I have dabbled in a little bit of video production, and on the (rare) occasion that I catch a few minutes of televised football, that yellow line always blows me away. Mentioning how difficult it must be to whoever is actually watching the game results in weird stares as if it's the most natural thing in the world, but I'm glad to read that I'm not going crazy. And I had no idea how extensive the setup actually was.

Yeah totally, having some experience with 3D motion tracking and green screens, I was thinking: wow, it's so smooth and so perfectly tracked, AND cuts between player/grass so well. I know how finicky it can be to key out elements, so it is quite amazing that whatever algorithm they have in play can do it so well (is there a jersey color that's close to grass color lol? Wonder what the edge cases are).

It's alluded to in the article that outdoor fields and green jerseys are the hardest. For outdoor fields the lighting is affected by the time of day, cloud coverage, players moving along the field, and plenty of other things I'm sure. It says they spent a lot of time creating a unique palette for the fields and that domed stadiums were preferred by the engineers on site. Indoor lighting probably reduced the shadows a lot.

I wonder if they use any tracking technology that looks at large groups of pixels over time. If you could track a player as a closed shape moving through space, you could mask out that player from the background even if pixels on the player's uniform were identical in color to the field's palette.

There are many teams that have very similar jersey colors to grass. Moreover, what is even more impressive, is there are some astroturf fields that aren't even green [1]! And the first down line works just fine on that as well.

[1]: http://upload.wikimedia.org/wikipedia/en/5/56/Bronco_Stadium...

I've had that same conversation with non-technical people. "Complicated? They're just drawing a yellow line on the screen!"

The first time I saw that line my first thought was along the lines of "holy shit, that technology is insane. How are they overlaying it like that?". I was seriously impressed with it.

I had a cousin who worked with Sportvision at the time. I recall he had mentioned that early on, the support team for the line was comprised of classic nerd engineers who couldn't care less about football. And ironically, they had somebody's dream job of being sent to every single football game being covered by the line tech.

> Hill liked the concept, of course, but Fox thought that the price, $25,000 per game, was too steep. [...] Our last stop was ESPN and the reception was decidedly different.

So interesting to see that FOX balked at the seemingly low $25k but ESPN didn't flinch and insisted on exclusivity for games they didn't even have rights to. Telling of how the networks viewed themselves and each other, but also how important it is to find the customers that are the true believers.

> insisted on exclusivity for games they didn't even have rights to

given the price per game and the market value for that, it seems like a low price to screw your competition. by insisting on an exclusive deal they couldn't use it in the playoffs - anywhere. what a well played "screw you buddy" move.

ESPN likely didn't flinch because they charge a cost per subscriber of ~$5/mo (as part of your cable bill), whereas FOX is charging significantly less. So ESPN can afford it.

ESPN is the 800# gorilla when it comes to TV programming and just about every cable network is trying to emulate them.

Reminds me of those Excite and Yahoo idiots who refused to buy Google for a cheap $1 million. Some people just don't deserve to be at the top.

But would google have ended up as google if they were bought at that price?

Cliche but, at least his, hindsight is 20/20

There is a recent interview with Marc Andreesen on EconTalk where he talk about how hard it would have been to realize that Google was a good buy at the time. Details here: http://www.econtalk.org/archives/2014/05/marc_andreessen.htm...

When this first debuted, my grandfather (born in 1917) thought that they had a piece of nylon material that was connected between the first down markers on the sidelines. That's when you know a technology is good. It's so seamless that it never occurs to a layman that it might even be possible.

I love this. I've spent a lot of time between plays imagining just what kind of engineering is behind that magic first down line. Of course, I should have known it came down to this: "Our engineering team... worked around the clock through the summer..." Sometimes I wonder if there is any such thing as a correctly scoped project that can be completed during daylight hours on weekdays.

Sometimes there are just hard deadlines you can't do a whole lot about. In this case, it was pretty much hunker down and get it done in 4 months, or wait another year. For a fledgling company, that extra 12 months was probably the difference between success and failure.

Yes, oddly enough they don't reschedule the Super Bowl for us if a job is going to take a few extra days.

"or wait another year"

Why? This quoted line above specifically.

Sadly, "around the clock" makes for a sexier story. I'm reminded of a previous job where working late and being super busy was worn like a badge of honor - while 75% of the day was filled with meetings. Also, many people didn't take much vacation time until the last half of December, because they were just too busy all year long to take it beforehand.

"Hey, how's it going?" "Super busy!"

I don't miss that part of the culture.

His mention of the glowing hockey puck made me realize that I haven't seen a glowing puck for awhile. Granted, it may be that I haven't watched hockey as much as I did when I was a kid, but I certainly didn't see it last year when I watched some games.

When I did see the glowing puck it made me feel like I was watching NBA Jam on Ice.

Not only was it a (blue) glowing puck, but it would also make a red-orange streak on screen when there was a slapshot (er... "one-timer").


I remember how unpopular it was when it came out. It made a lot of sense to try; not being able to follow the puck around really was a common complaint about hockey, and the NHL, as ever, was trying to expand its base. But if you watch even a fair amount of the sport, following the puck (even when it's out of sight against the boards) is really easy, so for the majority of the actual audience, it was distracting and unhelpful. To add to it, the implementation itself was corny and overstated. So part of it was problem was that it traded power users' comfort for new user friendliness, and part of it was execution. Compare to the yellow line in football, which is both helpful to everyone regardless of football watching experience, as well as simple and clean.

The thing is that most of the time the exact position of the puck isn't really the most important thing. Where the players are and where they're going to/from seems more important. Emphasizing the puck just distracts from the actual game. I don't think it even helps newbies very much for that reason.

In terms of helping novices understand the game better, maybe visual indications of the zone the puck is in could be interesting, or direction of play so it's easier to understand off side and icing calls would be more helpful.

> Where the players are and where they're going to/from seems more important.

Another way to put that is that you can tell where the puck is based on the behavior of the players, and that's what you're actually watching. I'm only a casual fan, but I think I'd know where the puck is even if it was actually invisible.

The adoption of HDTV has also negated the reason for the glowing puck. When I watch hockey on a non-HD feed I wonder how I ever did it.

It just makes so much sense. Of /course/ the puck should be instrumented, what century are we living in?

The posters below who say it was a terrible invention are missing the point. The technology needs to be more advanced, not less: visual effects on the puck should be at the option of the viewer.

This was a terrible invention.

it was a terrible invention. thankfully it didn't survive long.

Never seen hockey before and had a look on youtube.

The blue glowy bit is ok I think but that red tail comet is atrocious, it is completely distracting, covers up waaay to much of the screen.

Would you have been happy with just the blue glowy affect?

Before HD tvs it might have helped a lot, seeing a puck on an old screen is very difficult. But on the newer HD tvs, you can see the puck clearly enough I wouldn't want it, it probably isn't necessary.

> Would you have been happy with just the blue glowy affect?

nope, agree with the other follow up - HD is good enough that any "enhancements" are now a distraction.

The latest sport where Sportvision used this technology is sailing, called LiveLine there. During the last America's Cup [1] they used it to show the starting, finish and boundary lines, distance between the boats,...

[1] http://youtu.be/r0LH5cCuc_4

That was an incredible use of the technology. I'd watched sailing on TV before, but for the first time I could actually see and understand the scope of the race and the strategies of the skippers.

I have always thought that it was a complex computer vision problem that made this work, very interesting that it was more than just software and makes sense.

I am tempted to think that this was the first widely adopted (and accepted) instance of augmented reality. Very cool in that context!

A lot of this has changed since then, of course. No grass swatches anymore! We have a lot more computer horsepower to throw at the problem now, plus a lot of gained experience.

We have a bay in our lab with a to-scale football field where we can run indoor tests, but on the field it is computer vision that is doing it all. The cameras are instrumented to get attitude, zoom, and so on (we have to modify the lens to get the accuracy we need), and then of course you have to compensate for lens distortion. If you look closely, the "line" is not a line, but a curve. If we drew a line it would look stupid, and not reg well in most parts of the image.

Very cool. Are you guys doing statistical filtering at all or is it strictly palette matching? Seems like your data inputs are specific enough that you are trying to get where you don't need much buffer on either side of the wavelength.

I will have to ask. I don't work with this technology, and haven't touched the code. I work in computer vision for baseball, with a completely different set of challenges.

Is an automated "yellow box" to show the strike zone ever going to happen? I feel like that would have a huge impact on the viewing experience!

I dunno. I kind of like trying to judge it visually and waiting for the ump's call.

I can also imagine MLB having similar concerns as the NFL when, for instance, they insisted on removing the line as the ball is being spotted. That is, if they are showing the strike zone and the ump's call seems inconsistent, then it wouldn't be a "good look" for the sport.

There might be trouble with depth perception there. When I watch baseball, I can't easily tell at what point the ball crosses the plate in the behind-the-mound view. When a pitcher throws a breaking ball, I usually have to look at the non-overlay strike zone they show afterwards to know where the ball was when it passed the batter.

That might be an interesting approach. Instead of physically tracking the orientation of the cameras, you could place markers at set points around the playing surface. Then, as long as enough of the markers were always visible, the software could fit the virtual geometry of the surface onto the video. It's the same idea used in QR codes, where the three large squares mark three corners of the QR code. I suspect it could work, but would never be as reliable and precise as the real implementation.

Well the field just becomes a large marker at that point and you use the boundaries as easy reference points. This is what the real future of AR and CV combined are.

I work here, if you have any questions...

I don't actually know what this yellow line is, probably because I am from Australia. Have you got a sample video for it?

The yellow line in the 3 to 5 seconds mark in this video: https://www.youtube.com/watch?v=QtxoYaMebF0

Why is it so hard to find a single HD video with this line? I searched for 'football "yellow line"' and filtered only HD videos, and none in the first page actually showed some game footage with the line. The same for 'football "first down line"'.

Is there no HD footage for these events? Or do even short extracts get removed due to copyright issues?

You won't find it with those search strings because the American viewer (presumably who is posting those videos to YouTube) takes it totally for granted.

Try searching for just "first down" with football, or viewing some game highlights on NFL.com.

I was about to say the same thing. You wouldn't even need to search first down, just watch any football footage from the last decade. The first down line is going to be in pretty much every shot. Note that it isn't always yellow. Sometimes it's orange, and there are also other lines, namely the line of scrimmage, that are marked as well.

Which also shows the evolution of the yellow line idea: another line (usually blue) for the line of scrimmage, which turns out to be really useful. I think I've also seen a line for maximum field goal range.

And you probably also have seen advertisements inserted behind the end zone. I do not know how much use they make of it, but that allows the broadcaster to send different advertisements to different markets (the low tech variant that I have seen of this in soccer is to have one country show the main overview from one side of the field, and another country the main overview from the other side. That way, one can sell the main advertising space at the center of the field across from the main camera twice)

Also, in speed skating, the technology gets used to project a moving line indicating where a skater should be to reach some goal (win the race, world record, etc)

The concept was used a lot in the Olympics. I remember seeing national flags superimposed in swimmers' lanes, and they also used pace lines for swimming.

That is pretty cool, thanks!

Australians are pretty familiar with high tech sports enhancements, cricket probably uses them the most of any sport. You've got Hawkeye/Virtual Eye which are video systems that predicts where the ball would have gone if it hadn't hit the batsman being used by umpires for replays during the game, Snicko where a microphone is used to judge whether the ball hit the bat, and Hot Spot which uses infrared cameras to see if the ball made contact with the bat (or leg).

The yellow line is impressive because it's done on Live TV though not as a replay.




Us Aussies probably saw the same kind of tech first in swimming coverage. Where they used a moving line to show the world record times.

How did they actually get meetings with all of the Networks?

This comment provides some context:


They probably knew some people.

Sorry, don't have a clue, I joined in 2012.

Here's a decent youtube video that basically shows the tech in action. http://youtu.be/Vh9af_gXxlM

The yellow line really is amazing from a "user interface" perspective. It's so natural that I had forgotten there was a time when it didn't exist. And, as mentioned elsewhere in the comments, it seems everybody has an old uncle that swears it's just nylon between the down markers (or paint, or something else that isn't "magic").

"Gepner, a TV production expert, engineering guru Stan Honey and I (the business person) -- had been inspired by Stan's invention of the glowing hockey puck technology the previous year for Fox, technology which we licensed upon leaving the company in exchange for an equity interest for News Corp."

This article opens in May of 1998, just one month before the final use of the glowing puck in June of 1998. By this point everybody in that meeting must have known how hockey fans had reacted to the glowing puck. The interesting thing is that they bravely forged ahead anyways!

Execution may have been the key difference in fan acceptance. Fox made the puck into a giant glowing and color-shifting comet that practically screamed, "PEW PEW PEW!!!". These guys made a simple line that was meant to look, as much as the technology would allow, like a simple chalk line on the turf that some poor slob had to create (while erasing another) after every down. Would we still have puck-following tech in the NHL if Fox had been run by people with even the tiniest amount of taste or restraint?

I think the glowing puck would have still died when high-definition TV came along. I think HDTV saved the hockey viewing experience, because it made it a whole bunch easier actually watch the game. The problems solved by the glowing puck then stopped being problems.

Impressive technology, but perhaps someone with more football knowledge could explain why they don't just paint lines on the field?

The rules for American football can get pretty complicated, but an abbreviated explanation for the yellow line might go something like this:

The team that has the ball has four tries (i.e. downs) to advance the ball 10 yards. If the team that has possession is abl to advance the ball 10 yards or more within four tries, the marker is reset to the next 10 yards. That's why it's insufficient to mark the lines on the field: if I started at the 10 yard mark and advanced to the 12 yard mark, the line would have to be placed at the 22 yard mark. Painting lines would mean that the entire field would eventually have to be painted and unpainted.

I love American football but it's pretty much an acquired taste (lots of stoppage time, tons of instant replay, incomprehesible rules, heart-breaking inconsistency in the officiating, etc.)

The line moves on every play. Paint would be just a bit cumbersome :)

It represents the "line of scrimmage", which is where the ball starts at the beginning of each play.

The yellow lines marks the line to gain for a first down, not the line of scrimmage. Some broadcasts are showing the line of scrimmage now usually as red or blue IIRC, but yellow is used across the board for the line to gain.

Preceding his accomplishments at Sportvision, Stan Honey created, in 1985, an automobile navigation system showing your car's position on a moving map.

This was before Google Maps, before GPS, and before digital maps of cities. You couldn't just go out and buy them in 1983; Stan's company, Etak, had to create them itself [1].

Per [2], here's a description of the the hardware platform:

  The original Etak Navigator was a specially-packaged Intel 8088-based
  system with 256K RAM, 32K EPROM, 2K SRAM, and a cassette tape drive
  on which digital maps and some of the operating system were stored.
The 8088 has a 16-bit address bus, thus can directly address only 64 K. Must have been some bank switching to get to the full 256 K RAM. This for all the code and geodata.

edits for formatting


[1] Source: Personal conversation

[2] https://en.wikipedia.org/wiki/Etak

The 8088 can address 1 MiB with a somewhat wonky segment:offset addressing scheme.

The few times I've watched football, I've been fascinated by this technology. I imagine football fans take it for granted at this point, but I just want to say "wow, are you SEEING this!?" For a live broadcast to be augmented in that way is damn impressive.

I'd seen an interview[1] elsewhere with, I believe, the founder of SportVision where they indicated that their primary source of revenue now is selling the vast amounts of telemetry data they collect during events.

His example was of the MLB, SportVision has high-resolution data on every single pitch, the MLB teams are paying through the nose for that data to run through their analytics. SportVision indicated that, based on ball trajectory, they often know more about the state of a given pitcher's health than the team or the pitcher themselves.

[1] http://www.sportvision.com/media/hank-adams-presshere

American Football.

Exactly. Across the world, "football" usually refers to soccer; please have some consideration for the non-USA audience.

Strictly speaking that should be association football, although everyone calls it football outside the US of course.

"Football" existed before, but simplifying somewhat, just about all the modern games of football derive from rules codified at British private-education schools in the 19th century. So for example, rugby football originated at Rugby private-education school, which then developed into American football. Off the top of my head I can think of association football, American football, Canadian football, rugby football, Australian rules football and Gaelic football.


I was hoping they'd get into the later patent struggle and competitive technology. (But I guess it's really a piece about a single interview someone manager to score.)

Another company in the field (PVI) at about the same time used vision-based technology to similar effect. They got into a mutual patent infringement battle and ended up settling by cross-license.

I'm still trying to figure out why NBC (maddeningly) doesn't show the line of scrimmage. At one point I saw some vague report that NBC uses the third company in this space (SportsMEDIA) and that company has a license to the patent for the first down line but not the patent for the line of scrimmage. Which sounds silly and plausible at the same time. However, they advertise line of scrimmage as available: http://www.smt.com/products/smart/line-visual-insert-publish...

It is probably a policy decision based on a 2008 in-game controversy where the line of scrimmage was a yard off and one of the broadcasters (Madden) was using it as a basis for evaluating the call, and didn't know that the officials can't see the line.

In either case, they need to deal with it and get the line of scrimmage back in. It's a very annoying omission.

i'm not a huge football fan but i've always really loved the new technology they bring to the show. the yellow line has always impressed me, and the fly-by cameras they introduced are really cool too.

i think i've seen something similar, but it would be cool to have full on video-game style indications. player markers, etc.

Interesting from a startup perspective is that they raised a lot of money first and then tried to figure out if they could implement the idea. This definitely is an idea where if it isn't being developed in a lab at a large corp or by government grant you would need to raise money for.

This is amazing. I only had read about it but this one gives a first hand perspective. Hats off to the team! Can someone point to the algorithm/exact technology used in this? Would love to read more about the technology.

I can't see how exclusivity into the playoffs was a mistake. ESPN believed in the technology and they wanted to punish their competitors. I think the creators of the yellow line should be proud to reward ESPN for their foresight and take part in enjoying the Emmy together as opposed to sharing the limelight with another TV network who never believed in the company. Financially, I would think you're in better bargaining position too: well, this year the line is $30K/game since it's proven to work and you don't need to do R&D work like ESPN did with us.

I always thought it a neon lights in the actual football field.

That is not football. It is Egg Shaped Handball.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact