
The story behind football's innovative yellow first down line - nickgrosvenor
http://sportsillustrated.cnn.com/nfl/news/20130718/nfl-birth-of-the-yellow-line/
======
habosa
In the Eagles vs. Lions game earlier this year there was about 6" of snow on
the field and tons more coming down. The field barely showed any hints of
being green underneath. The photographers couldn't even use autofocus because
of the size and frequency of the snowflakes.

Due to the yellow line technology, or some other augmentation they were doing
live on the feed, all of the players would appear translucent for brief
periods. It was really weird, but it made me stop and think about how many
impressive live-processing algorithms get their hands on the video before it
gets to me. It was the first time I appreciated how tough it is to do the
"yellow line". This article is a very interesting insight to how it came
about.

~~~
function_seven
I remember that game. They were using the overlay to project all the yard
markers (numbers, lines were dug out). Detroit was wearing their whites, and
were sometimes mistaken as snow by the computer, causing them to appear as
apparitions beneath the markings.

That was my favorite game of the year.

~~~
StavrosK
Does anyone have a video of that?

~~~
zenojevski
I never watched the this sport, but I think it could be this one? [1]

When it fails, it makes people look life frosted glass [2], but actually it's
extraordinary that it's still working almost perfectly.

[1]:
[https://www.youtube.com/watch?v=E9qYZCl_F8I](https://www.youtube.com/watch?v=E9qYZCl_F8I)
[2]: [http://i.imgur.com/FExuLmY.png](http://i.imgur.com/FExuLmY.png)

~~~
StavrosK
Oh hum, that looks interesting, thanks!

------
jabelk
I have dabbled in a little bit of video production, and on the (rare) occasion
that I catch a few minutes of televised football, that yellow line always
blows me away. Mentioning how difficult it must be to whoever is actually
watching the game results in weird stares as if it's the most natural thing in
the world, but I'm glad to read that I'm not going crazy. And I had no idea
how extensive the setup actually was.

~~~
cclogg
Yeah totally, having some experience with 3D motion tracking and green
screens, I was thinking: wow, it's so smooth and so perfectly tracked, AND
cuts between player/grass so well. I know how finicky it can be to key out
elements, so it is quite amazing that whatever algorithm they have in play can
do it so well (is there a jersey color that's close to grass color lol? Wonder
what the edge cases are).

~~~
halisaurus
It's alluded to in the article that outdoor fields and green jerseys are the
hardest. For outdoor fields the lighting is affected by the time of day, cloud
coverage, players moving along the field, and plenty of other things I'm sure.
It says they spent a lot of time creating a unique palette for the fields and
that domed stadiums were preferred by the engineers on site. Indoor lighting
probably reduced the shadows a lot.

~~~
baddox
I wonder if they use any tracking technology that looks at large groups of
pixels over time. If you could track a player as a closed shape moving through
space, you could mask out that player from the background even if pixels on
the player's uniform were identical in color to the field's palette.

------
digikata
I had a cousin who worked with Sportvision at the time. I recall he had
mentioned that early on, the support team for the line was comprised of
classic nerd engineers who couldn't care less about football. And ironically,
they had somebody's dream job of being sent to every single football game
being covered by the line tech.

------
chaz
> Hill liked the concept, of course, but Fox thought that the price, $25,000
> per game, was too steep. [...] Our last stop was ESPN and the reception was
> decidedly different.

So interesting to see that FOX balked at the seemingly low $25k but ESPN
didn't flinch and insisted on exclusivity for games they didn't even have
rights to. Telling of how the networks viewed themselves and each other, but
also how important it is to find the customers that are the true believers.

~~~
AznHisoka
Reminds me of those Excite and Yahoo idiots who refused to buy Google for a
cheap $1 million. Some people just don't deserve to be at the top.

~~~
bbwharris
But would google have ended up as google if they were bought at that price?

~~~
Goopplesoft
Cliche but, at least his, hindsight is 20/20

------
brucehart
When this first debuted, my grandfather (born in 1917) thought that they had a
piece of nylon material that was connected between the first down markers on
the sidelines. That's when you know a technology is good. It's so seamless
that it never occurs to a layman that it might even be possible.

------
ksenzee
I love this. I've spent a lot of time between plays imagining just what kind
of engineering is behind that magic first down line. Of course, I should have
known it came down to this: "Our engineering team... worked around the clock
through the summer..." Sometimes I wonder if there is any such thing as a
correctly scoped project that can be completed during daylight hours on
weekdays.

~~~
chrismcb
Sometimes there are just hard deadlines you can't do a whole lot about. In
this case, it was pretty much hunker down and get it done in 4 months, or wait
another year. For a fledgling company, that extra 12 months was probably the
difference between success and failure.

~~~
RogerL
Yes, oddly enough they don't reschedule the Super Bowl for us if a job is
going to take a few extra days.

------
alelefant
His mention of the glowing hockey puck made me realize that I haven't seen a
glowing puck for awhile. Granted, it may be that I haven't watched hockey as
much as I did when I was a kid, but I certainly didn't see it last year when I
watched some games.

When I did see the glowing puck it made me feel like I was watching NBA Jam on
Ice.

~~~
frenchman_in_ny
Not only was it a (blue) glowing puck, but it would also make a red-orange
streak on screen when there was a slapshot (er... "one-timer").

[https://en.wikipedia.org/wiki/FoxTrax](https://en.wikipedia.org/wiki/FoxTrax)

~~~
icambron
I remember how unpopular it was when it came out. It made a lot of sense to
try; not being able to follow the puck around really was a common complaint
about hockey, and the NHL, as ever, was trying to expand its base. But if you
watch even a fair amount of the sport, following the puck (even when it's out
of sight against the boards) is really easy, so for the majority of the actual
audience, it was distracting and unhelpful. To add to it, the implementation
itself was corny and overstated. So part of it was problem was that it traded
power users' comfort for new user friendliness, and part of it was execution.
Compare to the yellow line in football, which is both helpful to everyone
regardless of football watching experience, as well as simple and clean.

~~~
stormbrew
The thing is that most of the time the exact position of the puck isn't really
the most important thing. Where the players are and where they're going
to/from seems more important. Emphasizing the puck just distracts from the
actual game. I don't think it even helps newbies very much for that reason.

In terms of helping novices understand the game better, maybe visual
indications of the zone the puck is in could be interesting, or direction of
play so it's easier to understand off side and icing calls would be more
helpful.

~~~
icambron
> Where the players are and where they're going to/from seems more important.

Another way to put that is that you can tell where the puck is based on the
behavior of the players, and that's what you're actually watching. I'm only a
casual fan, but I think I'd know where the puck is even if it was actually
invisible.

------
BioGeek
The latest sport where Sportvision used this technology is sailing, called
LiveLine there. During the last America's Cup [1] they used it to show the
starting, finish and boundary lines, distance between the boats,...

[1] [http://youtu.be/r0LH5cCuc_4](http://youtu.be/r0LH5cCuc_4)

~~~
snowwrestler
That was an incredible use of the technology. I'd watched sailing on TV
before, but for the first time I could actually see and understand the scope
of the race and the strategies of the skippers.

------
AndrewKemendo
I have always thought that it was a complex computer vision problem that made
this work, very interesting that it was more than just software and makes
sense.

I am tempted to think that this was the first widely adopted (and accepted)
instance of augmented reality. Very cool in that context!

~~~
RogerL
A lot of this has changed since then, of course. No grass swatches anymore! We
have a lot more computer horsepower to throw at the problem now, plus a lot of
gained experience.

We have a bay in our lab with a to-scale football field where we can run
indoor tests, but on the field it is computer vision that is doing it all. The
cameras are instrumented to get attitude, zoom, and so on (we have to modify
the lens to get the accuracy we need), and then of course you have to
compensate for lens distortion. If you look closely, the "line" is not a line,
but a curve. If we drew a line it would look stupid, and not reg well in most
parts of the image.

~~~
AndrewKemendo
Very cool. Are you guys doing statistical filtering at all or is it strictly
palette matching? Seems like your data inputs are specific enough that you are
trying to get where you don't need much buffer on either side of the
wavelength.

~~~
RogerL
I will have to ask. I don't work with this technology, and haven't touched the
code. I work in computer vision for baseball, with a completely different set
of challenges.

~~~
kanamekun
Is an automated "yellow box" to show the strike zone ever going to happen? I
feel like that would have a huge impact on the viewing experience!

~~~
unclebucknasty
I dunno. I kind of like trying to judge it visually and waiting for the ump's
call.

I can also imagine MLB having similar concerns as the NFL when, for instance,
they insisted on removing the line as the ball is being spotted. That is, if
they are showing the strike zone and the ump's call seems inconsistent, then
it wouldn't be a "good look" for the sport.

------
RogerL
I work here, if you have any questions...

~~~
KnightHawk3
I don't actually know what this yellow line is, probably because I am from
Australia. Have you got a sample video for it?

~~~
namityadav
The yellow line in the 3 to 5 seconds mark in this video:
[https://www.youtube.com/watch?v=QtxoYaMebF0](https://www.youtube.com/watch?v=QtxoYaMebF0)

~~~
dhekir
Why is it so hard to find a single HD video with this line? I searched for
'football "yellow line"' and filtered only HD videos, and none in the first
page actually showed some game footage with the line. The same for 'football
"first down line"'.

Is there no HD footage for these events? Or do even short extracts get removed
due to copyright issues?

~~~
snowwrestler
You won't find it with those search strings because the American viewer
(presumably who is posting those videos to YouTube) takes it totally for
granted.

Try searching for just "first down" with football, or viewing some game
highlights on NFL.com.

~~~
clarky07
I was about to say the same thing. You wouldn't even need to search first
down, just watch any football footage from the last decade. The first down
line is going to be in pretty much every shot. Note that it isn't always
yellow. Sometimes it's orange, and there are also other lines, namely the line
of scrimmage, that are marked as well.

------
taftster
Here's a decent youtube video that basically shows the tech in action.
[http://youtu.be/Vh9af_gXxlM](http://youtu.be/Vh9af_gXxlM)

------
alistairSH
The yellow line really is amazing from a "user interface" perspective. It's so
natural that I had forgotten there was a time when it didn't exist. And, as
mentioned elsewhere in the comments, it seems everybody has an old uncle that
swears it's just nylon between the down markers (or paint, or something else
that isn't "magic").

------
beloch
"Gepner, a TV production expert, engineering guru Stan Honey and I (the
business person) -- had been inspired by Stan's invention of the glowing
hockey puck technology the previous year for Fox, technology which we licensed
upon leaving the company in exchange for an equity interest for News Corp."

This article opens in May of 1998, just one month before the final use of the
glowing puck in June of 1998. By this point everybody in that meeting must
have known how hockey fans had reacted to the glowing puck. The interesting
thing is that they bravely forged ahead anyways!

Execution may have been the key difference in fan acceptance. Fox made the
puck into a giant glowing and color-shifting comet that practically screamed,
"PEW PEW PEW!!!". These guys made a simple line that was meant to look, as
much as the technology would allow, like a simple chalk line on the turf that
some poor slob had to create (while erasing another) after every down. Would
we still have puck-following tech in the NHL if Fox had been run by people
with even the tiniest amount of taste or restraint?

~~~
eropple
I think the glowing puck would have still died when high-definition TV came
along. I think HDTV saved the hockey viewing experience, because it made it a
whole bunch easier actually watch the game. The problems solved by the glowing
puck then stopped being problems.

------
morgante
Impressive technology, but perhaps someone with more football knowledge could
explain why they don't just paint lines on the field?

~~~
ken_railey
It represents the "line of scrimmage", which is where the ball starts at the
beginning of each play.

~~~
chrishas35
The yellow lines marks the line to gain for a first down, not the line of
scrimmage. Some broadcasts are showing the line of scrimmage now usually as
red or blue IIRC, but yellow is used across the board for the line to gain.

------
EEGuy
Preceding his accomplishments at Sportvision, Stan Honey created, in 1985, an
automobile navigation system showing your car's position on a moving map.

This was before Google Maps, before GPS, and before digital maps of cities.
You couldn't just go out and buy them in 1983; Stan's company, Etak, had to
create them itself [1].

Per [2], here's a description of the the hardware platform:

    
    
      The original Etak Navigator was a specially-packaged Intel 8088-based
      system with 256K RAM, 32K EPROM, 2K SRAM, and a cassette tape drive
      on which digital maps and some of the operating system were stored.
    

The 8088 has a 16-bit address bus, thus can directly address only 64 K. Must
have been some bank switching to get to the full 256 K RAM. This for all the
code and geodata.

edits for formatting

\---------

[1] Source: Personal conversation

[2] [https://en.wikipedia.org/wiki/Etak](https://en.wikipedia.org/wiki/Etak)

~~~
bitwize
The 8088 can address 1 MiB with a somewhat wonky segment:offset addressing
scheme.

------
Gracana
The few times I've watched football, I've been fascinated by this technology.
I imagine football fans take it for granted at this point, but I just want to
say "wow, are you SEEING this!?" For a live broadcast to be augmented in that
way is damn impressive.

------
cweiss
I'd seen an interview[1] elsewhere with, I believe, the founder of SportVision
where they indicated that their primary source of revenue now is selling the
vast amounts of telemetry data they collect during events.

His example was of the MLB, SportVision has high-resolution data on every
single pitch, the MLB teams are paying through the nose for that data to run
through their analytics. SportVision indicated that, based on ball trajectory,
they often know more about the state of a given pitcher's health than the team
or the pitcher themselves.

[1] [http://www.sportvision.com/media/hank-adams-
presshere](http://www.sportvision.com/media/hank-adams-presshere)

------
seanhandley
American Football.

~~~
oneeyedpigeon
Exactly. Across the world, "football" usually refers to soccer; please have
some consideration for the non-USA audience.

~~~
dragop
Strictly speaking that should be association football, although everyone calls
it football outside the US of course.

"Football" existed before, but simplifying somewhat, just about all the modern
games of football derive from rules codified at British private-education
schools in the 19th century. So for example, rugby football originated at
Rugby private-education school, which then developed into American football.
Off the top of my head I can think of association football, American football,
Canadian football, rugby football, Australian rules football and Gaelic
football.

------
jccooper
I was hoping they'd get into the later patent struggle and competitive
technology. (But I guess it's really a piece about a single interview someone
manager to score.)

Another company in the field (PVI) at about the same time used vision-based
technology to similar effect. They got into a mutual patent infringement
battle and ended up settling by cross-license.

I'm still trying to figure out why NBC (maddeningly) doesn't show the line of
scrimmage. At one point I saw some vague report that NBC uses the third
company in this space (SportsMEDIA) and that company has a license to the
patent for the first down line but not the patent for the line of scrimmage.
Which sounds silly and plausible at the same time. However, they advertise
line of scrimmage as available: [http://www.smt.com/products/smart/line-
visual-insert-publish...](http://www.smt.com/products/smart/line-visual-
insert-publisher)

It is probably a policy decision based on a 2008 in-game controversy where the
line of scrimmage was a yard off and one of the broadcasters (Madden) was
using it as a basis for evaluating the call, and didn't know that the
officials can't see the line.

In either case, they need to deal with it and get the line of scrimmage back
in. It's a very annoying omission.

------
catshirt
i'm not a huge football fan but i've always really loved the new technology
they bring to the show. the yellow line has always impressed me, and the fly-
by cameras they introduced are really cool too.

i think i've seen something similar, but it would be cool to have full on
video-game style indications. player markers, etc.

------
nikcub
Interesting from a startup perspective is that they raised a lot of money
_first_ and then tried to figure out if they could implement the idea. This
definitely is an idea where if it isn't being developed in a lab at a large
corp or by government grant you would need to raise money for.

------
shekhar101
This is amazing. I only had read about it but this one gives a first hand
perspective. Hats off to the team! Can someone point to the algorithm/exact
technology used in this? Would love to read more about the technology.

------
logn
I can't see how exclusivity into the playoffs was a mistake. ESPN believed in
the technology and they wanted to punish their competitors. I think the
creators of the yellow line should be proud to reward ESPN for their foresight
and take part in enjoying the Emmy together as opposed to sharing the
limelight with another TV network who never believed in the company.
Financially, I would think you're in better bargaining position too: well,
this year the line is $30K/game since it's proven to work and you don't need
to do R&D work like ESPN did with us.

------
AznHisoka
I always thought it a neon lights in the actual football field.

------
tn13
That is not football. It is Egg Shaped Handball.

