
Controllable Video Sprites That Appear Like Professional Tennis Players - tosh
https://cs.stanford.edu/~haotianz/research/vid2player/
======
gretch
As a tennis player, I think this is super impressive.

Their models actually do a decent job of replicating true tennis strategy, and
as they pointed out, even account for the quirks like the left handedness of
Nadal.

However, it's still a bit unrealistic due to the lack of full data.

There's 3 things that make a tennis shot what it is: placement (covered in the
video), pace (speed of ball), and spin (rpm and direction of spin). In this
method, they only use placement. Probably because pace and spin data don't
exist at this scale.

But there's a big difference between a slice, flat, and top spin shot to the
same placement on the court, and it directly affects the return shot. For
example, it's a very common and 'safe' play to return a slice with a slice

Would like to see the full extension one day

~~~
Austin_Conlon
The lack of full data is because tournaments keep Hawk-Eye
([https://en.wikipedia.org/wiki/Hawk-Eye](https://en.wikipedia.org/wiki/Hawk-
Eye)) data to themselves right?

~~~
gretch
Hmm from the look of it, it's not clear Hawk eye does everything.

Certainly it does pace. But from the article it's hard to say if it would
track spin - it's not clear to me that with the natural markings on the ball
(seam lines + ballmaker logo) and the lighting that exists in the stadium,
that would be enough fidelity for some algorithm to calculate the spin.

I think Hawkeye could be updated to easily do it though, and maybe if they
would be willing to draw 1 or 2 black dot markers on the ball in addition to
the natural markings.

~~~
ddek
It does track spin, they often show stats measuring rpm.

Additionally spin is a big part of the trajectory model - the Hawkeye cameras
are only 60fps, so the trajectory is interpolated.

However, I doubt it’s the cameras measuring spin. It’s more likely spin is a
free variable when they fit the trajectory.

~~~
glaberficken
Found this:
[https://commons.nmu.edu/cgi/viewcontent.cgi?article=1746&con...](https://commons.nmu.edu/cgi/viewcontent.cgi?article=1746&context=isbs)

Very interesting read, but I'm not knowledgeable enough to really give you an
accurate tldr, sorry

------
YeGoblynQueenne
Interesting work that demonstrates the benefits of the use of domain knowledge
[1] and of trying to understand a dataset, rather than throwing a bunch of
data at an and-to-end black box and hoping for the best. In particular, data
in the tennis point play domain is too sparse for approaches that rely heavily
on large amounts of dense data, like a neural network. This is good, old-
fashioned AI work and I mean that 100% as a compliment.

_______________

[1] Quoting from Section 10:

 _Finally, our work makes extensive use of domain knowledge oftennis to
generate realistic results. This includes the shot cycle statemachine to
structure point synthesis, the choice of shot selectionand player court
positioning outputs of player behavior models,and the choice of input features
provided to these behavior models._

A successful behavioural model of that kind is a contribution in and of
itself, useful beyond the task of simulation presented in the paper.

------
sarasasa28
I watched it and I felt like I was playing Mortal Kombat 3

technology really repeats itself

~~~
ekianjo
Mortal Kombat 3 (and previous ones) was just digitized sprites with no logic
in between animations, certainly nothing comparable to that.

~~~
reificator
I can very clearly see the sudden swap between (some) animations in this
example as well.

The MK analogy seems spot on.

------
jl6
It’s becoming increasingly clear that video by itself will soon no longer be
reliable evidence.

How long before it becomes a plausible criminal defence to say “the CCTV must
have been deepfaked, that’s not me”?

The indicator of reliability will be the chain of custody of the video data.

~~~
freyr
This is one of those technologies where I struggle to think of beneficial use
cases, but my mind fills with ways this could abused.

That's not to say it shouldn't be researched, or people more creative than me
won't think of beneficial use cases.

~~~
jaggirs
This whole post is a beneficial use case of deepfake-ish tech.

~~~
freyr
And that is what, exactly?

From the abstract: _" Our system can generate novel points between
professional tennis players that resemble Wimbledon broadcasts, enabling new
experiences such as the creation of matchups between players that have not
competed in real life, or interactive control of players in the Wimbledon
final."_

I don't believe academic researchers needs to justify their work by providing
real life applications. But if that were the extent of deepfake's utility, I'd
be underwhelmed.

~~~
pawelmurias
Interactive deep faking of real people in video games seems insanely
impressive at least to me. What more could you expect?

------
schemescape
I can’t get the paper to load, but I’m curious if they address the lack of
player shadows. Having shadows would have made the video much more realistic.

------
jcims
I think the player modeling is way more interesting than the visual
representation and could probably be competitively useful.

I wonder if you could use some of the recent advances in pose estimation to
rig a 3D model of each player rather than the rotoscope look of clipped frames
in the demo.

~~~
riidom
I can recommend "Two minute papers" for that kind of stuff. Here is an episode
about AI controlled basketball player movements:
[https://www.youtube.com/watch?v=pBkFAIUmWu0](https://www.youtube.com/watch?v=pBkFAIUmWu0)

~~~
jcims
Yes! have been a subscriber since nearly the beginning of the channel.

------
abalaji
I'm excited to see this applied to more popular video games such as FIFA,
Madden, and NBA 2k. The behavior modeling also likely has huge applications in
NBA film analysis and figuring out how a traded player might "gel" in a new
team.

~~~
totetsu
What about it's implications for things like ballet in a world where people
can't go to live venues and there is no money to support real dancers.. Or
music videos that already have auto-tune singers.. why not generated backup
dancers..

~~~
Animats
_why not generated backup dancers.._

It's been done.[1]

[1] [https://youtu.be/t9VYYhX3P1Y](https://youtu.be/t9VYYhX3P1Y)

~~~
y2bd
I’m not sure I understand the context of the video you linked, are you saying
it’s not real?

~~~
Animats
LiveLeak said it was CG.[1] But it may be real and the article may be fake.

[1]
[https://www.liveleak.com/view?t=9SDBX_1526444350](https://www.liveleak.com/view?t=9SDBX_1526444350)

------
symplee
I wonder if players will be able to use this to prepare for opponents. For
example, knowing where to hit and "seeing how they typically react" and then
predicting where they'll most likely to return the ball.

If so, this could be expanded to other sports, maybe even team sports, where
you can test set plays against the simulated defense.

~~~
chrisco255
Why wouldn't they just use actual replay footage for that and not some
generated sprite?

Simulated match ups have been done on pro sports video games for years. But
they're not that useful because a model is not reality.

~~~
renewiltord
Presumably:

* Hard to find all replays against that shot

* Time consuming to review

This would work as compression over that space. However, I don't watch tennis
and know nothing about sport. This may be an insignificant improvement.

------
woliveirajr
the title doesn't do justice: it's about the syntheses of tennis players in
generative video.

~~~
freyr
Synthesis of tennis player sprites.

------
jacobush
This is what I imagined video games would look like in the future... looking
from the 80s. Hi-res background, but static. Awesome life like sprites, but
basically the same video games we had on the C64.

It was somehow very soothing to watch that video. It felt like someone was
telling me a bed time story about a different brighter future that never came
to be. Very "Back to the Future".

------
Eduard
Video: [https://youtu.be/GnZUIuOzgQc](https://youtu.be/GnZUIuOzgQc)

------
tootie
This is amazing. I played this arcade game like 30 years ago that projected 3d
video that you could control. But instead of generative, it was a bunch of
little clips spliced together so you tap shoot and it plays the shoot clip.
Had lots of awkward seams but it was still really impressive. Now we get the
real thing.

~~~
robin_reala
Time Traveller?[1] It used a pepper’s ghost approach.

[1]
[https://en.wikipedia.org/wiki/Time_Traveler_(video_game)](https://en.wikipedia.org/wiki/Time_Traveler_\(video_game\))

~~~
daniellarusso
Could also be Holosseum?

[https://en.m.wikipedia.org/wiki/Holosseum](https://en.m.wikipedia.org/wiki/Holosseum)

I remember my local AMC 6-plex had one.

------
cm11
My head gravitates towards how visualization makes the behavior modeling more
palatable. The incredible technical feat on display is the video rendering,
not so much the behavioral modelling, but it seems conceptually
straightforward to add increasingly better "sabermetric" analysis to control
the player's choices.

The data might be just as “productive” as a spreadsheet or formula to inform
play, but it requires someone with a more specialized skill set to translate
its meaning. The HCI design, for lack of a better word, in rendering the data
visually makes it not only more entertaining but easier to "see"—for
mainstream users, pro players, or almost anyone. Design makes things visible.

------
goalexboxer
I was thinking at exactly this idea for over 2 years.

I even started with Python & OpenCV for basic background extraction, and as I
expected, the edges for the players are imperfect. But still, the result is
very very promising. I'm so glad someone did it.

Why I was thinking at it is because the end goal would be to apply frame
matching & transitioning to football. Tennis is the easier task, the camera
angle is almost fixed.

But from this result, to football, we aren't far. Even an approach based on AI
+ some human intervention would suffice.

~~~
goalexboxer
Part 2: Even though Fifa / PES are doing great things, no video game can match
the player personalities and add specific / individual animations for
everyone. By using the already recorded matches, we would have something very
authentic in terms of how players behave on the field.

------
d1str0
Yes but can it shoot a ball at a line judge’s neck?

~~~
Austin_Conlon
I’m sure Novak Djokovic would like to simulate an alternate reality where that
went slightly to the side, he didn’t get disqualified as a result, and then he
continued his undefeated 2020 run.

~~~
skocznymroczny
So in your universe it hit the judge and he got disqualified? Oh no, I jumped
into another reality again.

------
guram11
watching those realistic video sprites playing reminds me a lot of the old
tennis game on NES, it really makes you wonder how far and bizarre this
technology can become?

~~~
goatlover
Nick Bostrom might say to the point where we can no longer tell what's real.

------
andsens
I want this real-time generated on endless-tennis.tv

