
Remastering Star Trek: Deep Space Nine with Machine Learning - donbox
https://captrobau.blogspot.com/2019/03/remastering-star-trek-deep-space-nine.html
======
CharlesW
ST:DS9 was originally shot on film like ST:TNG, but the market performance of
the TNG remaster makes it very unlikely that DS9 will receive the same
treatment.

 _" Essentially, for the HD release of _Star Trek _, all people had to do was
scan each episode. For_ The Next Generation _, they would have to scan all
those original pieces of film and then edit together each episode again,
themselves. It’s more difficult, more expensive, and much more time-
consuming._

 _" Unfortunately, it wasn’t actually worth it. Sales of the extravagant_ TNG
_remaster—original retail price $118 for just season one—failed to reach CBS
and Paramount’s expectations. A similar process would have to be done for
both_ DS9 _and_ Voyager _—and would actually be even harder. "_

[https://io9.gizmodo.com/the-detailed-depressing-reason-
deep-...](https://io9.gizmodo.com/the-detailed-depressing-reason-deep-space-
nine-and-voy-1791962332)

~~~
stronglikedan
> original retail price $118 for just season one

Well, there's your problem right there. Expecting three figures for a single
season of _anything_ is just pure madness.

~~~
mjevans
I think my shock price is roughly in the ballpark of 15 to 20 USD per season,
and a maximum of roughly 100 USD total for an entire show.

Yes, this means if a show lasted more than 5 to 7 seasons I expect a discount
on the remastered version!

~~~
devoply
I want an online version that will exist for at least 10 more years preferably
20 for $100 with the DVD copy, sold by one of the giants with the budget to
sustain that.

------
crazygringo
Wow, so this is _fascinating_. Some observations:

\- Many of the frames look like someone just applied a "sharpen" filter --
there's (as expected) no real new information, it's just sharper... so it
doesn't seem like a big deal, like I could do it in Photoshop trivially

\- But then there are a few spots where new details are truly seamlessly
added... the fireball in the spaceship explosion, and forehead wrinkles.
Stunning, they're absolutely seamless and believable, with detail that is
simply not there in the original... that's literally magic

\- But at the same time, characters in the background that are slightly out-
of-focus get oversharpened when they're supposed to be blurry, like it can't
tell when moderate blurriness is due to resolution or focus

Overall, I'm pretty shocked that the effect is so seamless across frames -- I
totally would have expected this to produce weird discontinuities in time, but
I didn't see any at all. I mean, this actually seems like it's already
production-ready to throw into TV's or VLC. Which is _crazy_.

I wonder how much of this is "general-purpose", or to what extent this works
on this episode because it's trained on this episode? E.g. the neural network
is learning from a close-up of the Ferengi face or spaceship, to apply
specifically to a later smaller version. Or to what extent this will work well
across TV shows, across actors, across genres, without prior training, or with
sparse training?

~~~
ch_sm
He’s using an off-the-shelf commercial app called "AI Gigapixel", which in
turn uses a process called image super-resolution. It is general purpose
(works for all kinds of images) and fairly thoroughly explored by now. hope
this helps

------
ericsoderstrom
I don't see a huge difference between the two. Mostly the Ferengi forehead
creases are sharper I guess, but unclear that the overall effect is much of an
improvement. Cool proof of concept though!

~~~
__david__
Yeah I made sure the video was in 1080p mode and it was mostly subtle at best.
Not only that there were several places where it wasn't subtle, but the up-
rezzed faces had a super fake photo-retouched/cgi look to them. They had
crossed the uncanny-valley and didn't look like real humans.

~~~
eropple
I thought that faces were mostly fine. The spots that didn't look right, to
me, were when there was a face out-of-focus in the front of the frame (the bit
where Sisko's talking to that dude). I wonder if the network recognized it was
"supposed" to be out-of-focus, which it is, but then wasn't able to do any
edge cleanup to make it more consistent.

------
jaimebuelta
Interestingly, there's a project for a documentary about DS9 that's close to
completion.

[https://www.indiegogo.com/projects/what-we-left-behind-
star-...](https://www.indiegogo.com/projects/what-we-left-behind-star-trek-
deep-space-nine-doc)

It has been delayed due to master the old footage (it's one of the last
updates)

I'll leave this work it as a comment on the project.

~~~
themagician
I had no idea about this!

How did they manage to get the entire cast, EXCEPT Avery?

~~~
sdrothrock
Avery Brooks has distanced himself from DS9 intentionally; he considers that
part of his career done.

------
magduf
This is interesting, but it's still not the same as a proper remaster. You
can't get information out of SD video that wasn't there before with upscaling,
so the algorithm is just interpolating. It's too bad that Paramount is so fan-
hostile; they could just get the original film footage (assuming it was shot
that way; ST:TOS and ST:TNG were, which is why we have excellent remastered
versions today), scan it in as raw video, let the fans have at it, and they'd
make the high-def FX for free.

~~~
neuralk
No, it is not just interpolating. The underlying algorithm uses machine
learning by applying a trained deep neural network. So there is value added
besides a mere upscale.

You're ultimately right, though, and that a true HD is only going to come from
the raw film content. What the neural network gives us are essentially
plausible higher-res hallucinations.

Edit: as per the other comment, if the original exists only on video and not
film, perhaps this is the best we're going to get.

~~~
hinkley
That's still interpolating, by the definitions I know.

The main difference here is that the interpolation algorithm on your TV is
online. It's handling 30 frames per second, over 9 million pixels per second.
Doing the interpolation offline (ahead of time), you can take as long as you
want, look at multiple frames to try to make better guesses, try multiple
things and use some fitness measure to pick a winner, even a frame or a pixel
at a time.

It's still interpolation.

~~~
pbhjpbhj
No, if I interpolate a sequence 200, 400, 600, ... I might get 200, 300, 400,
500, 600, 700. I've not added info. If I look at real world situations and
find that whilst the figures fall at the even hundreds it's more realistic
that they fall in a range from 20 to 30 points below the hundred on odd-
hundreds. Then I have added information, albeit statistically, and the
resulting sequence like 200, 287, 400, 475, 600, 672 is no longer raw
interpolation.

In this case they're using machine learning to add additional information
about textures that isn't in the footage broadcast. They can add frames by
interpolation, but the ML texturising and detailing is not interpolation.

Starting with a blob, if you interpolate you get a smoother blob, with this
process you get a more structured figure.

~~~
roywiggins
It's more like hallucination than anything. You're just forward-projecting
your assumptions on what things _ought_ to look like and hallucinating detail
that just isn't there.

It can still look nicer than naive upscaling though.

------
Balgair
Pro-tip: You have to change the YouTube setting so that you are watching in
1080 HD, not the 360p that it can default you too.

You know, I think there is some real promise here!

With some HD remasters, you can start to really see the makeup, the little
pores in everyone's skin, the smudges and uncleanliness of real film-making.
Film-makers and directory choose the lighting and the focus with the end-
product in mind. They _know_ that the screen won't capture certain things and
so they know where they can skimp and save [0]. When you re-master it, you're
going against the 'vision' of the directors. Not in a _big_ way at all, it's
very subtle. But it's still there.

With ML techniques, you get the 'idea' that the director was going for,
without seeing all the screw-ups that they knew they could get away with. It's
crisper but the idea and vision are the same [0].

Peter Jackson's recent 'They Shall No Grow Old' is another great example of
using ML too. In that case it was to preserve the old WW1 footage, bring it's
frame-rates up to modern 24 fps, 3-D it, and colorize it. The results are
literally breath-taking. Personally, I gasped when it finally hits; it's that
good. Not to geek out too much here, but Jackson is _literally changing
history_ with that film. He changed the way we all view old footage, as
something all herky-jerky and grainy, to something that is modern and real.
Those 16 year old child-soldiers become _real people_ again.

Though Jackson's work is a lot different than this effort, I think we all know
that ML and the movies are here to stay. It's relatively cheap to update,
takes little time, and be profitable (Remember the Disney Vault gimmick?). How
long will it be before Chaplin's 'City Lights' is ML'd and remastered into 24
fps, 3-D, color, and with sound? Maybe 5 years?

Hell, I'd pay to see the best of old cinema brought back to modern standards
like that.

[0] I know no film-makers, this is supposition.

~~~
dractori
There's a Criterion Collection blu ray of City Lights that I would imagine was
done very well given their reputation and the reviews
[https://www.criterion.com/films/27558-city-
lights](https://www.criterion.com/films/27558-city-lights)

------
kriro
What is the licensing situation for remastering like? It would be a pretty
interesting competitive edge if a company like Netflix could improve their
entire library with some inhouse remastering magic. Or even a bargaining tool
for buying new licenses...sure, we'll add your stuff to our collection and
remaster it "for free".

~~~
tracker1
Would definitely get me more interested in handing over my cash... Though, in
this case, I think it would more likely be Amazon Video.

------
conception
People are using this same technique for old games -
[https://venturebeat.com/2019/01/18/machine-learning-is-
rescu...](https://venturebeat.com/2019/01/18/machine-learning-is-rescuing-old-
game-textures-in-zelda-and-final-fantasy/)

------
ocdtrekkie
I kinda hope that perhaps, if CBS doesn't find value in doing a TNG-level
remaster of DS9 and Voyager, that something like this might give them a "good
enough" version for a Blu-ray release. At the very least, one could imagine
using this technique where applicable to at least, massively cut down the
conversion cost in scenarios where re-cutting the scene would be particularly
cost prohibitive.

~~~
makerofspoons
I do wonder if having DS9 and Voyager available on so many streaming platforms
right now cuts down on the incentive to remaster them. I'm not sure if a
remaster would be enough of a draw to CBS All Access if you can already watch
all the episodes on the other major streaming services, but perhaps Star Trek
Discovery is successful enough they might consider it?

~~~
aetherspawn
I don’t think it’s fair to call ST: Discovery a part of the franchise at this
point.

It’s this bizarre smodge of minority group agendas, dark horror-like scenes
and modern political ideas disguised as the family friendly Star Trek that you
could watch with your kids that we all actually loved.

And are the plot lines intelligent? Do they make you think about interesting
(non overdone ie non-left) philosophical issues? Not really, and they’re
strung along in an awkward and long story arc that’s pretty dull and
predictable and takes away from the per-episode storytelling :/

~~~
ocdtrekkie
I distinctly recall every flavor of "it's not real Star Trek" when Enterprise
came out, and people still whine about the story choices in Voyager today.
Heck, some people who were fans of the original series would be loathe to call
TNG "real Star Trek". In fact, having people who complain it isn't real Star
Trek might be the best proof that Discovery is, in fact, real Star Trek. ;)

Star Trek has always been incredibly progressive/left-leaning. The Federation
is a communist utopia where everyone has everything they could ever want for
free. If you talk to people on the original series, plenty of them will talk
about things that they _wanted_ to do, in terms of gay characters and the
like, but that they couldn't get permission to actually _air_. But they have
always pushed the envelope of being about as far left as they can get away
with from a business side perspective.

~~~
krapp
Yeah... this is a series that had the first interracial kiss, a black woman
professional during a time when black women could only be nannies and
servants, and a Russian serving openly amongst Americans during the Cold War.

I've seen a lot of complaints about "liberal" and "feminist" agendas being
pushed in Discovery. It baffles me. Do such people not know what the
Federation is about? They're literally social justice warriors in space.

~~~
Chazprime
The funny thing is that the concept of inclusivity has always been at the core
of _Star Trek_ , so if anything, _Discovery_ is simply keeping with the status
quo.

------
cpeterso
"it took me about two days to get everything extracted, upscaled and put it
back together in a way that was pleasing. This resulted only in the first five
minutes of the episode being done."

Maybe someone at Google can use spare cloud time to remaster the whole series
as their 20% time project? This remastering process would be easily
parallelizable.

~~~
cpeterso
I forgot to add a :) because remastering Star Trek is clearly not Google's
business, but it could be _someone 's_ business: video remastering/upscaling
services for Hollywood. There's a lot of popular old film and TV content just
waiting to be sold again to fans on Blu-ray or steaming.

------
skybrian
I see a lot of comments from people getting excited about superficial quality
improvements, and I'm mystified. It's nice I guess, but the story and acting
are unchanged by this sort of thing, and isn't that what matters to Star Trek
fans?

~~~
luckydata
the low visual quality is becoming more and more visible as TV screens get
bigger and better. It would be great to have those old shows in a quality that
fits our new screens, that's all.

------
Jerry2
I recently tried to rewatch Babylon 5 but the resolution was terrible and it
looked bad on my TV so I gave up pretty quickly. I hope someone does the same
treatment to Babylon 5 episodes!

~~~
rosser
Unfortunately, that's probably the "last best chance" for a not-awful watching
experience. At some point after the show wrapped, they lost all of their 3d
models, so they will never be able to re-render a single ship.

EDIT: I didn't mention fan models as a possible approach because this _is_
WB/PETN we're talking about here. They aren't Paramount-dumb, but they seemed
take that as a challenge, at least _vis à vis_ B5.

~~~
coryrc
There weren't enough polygons on them to look good at higher resolutions;
they'd really have to be redone anyway.

~~~
Zardoz84
you are pretty wrong . Watch the original meshes at HD resolution :
[https://youtu.be/uHAuK_lDkk0](https://youtu.be/uHAuK_lDkk0)

~~~
coryrc
Did we watch the same video? It looks like a 2002-era video game when rendered
at 1080p. It does not realistic at all; it is nothing like improving a film
video of a model (film is basically ~5k equivalent and a model is..
infinite?).

~~~
boomlinde
I agree that it doesn't look particularly good. The question is whether it's
due to a lack of polygons.

------
MayeulC
As I see more of these machine learning applications, where a neural network
is trained on a dataset, then applied to a similar dataset, I wonder how well
it would help if the network was trained on part of the new dataset as well,
as some kind of "specialization".

One could take a generic "AI Gigapixel" net, and retrain it on some of the
newer Star Trek content, for which an upscaled/remastered "ground truth"
exists. My guess is that it would help a lot with features that are specific
to the Star Trek universe.

Taking this further, one could take the resulting samples, and "rate" them,
before feeding it back in the training engine. This would make some kind of
"adversarial" feedback loop, but instead of the GAN, humans are involved in
the loop (which could be used to train an adversarial network as well, that
said). My hope is that it would converge to much better results in a shorter
time, and with less input data.

My apologies if those are common concepts in ML. If so, I'd love to look at
some references that could further my understanding on these topics, and get
me to use the right terminology.

------
moultano
This seems like it would really benefit from a model trained specifically to
upscale video. There's a lot of information you can get from the structure
between frames that you can't exploit with a still image.

------
modzu
im not discounting ML being able to do this better. kudos to the op. but for
fun i tried it using conventional methods (NR+sharpen); similar results but an
order of magnitude faster. you be the judge:

[https://pasteboard.co/I6tvYmP.png](https://pasteboard.co/I6tvYmP.png)

------
AcerbicZero
On the one hand I'd love to see them remastered and touched up, but on the
other hand, I originally watched them on a ~27 inch, old mid 90's TV....so the
visuals aren't really the part which hits my nostalgia nerve.

------
entropicdrifter
Alright, now I want to do this with Babylon 5 since it seems like the HD
remaster from WB/Amazon petition has flopped. Heck, even just upscaling the
SFX shots from 360p to full DVD quality would be a worthwhile project.

------
nominated1
I'm curious as to why the author chose "AI Gigapixel". What made it a better
choice than say NNEDI3 or SuperRes or any other NN upscaler (there’s a bunch
of them out there)? Were others even considered?

------
dbcooper
Anyone compared this with MadVR?

------
bitL
Anyone knows how AI Gigapixel differs from DLSS? (outside Topaz Labs ofc).
Thanks!

------
pbhjpbhj
Are films/programmes now preparing for future re-rendering in any way?

------
craigmcnamara
Star Trek: Deep Space Nine aka, Cheers in Space!

------
microDude
Any Friends of DeSoto here?

------
sigi45
Holy shit, you did exactly what i wanted to do after i read about the final
fantasy results!11 :D

------
zackkrida
You may find the original blog post from the creator here:

[https://captrobau.blogspot.com/2019/03/remastering-star-
trek...](https://captrobau.blogspot.com/2019/03/remastering-star-trek-deep-
space-nine.html)

~~~
ukyrgf
The URL for this post should really be changed to that. It's nothing more than
a two sentence blogspam submission with a link to the original creator.

~~~
sctb
Done! Updated from [https://www.turingtribe.com/story/star-trek-fan-remasters-
st...](https://www.turingtribe.com/story/star-trek-fan-remasters-star-trek-
deep-space-nine-in-hd-using-machine-learning-yEHAXy7S7PxGZEQE9).

------
ChrisArchitect
[https://news.ycombinator.com/item?id=19406060](https://news.ycombinator.com/item?id=19406060)

