Hacker News new | past | comments | ask | show | jobs | submit login
Remastering Star Trek: Deep Space Nine with Machine Learning (captrobau.blogspot.com)
391 points by donbox on March 21, 2019 | hide | past | favorite | 195 comments

ST:DS9 was originally shot on film like ST:TNG, but the market performance of the TNG remaster makes it very unlikely that DS9 will receive the same treatment.

"Essentially, for the HD release of Star Trek, all people had to do was scan each episode. For The Next Generation, they would have to scan all those original pieces of film and then edit together each episode again, themselves. It’s more difficult, more expensive, and much more time-consuming.

"Unfortunately, it wasn’t actually worth it. Sales of the extravagant TNG remaster—original retail price $118 for just season one—failed to reach CBS and Paramount’s expectations. A similar process would have to be done for both DS9 and Voyager—and would actually be even harder."


> original retail price $118 for just season one

Well, there's your problem right there. Expecting three figures for a single season of anything is just pure madness.

I think my shock price is roughly in the ballpark of 15 to 20 USD per season, and a maximum of roughly 100 USD total for an entire show.

Yes, this means if a show lasted more than 5 to 7 seasons I expect a discount on the remastered version!

I want an online version that will exist for at least 10 more years preferably 20 for $100 with the DVD copy, sold by one of the giants with the budget to sustain that.

Back in the 90s, Trek episodes were released in the UK months before they were on TV. The price was typically £12-14 per video, which had 2 episodes on. That puts a season at £170. Inflation wise that's about £320 now, or $422 a season.

How many people bought an entire season for that price is another matter, but that was the asking price.

I did :) all of TNG Season 3-7 .. all of Voyager until DVDs came about

Voyager season 1

  US release: Jan 95
  UK VHS release: Jun 95
  UK TV release: Sep 96
I find it so amusing that Americans complain about having to pay $10 a month for Discovery

This. I bought the first trial disk for Star Trek TNG, which had a couple of episodes that had been redone in HD. They looked great, but when I saw the price for each season, I ended up not purchasing them.

That and remastered(or so I thought... I think they're in 4:3 aspect) episodes being available on Netflix makes it really hard to justify paying at all for discs.

The show was originally shot for 4:3, that didn't change when it was remastered. Thank goodness. It was very impressive remaster, as shown in various comparisons:


The film that was shot couldn't simply be overscanned to 14:9 or 16:9. It would work in some scenes, but in others there were bits of the set/booms/etc visible almost upto the actual frame that was cut out. The only way to convert is therefore

1) Throw the top/bottom parts of the frame away

2) Stretch the frame

Neither is acceptable.

Agreed. Of course, there have been more than a few remasters that didn't bother to concern themselves with such trivialities. The Buffy the Vampire Slayer remaster is an example of how you can thoroughly botch framing[0] when converting between aspect ratios. At that point, you might as well see what else you can screw up as well. At the opposite end of the spectrum, David Simon wrote a really interesting--and even-handed--blog post[1] about HBO's remaster of The Wire that converted to 16:9. He points out examples of scenes that work out better in 16:9 and those that...don't. Even when the conversion is made in conjunction with the original filmmakers, it's still a different version of what was originally intended and necessarily involves compromises.

0. https://www.youtube.com/watch?v=F28XcxHxH6k

1. http://davidsimon.com/the-wire-hd-with-videos/

An update to [0] was posted in the last day or so. https://www.youtube.com/watch?v=oZWNGq70Oyo

Showing vampires in broad daylight without catching fire, for one thing. (Not applying blue filter to simulate moonlight, for shots taken in daylight but which were supposed to be at night.)

Wasn't that the HD terrible remaster, rather than the 16:9 SD terrible remaster?


Depends on the market.

Japanese DVDs/Blu Rays for Anime sell for upwards of $40 and usually only include around 4 episodes.

Worth every penny I paid for them.

If it's an issue of fixed upfront costs, they can literally do a kickstarter and it will get funded.

Not a bad idea for a franchise like Star Trek. Basically judge whether market demand is high enough by asking for the whole production price upfront. Wouldn’t work for most media, but would for a known product.

It'd probably get thrown out by the lawyers for various nebulous (though valid) reasons

"What if we go over budget?"

"Does this mean they 'own' any rights to the remaster?"

Isn’t that answered by the nature of kickstarter? If your lawyers aren’t offering solutions to implement your idea get new lawyers.

The lawyers have to protect the shareholders, else the executives picking the lawyers are being kicked out.

I doubt paramounts hurting for money. Imagine they want to do a kickstarter and it's only 800 for the series.

But you could take the raw footage and the edited video and recreate the edit lists (which they should still have btw). Or just scan each film frame, then match the video to film frame by frame and re-assemble the list of frames. All software done by one person. No creativity or judgement needed.

The bulk of the work is cleaning up each of the scans, re-framing the picture, balancing the color, and making sure the correct takes are used, rather than simply applying an EDL.

It's possible ML could help with a lot of that, but the budget just isn't there.

While it was shot on film, the effects weren't, each has to be created again, not just the external scenes like the Defiant shooting something, but scenes like Odo morphing

Here's a comparison picture of the DVD vs film quality though


That's an incredible difference. Obviously NTSC was going to lose a lot of color detail, but it also lost quite a bit of vibrancy. In particular the warm highlight lighting from offscreen to the right is barely visible in the DVD version, but on the film version it really pops; look at his sleeve, the shadow cast by his hand, and the contrast of the bottle with the background. The value contrast is similar but the color contrast is greatly muted.


On second thought, considering the shirts that Jake wore on the show, it's probably better for the world that we don't have the higher vibrancy film version.,,

But also, compare the color of Garak's hand to his head. The high-quality one brings out more inconsistencies that break suspension of disbelief.

That's because of the orange light. Looks close enough to me.

> Obviously NTSC was going to lose a lot of color detail, but it also lost quite a bit of vibrancy.

Remember that those who do a lot of video call NTSC "Never The Same Color"

As opposed to "People Appear Lavender"?

When I played a broadcast engineer helper for one of the British TV companies in the 80ies NTSC feeds that we would sometimes get from AP would cause a total freak out. We would tweak the colors on a test feed and pray that the live AP feed would be at least somewhat close. We would hit it on the nose less than 1/3rd of the time.

Star Trek: The Original Series had the opposite problem. They wanted their aliens to be different colors (e.g. green Orions) but:

The technician over at the film lab would receive the film every day and run it through the development solution. As the image formed on the film, he kept saying to himself, ‘My God, this woman is green!’ And so he kept correcting the film developing process in order to turn her back to normal skin color again!

"Imagine everyone’s surprise, upon viewing the developed film the next day, to find the actress’ face just as normally pink skinned as ever! There was no trace of green."


Wow. The film looks so good. This upsampling, on the other hand, is to my eyes so subtle of an improvement to almost be imperceptible.

It has artefacts. If you look at some near similar defined geometrys, they stay unmoving for split seconds. Siscos upper head collapses in on himself, when he moves..

I think I saw that on the DVDs themselves, or at least on netflix. I suspect the source material was heavily compressed

Incidentally, it appears most of the CGI models still exist, at least in the hands of the original modelers. http://trekcore.com/blog/2013/05/deep-space-nine-in-high-def...

Send out a call for volunteers. I guarantee you, you'd have a group of fans with the know how to do it. And they would do it for free.

I guarantee you, you'd have a group of fans with the know how to do it. And they would do it for free.

The problem is they'd devolve into various camps bickering about the "right" way to do it, creating their own mini standards bodies, and forming a massive internal bureaucracy of tribalism.

It seems the more passionate a group of people is about something, the more convinced they are that they know the right way to do things.

You just described a prior workplace.

And if the technology doesn't exist, they'll probably invent it too.

This is Star Trek we are talking about. Their fans are dead serious.

~Just run the colorquads through the main optical processor, then reverse the polarity, compress the lightstream in the warp field, and blit the output texels through the main deflector dish. That creates a virtual lightfield holo-display, using the starfield as the light source, with practically infinite resolution. Then all you need to do is point an image recorder at it, and downsample back to 1080p at 60Hz.~

Don’t forget to account for the temporal variance created by high levels of tachyon emissions.

You missed a TNG buzzword... Dilithium Crystals. :P

But the parent used the reflector dish, any good hack always utilizes the RD.

Ya, I can't think of any show with such a long lasting, rabid fan base. Just look at that recent "fan film" that CBS tried to sue over. Production quality was off the charts for a fan film.

It’s pretty crazy. If the franchise were somehow freed from copyright control I have no doubt that several near-studio-quality fan productions would pop up in no time at all. People love their Trek.

Axanar was pretty impressive, but I love what the Star Trek continues crew did. The love and respect they showed to the original material was very impressive.

So was the budget, hence the court case.

We are, we also tend to skew more towards the technology end of the spectrum than the average TV show fans.

The fans will come, but there are limits to how far even they will go to create a product that will be owned and sold by someone else. Having toiled to edit together the HD version, id be screaming mad to see it sold for 100+ per season. If it was made by fans it should be sold at cost.

FWIW, they failed to recoup their investment in TNG's remaster, selling at 100+ per season. By definition, that arguably means it was sold below cost.

(I bought individual TNG remaster seasons as they came out specifically to support the remaster effort.)

> FWIW, they failed to recoup their investment in TNG's remaster, selling at 100+ per season. By definition, that arguably means it was sold below cost.

It means they charged at a level where sales volume times net (after variable costs) price per unit was insufficient to cover fixed costs.

But whether that's because they charged too much (variable costs for the disc sets were a small fraction of the price; it's quite possible they would make more profit with higher volume and lower price), too little (maybe the people willing to pay as much as was charged would have paid even more), or either would have worked and they priced in an unprofitable valley, or neither would have worked is speculation.

I mean, currently the entire series is $126 on Amazon: https://www.amazon.com/Star-Trek-Generation-Complete-Blu-ray...

It makes sense to price high to get maximum returns from the users who must have it now, and then catch everyone else later with lower priced offerings.

Commercial sense.

It makes cultural, or creative, sense to release the footage at cost and let people remix and use it for whatever purposes.

The profit for the show was already made, it should be public domain by now.

That's a rather naive view apparently based on a consumer-level underestimation of the complexities of asset management and funding on a big movie/show.

I'd guess from a studio's POV the trade-off is between spending money updating an ancient show in the hope of making a little cash now and a little more from residuals, and making new shows which are more or less guaranteed to bring a profit.

But that aside, I'm not sure there's any way to make something like Star Trek public domain. Never mind the studio - the actors, writers, and producers will still be relying on residuals for continuing income.

And even if that weren't true - what exactly do you make public domain? You can't just hand out the original unedited film stock to anyone who asks for it. How about the scores for the music? Or the audio mix files? Or the EDL? Or the various revisions of the scripts? Props? Set carpentry - if there's any left? Wiring?

Reality: very few elements are digital files that can be copied/shared, even if you wanted to allow the public to copy/share them.

You make it all public domain. It's been over 20 years, well past the founding father's original intent for copyright. The perpetual copyright garbage that exists today is culturally toxic and isn't seen in any other field of work. Should builders get paid on buildings that keep standing? Highway workers on roads that they make? No? Then why the hell should creatives get special treatment? Look at the house of mouse and the crap they've pulled. The fact that you can't take something you've loved for over 20 years and make derivations on it is theft from the public in my eyes. People being born today most likely won't be alive by the time it goes into the public domain. Think about that.

You both make compelling points

>That's a rather naive view apparently based on a consumer-level underestimation of the complexities of asset management and funding on a big movie/show. //

The situation is that the studio could do work on the footage to make it releasable, then sell the processed footage for profit.

The argument is that the processing of the footage makes it too expensive to do this, the studio might make a loss.

So, all the IPR, releases and such need to be in order for it to be possible to release the processed footage.

It's possible there is contractual obligation preventing release of rushes and other unfinalised footage, but it seems highly unlikely. I'd imagine the studio have rights to publish anything, thus enabling "making of" and "blooper reel" type videos.

So, if it's possible to arrange the IPR for the processed footage, then it's close to certainty that there's no IPR limitation on releasing the unprocessed footage.

Meaning finances are the only remaining issue.

Sure, the studio may not want to spend the money up front to arrange release of the footage; but that is likely to be primarily to avoid fan works based on the footage from competing with their own outputs.

>You can't just hand out the original unedited film stock to anyone who asks for it. //

You could hand it to an archive for digitisation and give them rights to sell it at cost. Or, burn it. Or leave it in a canister to degrade until it's unusable.

>How about the scores for the music? Or the audio mix files? Or the EDL? Or the various revisions of the scripts? Props? Set carpentry - if there's any left? Wiring? //

Well, IPR aside, why not. If you're paying to keep the set in storage, why not give it away and save your storage costs and get on with making new sets that are going to be used?

Primarily however, we the demos should be taking the question of what to do with unneeded IPR away from the studios. Make copyright terms shorter to match patent terms.

No. It means they charged too much. The fans were priced out of the market and, probably, acquired copies elsewhere.

The Blu-Ray sales can’t be the real story. Surely they expected the long term value to come from the significantly added appeal of HD content for streaming services.

IAE, I can understand the reasons something upscaled is all we’re ever likely to see of DS9 or Voyager. Primarily that as soon as CGI became the dominant effects technology, the frequency of use went way up, and all those scenes would require non-trivial reconstruction.

I am sure the TNG remaster will eventually be worthwhile to CBS, due to streaming, re-releases, etc. It's an asset they will monetize for decades to come. But studios do not think long-term like that, and are unlikely to attempt it again if they don't think they'll make their money back for twenty years.

As I noted in another comment, many of the effects artists from DS9 and Voyager have their original meshes and scene files, and some have posted insanely high resolution files on their blogs. It would be a frustrating effort to gather all of this, but arguably, CBS already owns the works in question, so compensation for retrieving these would be minimal probably, and the quality of the work done for the series was more than adequate if re-rendered in HD.

Similar situation with Babylon 5. There some defenders using the original stuff but in HD quality. Even using the original textures keeps looking impressive. Sadly, Warner not have idea of the golden goose that they have.

Does anyone have a high-resolution picture of the comet at the beginning of the DS9 title sequence?

IIRC the phrase they used was "failed to meet expectations", which is very different from 'failed to recoup their investment'. The idea that you can sell a remaster for 100 a season aside the second one is a factual statement. The first one is arbitrary. Maybe they expected 100 million profit and only made 90.

If I understand correctly, you would still have to overlay text, voice over, sound, music, etc. That's still a lot of work.

And even worse is special effects.

A bigger part of the issue with these remasters tends to be the conversion to 16x9 which most people expect with HD. Even if you have the 'missing' out of shot data on the origin film there are often problems where set equipment, cameramen, etc are just out of the shot so it requires manual digital touch up to fix. It also can require re-rendering any CGI.

When I saw the submission title I was actually hoping re-creating the out of shot image data is what the ML was being used to do.

The remastered TNG box sets kept the same 4:3 aspect ratio as the original standard definition.

Some TV networks have aired a 16:9 version of TNG (as well as an upscaled Voyager and DS9) but they just zoomed and cropped. There wasn't anything smart going on.

The zoomed and cropped from 480p, i.e. making it even worse quality?

Basically, yes. The upsampling algorithms they use are pretty good though so it’s passable for most viewers. But obviously there’s no substitution for having HD quality at source

> they would have to scan all those original pieces of film and then edit together each episode again, themselves.

Once they have manually scanned all those pieces, surely they can let machine learning edit together each episode again?

How would you go about doing that?

I’m guessing by using image recognition of the original digital / DVD releases against the newly scanned hi-def video?

You'd have to match the framing and position, and clean up the frame.

Then try to color balance (when the original information isn't there),

If you get ML to do that, congratulations, you'll make a fortune in the production industry.

The hype around machine learning has reached a point where every time someone sees a moderately challenging problem they leap to some nebulous "ML" solution.

No, we've long had the technology to solve most of this. Clip correspondence is content based image retrieval across a database of keyframes. Matching framing and position is an image registration problem (feature correspondence). Color balance seems like an almost trivial problem if you've solved the other ones, because the information _is_ there -- just modify each channel to match the histogram in the lower resolution image [1]

The challenge is that earlier stages in the pipeline need to be robust to inexact matches, and we don't want to rely on absolute color or pixel position. But I don't think that should be insurmountable for a slightly creative implementor - use local variation in color, gradient descriptors, pull in the motion vectors for an additional channel, etc...

Sure, you could go down the deep learning path by trying to reduce scenes to bags of labeled objects and semantic actions but that's bringing a water cannon to a squirt gun fight, which only makes sense if Google is giving you a free water cannon.

I'd probably try it if I really thought there was a fortune to be made here, but it's such a niche application. When something hasn't been done yet, there's usually a good economic reason, unfortunately.

1. First Google result for "histogram matching color balance": https://blog.mapbox.com/color-balancing-imagery-with-histogr...

Cropping a video to match another, and adjusting the color balance to match, seems simple enough, compared to asking a computer to do it without a reference. In other words, "make X look like Y" as compared to "make X look good".

Of course, the quality won't be as good as something done by a professional, but the question is that since we don't have a HD DS9, whether the version produced by an automated system is noticeably better than what we have.

I think you overestimate how difficult that is and how much it's worth.

I've never done it myself, basing it on the cost of the machines that professionals do, and the cost of employing people to do it (both do it and make the judgement calls), you're probably looking at $200k for feature film quality.

If you're aiming for network TV quality you can probably do an episode for $5k though.

Of course that's from scratch. The trouble is that using the original video as your source will have lost a lot of data, and that means making a lot of judgement calls about what the scene is meant to be doing, so you're not much nearer.

I wonder if it would be worth it for long tail revenues from streaming? I can’t imagine the HD widescreen remaster of The Wire selling lots of new DVDs (it was completed in late 2014 [0]), but it made the show a lot more appealing for continued rewatching over the years.

0: https://davidsimon.com/the-wire-hd-with-videos/

> original retail price $118 for just season one

Where, precisely, did that number come from? It doesn't happen to include marketing, does it?

Also, this kind of stuff of course can only be profitable in the long tail. So what means failed to reach any expectations? If it gets a reasonable price it will probably constantly sell in low amounts for 10+ years.

Wasn’t worth commercially, but as a fan I’m really happy they did it.

They should crowdsource that work- start operation Turktrek. (Sorry i couldnt resist).

Wow, so this is fascinating. Some observations:

- Many of the frames look like someone just applied a "sharpen" filter -- there's (as expected) no real new information, it's just sharper... so it doesn't seem like a big deal, like I could do it in Photoshop trivially

- But then there are a few spots where new details are truly seamlessly added... the fireball in the spaceship explosion, and forehead wrinkles. Stunning, they're absolutely seamless and believable, with detail that is simply not there in the original... that's literally magic

- But at the same time, characters in the background that are slightly out-of-focus get oversharpened when they're supposed to be blurry, like it can't tell when moderate blurriness is due to resolution or focus

Overall, I'm pretty shocked that the effect is so seamless across frames -- I totally would have expected this to produce weird discontinuities in time, but I didn't see any at all. I mean, this actually seems like it's already production-ready to throw into TV's or VLC. Which is crazy.

I wonder how much of this is "general-purpose", or to what extent this works on this episode because it's trained on this episode? E.g. the neural network is learning from a close-up of the Ferengi face or spaceship, to apply specifically to a later smaller version. Or to what extent this will work well across TV shows, across actors, across genres, without prior training, or with sparse training?

He’s using an off-the-shelf commercial app called "AI Gigapixel", which in turn uses a process called image super-resolution. It is general purpose (works for all kinds of images) and fairly thoroughly explored by now. hope this helps

I don't see a huge difference between the two. Mostly the Ferengi forehead creases are sharper I guess, but unclear that the overall effect is much of an improvement. Cool proof of concept though!

Definitely make sure you switch YouTube's AUTO setting from whatever default it has to 1080P. For a lot of scenes there is a HUGE difference.

Thanks for this, I also didn't think they looked any different.

Yeah I made sure the video was in 1080p mode and it was mostly subtle at best. Not only that there were several places where it wasn't subtle, but the up-rezzed faces had a super fake photo-retouched/cgi look to them. They had crossed the uncanny-valley and didn't look like real humans.

I thought that faces were mostly fine. The spots that didn't look right, to me, were when there was a face out-of-focus in the front of the frame (the bit where Sisko's talking to that dude). I wonder if the network recognized it was "supposed" to be out-of-focus, which it is, but then wasn't able to do any edge cleanup to make it more consistent.

> didn't look like real humans

Deep Space Nine. Mission succeeded?

You need to have both full screened on a 1080p screen to actually compare the two. Also video quality is difficult to determine from still frames. Video compression is more about how it moves from 1 frame to another. You would want to watch an actual clip of both to see the difference better.

Yeah that's probably it. Was wondering whether half a screen of 1080p would be enough to properly show the difference

I honestly really don't see the difference beyond that.

A lot of the CG (ships, etc.) looks sharper to my eye. I didn't notice at first, though, because YouTube defaulted to 720p (and since it's a 1080p render downscaled to be side-by-side, you do need those pixels).

The creator says that 4K-vs-480p renders (that is, rendering the YouTube video at 4K) are on the way.

Same. It's certainly not objectively "amazing" (to use the author's own verbiage)

What? Everything is sharper in the 1080p version, the effect is far from subtle. Is it possible you did not load the 1080p version of the Youtube video?

I agree. Resolution is 1% of what makes a TV serial. 99% is storytelling, the setting, character, drama, emotion, and so on.

Interestingly, there's a project for a documentary about DS9 that's close to completion.


It has been delayed due to master the old footage (it's one of the last updates)

I'll leave this work it as a comment on the project.

I had no idea about this!

How did they manage to get the entire cast, EXCEPT Avery?

Avery Brooks has distanced himself from DS9 intentionally; he considers that part of his career done.

This is interesting, but it's still not the same as a proper remaster. You can't get information out of SD video that wasn't there before with upscaling, so the algorithm is just interpolating. It's too bad that Paramount is so fan-hostile; they could just get the original film footage (assuming it was shot that way; ST:TOS and ST:TNG were, which is why we have excellent remastered versions today), scan it in as raw video, let the fans have at it, and they'd make the high-def FX for free.

No, it is not just interpolating. The underlying algorithm uses machine learning by applying a trained deep neural network. So there is value added besides a mere upscale.

You're ultimately right, though, and that a true HD is only going to come from the raw film content. What the neural network gives us are essentially plausible higher-res hallucinations.

Edit: as per the other comment, if the original exists only on video and not film, perhaps this is the best we're going to get.

The neural network is applying what it "knows" about photos and inventing new data for the missing pixels. It's "creative interpolation" ;-)

> Edit: as per the other comment, if the original exists only on video and not film, perhaps this is the best we're going to get.

I don't think that's quite right, at least it doesn't jibe with what the DS9Doc people have been doing (which consists partly of remastering pieces of DS9 scenes):


I think the footage really was on film, but the issue was that it was composited with low-quality CGI effects, or something like that. So you can rescan the film, but you have to redo all the compositing (and probably with your own models because I'm guessing the original CGI didn't look that good). That's why a DS9 remaster is so expensive.

That's still interpolating, by the definitions I know.

The main difference here is that the interpolation algorithm on your TV is online. It's handling 30 frames per second, over 9 million pixels per second. Doing the interpolation offline (ahead of time), you can take as long as you want, look at multiple frames to try to make better guesses, try multiple things and use some fitness measure to pick a winner, even a frame or a pixel at a time.

It's still interpolation.

No, if I interpolate a sequence 200, 400, 600, ... I might get 200, 300, 400, 500, 600, 700. I've not added info. If I look at real world situations and find that whilst the figures fall at the even hundreds it's more realistic that they fall in a range from 20 to 30 points below the hundred on odd-hundreds. Then I have added information, albeit statistically, and the resulting sequence like 200, 287, 400, 475, 600, 672 is no longer raw interpolation.

In this case they're using machine learning to add additional information about textures that isn't in the footage broadcast. They can add frames by interpolation, but the ML texturising and detailing is not interpolation.

Starting with a blob, if you interpolate you get a smoother blob, with this process you get a more structured figure.

It's more like hallucination than anything. You're just forward-projecting your assumptions on what things ought to look like and hallucinating detail that just isn't there.

It can still look nicer than naive upscaling though.

I see what you're getting at, but it still seems within the definition of interpolation. From wikipedia

> ... interpolation is a method of constructing new data points within the range of a discrete set of known data points

Is there any evidence for this? Showing bad 480p DVD rips alongside 1080p upresed video isn't really a fair comparison -- comparing it to a real TV upscaler's output would be fairer. And honestly, even the unfair comparsion doesn't show a whole lot of benefits to me.

Its not upscaling. Its taking what it knows about other non ST pictures and creating new texture and information.

Right, but how much does it actually add over a decent upscaler?

are you saying this specific program, or the idea in theory?


I'd be surprised to find a group more obsessed with specifics of what is and isn't quality, it would be the anime community.

There are entire catalogue of overlay comparisons of different releases, encodings etc. [0].

Example: http://compare.bakashots.me/compare.php?setId=3896&compariso...

[0] http://compare.bakashots.me/

Interpolation looks bad enough without completely sandbagging its color correction. Why resort to that kind of nonsense?

Are you asking why did I change the exposure? Because I did, I was playing around. It was the comparison I had uploaded, I have a copy without on a different computer.

Wow! That's a bigger difference than I expected.

> You can't get information out of SD video that wasn't there before with upscaling, so the algorithm is just interpolating.

In the fully general case of arbitrary video this is true, but in practice it isn't.

You can gather information over time to do superresolution, and if you want to get super fancy you can build a world model (e.g. get more information about what an actor's face looks like from a close up shot, and apply that knowledge to less detailed shots).

I expect ML based upscaling to eventually produce some truly stellar results.

a combination of a world model, along with non-source enhancements.

i imagine upscaling the Phantom Menace podrace to 4k, and giving the model a bunch of NEW rock texture info to use to create new detail.

Kinda of an automated way to combine these two ideas.



What you dont want is every upscale to start looking homogeneous, so it would be best for a design team to specifically map old to new texture sets, giving each upscale a unique look.

I would expect multi-view rendered models of ships/stations etc to also help significantly as well... vs. re-rendering sfx, it would be upscaling via AI with knowledge of detailed views/models for the actors... training stills could go a significant way.

I was thinking particularly about the space battle shots, where you could provide 3D models of each of the ships, so that the software had the higher detail information, and would just need to match position, movement, and lighting.

AFAIK, STTNG has one other advantage. They used real models and DS9 mostly switched to CGI. Old CGI is harder to beautify than models.

Here is an example of what can be done to old Babylon 5 CGI by just rendering them in 1080p. https://www.youtube.com/watch?v=uHAuK_lDkk0

"They’re using the original Lightwave scene files for camera and model movement, lights, etc. It’s also the original 3D models and textures used on the show – and nothing has been updated in any way other than being rendered out at 1920x1080. It’s the raw CGI without any post work."

In theory at least, a neural network with sufficient training in the right domain should be able to add the slight imperfections and subtle light/texture cues our eyes expect to see, even to too-perfect 1990s CGI scenes. This would, however, require a network that can maintain continuity between frames so these details don’t appear to jump around.

Um, that's not what I read at all. What I read was that TNG's FX was all rendered with computers at the time, at NTSC resolution, so it was all useless when they did the remastering, and they had to make all-new FX shots. The place where the film was invaluable was for all the live-action stuff (i.e., anything with actors and sets, or on-location). It's much like what they did with ST:TOS, except TOS actually did film real models, but they were so awful by today's standards they redid them with modern CGI.

That's true of some effects, but in TNG the core spaceship / space station models were shot on film, and then composited with other effects like phaser shots.

In DS9 / Voyager as I understand it, they transitioned from using models to directly generating the whole shot with CGI at NTSC resolution. See https://memory-alpha.fandom.com/wiki/CGI#Acceptance

This means that in TNG they just had to recomposite the film with a newly created high-res phaser shot; whereas for DS9/Voyager they would have to recreate the whole shot in high-res CGI.

while I can't disagree with your conclusions, I would add that some of those models were not necessarily built with HD in mind, and could look just as crummy as old CGI when viewed in very high definition. Not all, of course, but I imagine experienced directors and model-builders would have constructed it for the duties it was expected to perform on the media of the day and not build/shoot to perfection in the off chance it might be remastered someday in the future. Small imperfections accruing from time-constrained model-building would get dramatically amplified in HD.

That's a reasonable supposition, but having seen the remastered TNG and the documentary about it's creation, I don't think it's true. The models are very detailed and look great in HD. One could surmise that the model creators knew their models might be used in TNG feature films (as indeed some of them were). Or maybe they were just super-dedicated and liked making really detailed models.

They could generate higher detail model compositions to train the AI.

> You can't get information out of SD video that wasn't there before...

Well, that is the difference between regular upscaling and upscaling with neural networks. With a neural network, the additional information is being stored within the network during training and added to the video during the upscaling process.

Ultimately, you could argue, that this is just interpolating too, but the quality of the interpolation depends on the training material. If you would train it on an original and use such a neural network to upscale a lower resolution version you could end up with the original (a perfect interpolation).

So it all comes down to the quality of the training material and while AI Gigapixel seems to have quite good material, I wonder if the result could be improved by transforming the video as a whole and not just frame by frame, as that would give the NN even more information to interpolate on.

> With a neural network, the additional information is being stored within the network

I've seen a few people say this in the thread. It doesn't seem accurate to me. Information is being created/hallucinated/interpolated. Re-scanning filmstock gets new information. If not, it doesn't matter how sophisticated your algorithm is (naive upscaling or deep learning), you're still interpolating values.

In the blog post, it says:

"While the popular Original Series and The Next Generation were mostly shot on film, the mid 90s DS9 had its visual effects shots (space battles and such) shot on video.

While you can rescan analog film at a higher resolution, video is digital and can't be rescanned."

The Next Generation also had its visual effects shot on video, this was a major hurdle for their remastering efforts. They had to recreate all of the effects so it was a slow an expensive process. On the positive side, the effects end up looking much better since this isn't the 80s and we can throw a lot of CPU power at creating them.

Could you please expand on why CPU and not GPU?

Edit: why am I getting downvoted for asking a question? Sorry :-(

The parent was likely using "CPU" to just mean "processing power", not being prescriptive toward whether that processing is done on a CPU or a GPU.

I assume your downvoting is because people (uncharitably) believed you were being an annoying pedant instead of asking a genuine question.

For what it's worth, the implications are about the same... though in practice, it would likely be ray-tracing frame by frame, then overlays... though the process in the article could be trained with very high detail models and fill in the blanks from that.

> You can't get information out of SD video that wasn't there before with upscaling

But you can infer visual detail from the information that is already there. Especially because ML uses information from the training set to help make sense of the information that is already there.

e.g. if I show you a picture of a key then you can figure out what the lock looks like because the key contains that information, even if it’s not visible.

Pro-tip: You have to change the YouTube setting so that you are watching in 1080 HD, not the 360p that it can default you too.

You know, I think there is some real promise here!

With some HD remasters, you can start to really see the makeup, the little pores in everyone's skin, the smudges and uncleanliness of real film-making. Film-makers and directory choose the lighting and the focus with the end-product in mind. They know that the screen won't capture certain things and so they know where they can skimp and save [0]. When you re-master it, you're going against the 'vision' of the directors. Not in a big way at all, it's very subtle. But it's still there.

With ML techniques, you get the 'idea' that the director was going for, without seeing all the screw-ups that they knew they could get away with. It's crisper but the idea and vision are the same [0].

Peter Jackson's recent 'They Shall No Grow Old' is another great example of using ML too. In that case it was to preserve the old WW1 footage, bring it's frame-rates up to modern 24 fps, 3-D it, and colorize it. The results are literally breath-taking. Personally, I gasped when it finally hits; it's that good. Not to geek out too much here, but Jackson is literally changing history with that film. He changed the way we all view old footage, as something all herky-jerky and grainy, to something that is modern and real. Those 16 year old child-soldiers become real people again.

Though Jackson's work is a lot different than this effort, I think we all know that ML and the movies are here to stay. It's relatively cheap to update, takes little time, and be profitable (Remember the Disney Vault gimmick?). How long will it be before Chaplin's 'City Lights' is ML'd and remastered into 24 fps, 3-D, color, and with sound? Maybe 5 years?

Hell, I'd pay to see the best of old cinema brought back to modern standards like that.

[0] I know no film-makers, this is supposition.

There's a Criterion Collection blu ray of City Lights that I would imagine was done very well given their reputation and the reviews https://www.criterion.com/films/27558-city-lights

What is the licensing situation for remastering like? It would be a pretty interesting competitive edge if a company like Netflix could improve their entire library with some inhouse remastering magic. Or even a bargaining tool for buying new licenses...sure, we'll add your stuff to our collection and remaster it "for free".

Would definitely get me more interested in handing over my cash... Though, in this case, I think it would more likely be Amazon Video.

People are using this same technique for old games - https://venturebeat.com/2019/01/18/machine-learning-is-rescu...

I kinda hope that perhaps, if CBS doesn't find value in doing a TNG-level remaster of DS9 and Voyager, that something like this might give them a "good enough" version for a Blu-ray release. At the very least, one could imagine using this technique where applicable to at least, massively cut down the conversion cost in scenarios where re-cutting the scene would be particularly cost prohibitive.

I do wonder if having DS9 and Voyager available on so many streaming platforms right now cuts down on the incentive to remaster them. I'm not sure if a remaster would be enough of a draw to CBS All Access if you can already watch all the episodes on the other major streaming services, but perhaps Star Trek Discovery is successful enough they might consider it?

I don’t think it’s fair to call ST: Discovery a part of the franchise at this point.

It’s this bizarre smodge of minority group agendas, dark horror-like scenes and modern political ideas disguised as the family friendly Star Trek that you could watch with your kids that we all actually loved.

And are the plot lines intelligent? Do they make you think about interesting (non overdone ie non-left) philosophical issues? Not really, and they’re strung along in an awkward and long story arc that’s pretty dull and predictable and takes away from the per-episode storytelling :/

> It’s this bizarre smodge of minority group agendas, dark horror-like scenes and modern political ideas disguised as the family friendly Star Trek that you could watch with your kids that we all actually loved.

Uh, every Star Trek series was, in its time, a “bizarre smodge of minority group agendas, dark horror-like scenes and modern political ideas disguised as” something else (“Wagon Train to the stars”, in the case of TOS.) It seems tamer retrospectively because it's usually been on the side that became the accepted standard on the issues, and because the presentation of horror evolves.

OTOH, the pervasive long story-arc thing hasn't been constant throughout Trek, but it's been a growing trend within Trek since at least DS9 (and also, within adult TV generally; static-background episodic fare outside of children’s shows seems to have been progressively going out of fashion for decades.)

> I don’t think it’s fair to call ST: Discovery a part of the franchise at this point.

It is, because CBS says it is, and your opinion on the matter is not relevant[0].

It's canon. It's part of the franchise. It takes place in the same timeline as TOS. Accept it.


> modern political ideas

> family friendly Star Trek

Star Trek has always been preachy. Did you forget about the TOS episodes with the half-black-half-white and half-white-half-black guys who hated each other?

I distinctly recall every flavor of "it's not real Star Trek" when Enterprise came out, and people still whine about the story choices in Voyager today. Heck, some people who were fans of the original series would be loathe to call TNG "real Star Trek". In fact, having people who complain it isn't real Star Trek might be the best proof that Discovery is, in fact, real Star Trek. ;)

Star Trek has always been incredibly progressive/left-leaning. The Federation is a communist utopia where everyone has everything they could ever want for free. If you talk to people on the original series, plenty of them will talk about things that they wanted to do, in terms of gay characters and the like, but that they couldn't get permission to actually air. But they have always pushed the envelope of being about as far left as they can get away with from a business side perspective.

Yeah... this is a series that had the first interracial kiss, a black woman professional during a time when black women could only be nannies and servants, and a Russian serving openly amongst Americans during the Cold War.

I've seen a lot of complaints about "liberal" and "feminist" agendas being pushed in Discovery. It baffles me. Do such people not know what the Federation is about? They're literally social justice warriors in space.

The funny thing is that the concept of inclusivity has always been at the core of Star Trek, so if anything, Discovery is simply keeping with the status quo.

> minority group agendas, [...] and modern political ideas

could you refer me to an analysis that elaborates on this claim?

On the contrary, I'd think that a remaster would give them a reason to charge all those streaming services more for the shows. No reason to hold exclusivity over a twenty year old show.

I think that changing the price is sensitive, but having better quality content would entice people to watch it (reducing pressure on producing new more costly content)

As a fan I would be happy with a hybrid approach, where they use the original 35mm negatives where possible and upscale the VFX shots.

Honestly the worst part of the currently available DVDs is the compression artifacting.

I know they're rescanning a few scenes in HD for the DS9 documentary. It'd be pretty interesting to compare the ML-upscaled version with it.

One of the other interesting things is that while CBS may not have all of the source files for most of the VFX shots: The original artists do have many of them, and would likely offer them up in a heartbeat. The original 3D models and scenes for DS9 and Voyager are incredibly detailed, and would be more than adequate in HD. In many cases, it'd just be a matter of loading up the files, changing the output resolution, and mashing the "render" button.

> The original artists do have many of them, and would likely offer them up in a heartbeat

Indeed. I think its TrekCore that has some HD images of re-renders of the 3D model of Voyager used on the show. They are so detailed, you can start to see the where the 3D artists copy/pasted the same empty sickbay stock photo in various places around the ship to show there is something inside the windows.

I don't think the ML-upscaled scenes will compare favorably to the new 35mm scans. They don't really look a whole lot better to me than the DVDs upscaled with a normal high quality scaling algorithm.

"it took me about two days to get everything extracted, upscaled and put it back together in a way that was pleasing. This resulted only in the first five minutes of the episode being done."

Maybe someone at Google can use spare cloud time to remaster the whole series as their 20% time project? This remastering process would be easily parallelizable.

I forgot to add a :) because remastering Star Trek is clearly not Google's business, but it could be someone's business: video remastering/upscaling services for Hollywood. There's a lot of popular old film and TV content just waiting to be sold again to fans on Blu-ray or steaming.

I see a lot of comments from people getting excited about superficial quality improvements, and I'm mystified. It's nice I guess, but the story and acting are unchanged by this sort of thing, and isn't that what matters to Star Trek fans?

the low visual quality is becoming more and more visible as TV screens get bigger and better. It would be great to have those old shows in a quality that fits our new screens, that's all.

I recently tried to rewatch Babylon 5 but the resolution was terrible and it looked bad on my TV so I gave up pretty quickly. I hope someone does the same treatment to Babylon 5 episodes!

Unfortunately, that's probably the "last best chance" for a not-awful watching experience. At some point after the show wrapped, they lost all of their 3d models, so they will never be able to re-render a single ship.

EDIT: I didn't mention fan models as a possible approach because this is WB/PETN we're talking about here. They aren't Paramount-dumb, but they seemed take that as a challenge, at least vis à vis B5.

They used a fan-created model of Babylon 5 for The Lost Tales, and there likely exists fan meshes for most of the rest of the models on the show. Arguably, even rebuilding the models of B5's relatively simple models is not that big a deal compared to all of the editing work.

The biggest reason you'll never see a remaster of Babylon 5 is that it lacks the continued financial interest of it's backers or widespread market appeal. Star Trek is a wildly popular 50 year old franchise, which is still seeing new work today, and even then, CBS decided it wasn't worth it to remaster two of it's shows.

There's probably six million recreations of every 3D model in B5 made by fans, just like it is with Star Trek. Lack of 3D models isn't really a show-stopper, just ups the labor requirements. Hell, if respective owners relaxed their grips on IP, remasters would've been done already by fans of both Star Trek and B5 (I'd happily participate).

There weren't enough polygons on them to look good at higher resolutions; they'd really have to be redone anyway.

you are pretty wrong . Watch the original meshes at HD resolution : https://youtu.be/uHAuK_lDkk0

Did we watch the same video? It looks like a 2002-era video game when rendered at 1080p. It does not realistic at all; it is nothing like improving a film video of a model (film is basically ~5k equivalent and a model is.. infinite?).

I agree that it doesn't look particularly good. The question is whether it's due to a lack of polygons.

Some of the CGI/composite shots in Babylon 5 are particularly horrible.

If I remember correctly, the show was originally shot at an aspect ratio of 16:9. Because it was only going to air in 4:3, the special effects were done in 4:3. For the DVDs (which I think are the same version that Amazon streams), the 4:3 shots were enlarged to fill a 16:9 screen, making the picture look fuzzy. It would be interesting to see how well this technique would work on those types of shots.

B5 is even worse off--anything that has an effects shot basically only exists in tape quality, and the original CGI assets have disappeared.

According to JMS, Warner Bros has zero interesting in anything Babylon 5 because it was created under the weird PTEN thing and not by Warner proper. https://en.wikipedia.org/wiki/Prime_Time_Entertainment_Netwo...

https://twitter.com/straczynski has stated that due to the politics between Paramount and PTen, there will NEVER be any further work on B5 related works. Someone like Disney would have to purchase the rights to change this.

First season renders look pretty bad, but improve by the second season.

As I see more of these machine learning applications, where a neural network is trained on a dataset, then applied to a similar dataset, I wonder how well it would help if the network was trained on part of the new dataset as well, as some kind of "specialization".

One could take a generic "AI Gigapixel" net, and retrain it on some of the newer Star Trek content, for which an upscaled/remastered "ground truth" exists. My guess is that it would help a lot with features that are specific to the Star Trek universe.

Taking this further, one could take the resulting samples, and "rate" them, before feeding it back in the training engine. This would make some kind of "adversarial" feedback loop, but instead of the GAN, humans are involved in the loop (which could be used to train an adversarial network as well, that said). My hope is that it would converge to much better results in a shorter time, and with less input data.

My apologies if those are common concepts in ML. If so, I'd love to look at some references that could further my understanding on these topics, and get me to use the right terminology.

This seems like it would really benefit from a model trained specifically to upscale video. There's a lot of information you can get from the structure between frames that you can't exploit with a still image.

im not discounting ML being able to do this better. kudos to the op. but for fun i tried it using conventional methods (NR+sharpen); similar results but an order of magnitude faster. you be the judge:


On the one hand I'd love to see them remastered and touched up, but on the other hand, I originally watched them on a ~27 inch, old mid 90's TV....so the visuals aren't really the part which hits my nostalgia nerve.

Alright, now I want to do this with Babylon 5 since it seems like the HD remaster from WB/Amazon petition has flopped. Heck, even just upscaling the SFX shots from 360p to full DVD quality would be a worthwhile project.

I'm curious as to why the author chose "AI Gigapixel". What made it a better choice than say NNEDI3 or SuperRes or any other NN upscaler (there’s a bunch of them out there)? Were others even considered?

Anyone compared this with MadVR?

Anyone knows how AI Gigapixel differs from DLSS? (outside Topaz Labs ofc). Thanks!

Are films/programmes now preparing for future re-rendering in any way?

Star Trek: Deep Space Nine aka, Cheers in Space!

Any Friends of DeSoto here?

Holy shit, you did exactly what i wanted to do after i read about the final fantasy results!11 :D

You may find the original blog post from the creator here:


The URL for this post should really be changed to that. It's nothing more than a two sentence blogspam submission with a link to the original creator.

Interesting, I wonder if we could create a web-based system where people could upload videos and get 4k footage back...

Edit: Okay, new plan, anyone wanna make a cli for AI Gigapixel?

Sounds intriguing. What's your experience with AI Gigapixel so far?

Download the trial and upscale a photo. Its pretty cool, it seems a little better than old fractile upscale/sharpening techniques.

I changed the exposure too, but this is what i was able to achieve using both their JPEG to RAW and their Gigapixel, along with some color correction. (theres still a tiny bit of blocking id like to clean up, but its not bad for a test run.)


If anyone has used Photoshops Match Color feature before, I do expect a day where I can open some textures and apply them to similar low rez textures in a photo, and use the samples to repaint the picture. Similar in end result to how texture packs work for games.

I also think by taking HDR footage and compressing it to 6-8bit color space, and training a model to reproduce the original, that AI will be pretty good at upconverting older footage into high color depths.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact