
Online streaming sites and URL obfuscation - Jonhoo
https://thesquareplanet.com/blog/resistance-is-futile
======
stabilo
Is a shorter solution possible?

Is decompiling SWF and/or de-obfuscating Javascript really necessary?

Years ago, I approached these sites like the author in the blog post. I spent
hours reading about Flash and figuring out the schemes websites used. I
decompiled swf. I also tried rtmpdump, livestreamer and other all-in-one
solutions. I used early youtube downloaders (clive, etc) and later quvi. I
also used /dev/bpf to capture HTTP requests. I tried it all.

I can relate to what the author is trying to do. I never settle for
"streaming". Download only.

However I found over the years either the websites have made things easier or
I was simply trying too hard. Or maybe I just am not interested in the type of
video that is served in this way. For some reason it takes much less effort
now.

To make sure I am not imagining this, I decided to try one of the author's
examples.

I chose projectfreetv.so

1\. I followed a link on the main page for some TV show I have never seen.

2\. I read the HTML.

3\. I noticed /watch/?aff_id= URL's.

4\. I chose the first one, 493165, and followed it.

5\. I read the HTML.

6\. I noticed a URL pointing to a website that serves TV shows and followed
it.

7\. I read the HTML.

8\. I noticed a URL pointing to an MP4 file.

9\. I downloaded the file.

I assume I got the TV episode on offer. The screen size is a little small but
the file was about 89MB and appeared to be the full episode.

Total time: less than 10min

What did I use? sed, netcat, less and ftp for the download. The entire process
is a one-liner.

If anyone has some more examples to try, please list them. I would welcome the
challenge.

~~~
Jonhoo
Author here: for some sites, that approach works well, and this is what I
originally did (and why I decided to write streamsh in bash). Have a look in
the "tricks" directory and you'll see some of the much simpler solutions that
are tried first. However, an increasing number of sites try to hide the URL,
and through increasingly complicated means. The article is about vidbull.com,
but there are others. They often also layer multiple obfuscators (vidbull uses
both JS obfuscation and URL encryption for example). While it's true you can
get pretty far just by scanning for mp4s on some mirrors, they are becoming
fewer and farther between.

~~~
stabilo
vidbull looks more like the old "file storage locker" sites that were popular
before the megaupload crackdown. Isn't this the same content you can find
using torrent sites? Is it likely that vidbull would have something for which
no torrent would ever exist? Sorry for my ignorance.

~~~
Jonhoo
No no, there are torrents for all of these videos (at least usually). However,
torrents are slightly less convenient in that you can't play the file until
the download completes. For me personally it's also an upside that HTTP is
easier to proxy, and there's no need to seed. That said, torrents will
probably be a better solution for many people. If you're interested, I wrote a
tool a while back that scanned torrent sites for the newest episode of a
particular show and automatically downloaded and started the torrent:
[https://github.com/jonhoo/smore](https://github.com/jonhoo/smore). No idea if
it still works, but shouldn't be too hard to get it up and running again.

------
hackerboos
rtmpdump [1] and livestreamer [2] are other good tools for this type of thing.

I've always had trouble trying to rip JWPlayer videos that sit behind a login.
I did manage it but I had to stream (second-by-second) the entire video.

[1] - [https://rtmpdump.mplayerhq.hu/](https://rtmpdump.mplayerhq.hu/)

[2] - [http://docs.livestreamer.io/](http://docs.livestreamer.io/)

~~~
Pengwin
Youtube-dl [1] is my go to tool for pulling video content out of a page, and
its source [2] is a nice lesson in de-obfuscating the sites it supports

[1] = [https://rg3.github.io/youtube-dl/](https://rg3.github.io/youtube-dl/)
[2] - [https://github.com/rg3/youtube-
dl/tree/master/youtube_dl/ext...](https://github.com/rg3/youtube-
dl/tree/master/youtube_dl/extractor)

------
anseljh
"streaming freedom" is a really clever euphemism.

------
tobltobs
At the end he says something like "We need Encrypted Media Extensions to get
ad free streaming...". Hmm, never thought about that this way. Maybe he is
right, but couldn't it be that with EME the ads will not be sitting next to
the player, instead the streamed video will contain the ads? Or does anybody
believe (enough) people will pay for streaming on youtube or similar pages?

~~~
madeofpalk
> the ads will not be sitting next to the player, instead the streamed video
> will contain the ads?

This tech already exists, and it's called 'Server Side Ad Insertion', with
Brightcove, The Platform, Adobe and Google all offering solutions.

It's actually pretty cool - it works by instead of streaming a baked file, the
client gets a 'live stream' URL and ads are inserted into that stream,
dynamically, just like regular video. It's like having `badge.svg` image
that's actually a PHP script that dynamically creates an SVG.

I'm not overly familiar with video tech, so I've definitely butchered the
exclamation.

~~~
TheLoneWolfling
And then ad blockers will snip out the ad portion of the stream and the cat
and mouse game will continue.

~~~
userbinator
VCRs did that long before streaming became popular, as in those days the only
way ads could appear were as directly in the video signal.

[https://en.wikipedia.org/wiki/Commercial_skipping](https://en.wikipedia.org/wiki/Commercial_skipping)

Not surprisingly there was some legal action, but I think it's rather
unenforceable in practice, and saying it's illegal to do so is nearly
equivalent to saying that you do not have the freedom to change the channel,
look away, or even close your eyes - a rather scary situation.

------
tlow
For a long time I've used the by hand method of URL extraction either
inspecting source code and searching for .flv or .mp4 filetypes (or simply
"file:*"). The other method I use is the built in developer tools "network"
tab of chrome under "inspect element". So far, I've only encountered one host
(sorry I cannot remember the name) which I wasn't able to extract the source
file via one of these two methods.

~~~
Jonhoo
Yes, the second method I used to use quite frequently, but it means you have
to open your browser, click play to get the video to load (which will spawn a
torrent of ad pop-ups and captcha prompts and whatnot). This way I can get the
download straight from the command line.

------
NickHaflinger
How do you use streamsh?

[https://github.com/jonhoo/streamsh/blob/master/tricks/jwplay...](https://github.com/jonhoo/streamsh/blob/master/tricks/jwplayer.sh)

~~~
Jonhoo
stream.sh <video URL (preferably the embed URL)> <filename to download to
(without extension)>

------
wodenokoto
Many of these sites have started supporting iOS. Wouldn't it be easier to
spoof iOS Safari and then only have to dig around in JavaScript?

