
Why PS4 downloads are so slow - kryptiskt
https://www.snellman.net/blog/archive/2017-08-19-slow-ps4-downloads/
======
ploxiln
Reminds me of how Windows Vista's "Multimedia Class Scheduler Service" would
put a low cap on network throughput if any sound was playing:

[http://www.zdnet.com/article/follow-up-playing-music-
severel...](http://www.zdnet.com/article/follow-up-playing-music-severely-
degrades-network-transfer-performance-in-vista/)

Mark Russinovich justified it by explaining that the network interrupt routine
was just too expensive to be able to guarantee no glitches in media playback,
so it was limited to 10 packets per millisecond when any media was playing:

[https://blogs.technet.microsoft.com/markrussinovich/2007/08/...](https://blogs.technet.microsoft.com/markrussinovich/2007/08/26/vista-
multimedia-playback-and-network-throughput/)

but obviously this is a pretty crappy one-size-fits-all prioritization scheme
for something marketed as a most-sophisticated best-ever OS at the time:

[https://blogs.technet.microsoft.com/markrussinovich/2008/02/...](https://blogs.technet.microsoft.com/markrussinovich/2008/02/04/inside-
vista-sp1-file-copy-improvements/)

Many people had perfectly consistent mp3 playback when copying files over the
network 10 times as fast in other OSes (including Win XP!)

Often a company will have a "sophisticated best-ever algorithm" and then put
in a hacky lazy work-around for some problem, and obviously don't tell anyone
about it. Sometimes the simpler less-sophisticated solution just works better
in practice.

~~~
cm2187
Reminds me of Apple's Jim Reekes "engineers are retarded".

[https://m.youtube.com/watch?v=C5d151lqJsA](https://m.youtube.com/watch?v=C5d151lqJsA)

(The annecdote really starts at 2:30).

~~~
Systemic33
Haha, what a great video.

To some extent he's onto something. Sometimes you are in this zone when
everything will work if you just do this one quick little hack, and then the
hack ends up not getting replaced with a proper solution, and in the end you
end up with a harddrive right next to a speaker >.<

~~~
btschaegg
There actually seem to be people whose default mode of operation seems to lie
entirely in that zone though. I'm pretty sure everyone arrives at a similar
conclusion if they realise that attempts to fix a certain person's code always
end in a express trip to the 6th stage of debugging [1]...

[1]: [http://plasmasturm.org/log/6debug/](http://plasmasturm.org/log/6debug/)

~~~
jzwinck
Definitely. And that "quick hack by default" person will often end up being
the favorite of management because they get so many things "done."

So it's important to figure out what kind of manager you have, and try not to
be the slow guy on a team full of quick-hack people. You won't look good and
nobody will be happy.

~~~
chii
> the slow guy on a team full of quick-hack people

isn't that called 'cultural fit'?

~~~
lostlogin
That assessment is best made by the People and Culture team, not you.

------
andrewstuart
Its bizarre because I bought something from the PlayStation store on my PS4
and it took DAYS to download.

The strange part of the story is that it took so long to download that the
next day I went and bought the game (Battlefield 4) from the shop and brought
it back home and installed it and started playing it, all whilst the original
purchase from the PlayStation store was still downloading.

I ask Sony if they would refund the game that I bought from the PlayStation
store given that I had gone and bought it elsewhere from a physical store
during the download and they said "no".

So I never want to buy from the PlayStation store again.

Why would Sony not care about this above just about everything else?

~~~
colechristensen
When I read stories like this I'm bothered by the fact that the author didn't
go to their credit card company and chargeback the unwanted purpose. Your
credit card is a powerful tool to mitigate retailers not giving you what you
expect, using it motivates bad actors quite well. Chargebacks are expensive
and too many chargebacks leads to unfavorable terms with the credit card
companies which are inescapable to the retailers. (just like PlayStation has a
captive market, so does Visa)

~~~
cuckcuckspruce
Sony bans your PlayStation Store account and you lose all your games when you
do a chargeback[1]. Yet another reason to never buy digital downloads from
Sony.

Edit: Steam puts your account in a restricted mode when you issue a
chargeback[2].

[1]
[https://www.reddit.com/r/PS4/comments/2ivsz3/beware_if_you_s...](https://www.reddit.com/r/PS4/comments/2ivsz3/beware_if_you_successfully_charge_back_sony_over/)

[2]
[https://support.steampowered.com/kb_article.php?ref=6687-HJV...](https://support.steampowered.com/kb_article.php?ref=6687-HJVM-8966)

~~~
colejohnson66
Why is that even legal? Consumer protection laws are supposed to protect you
from terrible retailers. But, if I issue a chargeback against Sony for
recourse, but still lose, that defeats the purpose of a chargeback.

------
erikrothoff
Totally unrelated but: Dang it must be awesome to have a service that people
dissect at this level. This analysis is more in depth and knowledgable than
anything I've ever seen while employed at large companies, where people are
literally paid to spend time on the product.

~~~
PaulHoule
It is astonishing how oblivious people who are "paid to spend time on the
product" can be about the products they ship.

Case in point, Microsoft shipped a EXPLORER.EXE with Windows 10 that is 20%
whitespace because of an Adobe invented metadata format that I bet nobody
involved with Windows ever thought about it. See

[http://ontology2.com/essays/LookingForMetadataInAllTheWrongP...](http://ontology2.com/essays/LookingForMetadataInAllTheWrongPlaces.html)

~~~
bitwize
The PC release of Sonic Mania was delayed because of slowdown issues.

Slowdown issues. On a game that is _literally_ nothing but 90s graphics.

~~~
coolsunglasses
The game is more graphically intensive than you might think and the people
that care about this kind of game very much value a solid 60 FPS frame rate.

~~~
quazeekotl
yet somehow it ran at 60fps 20 years ago on hardware that was 10000 times
slower

hmm

~~~
jhasse
The game includes garphics which wouldn't be possible on the Mega Drive /
Genesis. For example alpha blending or lots of sprite rotating.

Also Sonic on the Mega Drive had multiple framerate drops, e. g. when you lose
lots of rings.

~~~
TulliusCicero
It would've been possible on the Saturn, and on the Dreamcast at 60 fps,
easily. How much more powerful are even mediocre modern PCs compared to the
Dreamcast?

~~~
Lorkki
Saturn would most likely choke on memory requirements, given how much is going
on in the levels, as well as lack of transparency support. Symphony of the
Night was the state of the art 2D platformer for that hardware generation and
it has frequent loading pauses, despite being much more static.

Dreamcast would probably work, given enough effort by people who know the
platform inside and out. But then, a much lower barrier to entry is probably a
significant reason as to why Sega was willing to fund another 2D Sonic game in
the first place.

------
g09980
Want to see something like this for (Apple's) App Store. Downloads are fast,
but the App Store experience itself is so, so slow. Takes maybe five seconds
to load search results or reviews even on a wi-fi connection.

~~~
ex3ndr
Same with Google Play since day 0.

~~~
pier25
I use both iOS and Android on a daily basis. In my experience the Play store
is consistently faster.

I'm not surprised, Google is better at these kind of things.

------
cdevs
As a developer people seemed surprised I don't have some massive gaming rig at
home but there's something about it that feels like work. I don't want to sit
up and be fully alert - I did that all day at work I want 30 mins to veg out
on a console jumping between Netflix and some quick multiplayer game with less
hackers glitchin out on the game. It seems impressive what PS4 attempts to
accomplish while you're playing a game and yet try and download a 40gig game
and some how tip toe in the background not screwing up the gaming experience.
I couldn't imaging trying to deal with cranking up the speed here and there
while keeping the game experience playable in a online game. Chrome is slow?
Close you're 50 tabs, want faster PS4 downloads, close your games/apps. Got
it.

~~~
dcow
But it's just one app and boom 7kb receive window. The author goes into how
terrible the impl is. I'm sure it might seem crazy if you're not familiar with
kernel networking. Also I hope you get more than 30 min of free time each day
(=

~~~
cdevs
I know my comment came off as putting the article down but I always have
praise for anyone on the hunt for issues of why Firefox does this or why PS4
does this. I shoot for 30 mins of game time but we all know how that turns
into 2-3 hours when I ment to start some programming side project. Games are
the devil.

------
ckorhonen
Interesting - definitely a problem I've encountered, though I had assumed the
issues fell more on the CDN side of things.

Anecdotally, when I switched DNS servers to Google vs. my ISP, PS4 download
speeds improved significantly (20 minutes vs. 20 hours to download a a typical
game).

~~~
jsnell
Yeah, there are definitely other possible reasons. E.g. one workaround I found
had Australian users block a specific IP in their firewall. Turns out the DNS
configuration was bad, and they were getting an overseas CDN server every now
and then. Due to the way the PS4 schedules the downloads (chunks downloaded
serially, chunk boundaries at static byte offsets rather than based on time)
having even one bad server in the rotation can really ruin performance.

But CDN configuration errors can happen to anyone. I think this client-side
mess is a much more creative way to screw up :)

~~~
FridgeSeal
I don't suppose you'd know what that ip is and if this fix still works do you?

------
mbrd
This Reddit thread also has an interesting analysis of slow PS4 downloads:
[https://www.reddit.com/r/PS4/comments/522ttn/ps4_downloads_a...](https://www.reddit.com/r/PS4/comments/522ttn/ps4_downloads_are_notoriously_slow_i_might_have/)

------
lossolo
DNS based GEO load balancing/CDN's are wrong idea today. For example if you
use DNS that has bad configuration or one that is not supplied by your ISP,
then you could be routed to servers thousands km/miles from your location.
Last time I've checked akamai used that flawed dns based system. What you want
to use now is what for example cloudflare uses which is anycast IP. You just
announce same IP class on multiple routers/locations and all traffic is routed
to the nearest locations thanks to how BGP routing works.

~~~
toast0
As with all things, there are tradeoffs. As you say, GEO DNS doesn't work very
well when the authoritative server doesn't get enough information to give a
good answer; for example, if all of a nationwide ISP uses the same pool of
recursive DNS, and doesn't proved EDNS client subnet, the authoritative server
will not get any signal about where in the country the users are. Or, if you
use DNS servers in another country for some reason, you may be assumed to be
nearby to that country.

Anycast isn't perfect either though. It may be more likely to get you a route
with fewer hops, but that's not necessarily the lowest latency link. And if
there's a congested link in the path, it's much harder to steer a portion of
users, but not all to alternate servers. You also have to deal with the
potential of routing changing from packet to packet, and potentially different
locations getting different parts of the same connection. It's not 'just
announce the same IPs from multiple locations' and take a nap.

Anyway, unless you have thousands of datacenters well distributed throughout
the globe, a lot of people are going to have high latency to whatever server
is chosen, and high latency and very small windows == poor throughput,
regardless of how you get to a server.

~~~
mjevans
The DNS server is the wrong place to make that decision.

If there is a set of choices and a decision to make, letting the client choose
would be the best option.

I think I'd start with a client shuffling the list of results and asking for
small fragments of the dataset from each. Based on the latency, bandwidth
detected, and the size of the requested data it would then pick the most
optimal set of servers. This also allows for fallback in case servers are
unresponsive.

------
Reedx
PS3 was even worse in my experience - PS4 was a big improvement, although
still a lot slower than Xbox.

However, with both PS4 and Xbox One it's amazingly slow to browse the stores
and much of the dashboard. Anyone else experience that? It's so bad I feel
like it must just be me... I avoid it as much as possible and definitely
decreases the number of games I buy.

~~~
rado
Not network related (?), but the slow and choppy PS4 UI/dashboard is one of my
"favourite" features. What were they thinking?

~~~
jhasse
"JavaScript is just as fast as C++ nowadays."

------
jcastro
Lancache says it caches PS4 and XBox, anyone using this?
[https://github.com/multiplay/lancache](https://github.com/multiplay/lancache)

(I use steamcache/generic myself, but should probably move to caching my 2
consoles as well).

------
foobarbazetc
The CDN thing is an issue too.

Using a local DNS resolver instead of Google DNS helped my PS4 speeds.

The other "trick" if a download is getting slow is to run the in built
"network test". This seems to reset all the windows back even if other things
are running.

~~~
jsnell
> The other "trick" if a download is getting slow is to run the in built
> "network test". This seems to reset all the windows back even if other
> things are running.

I did not see that in my testing. The built in speedtest runs with a large
receive window, but the store downloads are not affected. (You can see an
example in the last graph; there's a speedtest early on that has a receive
window 100x larger than the PSN downloads that are crawling along).

It's probably just a placebo.

~~~
matwood
> It's probably just a placebo.

Possibly, but I think the OP above was onto something about it 'resetting' the
windows. I just tested downloading a large game while I had another game
running. The PS4 showed it was going to take 6 hours. I closed the other game
I had running, and it still showed 6 hours. I ran the speed test, and now it
shows it's going to take 50 minutes.

BTW, thanks for running these tests as I have some stuff to try instead of
waiting hours for a game to download :)

~~~
jsnell
My experience is that the finish time prediction isn't reliable. It's based
average speed over the whole connection, not the current speed. So the
projection will lag behind if the speed changes dramatically. Maybe the
speedtest somehow resets the projection?

The measurement really needs to be done on the wire.

------
Tloewald
It's not just four years into launch since the PS3 was at least as bad.

------
tgb
Sorry for the newbie question, but can someone explain why the round trip time
is so important for transfer speeds? From the formula I'm guessing something
like this happens: server sends DATA to client, client receives DATA then
sends ACK to server, server receives ACK and then finally goes ahead and sends
DATA2 to the client. But TCP numbers their packets and so I would expect them
to continue sending new packets while waiting for ACKs of old packets, and my
reading of Wikipedia agrees. So what causes the RTT dependence in the transfer
rate?

~~~
preinheimer
I think this is a good question, I did some reading myself!

The blog post talks about "receive windows" which comes into play here. The
server will send up to the number of bytes specified in the receive window
before needing to start seeing some ACKs.

So the shorter the round trip time, the less likely the server is to spend
time twiddling it's thumbs waiting for an ack, because it's exhausted its
receive window. Of course, increasing the size of the receive window would
also help.

(I read this answer to get here, I could be way off:
[https://stackoverflow.com/questions/9613197/what-
determines-...](https://stackoverflow.com/questions/9613197/what-determines-
number-of-data-packets-before-an-ack-is-sent-net-sockets-on-se) )

~~~
syncsynchalt
That's right. In the case of a download, the receive window can also be
visualized as a limit on the number of packets that are allowed to be "in
flight" at once.

Shrinking the window is the wrong tool for the job here, like hammering in a
screw. It's sensitive to round-trip-time in a way that makes it impossible to
equate a given window with a desired speed. A better method might be to
monitor download speed and delay ACKs when it exceeds the limit, but
traditionally Unix doesn't have a standard API to do this.

------
Companion
I actually dread downloading patches and whatnot from PSN for this reason. I
have a 500 mbit connection, that works perfectly well on all my other devices,
but my Ps4 Pro, well, it's incredibly fickle. There'll be days where download
speeds are good, and then there'll be days where even downloading 200mb is a
challenge. It's all wired, so it's not a wifi related problem. I went through
different routers and even changed ISP's once and the problem still persisted,
so I think I have ruled it out as being on my end. It seems to be some weird
QoS feature of the PS4, or possibly the PSN not being up to scratch - I don't
know. Stuff like closing all background apps, or changing DNS, they don't
really seem to do anything for me. Sometimes pausing/unpausing does help,
though..

------
sydney6
Is it possible that lacking TCP Timestamps in the Traffic from the CDN is
causing the TCP Window Size Auto Scaling Mechanism to fail?

See this commit:

[https://svnweb.freebsd.org/base?view=revision&revision=31667...](https://svnweb.freebsd.org/base?view=revision&revision=316676)

------
tenryuu
I remember someone hacking at this issue a while ago. They blocked Sony
Japan's server, of which the download was coming from. The Playstation the
fetched the file from a more local server, of which the speed was considerable
faster.

Really strange

~~~
motoboi
Closer server means lower rtt. All things equal, this translates to a faster
transmission.

------
lokedhs
As one piece of information I offer my own experience with PSN downloads on
the PS4.

I'm in Singapore and my normal download speed is around 250 Mb/s, sometimes
getting closer to 300.

However, I sometimes download from the Swedish store as well, and those
download speeds are always very slow. I don't think I've ever gone above one
tenth of what I get with local downloads.

That said, bandwidth between Asia and Singapore are naturally more
unpredictable, so I don't know if I can blame Sony here. My point is that PS4
downloads can be very fast, and the Singapore example is evidence of this
fact.

~~~
rohmish
I've experienced similar thing with PS3. That said, unlike the author I am too
lazy to inspect traffic on that.

------
jumpkickhit
I normally warm boot mine, saw the speed increase with nothing running before,
so guess I was on the right track.

I hope this is addressed by Sony in the future, or at least let us select if a
download is a high priority or not.

------
deafcalculus
Why doesn't PS4 use LEDBAT for background downloads? Wouldn't this address the
latency problem without sacrificing download speeds? AFAIK, Macs do this at
least for OS updates.

------
hgdsraj
What download speeds do you get? I usually average 8-10 MB/s

~~~
lqdc13
I've gotten anywhere between 100KB/s and 60MB/s on the same ISP.

------
galonk
I always assumed the answer was "because Sony is a hardware company that has
never understood the first thing about software."

Turns out I was right.

------
bitwize
This is so that there's plenty of bandwidth available for networked play.

The Switch firmware even states that it will halt downloads if a game attempts
to connect to the network.

~~~
jonny_eh
Even if the game is suspended? It seems reasonable that you're right, but it
also seems like an oversight to not lift the throttling when games are
suspended, or if the game's not being played online.

------
frik
PS4 and Switch have at least no peer-to-peer download.

Win10 and XboxOne have peer-to-peer download - who would want that, bad for
users, wasting upload bandwidth and counts against your monthly internet
consumption.
[https://www.reddit.com/r/xboxone/comments/3rhs4s/xbox_update...](https://www.reddit.com/r/xboxone/comments/3rhs4s/xbox_updates_can_we_disable_p2p/)

~~~
deelowe
Do people really have issues with bandwith limits on uploads? I've always
heard the opposite.

Is it P2P or more like torrent where there are multiple peers? That would make
more sense.

At the end of the day, isn't the switch and xbox faster at downloading? That's
what really matters.

~~~
Spooky23
My network has multiple iOS devices, two computers and a bunch of appliance
devices I also use voip.

Maxing out the standard 3mb uplink is easy when photos sync or any number of
other use cases.

~~~
kuschku
3mb uplink? That’s... I’m sorry for you.

Maybe you can find another ISP, I’m in northern Germany and my ISP provides
100mbps down, 40mbps up via DSL.

~~~
Spooky23
Now I pay extra for "enhanced" service. 50/5!

Here in America we believe that competition involves 1 cable company and one
legacy phone company who offers 2/0.5 DSL for $70/mo, and will not actually
install it if you try!

