
The Mystery of the Slow Downloads - Doubleguitars
https://panic.com/blog/mystery-of-the-slow-downloads/
======
mrbill
As a lifelong sysadmin and former ISP person, I would have never thought that
I would be impressed with Comcast. However, after I lost a sweet free colo
deal (company I used to work for, got sold) a few years ago, I decided to move
my main system back to the house. I signed up for Comcast Business, because
AT&T's only "business" option with static IPs involved 1:1 NAT through their
crappy 2Wire-brand gateway.

Anyway, years later - I'd give Comcast Biz a score of 95/100\. When I called
to get PTR records added for my five static IPs, I ended up on the phone with
the guy who was typing them in. For the very few times I've had to call them
(less than once a year), I've never had a tech try to run me through a
"script", or argue that I didn't know what I was talking about when I tell
them I've already done all the debugging and the problem was on their end.

Had upstream issues once (even to their own speedtest server, so it wasn't
going outside of the Comcast network), and after a couple of calls I woke up
one Saturday morning to two techs and a manager on my front porch. "We brought
an unlocked modem and everything we might need, not leaving until this is
fixed." They were at my place for about an hour, then went down the road to
fix something on a junction box, and I had full speeds back.

Couldn't get them to leave that uncapped modem, though...

~~~
joecool1029
Comcast Business Tier support had been decent. However, now they threaten a
$99 service call charge if they can't replicate the issue or it goes away by
the time the tech shows up. Response time is usually within a day but for
something up the network a bit they'll just throw blame at the customer and
bill them.

Otherwise, I will say that lately their commitment to keeping equipment up on
the east coast has been admirable. My house has been without power since
Friday but they threw generators on all the fiber nodes' power supplies to
keep things up. pic:
[https://pbs.twimg.com/media/DXjUdljXUAElsRX.jpg](https://pbs.twimg.com/media/DXjUdljXUAElsRX.jpg)

~~~
lilbobbytables
Interesting. Is a power supply required for cable based internet? If not, then
they certainly would need to do that in order to maintain a level of feature
parity.

It would be pretty annoying to have a generator running at home, but no
internet (although it certainly still happens with downed wires). Or worse,
you have power, but the fiber box doesn't.

~~~
joecool1029
So that magical box on the pole has batteries inside. Depending on how they
have it set up, you get around 24-48 hours of battery power before the fiber
node converting optical to coaxial goes dead. They often have a LED on the
outside showing red or green depending on status of the node.

A mini generator solves the issue of powering the node if your power utility
is worthless and can't restore things after a few days.

------
ngrilly
I have a similar issue - very slow download - with my ADSL access provided by
SFR in France.

I did similar tests, by downloading the same file from servers hosted at
different providers, and the bandwidth was always very slow, except when I
changed the HTTP port from 80 (the standard) to an arbitrary chosen number!
This change alone made me able to use the full bandwidth provided by my
physical link! It was the proof that my ISP was running some kind of traffic
shaping, which is a shame.

More information if you read French:
[https://twitter.com/ngrilly/status/756453318113783809?s=19](https://twitter.com/ngrilly/status/756453318113783809?s=19)

~~~
lloeki
No such issue over here (SFR FTTH), bandwidth is extremely stable and through
the roof.

 _Free SAS_ is absolutely _terrible_ in that regard. I ran a few experiments
(at home - when I was still a customer - and at friends') over the years
similar to that blog post, and starting 17:00 things get throttled down to
death due to clogged pipes. It's almost on the clock. The behaviour is
immediately obvious to anyone having some background in network engineering,
and support is tone deaf to the issue.

~~~
jrimbault
Same conclusion from me.

Free _will_ throttle everything from 17:00 to 00:00 and during the week ends
these last few weeks.

On the other hand I don't know any other provider who just gives a fixed IP.

~~~
lloeki
> On the other hand I don't know any other provider who just gives a fixed IP.

SFR does (at least on FTTH) and you also get native IPv6 (not 6rd) and (IIRC)
a /48\. That is, unless you're subscribed via RED (which I currently am) in
which case the IPv6 feature is disabled on the remote end (you can enable it
on the NB6 router but it just sits there waiting for the prefix to be
announced). On cable the router is different but they're on DOCSIS 3.0 mostly
by now and I hear at least some people do get IPv6.

Orange still leases dynamic IPv4, dunno if they enable IPv6 by default now or
if it's getting tiered like SFR.

~~~
wut42
>Orange still leases dynamic IPv4, dunno if they enable IPv6 by default now or
if it's getting tiered like SFR.

AFAIK it is now enabled by default -- but maybe only on new boxes.

But they leases dynamic ipv6.

Exactly like static v4 if you want a static v6, you have to pay for it.

------
joshribakoff
> sends the user to run the same script on the control server, which we chose
> to host with Linode

Just went through a 3 month long issue with Linode, slow download speeds at
peak hours on Cox cable. They said to call Cox, Cox escalated me to a guy who
did not understand mtr tool. On Linode's end, they would not acknowledge any
issue & burned up many hours of time trying to convince me there was no issue.

Ended up switching to Digital Ocean since Linode refused to call Cox on my
behalf after I failed to get through. We also had the slowness issue with
Linode with users in Dallas, where our servers were located. I suspect this
will only get worse as net neutrality is repealed. I'm not saying these ISPs
are slowing things on purpose, but if they do so accidentally, then imagine if
they have green light to do so on purpose.

Linode also uses Cogent, FYI.

------
jlgaddis
To any of my fellow network engineers:

Although you very well might be, do not be tempted to stop reading after you
get to the sentence, " _The Panic web servers have a single connection to the
internet via Cogent._ "

Believe it or not, it's not (just) Cogent this time. It's Comcast, too. Or,
more specifically, saturated peering links between them.

To the others: this will likely not come as a surprise to any neteng who has
had to deal with either of them.

------
mschuster91
Deutsche Telekom is actually doing this on purpose here in Germany in order to
extract money from service providers. Hetzner has documented this here:
[https://wiki.hetzner.de/index.php/Double_Paid_Traffic/en](https://wiki.hetzner.de/index.php/Double_Paid_Traffic/en)

------
Maultasche
That was interesting. I do believe that's the first time I've heard a positive
story about Comcast. It's amazing how much cooperation has to happen to make
the Internet work.

~~~
supergarfield
I know it's completely anecdotal, but I've been with Comcast for two years now
and I've always had a good customer service experience with them. That said,
I'm in an area with a fair amount of ISP competition, so it's possible they're
more careful here.

~~~
zwily
Same here... All my friends complain about Comcast all the time and I’m
thinking there must be something wrong with me cause all my interactions with
them have been positive, and my service and price are decent.

(I promise I’m not a paid comcast shill...)

~~~
grzm
My experience with Comcast has been decidedly split. I know a few people who
work on the engineering side at Comcast and they—to an individual—have been
great. Comcast also has sponsored engineering meet ups in Philadelphia. The
three or four people who have come out to work on issues with my service have
likewise been great.

At the same time, every one of my tens of interactions with customer service
at Comcast have been a disaster. Unfortunately, the customer service
experience can be the one that overshadows the rest, and I'm sure there are
plenty of customers who don't have the positive in-person experiences to
balance out the poor customer service experiences. Tough to keep it all in
perspective. Also important to note that I haven't been using Comcast since
early 2017, so things could have changed in the interim.

~~~
DrScump
Since Comcast offshored its support call centers, every single contact I've
had with anybody there (including "supervisors") has been excruciatingly, um,
unhelpful.

------
jerkstate
> The Panic web servers have a single connection to the internet via Cogent.

Say no more. Hope this company gives their NetOps department the requested
budget next time. Would have been extra credit if they started monitoring
their download servers for TCP retransmits and grouping it by netblock to look
for trends (or just move it to the cloud where it's someone else's problem)

~~~
culturestate
Panic has something like 20 employees; I would be mildly surprised if there’s
_anyone_ dedicated to this area full-time.

~~~
scurvy
That's a shame. They should have picked up someone for this around employee
number 8-10.

~~~
wtallis
Panic's business is making apps that run on your local machine. Most of their
functionality isn't cloud-based. There's no way a full-time server admin
should be one of the first ten employees; it's not even obvious that they have
enough work to keep a sysadmin busy full-time doing stuff that isn't
contributing directly to the development of their apps. When you only have ten
employees, you don't pay one of them to spend all day watching log files.

~~~
imhoguy
After learning what they do I am even surprised the author did extra mile to
fix the issue this way. I would probably just go with some CDN.

~~~
michelb
But why increase operating cost if you can try to get it resolved otherwise?

~~~
imhoguy
All right, once resolved it wouldn't make much gain. My lazy attitude excludes
human route usually but it paid off to the author. Lesson for me here: to give
a try with people more often.

------
tbyehl
See lots of posts in here about Comcast customer service issues. Here's a link
to the executive escalation form:

[http://customer.xfinity.com/help-and-support/vp-contact-
form](http://customer.xfinity.com/help-and-support/vp-contact-form)

In my experience, the people responding to those tickets get shit done.

~~~
Serow225
Thanks!!

------
darkengine
I've been fighting dreadful peering between CenturyLink and a variety of
endpoints. I can saturate my gigabit connection on speedtest.net with a
directly-peered test host, but YouTube videos at peak times of day buffer at
720p, and downloading from any USA OpenBSD mirror (of which there are only a
few, and none in Seattle) nets me about 4mbps, compared to 450mbps or so I can
get on a Comcast connection.

What's funny is that using a VPN or tunnel to which CL has a good connection
resolves the issue, because the VPN/tunnel host has better peering than CL
does.

The difference between Panic's anecdote and mine, is that CenturyLink support
won't even acknowledge the issue exists.

~~~
tkone
I live in Seattle and use Centurylink and experience none of this. My children
watch youtube all evening with no degradation of quality, even while my wife
and I watch netflix. I have never experienced slowness with centurylink -- I
wonder if there is some bad hop that the vpn is avoiding?

~~~
darkengine
I notice some videos play fine at 1080p60, while others auto-throttle down to
360p. I wonder if this indicates that YouTube videos can be stored on a
variety of datacenters (with a variety of paths between them and CenturyLink).
Google's video quality report also rates CenturyLink in Seattle as "standard
definition" [1]. I don't have Netflix, but I just scored 910 Mbps on fast.com
(Netflix's speed test), so I think their connection to Netflix must be good.

A quick Google search reveals a thread of similar stories:
[https://www.reddit.com/r/centurylink/comments/5y1jzy/does_ce...](https://www.reddit.com/r/centurylink/comments/5y1jzy/does_centurylink_have_peering_issues_with/)

[1]
[https://www.google.com/get/videoqualityreport/](https://www.google.com/get/videoqualityreport/)

~~~
jerf
I don't have an in to Google, but a bit of fairly simple logic would suggest
that while YouTube presumably puts the short tail of popular videos on all
nodes, the long tail will be much more distributed and less replicated; the
costs inevitably force them that way. I can't prove it but as a long-tail
consumer myself I believe I've witnessed some videos getting pulled in from
some slow long-term storage before, things like conference videos from five
years ago in the low hundreds of views.

------
Pilfer
The real takeaway is that Cogent over-promises and under-delivers. If you want
reliable transit, switch providers.

~~~
chuckgreenman
I'm not sure that's something that you can take away from the article, Cogent
was performing well on other ISPs, Comcast was not providing enough bandwidth
for Cogent due to a disput external to Panic.

While we can't know I'd be willing to be the "unspecified" traffic re-
engineering was to prioritize Cogent's Panic traffic specifically. Comcast is
still the bad guy.

~~~
michaelmcdonald
Working in the field of networking I can say it's highly unlikely that Cogent
is performing traffic shaping specific to Panic traffic. That's a ton of
changes to a ton of hardware with a risk for screwing things up that just
isn't worth it. Cogent isn't going to go to those lengths for a client that
has a single connection and hosts a small / moderate site. Beyond that: the
connection that Panic has to Cogent is most likely through the data center
they're colocating with, not a direct connection to Panic equipment. That's
not for certain, it's entirely possible that Panic paid for a drop from
Cogent; however I'm guessing that's not the case.

What I find odd is that they would colocate their hardware in a DC with only a
single provider. The company I work for has four transit providers so as to
ensure uptime / reliability (don't get me started on how often we experience
unplanned maintenances from our upstream providers). Seems like Panic may want
to consider a different host for their colocation or examine (provided they're
dealing with Cogent directly) a secondary ISP.

~~~
tw04
>What I find odd is that they would colocate their hardware in a DC with only
a single provider.

I think it's extremely unlikely the provider only has cogent. More likely is
the fact that the link between the provider and comcast preferred cogent -
which panic would have no way to change.

~~~
gcbirzan
That's not true. panic.com has an A record (and www is a CNAME to @) that
points to an IP address that's not PI and advertised only as part of
38.0.0.0/8, obviously, only from Cogent.

------
stevoski
FTA:

> We colocate our own servers, rather than using AWS or any other PaaS, and we
> also don’t currently use a CDN or any other cloud distribution platform.

I wonder if Panic would never have encountered this slow download problem if
they did use AWS or another PaaS.

~~~
toomuchtodo
Their bandwidth costs would skyrocket. AWS and other cloud providers
overcharge considerably on outbound transfer.

~~~
stevoski
>Their bandwidth costs would skyrocket.

I don't think they would. Or at least, the costs would still be tiny.

Like Panic, my company mostly sells desktop apps. We're smaller than Panic,
but we still have a considerable quantity of downloads each day. We use AWS S3
+ AWS CloudFront to host our app downloads, and the cost is so small as to be
irrelevant to my business.

~~~
toomuchtodo
It’s possible I’m over exaggerating. Some of the AWS service price gouging
grinds my gears.

S3 and Fastly (or some other very inexpensive CDN) would be fine.

~~~
mfjordvald
You're not really.

My costs: $799 dedicated on 1gbps connection (overkill)

AWS Costs: $6000+

AWS is simply impossible for me.

------
NKCSS
The TLDR as video (have not watched it; at work) is a great idea :)

------
bewo001
I read 'Cogent' and knew it was going to be a peering issue. They are
'special'.

------
tgtweak
You should really use a cdn, your users abroad will have extremely poor speeds
connecting to cogent overseas from their residential connections.

It's almost mandatory unless you have 3-4 POPs strategically placed
geographically.

------
shmerl
_> And then, the craziest thing happened… They wrote back quickly._

Now they are responsive, because of the whole Net Neutrality repeal backlash.
They are scared to provide factual evidence of foul play - it will help the
courts to blast FCC for their decision.

But if they win in courts, Comcast and other crooks will definitely double
down on starving peering, as expected to extort money. And obviously, they
won't ever write back on this topic or will write that there is nothing wrong
on their side.

The only way to prevent that peering extortion by Comcast and the like, is to
have explicit rules that forbid it.

------
voltagex_
Isn't Cogent a "low tier" bandwidth provider?

~~~
scurvy
They are, but even the larger players get into peering disputes with eyeball
networks. Level3 and Comcast had a huge peering spat a few years ago (via
Netflix). Everyone's ports run hot with China Telecom and China Unicom.

It's almost always political (revenue) problems and rarely technical.

That said, I wouldn't pick up Cogent with their negative attitudes towards
IPv6.

~~~
ancarda
>I wouldn't pick up Cogent with their negative attitudes towards IPv6.

Ignoring and not adopting IPv6 is bad enough, but being negative about it?
What did they say?

Sometimes I think we’re never going to fully adopt IPv6. It’ll just be layers
of CGNAT.

~~~
scurvy
Google offered all free IPv6 peering to all major transit carriers. Cogent
turned them down and said, no, you need to pay Cogent for this. They were the
only carrier to do this. So Cogent uses other transit carriers to reach
Google. If you're on Cogent, you don't have a direct peering connection to
Google. They probably did this to balance out their ratios with other
carriers, but it's bad from a customer perspective.

You're better off using another tier 1 carrier than Cogent if you use a fair
amount of Google.

~~~
lawguy
I'm not seeing _any_ connectivity between Cogent and Google. Try Cogent's
Looking Glass[1] and test connectivity to www.google.com, for example.
("Destination unreachable: No route")

[1] [http://www.cogentco.com/en/network/looking-
glass](http://www.cogentco.com/en/network/looking-glass)

------
Hello71
> why me? Why was I able to get this corrected with an e-mail when Cogent
> couldn’t?

Because Panic doesn't compete with Xfinity.

------
amelius
Speaking of slow downloads, when I save a webpage in Chromium, it seems as if
it downloads the page byte by byte, reporting progress after every byte. And
another quirky thing is that if I close the tab during a download, then the
download is cancelled. Did anyone else notice this?

~~~
sirtel
It's common. Is it because of the protocols the website/connection is using?
If it is like stream, the total size won't be known until the end.

~~~
icebraining
Usually a missing Content-Length in the response header.

------
hiccuphippo
I'd love a service that gave me the right contact for the right problem in all
the businesses possible. Finding the right person while googling seems like a
matter of luck.

------
rsre
Surprising to see an ISP listening to its clients for once.

------
drefanzor
I've worked with Comcast for quite a few years though Contingent Technology
Solutions, and I have noticed a change in their quality lately. It's almost
like they care a little bit more. Don't quote me on that, though.

------
ElijahLynn
It is still a mystery after watching that.

------
dkuebric
I'd be surprised if it was an issue related to real capacity as opposed to
traffic shaping. This reads like the exact kind of thing that net neutrality
is meant to prevent.

~~~
toast0
There's no reason to use traffic shaping to get the before situation, when you
can simply not upgrade peering connections.

The real question is why Comcast decided to upgrade the connections? Not
upgrading Cogent peering is a pretty defensible position [1], although maybe
they didn't want to look that much worse than their competition.

[1] See the previous results of Cogent peering disputes, which is about half
of all the highly publicized peering disputes.

