
Comcast: Simulating shitty network connections so you can build better systems - ot
https://github.com/tylertreat/Comcast
======
api
Nice.

It'd be neat to include options to simulate other pathological conditions
often encountered in the field like:

\- Multiple layers of NAT - overlapping un-synchronized timeouts are one thing
... but the REAL fun comes in when intermediate layers have the same IP ranges
as IPs you are trying to reach on the "outside" of the NAT sandwich. All kinds
of "interesting" things can happen, like the "software laser":
[http://catb.org/jargon/html/S/software-
laser.html](http://catb.org/jargon/html/S/software-laser.html)

\- Stateful NATs/firewalls with bizarrely short connection / UDP association
timeouts or that randomly forget connection state

\- Shitty NATs/routers that go into bizarre failure modes when there are too
many open connections, like forgetting early ones because they remember
connections via a small ring buffer

\- People who block all ICMP "because security."

\- NAT + short DHCP leases = musical external IP addresses

\- Small MTUs (<1500) like those imposed by PPPoE and other nasty
encapsulation protocols, sometimes without ICMP errors for big packets because
some jackhole blocked ICMP "because security."

All these things are common in the field. WHYWHYWHYWHYWHY

~~~
Spooky23
My favorite enterprise network pathologies:

\- Randomly block specific ports "because security"

\- Limit RPC port ranges to some random small number.

\- Strange network optimizations. Example: Optimize TCP Window Size to support
NT4 clients on 56k frame relay circuits.

\- Very Slow DNS response

\- Optimize core switching rules to fully utilize switch CPU. Avoid
configurations that take place in an asic.

\- Long DHCP Leases + Undersized DHCP scopes.

~~~
wallflower
It makes me wonder at how Skype manages to tunnel though almost anything.

~~~
api
Primer I wrote a while back:
[https://www.zerotier.com/blog/?p=226](https://www.zerotier.com/blog/?p=226)

Skype has abandoned P2P, though I'm not sure why. One possibility is that MS
now has middle boxes deployed at so many interchange and peering points that
there's no benefit in maintaining the added complexity. Another is that it was
to comply with surveillance/tapping requirements.

~~~
ot
IIRC they said that it's because of the shift to mobile. You can't really do
P2P when the majority of your clients sit on phones.

EDIT: [I can't reply to the comment below, so I'll add here] It's most likely
not a technological problem, but rather than data usage is limited on mobile
and you don't want your user to pay for traffic they didn't use.

~~~
api
I keep hearing a categorical "you can't do P2P on mobile." I intend to fling
myself at this problem like a drunk seagull soon, so I'll blog on it. I think
it's possible, but with a number of special considerations around wakeup-
quantization and general power management, some protocol augmentation, and a
high tolerance for nodes appearing and disappearing.

Primer concept:
[http://en.wikipedia.org/wiki/Interrupt_coalescing](http://en.wikipedia.org/wiki/Interrupt_coalescing)

~~~
exelius
The problem isn't power (well it is; but that can be overcome), it's spectrum
and pricing.

Pricing: Most people in the US are charged for the data they use on their
mobile devices, and thus would not want P2P used on their phone because it
costs them money.

Spectrum: P2P is not a very efficient distribution model in a world where most
clients are on asynchronous connections. Asynchronous connections exist
because transmission spectrum is limited, so to maximize spectrum usage,
telcos allocate more spectrum for downstream transmissions than upstream. But
if everyone's phone is chattering all the time with P2P traffic, you're going
to saturate the spectrum and reduce overall data speeds. This is why mobile is
still charged on a usage basis: it discourages overly chatty applications.

~~~
api
P2P doesn't necessarily mean cooperative relaying or swarm distribution like
BitTorrent. I agree that those applications are mobile-unfriendly with current
batteries and cell networks. It just means you are talking directly to your
peers instead of back-hauling to the cloud. You can have P2P where the only
traffic you handle is your own. In that scenario total aggregate bandwidth
shouldn't be that different from a cloud-backhauled app -- the only difference
you'd see is in how many endpoints you're talking to. So instead of seeing
25mb transferred to/from one IP, you'd see it sprayed across a few dozen IPs.

The big hurdles I see are (in no particular order):

\- Connection maintenance and keep alive. Basically this is bad, so you want
to be more aggressive about shutting down unnecessary P2P links on mobile than
you need to on desktop/server. Keep alive requirements generally suck anyway,
and are one way NAT murders kittens.

\- Restrictions around background tasks on mobile OSes (iOS is particularly
onerous).

\- Squelching inbound traffic from badly behaved or broken peers to avoid
inbound flooding.

\- Battery life and related concerns.

I understand it this way: cellular networks currently have their own
"netiquette." It includes things like don't be too chatty, try to coalesce
instead of spewing packets at excessively random times, etc. These things are
less important on wired networks since they don't have the same resource
constraints or bandwidth issues.

I guess a related question is why you would do P2P and not backhaul to the
cloud? I can think of many:

\- Reduced latency for things like AR and VR where latency matters a lot.

\- Reduced bandwidth cost due to lack of back-haul, and back-hauling a pic
being sent between two people in the same city 2000 miles to a cloud server is
just offensively stupid anyway.

\- Privacy and security.

\- With P2P you could have a more open app model where apps aren't wedded to
proprietary cloud infrastructure. They still work even without someone's
cloud, etc.

~~~
exelius
If you're talking about P2P over cellular, I think it's just in general a bad
idea. The way cellular networks are operated, you can't do device-to-device
connections - they have to go through the tower for a number of (very good)
reasons. That's where the latency comes in, and the wireless transmission
latency can be an order of magnitude higher than the latency on a cross-
country connection. Furthermore, the resource that is constrained is the
amount of available spectrum - so we need to optimize for that. Bandwidth is
effectively infinite from the tower to the cloud, but bandwidth from the tower
to the device is the constraint.

P2P over non-cellular (i.e. Wi-Fi or Bluetooth) where you can actually make a
direct device-to-device connection seems like it may have some use cases
(messaging apps, etc.) But they're edge cases and by no means a common use
case because it just isn't reliable enough.

------
ejdyksen
If you're on Mac OS, you can also install the Network Link Conditioner from
the Apple dev tools, which is a prefpane that serves the same purpose:

[http://nshipster.com/network-link-conditioner/](http://nshipster.com/network-
link-conditioner/)

~~~
athenot
And on the command line, you can use the dummynet(4) command (can be useful to
automate testing under various network conditions).

------
kozhevnikov
There is also clumsy with what possibly is the best demo gif ever made.

[https://jagt.github.io/clumsy/](https://jagt.github.io/clumsy/)

~~~
14113
Regardless of how good the tool is - that was an excellent demonstration clip!
Within ~30 seconds I knew exactly what the tool did, why it was cool, and what
I might want to use it for.

Startup landing pages could learn a lot from that single gif.

------
chatmasta
People talk about "fuck you money," meaning the amount of money you need to
make before you can never work again. Personally, my "fuck you money" isn't
money _per se_ , but the fall of Comcast. When I build enough value in the
world to displace Comcast, I will retire.

We need to see more work in the areas of mesh networking, layer-3 routing, and
consumer networking in general. This tool is a good step. Personally I'm
hacking on some openwrt routers right now -- I recommend everyone try it. The
documentation is dense, with a friendly community writing it.

~~~
Fuxy
You know your company sucks when its name has become synonymous with high
latency, dropped packets and a all round shitty network.

Unfortunately i suspect they will just send a trademark claim to Github to get
it removed.

~~~
shanemhansen
I hate comcast customer service with a passion, and their pricing models are
super hostile to existing customers.

But their internet service has been very reliable and very fast for several
years for me. I rarely have internet outages.

~~~
e40
Just yesterday I noticed my bill had gone up to $147. I called and did the
yearly "I'll cancel if you don't give me a discount" nonsense. An hour out of
my day.

Btw, they had raised the price of my 5 static IPs to $24.95 (from $9.95). I
have to lease their modem ($12.95) because of the static IPs. I did research
and found out that my registrar (namecheap.com) has free dynamic dns, so I'm
going to switch to that and give up my static IPs.

So, after the $40 discount I got yesterday, plus the extra $38 for the
modem/IPs, I should be down to a reasonable rate again.

I fucking hate Comcast.

~~~
phil21
> I have to lease their modem ($12.95) because of the static IPs.

This isn't actually true, assuming you are on the consumer side of the fence.
No experience on a business line though.

Source: Has extra IPs on his owned modem, after fighting with them for a
couple weeks.

~~~
e40
It is true, what I said. I have Business Internet and they will not allow
activation of a non-Comcast-owned modem when you have static IPs. I know, I
tried to do it __twice __.

~~~
phil21
Ah yeah, business side I have no experience with. Just wanted to make sure
anyone buying on the consumer side knows this, as they tried to pull the same
crap with me when it was not true.

~~~
e40
You can't get static IPs on the residential side, btw.

------
SEJeff
You really shouldn't do something like this without at the very least
mentioning Kyle Kingsbery and the Jepesen test suite he wrote to validate how
distributed datastores actually follow CAP or not.

For any devops / sysadmin / systems engineer: I highly suggest reading this if
not only to understand failure conditions better:

[https://aphyr.com/tags/jepsen](https://aphyr.com/tags/jepsen)

The entire "Call Me Maybe" blog series is because that is Carly Rae Jepsen's
stupid pop song, for which he named his test suite after. That and Kyle's
posts are absolutely hilarious to read. Seriously, read them.

------
cordite
Would something like this be DMCA'd or taken down because of the name? (Not
that it is violating copyright by its contents)

~~~
bobmagoo
I had that thought as well. From poking around various IP law sites, DMCA is
purely for copyright infringement, and can possibly open you up to more
liability if you try to use it for trademark claims (what this would be), see
[http://www.lexology.com/library/detail.aspx?g=13f9814f-b56e-...](http://www.lexology.com/library/detail.aspx?g=13f9814f-b56e-4314-8e1b-95215ce60a6d)
for an example.

If Comcast wanted to take this down, it would be through a trademark
infringement claim.

(IANAL, etc, etc)

~~~
Florin_Andrei
So... the project is infringing on Comcast's ownership of shitty networks?

------
blueskin_
On linux, tc can do similar things, such as dropping a percentage of packets
or adding random delays.

[http://linux.die.net/man/8/tc](http://linux.die.net/man/8/tc)

~~~
lucb1e
As said in the readme:

> you can use tc which supports some additional options.
    
    
         $ tc qdisc add dev eth0 root netem delay 50ms 20ms distribution normal
         $ tc qdisc change dev eth0 root netem reorder 0.02 duplicate 0.05 corrupt 0.01
    

> To reset:
    
    
        $ tc qdisc del dev eth0 root netem

~~~
Florin_Andrei
Yeah, I used to do stuff like this with tc directly back in the day, but I
could never remember the options. This wrapper seems to make it a bit more
obvious what goes where.

------
copsarebastards
I don't need to simulate this, I have actual Comcast service.

------
jvehent
I don't need this, I have comcast.

    
    
        $ sudo ping -i 0.02 8.8.8.8
        [...]
        64 bytes from 8.8.8.8: icmp_seq=578 ttl=53 time=15.5 ms
        ^C
        --- 8.8.8.8 ping statistics ---
        579 packets transmitted, 364 received, 37% packet loss, time 13854ms
        rtt min/avg/max/mdev = 15.009/19.308/50.884/6.152 ms, pipe 2

~~~
YuriNiyazov
It's so weird to hear so much complaining about Comcast here. My stats with
that same command:

    
    
         --- 8.8.8.8 ping statistics ---
         615 packets transmitted, 612 packets received, 0.5% packet loss
         round-trip min/avg/max/stddev = 21.332/35.875/95.576/11.248 ms
    

I have never seen it be anything different.

~~~
sarciszewski
> I have never seen it be anything different.

If you parse this statement literally, it has amusing consequences.

------
azdle
I'm curious if anyone knows of tools like this that are more deterministic?

I'm in the process of writing a library at work that is basically implementing
the CoAP protocol from the ground up and having something that I can script to
force the same packets to be lost or delayed is something that I would find
very useful for testing. I've been using Apple's network link conditioner, but
when I do discover a problem, it can take a long time for the same scenario to
happen again which makes testing the fix quite difficult.

I'm currently trying to think through how I'd write this to make the filters
easy to setup [0], but if anyone knows of something that already exists,
please let me know.

[0] [https://github.com/Azdle/dolos](https://github.com/Azdle/dolos)

------
yincrash
Was this made in response for more real version of
[https://theonion.github.io/comcastifyjs/](https://theonion.github.io/comcastifyjs/)
?

------
xtrumanx
Google Chrome's device mode has a subset of this functionality where you can
make Chrome throttle the connection simulating different conditions (GPRS,
DSL, offline, etc.).

Access device mode by clicking the smartphone icon on the top left of the
Developer Tools (F12) right next to the Elements tab.

~~~
bduerst
It's very useful. I discovered this the other day when testing some client
side scripts. I was able to simulate a 3G mobile connection and determine
which parts were causing the intolerable load times.

------
bas
This is a beautifully-named project.

------
nexneo
`--device=eth0 --latency=250 --bandwidth=1000 --packet-loss=0.1`

This could have been named `--i-am-in-india`

------
mml
Did this with dummynet on FreeBSD around 1999. Made it a bridge though, so
everyone on the test network could "enjoy" 28.8kbaud dialup experience! Great
fun.

Called it "the molasses network", which is what they should change the name to
before Comcast smokes them.

------
nwilkens
SmartOS has this nice utility - ipdadm which does something similar.

ipdadm - administer the Internet packet disturber

[https://smartos.org/man/ipdadm](https://smartos.org/man/ipdadm)

------
alexcasalboni
Very nice.

How's this better/different than Crapify?

[https://www.npmjs.com/package/crapify](https://www.npmjs.com/package/crapify)

------
jpetersonmn
Id change the name, not because of Comcast coming after you, but I guess it's
just not very clever IMO. Good luck on the project though. :)

------
pc2g4d
Prediction: name of this project will be different within 12 months due to
trademark infringement.

------
GrinningFool
Nice, and the detailed README is appreciated.

I do have to wonder if the name won't cause you trouble though...

------
spiritplumber
I wonder if this can be used to simulate speed-of-light delay for a Mars
mission.

------
mpu
Written in Go for nothing, you might as well hack a cheap shell script.

~~~
scott_karana
Who cares? It also could have been written in Perl, csh, C, APL, Pascal, or
Fortran, and it would work exactly the same.

Can we stop shitting on cool things just because the technology being used is
"trendy", or you're sick of hearing about it?

------
mchristoff
This is great. I would love mobile support.

------
dangerboysteve
The project name is pure gold!

------
exabrial
In the words of the great poet Macklemore:

"This is fucking awesome"

------
farresito
I have a feeling that we will see an incoming DMCA soon.

~~~
farresito
Getting downvoted. The lack of humor in Hacker News is astonishing.

------
knd775
Upvote for the name.

