

How a Potato Saved Shopify's Internet - Wisqnet
http://www.shopify.com/technology/3754222-how-a-potato-saved-shopifys-internet

======
hannibalhorn
Sounds like the time invested would have easily justified the cost of a
fractional T3 or similar, at least in the states?

And you still have the issue of different connections from the same machine
will show up from different IP addresses, which could certainly cause issues
with some of the more paranoid sites.

~~~
Wisqnet
The problem was that all cabling had to be done underground at our place, and
phone lines were the only thing we had in abundance. Coax, fibre, anything
else basically meant busting up a path to the road in a major urban area. Plus
it was the middle of winter which makes road work even harder.

Not every internet provider realised this immediately. We started down the
path to better internet a lot of times with companies that said "no problem,
we can get a wire to your building just fine" -- and then backed down weeks
later when they saw our situation.

After one or two of these delays, we applied the shotgun approach, asking
every single internet provider to give us an estimate all at once, to make the
delays concurrent rather than successive. Every last one came back negative,
or just stopped returning our calls(!). So Potato was my response to our
complete lack of options.

The multiple IPs thing was an issue for some sites, but it turned out to work
fine in general. If there was a site that really had a problem with it, it was
also typically a low volume site, and didn't need heavy load balancing. So we
had one of our proxies set up to not be balanced and just pick a single
working link, and used that for those sites.

------
nwmcsween
How did Linux bonding not work? Did you try balance-xor and did you set miimon
and arp_interval to something besides the default 0?

~~~
Wisqnet
The problem is that DSL around here is PPP over Ethernet, rather than just
plain Ethernet. So there's no MII, no ARP, no regular ethernet link bonding.
It's hard to escape PPPoE for DSL around here due to phone company politics.

The type of bonding I talked about in the article was all PPP link bonding,
which apparently still has a lot of issues to work out in Linux. Our two-line
bond was pretty unreliable, and our attempts at a three-line bond ended up
with a lot of return packets going nowhere. Didn't matter which lines we
picked, either.

------
ssoroka
Brilliant, very impressed. Though I'm not sure I understand it even after
reading how he did it. ;)

