Hacker News new | past | comments | ask | show | jobs | submit login
Skype Outage – I run a supernode and here’s what I think happened (voxeo.com)
99 points by jdupree on Dec 24, 2010 | hide | past | web | favorite | 43 comments



Maybe this will cause Skype to take Linux and Mac more seriously. If you've used the Windows version, you know almost all development has occurred there, whereas the Mac and Linux clients have looked the same for the last four or five years. The Linux client did have a major version bump about a year ago iirc, and that brought some needed features, but it was mostly the same.

Also, hopefully this will teach Skype to do more with open-source. I really hope they open the client up. This bug may have been caught, and things would definitely have turned out differently if Skype ran freely on other platforms. Maybe someone could even factor out a "Skype server" instead of an exclusive policy of client supernodes. Even serious torrenters rent a server somewhere to host their torrents -- P2P doesn't have to be strictly consumer-level connection, and really shouldn't be.


Or, maybe this will cause users to take Jabber/XMPP more seriously and stop using proprietary technology for corporate IM.

I've worked at places where management is completely gaga over Skype and would push me to support it despite the fact that I had no ability to block spam, troubleshoot messaging problems or integrate our IM system into existing Asterisk, SSO, monitoring and collaboration solutions.

IMHO, Openfire is far more flexible, extensible, secure, reliable and most importantly - manageable as a service.

http://www.igniterealtime.org/projects/openfire/

disclaimer: I do not work for Ignite and have no vested interest in their business.


When an XMPP solution that's as easy to use as Skype comes out I'll be all for this. As it stands, even Google's video chat is much more difficult for end-users to install and use.

I used to use Ekiga for video chat and it was much shoddier than Skype, constantly dropping calls, refusing to release the audio or video device so that we couldn't call back, and other serious bugs. Skype "just works".

I really hope that someone comes up with a decent free software competitor, but it doesn't really exist at this point. The fastest way to solve this problem would be for Skype to become free software.


Agreed, I've always thought that Skype would one day roll out an enterprise type "Skype Server" that would explicitly make performance better within corporations.


Why would they want to fragment the largest botnet in history? :)


They've already said that they're going to open source the client (http://blogs.skype.com/linux/2009/11/skype_open_source.html)... they just haven't actually done anything about it yet.

They'd better get a move on though, otherwise someone else (http://www.enrupt.com/index.php) is going to do it for them: http://www.enrupt.com/index.php/2010/07/07/skype-biggest-sec...

The trouble is that Skype is so closed and has been seemingly uninterested in open source and Linux, it's hard to find open source dev's who are interested in developing Skype related stuff.


Their opening of the client is more of an opening of the interface -- all of the real work will still be performed by a proprietary blob that the open interface interacts with. I don't think that will be very useful. I'd prefer if they released something that was _really_ open-source.


It would be more useful then the current state of affairs. With a proprietary blob to talk to, it would be possible for people to do things like write text-based clients in curses, or the suggested 'server' version of Skype specifically for running as a super node or relay node.


We don't know how flexible the blob they give us will be. Since the current interface has no provision for toggling or dealing with one's status on the P2P network, there's no reason to believe that that will necessarily be a feature that the blob exports; Skype may still keep the network communications totally opaque, all handled behind the black magic of the blob, which would simply export the basics like contact list, etc.


Not necessarily. We already know that things like supernodes are selected based on network conditions. The 'server' interface could just be a reduction of dependencies so that you could run it as a daemon process on a server. The Skype Network would then select it as a super node so long as the server had the right conditions (no NAT, no firewall, fat pipe, etc)

It presumably would still need a Skype login, but that could also be used to control it. Accepting commands from a white-list of Skype Ids.


Just checked my Windows event logs... as I also run a supernode and yup, one Application Error at 2010-12-22 15:51:10 from Skype.

Tech data shows version 5:

    Faulting application name: Skype.exe, version: 5.0.0.152, time stamp: 0x4cb31516
    Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time stamp: 0x4a5bdbdf
    Exception code: 0xe0fafafa
    Fault offset: 0x0000b727
    Faulting process id: 0xd20
    Faulting application start time: 0x01cba1ea3b70b5ef
    Faulting application path: C:\Program Files (x86)\Skype\Phone\Skype.exe
    Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
    Report Id: 4b2781d7-0de3-11e0-bbdc-005056c00008
I didn't upgrade voluntarily.


Nice to see the CEO of a large company that knows how to walk the talk. I've also seen him post on HN. Does Steve Ballmer know the difference between a micro and monolithic kernel?


I was thinking the same thing. What other big company CEO's would even think of investigating why their Skype was more jittery in some hotels, let alone actually know how to do it.


probably busy running big companies


CEOs have no free time? They don't eat, sleep, or rest. They will not stop until the stock prices surge?


Wow, Ballmer must be pretty tired by now


I am also surprised that a company as large as his, uses Skype instant messaging for their communication.


We're a very distributed company, with some projects being worked on by people in a dozen locations, so some form of IM is needed.

I'm a big XMPP proponent (and Voxeo acquired my xmpp-based company) and we do use XMPP for a lot of things inside Voxeo (like Phono, our jquery softphone: http://phono.com/).

Every voxeo employee has an XMPP ID, but they don't tend to get used. People gravitate to Skype naturally. It's the user experience on the client side of XMPP that keeps people from using it extensively for internal communications. At any given time I've got 40+ group chats going on and we create and destroy group chats many times a week for specific needs. The ability to make a voice call to one or more participants of the chat is also a winner.

The fact that everyone on the planet has a Skype ID also helps. We can easily pull partners and customers into chats as needed. They're already on Skype, so we don't need to teach them anything new.

It's a lesson in usability, certainly. Create a client that allows users, without any assistance, to create groups, move seamlessly between text, voice, and video, and has a foolproof signup and setup process, and folks will use it.


tl,dr; a forced software upgrade kept crashing all skype windows clients, taking down supernodes as well.


Well, the big question is not if Skype went down becase lots of supernodes crashed. The question is why they crashed, and if the trigger was external. There are messages on p2p-hackers list arguing just that:

  I have evidence that suggests this is being done 
  by a ddos attack on the supernodes' object list 
  cmd parameter.


I had the same issue: Shortly before Skype went completely offline, my Windows Skype client suddenly started crashing every few minutes. The Mac client didn't have that problem.

Some bug in the new version is probably the likeliest explanation, but if someone deliberately attacked the Skype network, it would probably look similar, right?


You're not going to get full client crashes without a major bug. Some amount of attack is possible but I've been getting crashes for a couple weeks now and skype should have fixed things.


...or vice versa. a bug could easily look like an attack.


Nice write up, pretty interesting that Skype has not come out and said anything about WTF happened. Also ironic that their security blog's last posting it titled "The importance of updating" :) ( http://blogs.skype.com/garage/ )

-John


A wonderful reason to have pre-release testers for every piece of software that communicates with other software, any time you plan to push updates. With pulls, your early updaters are your test beds; with push, you have to create your testing groups.


The problem with a big adaptive p2p network is that it is close to impossible to test under real production loads.


My guess is that the average supernode is on something like 6M/1.5M DSL. This is easy to simulate.

Now, if most of your nodes are on machines directly connected to 10GbE, then that's a problem. But most Skype nodes aren't. (I do imagine there are a few connected to 100M+ connections. But only a few.)


I imagine quite a lot in countries like Sweden. I had 100Mbit in my apartment for cheaper than my shitty comcast which barely breaks 5.


They could have setup their 'mega-supernodes' running the new client in a production environment, so that if it crashed horribly, it wasn't a client machine that failed (or their entire network).

Upgrading (relatively) untested software network-wide on what is essentially their critical infrastructure is bad news. If this didn't happen now, it would happen sometime. It was just a question of when.


Push software is even better about testing: you can forcibly upgrade 1% of installs, monitor them intensely, and let the (non-upgraded) system executive roll back the version if something bad happens. The system executive can even automatically roll back if it loses contact with the vendor, in case the upgrade goes so horribly wrong that it brings down networking.



Are Skype relying on third parties with no contractual relationship for an essential component of the network?


Yup. But they do mention it when installing. You can also disable it.

If you are not firewalled or NATd you are a supernode automatically - on the plus side you get better sound quality for voice calls.

Normally the bandwidth used is pretty low since you are mostly forwarding text messages.

When they said mega-supernodes they meant machines controlled by them that do nothing else, and are on high bandwidth connections. (My bet is lots of amazon instances.)


I've looked around for the ability to turn off supernode on Linux and Mac clients for a while but have come up empty. Do you know which section of the preferences/settings it's supposed to be in?


I don't think there's a UI element for it, but you have to add a registry key, see here:

https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration...


How pray-tell, does a Windows registry entry disable super node functionality on Linux and/or MacOSX?


Well barring effective virtualization, booting into Windows disables supernode functionality on Windows and MacOSX in the same sense that a brownout does


Skype's EULA pretty clearly states that this is the case:

3.3 Utilization of Your Computer: Skype Software may utilize the processor and bandwidth of the computer (or other applicable device) You are utilizing, for the limited purpose of facilitating the communication between Skype Software users.


I, too, run a Skype supernode - completely unvoluntarily, barring the possibility of something hidden behind an asterisk in the EULA. Does anyone know if there is a way to "opt out" of this? Everytime I start skype up it makes a million connections to everywhere and without even having a call on the line my measly 2.5 mbps upstream is permanently choked.



Sorry, I should've mentioned that I'm an OS X user :)

With that said, I've had a look-see in Skype's .plist and its various Application Support files, but oddly there's nothing "grep'able" to be found anywhere...


I seriously doubt the Skype supernodes actually run in regular clients on people's desktops. I'm pretty sure these are special servers placed strategically on the net.

If you see thousands of connections on your Skype at home then there probably is some weird P2P problem going on.


I'm sorry to be the one who has to point this out, but you have no idea what you are talking about.

http://www.skype.com/intl/en-us/security/universities/




Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: