Hacker Newsnew | comments | show | ask | jobs | submit | josh2600's commentslogin

If you use Terminal on top of AWS (one deployment option) we can just migrate your workloads without rebooting.

The way it works is that you read the RAM pages from one machine to another in real time and when the RAM cache is almost synchronized you slam the IP address over to the new box (and then you let Amazon reboot your old box and then migrate back post-upgrade if you want to).

You can try it out on our public cloud at terminal.com if you'd like to (we auto-migrate all of our customers off of the degrading hardware before it reboots on our public cloud, but you can control that if you're running terminal as your infrastructure).

reply


... how?? That is seriously nifty.

Are you migrating just a process tree / other contained environment, or the entire machine?

Are you using CRIU or similar? Do open TCP connections survive the transfer?

reply


We wrote a bunch of hacks to the linux kernel to do it.

Custom container implementation, custom networking, custom storage.

It's just really good hardcore kernel engineering.

If you wanna talk more and you're in SF, come to our meetup on the 10th: machinelearningsf.eventbrite.com.

Edit: the whole machine including RAM cache, CPU instructions, IP connections, etc. is carried over. We can also resize your machine in seconds while it's running.

reply


Is this somehow different to Xen Live Migration/VMware Vmotion/etc?

reply


Yes. VMWare VMotion and Xen Live Migration are both VM migration tools, not containers.

The difference is subtle, but important. VMs have overhead because of virtualizing the kernel, Containers don't (or rather containers benefit from kernel performance much more than VMs).

In other words, you can achieve the same thing with VMotion, but it's slower and more overhead and harder to manage.

reply


Ah I didn't even know you're a container based shop. So you're moving live containers between aws provided xen vms.

-----


Wow, that's pretty sweet. Any plans for trying to get that upstream?

reply


Don't know that we have plans around that at this time. I'll try to dig and see if I can flesh out our story around this.

reply


With TCP_REPAIR, presumably they could...but both ends need to implement the REPAIR option I think, so maybe not in practice yet.

Or if the SDN of your cloud is good enough, even TCP_REPAIR might not be needed!

reply


It's a custom SDN layer we wrote.

reply


Are you running your VMs inside Amazon VMs? Or are you running containers instead, to avoid the overhead of having 3 nested OSs (the Xen host > the Amazon Xen guest > your VM)? If you run containers, how do you guarantee isolation of tenants (it is generally considered to be very difficult to achieve)?

reply


We are running a custom container implementation. The goal of our implementation is containers that perform like VMWare.

Process isolation is hard, but we've achieved it. We currently have some tens of thousands of users on our public cloud with zero container breakout, and while no security is perfect, we're constantly trying to improve our offering through White Hat bounties and constant security testing. In this case, I can tell you heuristics with which you can infer security, but I can't blanket label something as secure. I would say I think it's the most secure new virtualization tech, but I would also note that's a matter of personal opinion. Again, zero container breakout is probably the main point.

You can run our virtualization inside of Amazon, in which case you only really have the pain of Xen host + Amazon Xen, but it performs faster on bare-metal (as one might expect).

reply


Isn't your ability to "migrate workloads without rebooting" similar to Google Compute Engine transparent maintenance and to the live-update capability that Amazon is progressively deploying (which is explained in the post)?

How is it different from Xen or KVM live migration?

reply


It's much faster and doesn't use VMs.

reply


I don't see anything on your web page about running on top of AWS...? It looks like you guys only run your own cloud. Can you point me at some docs or anything about running on AWS?

reply


I don't have docs yet because I haven't written them, but it's running on AWS right now.

It runs inside of any hypervisor or on bare metal.

Feel free to email me at josh[at]terminal[dot]com if you want to talk more. I can peel back the kimono quite far (we're also in SF if you wanna meet up).

reply


Neat! I'll probably take you up on that. :-)

I was definitely impressed by the pulldocker err... now pullcontainer project and think it'd be great to see how you secure your containers and handle networking.

reply


I mean, I'm a little biased because I work there, but Terminal gives you multi-tenancy of compute and data workloads by default, and you get almost all of the properties of VMWare on containers (including VMotion style migrations).

You can make an account and boot apache Spark in about 30 seconds using this link [0]. It's running in production right now for a lot of people, and you can run Mesos on top of Terminal if you want it [1].

Again, I'm not trying to push this on you if you're happy with how stuff works today, but I think we've made a PaaS that solves a lot of these problems (and we'll let you run it on your own metal too if you want it). Check it out at terminal.com if you have some free time.

[0]https://www.terminal.com/snapshot/c81e6215eba5799335a45b6936... [1]https://www.terminal.com/snapshot/44d4ee043422afec75dfd3bdaa...

reply


This post would be a lot less spammy if you actually added some content, like how you do things differently, or what stack you use, whatever. Anything except "look at us, we are great at this"

reply


We hacked the linux kernel to provide better checkpointing of RAM state and then we write the state to disk. Anyone can then summon the state from disk in the time it takes to read the SSD.

It's kinda cool, or at least I think so.

reply


If anyone wants to test out this setup, you can boot owncloud 8 onto a running box in about 30 seconds using this snapshot: https://www.terminal.com/snapshot/728c65da91ab44fd8616212385...

reply


You can take a look at the work we've done with containers if you want over at Terminal.com. You can run it on your own metal too if you'd like.

We wrote a blog post about running docker containers on it too a while ago: https://blog.terminal.com/docker-without-containers-pulldock...

-----


That looks like a useful tool

We're running on (mostly) raw lxc, with networking via openvswitch, cgroups, yada yada. so I don't think it's applicable to us at this point

A containerized world makes a lot of sense, but it still seems like a really young ecosystem. It's really the 'wild west'at this point.

To be honest, I'd rather back an accepted standard, then a specific implementation.

Don't get me wrong, Tools like this are super valuable, and generally make my day to day life easier

-----


Every time I see the Go gopher I think of Gopher[0] immediately.

[0]https://tools.ietf.org/search/rfc1436

-----


I made a snapshot of a CentOS 6 box with CoreCLR cloned if anyone wants to play with it: https://www.terminal.com/snapshot/f34341a1b529a9141529cda006...

Note: You'll need a terminal account to boot it, but it only takes 10 seconds to come online once you do that.

-----


Minor point but one that I think is valuable. Decimating something is to reduce it by 1/10th. You are thinking of it as an analogy for destroy which it is not.

-----

[deleted]

Have you read that link? Do you know what is under discussion here? The actual misunderstanding is between reducing something by a tenth and reducing it by a great deal. In that context, "decimate" does indeed mean to reduce by a tenth. Even the most pessimistic legalization forecasts see greater changes in trafficking than a reduction of a tenth.

Maybe you're not a churchgoer, but the additional meaning of "tithe" discussed in your link confirms even further that a tenth is what "decimate" is about, because those who tithe, typically tithe a tenth of their incomes.

[EDIT:] Language changes, and attempts to avert such changes, are both part of what "language" means. Don't even get me started on "penultimate". b^) I wouldn't have commented, except that parent (since deleted) was a fairly aggressive, specific, and substantially incorrect ("nuh-uh! decimation can also refer to tithing!") criticism of josh2600's comment.

-----


Language changes whether you like it or not. There are plenty of changes happening that I don't like, but I realize that in most cases it is a waste of time to fight them.

-----


Every dictionary I can find defines decimate as "destroy a large part of." A few of them mention the 1-in-10 definition as obsolete.

-----


Wonderful piece, but the author is wrong about one point.

The idea that Piano rolls predate all other programmable storage medium is factually incorrect. Surely the Jacquard loom and its punch-card system, patented in 1801, pre-date the piano rolls of the 1900's?

Other than that, a great piece, but I would be remiss if I missed a chance to remind people of how amazing (and early) the Jacquard loom must've been at the time.

-----


Carillons (http://en.m.wikipedia.org/wiki/Carillon) had control drums ('speeltrommel' is the better search term) in the sixteenth century (oldest one I could find is from 'before 1542') that allowed one to program in a melody to play. Example of programming at http://youtu.be/kHuvTKxZwr0

-----


Well, there's a reported cylinder-based musical automaton in the 850 CE "Book of Ingenious Devices": http://en.wikipedia.org/wiki/Book_of_Ingenious_Devices So it looks like mechanical musical instruments may have been the earliest to use storage media, even if not exactly piano rolls.

-----


It's still just as amazing today, even more amazing that they are still in use!

-----


And also that it was considered a diabolical device designed to steal from artists who deserved simply to make a living when it was introduced.

-----


Good point. I probably could have done more research on that. The little that I DID do placed the piano roll at late 1800s. I'm not a historian. ;) I absolutely find all these ancient systems remarkable.

-----


I think programmable medium must handle conditions. Anyway ancient water clock, antikythera, or even Stonehenge could classify as 'programmable storage medium'.

-----


Look, I normally agree with almost everything you say, but this is just not true. Wheeler was at the NCTA until divestiture, then he saw the writing on the wall and went to CTIA and CWA. After CWA he worked for Core which is a very industry heavy VC.

Wheeler was a huge lobbyist for over 20 years, talks to Meredith at CTIA on a first-name basis and knows all of the people who run the telcos and their lobbying organizations. In some narrative, he's the perfect person to lead the FCC and in another, he's not.

My opinion is that, YES, he was a lobbyist and YES he is anti-neutrality because that was the position that made the most sense at the beginning of his tenure. I'm bullish that he'll come over to the Network Neutrality side (and gave indications in that direction at CTIA Super Mobility week this year) but I think there's a chance he'll do title 2 with lots of caveats that will make this, let's say, complicated.

You are right that memes aren't helpful, but Wheeler was actually the biggest lobbyist in the history of telecom, IMHO, so that much is quite true. Whether he is anti-neutrality now is up for debate, but when he took the office, well, I think there's little evidence to show he was in favor of neutrality then.

-----


I take your point, that Wheeler was a lobbyist for network operators for most of his career. I think "cable company lobbyist" is still a particularly dumb way to sum him up, but can see why lobbying for telcos is also scary for someone arbitrating net neutrality.

-----


>I think minimal infrastructure should be free.

I think you mean minimal infrastructure should be subsidized. There is no free in the physical world. The internet you are using comes in on big copper (or glass) cables and tearing up the ground costs lots of money. Overhead wireless internet is feasible, but not at the scale of an entire city, let alone a nation.

It's not like you can wave your hand and have a network magically appear. People have to dig up the streets, put the cables in the ground and connect things. There are servers in big datacenters and routers and all sorts of technical devices that have to be configured. It is not as simple as pushing a button and making your code open-source; building networks is construction not publishing.

If you think the internet should be subsidized (or nationalized) that's a conversation, but asking private companies to give things for free is ludicrous (you either mandate it to be free or accept the private status quo). If you really want free wifi, lobby for it.

Is the Internet a business or not? Is telecom a business or not? The highways are nationalized, and I'd argue that they work pretty darn well. It's very politically difficult to go from a privatized industry to a nationalized one, particularly in America (it has only historically happened during times of war or massive banking crises and even then quite rarely).

The fundamental question is this: Are these networks public or private? If they're public, most of these questions are non-issues. If they're private, again, most of these questions are non-issues. The reason we're talking about this stuff is because the line between public and private is hard to define.

-----


Yes, I meant free in the sense that a certain basic level would be free OF CHARGE to everyone physically using it. Free basic level of food, water, education, medical care, internet, police, etc. There are good economic reasons for this.

Subsidized isn't exactly right because it sounds like the customer still has a copay. I mean wealth redistribution, but only to provide the basics for everyone.

One can probably achieve this by a basic income calculated in terms of the "minimal cost of living" in a certain area. If people want more than that, then they can start a company, pay taxes, go work etc. But those taxes would go towards ensuring the minimum.

In short I'm arguing for a very well-defined minarchist position that I can defend.

-----

More

Applications are open for YC Summer 2015

Guidelines | FAQ | Support | Lists | Bookmarklet | DMCA | Y Combinator | Apply | Contact

Search: