This seems risky from a business perspective: it's voluntary vendor lock-in.
What if Apple decides to change the Mac Pro form factor for the next iteration? Then you have to retool and are left with a bunch of incompatible chassis. What if Apple stagnates with hardware upgrades? You'd be stuck running obsolete hardware. What if Apple discontinues the entire Mac Pro line? Not to mention the price premium of Apple hardware itself, then the time and expense incurred to design and fabricate this.
The fact that their software depends on Apple's graphics libraries doesn't seem like a good justification for doing this. What it says is they are willing to throw a ton of money and effort towards (very cool) custom hardware, but are unwilling to hire a person to write optimized OpenGL shaders for Linux, which would work on pretty much any other server they choose to build/buy/lease/cloudify. Certainly there will be other "debt" to overcome, especially if much of your codebase is non-portable Objective-C or GCD, but that has to be weighed against the possibility of your only vendor pulling the rug out from under you. And looking at Apple's history, that is a very real possibility...
Owning your hardware like this makes complete sense if your core business is directly tied to the platform itself, eg an iOS/OSX testing service. But as far as I can tell, imgix does image resizing and filters... their business is the software of image processing, and they're disowning that at the expense of making unrelated parts of the business more complicated. Not a good tradeoff, IMO.
This was pretty much my thought as well. The results of this cost-benefit analysis make me raise my eyebrow, and, same as you, I can only assume that OS X permeates the infrastructure from top to bottom, to an extent that makes pulling it out too painful to even contemplate. Image processing shaders of this type aren't that hard to write.
If you're worried about them not matching some piece of client software exactly (Quartz Composer, Photoshop, etc.), you still have options. And those options - e.g., webapp for previewing/something else/etc. - you'll probably want anyway, for the benefit of designers that don't run OS X.
(The filtering aspect of the system I find a little surprising anyway - the idea of an image-focussed client-aware DPI-aware CDN makes sense to me (and I like it!), but something that does your Photoshop filters in the cloud sounds less compelling. I would have expected people to prefer to do that locally, and upload the result. But... while I've worked with many artists and designers, I'm not one myself. So maybe they'll go for that. And/or maybe a lot of their customers take advantage of the fact that the processing appears to be free. And I'm prepared to contemplate the possibility, however unlikely, that they might know their customer base better than I do.)
Uploading pre-edited images takes time/resources, and in general a lot of our customers rely on us to do all of their image processing so that they don't have to.
Additionally, creating edited versions of images in advance presents two problems: 1) Any future site redesigns or edits must now be applied en masse to the existing images or risk older images not complying with the new scheme, and 2) Instead of only managing the one original source image in the origin, now we're talking about maintaining all of the different edited versions, which is very inefficient from a storage and image management perspective.
There are many advantages to applying all of the image transformations on-demand, rather than in advance. Keep in mind that we are not simply photo filters, but a full end-to-end image processing offering (which applies everything from simple photoshop edits like cropping, face detecting, color correction, watermarks, etc. to automatic content negotiation and automatic resizing/responsive design) that works on the fly; this means that our customers now can make batch edits to their entire corpus of images through a few simple code edits.
This can become extremely cost-effective, but also helps in reducing page weight significantly.
I work at Hemnet, a Swedish real estate site, where we currently have about 300 000 - 400 000 new images each week. We recently investigated the on-demand approach but found that JPEG decompression was the biggest time consumer in our use case of scaling fairly large images down to sizes appropriate for web and mobile. This made us decide to do scaling in advance to all our sizes which resulted in overall CPU time savings.
It would be interesting to hear how img.ix solves this, since you are arguing for the resources savings in the on demand approach.
It is true that the time to fetch, render, and deliver an image the first time it is requested can be a bit longer, because of all the processing we're doing in the case of that single request, but because we then cache that fetched image, all subsequent requests are delivered an order of magnitude faster (50th percentile 45ms).
The majority of the time taken in the first request is actually in fetching the image from the origin source, so once it's cached in our system it becomes a much faster operation: and of course delivering the cached image without it traversing our stack is even faster.
So yes, the initial request can take time, but all of the subsequent requests of that image are much faster than the alternative. And when you take into account that our service makes it possible to send the correctly sized image for the display (instead of loading a preset size and displaying it smaller), and optimal file types based on device and browser (webp for chrome, etc), load times/page weights on all of those requests are significantly improved.
In general, anyone who serves multiple requests for their images over time will see a marked improvement on page weight/speed, compared to rolling their own solution where they have preset image sizes and deliver jpeg only.
Look at the current state of Mackintoshes. People are having kernel panics and struggling to keep their machines running with current software. OS X moves pretty fast, possibly faster than Linux, and Apple builds it to support Mac hardware.... the teams who are porting hackintosh code have to support a lot more hardware variety and they have less resources than, say, linux.
Running mackintoshes in production makes no sense.
And I challenge the claim that you would save money.
Looking just at off the shelf costs of low end hardware does not tell you the TCO of serious machines that need to be running all the time.
To get comparable hardware quality to Apple products you have to spend more, generally, when going with "commodity" hardware.
The idea that Apple is expensive is a myth, born of two things-- people perpetuating it since 1980 (yes, 35 years this myth has been spread), with the vested interest of rationalizing their dislike of Apple... and the fact that Apple doesn't compete at the very low end.
In production, TCO is much more about reliability and other things than initial hardware cost.
> People are having kernel panics and struggling to keep their machines running with current software.
Not here. My Mac is at about 11 days of uptime and it's under constant use. At this moment, I can't say it's less reliable then my Linux machines.
In this specific case, however, I'd consider ditching the enclosure and ducting cold air through the internal chassis/heatsink. A Macpro is, essentially a heatsink with boards mounted on it and I'd just let the chassis do that part.
It's hardly the uptime of someone who's struggling with kernel panics and random crashes. Last power down was when I embarked on a trip. The previous one was a system update. In fact, I never saw a kernel panic with this machine.
> People are having kernel panics and struggling to keep their machines running with current software.
A guy I work with has been running several heavily used hackintosh servers without issue. They have been very reliable and he's happily converted existing Linux servers to hackintosh. He's been doing this for a while and knows exactly what hardware to use.
It is appealing in a way, but I also think it would put the business at legal risk in a way that is totally unconscionable. We cannot run a Hackintosh in production for a single moment, the long term ramifications could be immense.
The chassis we designed represents my attempt to re-house the Mac Pros in something more suitable for the datacenter.
It'd be interesting to see someone rip apart a Mac Pro and build an entire form-factor around its setup.
Don't get me wrong, what you guys have done is extremely beautiful in all ways, but I can't help to think that if someone wanted to do this with say a Mini... say you take a 1u rack, drill some new holes into it... hmm.
The Minis are incredibly simple machines inside, and their airflow is not great. You could pop their logic boards out and run them without the chassis. With some thought it wouldn't be too tough to significantly improve their airflow for this environment.
I do have an existing rack design that holds 64 of them (and other people have gone denser, with operational compromises I prefer not to make), so there's no great impetus on my side to rip them out of their enclosures.
Thanks for the detailed response. Really cool stuff you guys have going on.
What other companies utilize Apple hardware in this way at this kind of scale? While not "out of this world" in comparison to some of the big players who have tackled scaling, it's definitely significant considering Apple hardware.
I'm not aware of anyone who specifically uses Mac Pros (although I've heard some private rumblings). I suspect part of the issue is that the old form factor was not very rack-friendly, and people haven't gotten comfortable with the new form factor. Maybe this chassis design will help move this forward.
Mac Minis are a little more common than it might seem at first glance, particularly for use cases that some other people have outlined in their comments throughout this thread. Mozilla uses them to test Firefox builds on OS X for instance. I would imagine that places like Sauce Labs must have a Mac Mini farm to facilitate browser tests on OS X.
I'm not aware of any other service that operates in the same space as imgix that runs outside of EC2, so they definitely aren't using OS X there. I think in general there's a sort of disregard for the particular graphics processing benefits that OS X provides (as evidenced by some of the comments in the thread).
I would also be remiss to not mention Mac Mini Colo (http://macminicolo.net/) who do co-located hosting. imgix started out with them, and they did a great job.
There's another interesting use case where you need to have OS X (or iOS): when you want to display photos taken on iOS devices with their applied filters (the images are stored pristine, and the filters are applied on top when you view them). To recreate these photos exactly as they were on the device, you ideally need to render it within an Apple environment. You can probably imagine the use case for a service that stores a lot of user generated photography, in a world where iPhones are the most popular cameras (https://www.flickr.com/cameras).
I also heard through the grapevine today that a certain film studio is interested in getting one of these chassis to test out, because they saw this article. That's pretty exciting to hear, even though we don't profit in any way from the sale of these chassis.
Why pay 5-10X as much to host on AWS? It's not for free.
Hosts have a really nice markup, compared to hosting yourself. Hosts make a lot of sense for small companies who can benefit from the aggregated demand and capital costs being spread over many clients.... but not when you're at the level of building your own datacenter, or even using a full rack.
It's funny how since 1980 people have been talking negatively about Apple as "vendor lock in". For most of that time it was advocating vendor lockin to windows.
The thing is when you build your system on an OS or hardware choice you're making "vendor lockin" to that platform. Build on Linux and you're locked into just Linux, unless you port.
There is little risk being "locked in" to the largest most successful company in the world. Plus the costs being dramatically lower given the rabidly higher performance of Apple's technology for this particular service more than covers the cost (in fact I think one Mac Pro probably replaces 4 or 5 Linux boxes doing this.)
If you think Optimized OpenGL shaders would do this, you're not understanding what it is that they are doing. You're just assuming it's a trivial problem, it is not.
Owning your hardware makes a great deal of sense when you are operating at scale.
You have no idea what the comparison is, and I don't either. But again, the criticisim is around running a business off of a bunch of Apple "trash cans".
>advocating vendor lockin to windows.
Linux is no lockin, Windows lockin via software vs. Apple for hardware and software.
>"vendor lockin" to that platform
Java, Scala,or any other JVM language protects from that, and to a lesser degree Python, PHP does as well.
>There is little risk being "locked in" to the largest most successful company in the world.
Price gauging? Deciding not to support your platform anymore? Forcing you to upgrade?
>Build on Linux and you're locked into just Linux, unless you port.
Only that there are a bunch of Linux options to choose from, they are all open source so you can do whatever you want as far as upgrade paths and support, and if you use the JVM languages this isn't an issue.
>in fact I think one Mac Pro probably replaces 4 or 5 Linux boxes doing this.
There is no fact there, that's your delusional opinion.
>If you think Optimized OpenGL shaders would do this, you're not understanding what it is that they are doing. You're just assuming it's a trivial problem, it is not.
It's a CDN + image manipulation tool, you don't need 3D libraries. And if you use exiting libraries or tools, it is quite trivial. Here is their API: http://www.imgix.com/docs/reference
Platform lockin isn't exactly vendor lock-in, but there's a kind of lockin nonetheless. You're going to be dependent to some extent on your platform whatever your platform happens to be.
>Platform lockin isn't exactly vendor lock-in, but there's a kind of lockin nonetheless. You're going to be dependent to some extent on your platform whatever your platform happens to be.
So you making your own chips off of beach sand or something? /s After a certain point you get ridiculous.
JVM and C/C++ (python and other scripting languages to some degree) are the options if you want cross platform environments.
Jeez, this post is brimming with strawmen. (Why am I even bothering...)
> Why pay 5-10X as much to host on AWS?
Nobody said anything about AWS...
> Hosts make a lot of sense for small companies
Sure, nobody is disputing their choice of colocating themselves.
> OS or hardware choice you're making "vendor lockin" to that platform
It is abundantly clear that the vendor lock-in refers to single sourcing your hardware. That problem is nonexistent on Windows, Linux, BSD, etc.
> I think one Mac Pro probably replaces 4 or 5 Linux boxes
Oh come on, now you're just talking crazy... see other posts in this thread for a cost/performance comparison.
> you're not understanding what it is that they are doing
On the contrary, I think I understand better than you. Do you perform a lot of image processing work on various platforms (including OSX and Linux)? I do.
> This seems risky from a business perspective: it's voluntary vendor lock-in.
If you want to run a business that builds/tests using the osx/ios ecosystem this is the only way to do it legally. Apple's licensing terms enforces this. Otherwise we'd be running OSX on generic pizza box servers since Apple's hardware is truly overpriced and not built efficiently at all for the datacenter (they work fine on desks). Apple really gimped the 2014 mac mini's btw. They perform worse than the high end 2012 mac minis.
You have to look at it as an entry barrier that protects them against competitors too. If they are the only ones offering this setup, they can charge very high prices and recover the initial investment relativelly quickly.
> look at it as an entry barrier that protects them against competitors too
What barrier to entry? Their customers don't care that OSX is running under the hood. You can offer an image processing service using any platform today. Sure, on Linux it probably wouldn't be as efficient, but it doesn't have to be. Scaling is a Good Problem to have.
Basically, as you grow, it helps to take a critical look at risk factors and the technical debt which contributes to that risk. The longer you wait to pare down that debt, the more expensive it is, and the more exposed you are to that risk. A little more work up-front saves a lot of work later on.
(I'm the datacenter manager at imgix, and I wrote the article)
I completely agree with your concerns, and I'm constantly evaluating our business for operational risks and inefficiencies. There's a lot of stuff that I can't share in public about this, but what I can say is that the math works out (for now): OS X graphics processing is worth the downsides. It may not always be the case, and we're built in a flexible way where we can make a change when it makes sense to do so.
How different is it? Aren't they dependent on OpenGL and Nvidia/AMD GPUs/Drivers themselves? Wouldn't it make better sense/efficient to invest in becoming platform agnostic and optimize this.
I only say so cause it seems like Imgix could massively benefit from such a move and maybe look into other solutions which you currently can't consider (Custom ARM silicon - PowerVR-based servers, Professional AMD/Nvidia GPUs, etc)
Without going into too much detail about our stack, there's quite a bit more to Core Image than OpenGL+graphics card driver.
Those are important components, and we're not talking about splitting the atom here, but Apple has had a number of smart people working on graphics technologies for a long time now. imgix also has a bunch of smart people working on this, but for a much shorter period of time.
> What if Apple decides to change the Mac Pro form factor for the next iteration?
It seems as though they're prepared for this. This version 2 of their process is already moving away from an existing Apple form factor to a new one. It doesn't seem to be a leap in logic to consider that, should a new form-factor be released, they'll modify their rack cases again.
"What if Apple decides to change the Mac Pro form factor for the next iteration?"
Given recent history, that's not going to be for a number of years.
"What it says is they are willing to throw a ton of money and effort towards (very cool) custom hardware, but are unwilling to hire a person to write optimized OpenGL shaders for Linux, which would work on pretty much any other server they choose to build/buy/lease/cloudify."
Hardware will almost always cost less than engineers.
> Given recent history, that's not going to be for a number of years.
That is something that no one outside of Apple can say for certain. It doesn't even have to be a major change, but something like rearranging ports, adjusting taper or extrusions on the chassis, etc. Those kinds of adjustments happen all the time on consumer hardware, and most people don't notice, but may be an issue if you're trying to fit into precision machined slots.
> Hardware will almost always cost less than engineers.
For commodity, off-the-shelf hardware, absolutely. This is anything but, and still requires engineering effort to design, fabricate and assemble. And it's not always about the immediate dollars: sometimes a fundamental reworking means sacrificing short-term savings in favor of the long-term: flexibility, risk mitigation, reduced operational complexity, and cost over successive generations of hardware.
There are definitely risks, but I do want to gently re-iterate that we're not blind to them.
About the only thing that Apple could do that would render this chassis obsolete is to substantially change the exterior dimensions of the Mac Pro. Obviously if it's a different shape, we would have to adjust things.
If they kept the same shape but modified it somehow, the only dimensional change that would be truly tough to accommodate is an increase to circumference. This is the dimension with the least wiggle room built-in, and it would cause some headache. We would probably have to sacrifice some density by removing 1 chassis from the rack.
Otherwise, changes to ports or minor adjustments to the length of the chassis can all be accommodated for in this chassis design.
> There are definitely risks, but I do want to gently re-iterate that we're not blind to them.
Yep, I was just responding to the assertion that it wasn't a risk.
For what it's worth, it does sound like you've thought this through really carefully, and thanks for taking the time to explain so thoroughly and respond to everyone.
What if Apple decides to change the Mac Pro form factor for the next iteration? Then you have to retool and are left with a bunch of incompatible chassis. What if Apple stagnates with hardware upgrades? You'd be stuck running obsolete hardware. What if Apple discontinues the entire Mac Pro line? Not to mention the price premium of Apple hardware itself, then the time and expense incurred to design and fabricate this.
The fact that their software depends on Apple's graphics libraries doesn't seem like a good justification for doing this. What it says is they are willing to throw a ton of money and effort towards (very cool) custom hardware, but are unwilling to hire a person to write optimized OpenGL shaders for Linux, which would work on pretty much any other server they choose to build/buy/lease/cloudify. Certainly there will be other "debt" to overcome, especially if much of your codebase is non-portable Objective-C or GCD, but that has to be weighed against the possibility of your only vendor pulling the rug out from under you. And looking at Apple's history, that is a very real possibility...
Owning your hardware like this makes complete sense if your core business is directly tied to the platform itself, eg an iOS/OSX testing service. But as far as I can tell, imgix does image resizing and filters... their business is the software of image processing, and they're disowning that at the expense of making unrelated parts of the business more complicated. Not a good tradeoff, IMO.