Ok, so first and foremost I am incredibly sorry, Jacques, for the experience you had. I can say with confidence that your experience with WP Engine is out of the ordinary, but no excuses. I’m writing this as a way of owning up to mistakes that were made on behalf of our company, as well as transparently engage the discussion here. There's a couple of points I'd like to make, if I may...
We're in the process of moving to complete 24/7 support right now (we do have emergency 'site down' 24/7 cover) - as any entrepreneur on HN will tell you, getting your startup to take off is hard and scaling the human aspect is perhaps even harder. With mostly US-based business customers it wasn't as much of an issue to begin with but we're now addressing that.
Obviously if we'd had that 24/7 support in place we could have addressed this more immediately.
However, one of the reasons that your site experienced problems was that there were some aspects of the site development that simply didn't scale. Like any scale-orientated PaaS, we can provide the infrastructure to enable you to scale but your code needs to work in partnership to fully achieve that.
As Sean, my head SysAdmin wrote on a ticket to you, some of your pages were performing a sort on 188590 rows in memory each time. That just doesn't scale. From your post, I'm guessing this is the issue that also occurred when you tried Pagely. While it's awesome that the good folks at Pagely were able to work with you a little to try to address your problem, like most PaaS providers (eg AWS, Heroku) we don't provide consultancy services - there is an amazing WordPress community of consultants and dev shops out there and its just not what we want to get into. Just like Heroku won't fix your Ruby code.
We have many, many clients who are very happy with WP Engine but as a growing business there will always be customers who find that the service isn't right for them, and even occasionally have a bad experience.
So lets try to make this right. I can see that we've already refunded you all the hosting fees you paid for your account, and I'll also make sure we pick up the $500 migration bill that you incurred. If there is anything else we can do please let me know.
I will also take time once I've finished my coffee and out of my PJ's to analyze your case further and see what else we can do internally to ensure this doesn't happen again.
Ben, co-founder WP Engine
Ben, I really appreciate you taking the time to post this here and demonstrate that you take this seriously, but your cofounder is currently publicly ridiculing customers who have paid good money and been dissatisfied with your service.
As a current customer of WPEngine (who has had issues), nothing makes me want to bail on you guys as much as those two tweets. From the CEO.
EDIT: This is more what I expected to see from Jason, though even this has a weird tone: http://news.ycombinator.com/item?id=4086553
I'm neither justifying nor criticizing what he said, just let's be careful not to burn someone or something based on one mistake that we all can one day find repeating.
His clarification tweets:
I'm sure he'll be more careful on public tweets next time, but I think jumping to conclusions on his company etc is going too far.
He made a mistake, obviously seems he learned from it, and we all learned a nice little lesson in customer service, public relations and social web dynamics.
@billmcneely yup, there's always someone bitter and hateful. Sometimes they even have a point, but choose to make it in a destructive way.
@P_Doubleya @gozmike thanks guys. That's also why I ignore HN. There's always haters.
Of course you may disagree with the tone, or the content, but disgusting is a little strong isn't it? Can we dial down the hyperbole a little?
I'll grant you that "ridiculing" is probably not the right word...maybe dismissive or condescending would have been more accurate.
And yes, I do find it disgusting that he's publicly behaving this way towards a customer.
It's fair to criticise those tweets and I don't feel they are appropriate really, but I don't think they should be classified as disgusting - misguided and inappropriate at this particular time, perhaps, but not disgusting. Isn't it better to deal with situations like this without escalating emotions further and degenerating into both sides insulting each other?
I think a great heuristic here is this: your customers can be publicly negative, but you cannot.
If you publicly flame a customer then, no matter how deserving the customer was, some future potential customers will look at that and say "That could be me." Most businesses, I think, can't afford to risk that.
There are some notable counterexamples, in my opinion they are the exceptions which prove the rule, i.e. when the business wants to fire the customer and make an example of him/her for others.
The best example of this is the Drafthouse movie theater in Austin, who created a great PSA wherein they ridicule a customer for insisting she had the right to use her cell phone during a movie: http://adland.tv/content/drafthouse-movie-theatre-makes-keep...
I agree 100%, the tweets are not helpful, but I disagree with the "disgusting" description and felt it was a little over the top.
All I want to see from the CEO in a situation like this is: "I'm sorry. We fucked up. Here's how we'll make it right, and here's how we'll make sure it doesn't happen again."
And if you genuinely can't say that because you don't think the customer is right and you're not willing to do anything for them, just keep your mouth shut.
The completely different service refund wouldn't be necessary if the not different service had been up to expectation in the first instance.
I feel absolutely justified in being refunded that amount by WPEngine.
Would you consider it reasonable to be the subject of the manager's tweet, or open letter or any other kind of public message calling you someone who cannot be pleased, who has an agenda against the store, is disposed toward bitterness, and is communicating destructively.
How is that not disrespectful?
I actively ignore Twitter for the same reasons some might avoid HN, but that doesn't mean that there aren't valid and intelligent conversations to be had there, particularly if you have to deal with customer service and PR.
FYI, the problem is with the Recent Comments widget code that ships in the Wordpress base install. Page.ly had the same database connection dropping issues you did and I presume for the same reason. (edit: By the way, use the db-error.php facility to put up nicer error pages. It's also a really obvious point to insert some monitoring logic -- I think customers would like it if you noticed their problems before they do).
The core issue, as I said elsewhere in this thread, is that unless you partition the underlying wp_comment tables, performance becomes steadily worse as the total number of comments rises.
The only reason that this was never a problem for me on my own servers is that I am loltastically over-provisioned for my requirements. I just never saw it.
You will see it more in future. Like I said: this is not a badly behaved 3rd-party plugin. This is mainline code. Your customers can reasonably expect that you can run Wordpress out of the box without degradation.
None of changes the fact that my original migration experience was a total disaster. I know you're experiencing growing pains.
Well here's one of them. I hope it serves you and your future customers well.
Yet in my pinboard I have this bookmarked: http://pento.net/2011/04/28/partitioning-the-wordpress-comme...
Yup, I had to partition the comments table.
I've been watching this too: http://wordpress.org/extend/plugins/postgresql-for-wordpress...
I'm willing to accept fractionally slower page loads, as for the most part they're cached heavily post-generation. I'd feel far more comfortable partitioning in Postgres not least because I'm far more familiar with it.
Since our customer was looking for a more agile development & deployment workflow I personally recommend them switching from us to WPengine based only on Jason reputation and your technical expertise.
Almost a month has passed since they signed in for the service, $500 in fees and ~ $700 on WPmovers afterwards, they keep hosting with us and couldn't arrange a 0 downtime migration at all, neither provide a fixed quote for the migration.
Our case right know is being handle by Sergio, maybe you can talk with him and our customer to find a happy ending?
I've always been an advocate of WPengine and big fan, but after our bad experience and seeing others having similar issues too, I'm probably not recommending WPengine in a while..
The Most WordPress Experts Per 1000 Customers…
Great support? Many hosting companies claim this. But when you’re hosting WordPress, you need specific support. If a plugin breaks, or a theme file doesn’t work, and you’re hosted with a typical hosting company, they’ll throw up their hands and say “Not responsible! We just provide the server and the bandwidth, buddy.”
Not so at WP Engine. Our staff consists entirely of WordPress experts, so we can help you track down and troubleshoot WordPress issues.
Which is fine, but then you website says this:
Which is totally misleading, and exactly opposite of what you say. I wouldn't blame the customer for dissing.
Edit: Quote formatting
You quote 'we can help you track down and troubleshoot WordPress issues.' - that's exactly what the support team helps customers with all day long.
But we don't get into consultancy around helping to build custom code, etc.
There's always going to be a point where a hosting provider will not be able to support you further. Many dedicated providers and VPS providers will ensure that the machine is powered up and has connectivity and leave you at that. What we're trying to communicate is we go a lot further into supporting your use and configuration of WordPress - but not into consultancy.
I know Jacques didn't mention that the problem was with the stock installed-with-Wordpress Recent Comments widget until a couple of hour after you posted that comment, but _surely_ someone at WPEngine knew already that this wasn't a "custom code" issue?
From what I'm reading (and please correct me if this is wrong) a stock WP install with no plugins or custom themes, but with several hundred thousand rows in the comments table and the Recent Comments widget running - will crash hard on WPEngine (while running just fine on a "loltastically over provisioned Linode VPS).
In my view - that's a "Wordpress support issue" suitible for diagnosis by a "Wordpress expert", and a long way from "consultancy" for "building custom code".
It's true there needs to be a limit to what customers expect from hosting companies, but like it or not, you've set the expectation bar _very_ high with the marketing copy on your homepage, and you've got a wonderful social media reputation based on those expectations. At some stage, you need to suck it up and dedicate the time/expense to live up to that reputation, or inevitably social media will bring your reputation back down in line with your business behaviour/policies - and the wider the gap between the expectations you've set and the service you deliver, the more ammunition you're handing over to "the haters" to shoot you with.
Had to move our highest traffic site back to GoDaddy (which makes me cringe, don't get me wrong). Doing the same with our other two sites.
That said, I believe in what you guys are doing, think there's space in the market for someone like WPEngine ... it's just that, in my experience, you guys are executing poorly and dropping the ball quite often.
...if I were in your place I'd drop the "solving this while yawning in my PJs" relaxed attitude and prepare to bring a sleeping bag to work and stack a fridge full of red-bull (or whatever makes you "fly") ...or try and hire someone like Ledorf ...'cause all the other "PHP gurus" are at Facebook now ...tough luck. I'd hate to be in your place, but no matter how much I hate the "PHP ecosystem" I hope you guys manage to make this work
Ben, I don't know Jacques and don't pretend to speak for him, but I gather from his post that the scaling issue, and you not being able to accommodate him, would have been no big deal if he had not already been through the nightmare of migrating with bad documentation and support not handling that properly.
I think cascading frustration is what got him.
I'm incredibly proud of the business the team has built, and know we are still an excellent place to host WordPress. But yes, incidents like this give us opportunity to revaluate aspects like our documentation (and support hours) to double down and make things even better one we resolve them.
As Sean, my head SysAdmin wrote on a ticket to you, some of your pages were performing a sort on 188590 rows in memory each time. That just doesn't scale.
From a customer service perspective, this comment leaves a really bad taste in my mouth.
You guys say right on your home page that you're "infinitely scalable." And then if you click through to http://wpengine.com/scale-to-millions-of-hits-a-day-or-hour/ you read that WPEngine features "secret sauce" made of "one of the most scalable WordPress architectures on Earth" that makes you "faster and more scalable all the time." The (strong) implication being that any WordPress configuration will perform better on WPEngine than on an alternative hosting platform.
So if someone comes to you with a WordPress configuration that you can't make "faster and more scalable" by running it on your platform, isn't The Right Thing To Do to say "oops, our bad, sorry we misled you" rather than blaming the customer for having a bad WordPress configuration? Because that's what you're doing here -- blaming the customer.
In fact your comment basically boils down to three parts, only one of which is reassuring:
* Paragraphs 1-4 translate to "we're a growing startup and growing customer support is hard," which frankly if I were a customer I wouldn't care about in the least. Scaling your business is your problem, not my problem. We know it's hard, but it's like Tom Hanks said in A League of Their Own (http://www.youtube.com/watch?v=tPqYnC-SW5w) -- there's no crying in business.
* Paragraphs 5-7 translate to "your web site sucks, no wonder we couldn't scale it, nobody could," which is the blaming the customer part. I don't doubt that that is true (WP can be a real hog, especially if it's hacked up), but if you're advertising yourself as The People Who Can Scale WordPress, you shouldn't be surprised when people start coming to you with funky, hacked-up, hard-to-scale WordPress sites. Who else would someone with a site that can't scale go to but the scaling experts? When that happens it's OK to step back and say "we can't scale this," but it's not their fault for taking your marketing materials at their word.
* It's not till Paragraph 8 that we get to the "let's try to make this right" part, which is where this comment should have started.
I don't say this to be a pain or to make what I'm sure is already a difficult day worse. I'm saying it because I like what you guys are trying to do, and responses like this hurt you as much (if not more) as they help.
I've had to deal with situations like this myself, and no matter how tempting it is to blame the customer -- even when the customer is manifestly at fault -- it never, ever, ever improves the situation. It just makes things worse, first by giving the angry customer something new to be angry about, and second by deflecting you from the more important question of what you can do to prevent the situation from happening again. In this case, for instance, better screening of incoming customers to ensure their sites really can be scaled on the WPEngine platform would filter out problems like this before they become problems. (IIRC you've already started doing this, by blacklisting a small set of known-bad plugins. Maybe that list needs to be beefed up?) But time spent sniping at customers is time not spent making those improvements happen.
No, there's no inference or implication that any (ie regardless of how its built) WordPress configuration will scale on WP Engine. As I wrote in my prior comment, scalability is a partnership between infrastructure and code. That's the case irregardless of framework and PaaS provider.
We like car analogies at WP Engine - if I sell you a racing car and you're only a moderately good driver then you're going to crash when you approach the corner at 200mph. There's little I can do about that other than teach you to drive, and I'm in the business of building the cars. There are plenty of folks who can help you improve your race-craft out there.
To say that we've missled someone is an unfair assertion.
If I analyze the suggestions you're making in your comment here...
* Jacques complained about lack of 24/7 support, but I shouldn't have tried to reply and explain the reason for that because 'there's no crying in business'. ?????
* The WP Engine website shouldn't mention scaling WordPress because it's not technically possible to scale all WordPress code regardless of how it's written. (What should our site say? What should any scale-orientated PaaS website say given the same issues apply to anyone in this space)
* WP Engine should vet and screen it's customers so that they must prove their code can scale before we sell them an account? ?????
I'm sorry, but if that's the advice you're offering then, with respect, I don't agree.
That being said...
No, there's no inference or implication that any WordPress configuration will scale on WP Engine.
I would humbly suggest that your entire Web site is a refutation of this assertion. Even moving beyond the copy on the home page to the fine-print "Common Questions" section of http://wpengine.com/pricing/ , there's no disclaimer or other "are there WordPress configurations that WPEngine can't scale?" fine print.
My point here isn't that you shouldn't promise scaling, it's that it's not hard to see how a mismatch could arise between your expectations (scaling is hard, customer code can make it harder) and customers' expectations (scaling is easy, WPEngine has figured it out).
Jacques complained about lack of 24/7 support, but I shouldn't have tried to reply and explain the reason for that because 'there's no crying in business'. ?????
Yes. If a customer complains about a lack of 24/7 support, you say "I'm sorry, but we're not in a position to provide that level of support at this time." And nothing else. The reason why you're not in a position to provide it is irrelevant to the customer, and risks sidetracking the conversation into a blind alley about the reasons.
It also feels a little bit like pleading for understanding -- "cut us some slack, this stuff is hard!" -- which is off-putting. Of course it's hard, that's why people pay you to do it for them.
WP Engine should vet and screen it's customers so that they must prove their code can scale before we sell them an account? ?????
As I said, you already do some vetting by disabling plugins you know to be performance-killers. Look at this list: http://support.wpengine.com/disallowed-plugins/ , there's an entire category called "CPU/MySQL Thrashing Plugins."
I'm not suggesting that you need to give your customers a full CIA background check before bringing them on board, but you've already moved beyond just taking whatever code customers throw at you. If there's other things that need to be disallowed to ensure that WP sites work on your platform, the thing to do is to find out what those things are and add them to the blacklist.
Again, just my $0.02, IMHO, YMMV, IANAL, etc.
Use some common sense.
"infinite scaling" means _exactly_ that. If you don't want people to use accepted dictionary definitions of words like "infinite", you should not use them in your marketing.
A few telcos here in Australia have recently lost court cases about their use of the word "infinite" in their cellphone plans/marketing, and then cutting off or charging extra to customers who "used too much". The court, rightly in my opinion, said that the meaning of the word "infinite" is clear and well known, and any claim that it included "limited, of course, by common sense" is bogus. Be careful of letting your marketing team write too much hyperbole - occasionally you'll be held legally accountable for it.
I suspect if the amount in dispute had been an order of magnitude or two larger, Jacques may well have had a strong enough case to find a no-win/no-fee lawyer to go into bat for him... I know I'd prefer not to have to explain to a judge how those first two boxes in the first column on the WPEngine homepage didn't cover all of Jacques complaints.
I can create an infinitely scalable server and you can come in and drop a sledge hammer on it...does that mean I shouldn't use the word.
You have to understand the layperson won't know anything about tables etc and stating "infinitely scale" is a HUGE selling point for potential customers. If they wanted to be truthful they wouldn't say that, or they would say we CAN provide infinite scaling IF the following conditions are met.
The way I see it, you are not selling the car, but providing a driver so that I don't have to worry about how it should be driven.
I should sit in the back and relax while you make sure that my car is driving safely and comfortably.
Maybe your message isn't clear enough if you think that your product is the car. From your website:
QUIT WORRYING AND LET US RUN WORDPRESS FOR YOU
QUIT WORRYING AND LET US DRIVE YOU CAR FOR YOU
It seems to me wordpress.com is the "we supply the racing car and driver" solution provider. You don't have to worry about how it gets driven, but you also have only limited say in what car you get and which lines the driver takes through corners.
In my version, WPEngine are providing the race track - to which you bring your own car and driver. They go to a bit of effort to improve the safety, mostly by ensuring other track users don't do too much to affect your performance, but you're free to bring your own F1 car, MotoGP motorcycle, Nascar, or go kart; and run at whatever pace you think is appropriate.
Seems to me the OP's problem was that he wanted to run a top fuel drag rail. From his perspective, he made assumptions about what "a race track" meant that didn't include "decreasing radius downhill hairpins". Meanwhile WPEngine's website/marketing didn't anticipate that class of user, and didn't think they needed to point out that their track isn't an ANDRA approved 1440' long straight with a mile or two of slightly uphill runoff at the end.
With my sympathetic hat on, I can easily understand why both parties assumptions were sensible to them when they made them.
With my cynical hat on, it's pretty obvious that you could install a Wordpress theme or plugin on anybodies hosting platform with an O(m^n) or O(n!) function it it somewhere, and no magical cloud autoscaling pixiedust is going to solve your problem. At the same time, the expectation that a company claiming "Insanely Fast. Infinitely Scalable." and "At WP Engine, there’s no “first level” of support–our entire staff are WordPress experts, so you never hear “We don’t know how to do that.”" on their homepage should be able to do a better job keeping a Wordpress site up than some time-limited non-technical guy self-hosting on Linode.
Edit: and Wordpress is a monster. But that's why I wanted to pay you guys in the first place.
FWIW, it's clear that you _did_ mislead Jacques.
Whether the assumptions he made were valid or not (and frankly, based on the content of your homepage, it's hard not to have sympathy for his point of view), he clearly expected you guys would do a better job hosting Wordpress that he could do himself on his own Linode VPS. He also clearly expected that if there were problems, your "Expert Wordpress Support" would provide the solutions. Instead he found your support guys effectively saying "we don't know how to do that" and flicking him on to WebSiteMovers, who _also_ end up saying "we don't know how to do that" and charging him $500 for failing.
So I think you did let Jacques down, and I think he has every reason to feel missled.
You should probably acknowledge, at least prominently enough that anyone researching your service with a view to becoming a paying customer will find it, that there are possible pathological cases and configurations which WPEngine's technology/infrastructure/service will not solve. If you don't want to feature this on the homepage (and I can understand you not wanting too), I'd suggest a note near the top of the Disallowed Plugins page - and probably re-wording that page to make clear that avoiding everything on the "complete list" of disallowed plugins doesn't guarantee a successful deployment, and also that it's possible for themes with certain code patterns in them to cause a WPEngine deployment to fail to perform as advertised.
Secondly (and I suspect you've already got this one in hand), you need to train your support staff to recognise more quickly installations with pathological performance things going on, and apologise and extricate yourselves and inform the customer as early in the process as practical. I don't know what the tone/implication in Seans message about the ~188k row sort was, but from reading your defence here I can't help but think there was at least a bit of "you're doing it wrong!" - which even when it's true, it's not the wording you want to feed an unhappy customer.
(Note: this is written as a happy WPE user, with some curiosity about exactly what went wrong, and why some guy with a Linode VPS can, in some cases, keep a Wordpress site up better then WPE. I'm suspecting he's got define('WP_MEMORY_LIMIT', '2048M'); or something similar covering up some quickly hacked together "it works for me" not-very-well-though-out-code - which you can get away with on your own hardware/VPS, but not generally on shared hosting...)
You guys say right on your home page that you're "infinitely scalable." [...] The (strong) implication being that any WordPress configuration will perform better on WPEngine than on an alternative hosting platform."
I agree with your general point that, from the standpoint of the entrepreneur, "the customer is always right" should be words to live by. On the other hand, in the paas industry, it should be axiomatic, i.e., understood by all competent parties, that horizontal scaling (the "infinitely scalable" bit) manifestly will NOT address problematic architectures, n+1 queries, full table scans, poorly/non-indexed schemas, etc.
This, to me, is not blaming the customer, it's laying out in reasonable terms assumptions any competent developer or software entrepreneur should understand long before they employ a paas.
Your larger point, though, is valid: even after bracketing the given assumptions of what the responsibility of a paas truly is, better communication up front could temper expectations as to what "infinitely scalable" (a bold phrase, to be sure) really means to the individual customer. It doesn't mean a paas can fix a application that locks the datbase for a full second per query, etc. It does likely mean (I'm assuming here, I have no experience with WPEngine) spinning up new instances and handling increased loads should be Heroku-simple.
In this case, I think WPEngine is taking the hit for what they should (bad communication on the support side) and explaining in real terms what paas's can and can't do. There's a difference between "blaming the customer" and simply being honest about what the bottom-line issue is. NOT addressing this side of it may have long-term implications if another customer comes along with similar expectations. Being clear about what the client's responsibility is doesn't necessarily equate to "blame."
The problem with this is that the original post doesn't say "I have a site that was crashing and locking up all the time, so I took it to WPEngine." It says "I have a site that was running fine on a VPS at Linode, but I got tired of the hassle of managing the VPS, so I took it to WPEngine."
The original post is written by the complainant, of course, so it's possible it's subjective or slanted. But if it's correct on this point, I don't think it's unreasonable for a competent party -- in this case, a person with a WordPress site that runs acceptably on a commodity VPS -- to expect that it would run acceptably on WPEngine as well. Presumably performance killers like the ones you cite would be just as bad on the VPS -- if not worse, since it's unlikely that a VPS' operator will be able to tune MySQL, PHP, Apache, etc. to the degree that an expert at a place like WPEngine would.
There's a big difference between being infinitely scalable, and being infinitely foolproof. If your PaaS exposes enough power to the user, they'll find a way to shoot themselves in the foot occasionally.
I can understand that the customer is frustrated with having to change his code to move from a VPS environment to a cloud environment, but that is the unfortunate truth when scaling. Either you 1) get closer to the real hardware and throw resources at it (more ram, vps, or dedicated machine) or 2) change your code to be horizontally scalable on light cloud instances.
Maybe this is chance to create a help-page that explains to users what resource limits they should stay below (memory, cpu, queries) to handle peaks in traffic and how to identify problem points (bad SQL queries, hungry plugins).
A friend of mine once said, "a customer will defend his bad decision to the death if you point it out to him."