For all who are thinking about moving to WPEngine:
Your staging server does not use WPE's caching technology, which your production server will use. WPE will not give you details on how their secret-sauce caching works. This means that things that work on staging will break on production. WPE also refuse to turn off their caching on production, but also refuse to turn ON their caching on your staging server. This means there is no real way to test. Things that work with their staging server, and even work with the standard WP caching plugins, will break in WPE production.
Their solution? Buy another production site to test on.
That website is now moving from WPEngine after having its launch botched by the aforementioned issues, and after not being able to make a basic paywall work with their caching. Once they realised the use case, WPE simply suggested moving elsewhere without making any further effort to accommodate.
I had a similar reaction to the post yesterday by asmartbear as Jaques, but he made the extra effort to actually write the blogpost, so bravo to him. WPE enjoys this sterling reputation that misleads a lot of people into using them, but their comfort zone it seems is limited to vanilla WP installs with little deviation.
So we don't apply the caching stuff to the staging server because the purpose of the staging server is to stage content and test new plugins and themes.
We invented this feature in response to bloggers and site owners who wanted to be able to test a new plugin before adding it to the live/production version of their site. We're also the only managed WP platform to offer such a feature as standard with all accounts.
I can see that in a true dev/staging/production test suite environment, it does't stand up as a true mirror of product for the reasons you stated - but then neither does the staging/local version of Google App Engine or Heroku offer the same exact environment of their production platform.
If you need true like-for-like mirroring of production and staging, then yes, you need to create another WordPress instance - that's the case for most providers. Also its wroth nothing all WP Engine accounts, other than the Personal plan, come with multiple WordPress instances, so rarely that means actually buying anything further.
The solution with Heroku is to create a separate app and use that. The app will be identical except for having fewer (and optionally different) resources. Because it's billed hourly, it doesn't cost too much to run it at full power for a short while to answer a question, before scaling it back down. The reason you needed to "invent" this feature is because your environment isn't as self-service as it could be.
This is not true for App Engine. You can create a "staging" version of your app and access it at staging.my-app.appspot.com. It all runs on the same infrastructure that your main version runs on, and it's easy enough to make it admin-only. Once your satisfied that it's safe to deploy, it's two clicks to change which version is serving.
You may be the only managed Wordpress platform to do so, but App Engine has had this functionality for a long time.
I was helping a friend debug something on WPEngine. We learned that the caching doesn't allow you to set cookies it would seem (from PHP). Had to do it all using JS instead. I thought I was going crazy trying to debug the code when it was definitely an issue with the way they cache.
Not saying it was right or wrong, it just wasn't a fun experience thinking I was going crazy. And hopefully if anyone else is trying SETCOOKIE() and it's not working they also realize they will have to use JS.
That was my issue, and yes, it absolutely is one of the many reasons why staging instances should have a caching on/off switch that mirrors production. It is absolutely 100% feasible that WPEngine's caching will break your site, and you need to know this before you push to production.
Don't get me wrong, as a contractor to a company that uses WPEngine, we have had a great experience with it - but e.g. in the case of setcookie() being completely useless for unauthenticated users, some documentation / warnings / big red flags would be nice.
In my case I was trying to build a simple site selector for unauthed users that would forward them to their selected site every time they hit the main domain. Nothing exotic by a long shot.
It sounds like you and ohashi don't understand how caching works. WP caching serves up static copies of your pages, bypassing the code and DB calls that would normally be required. Since your code is bypassed cookies won't be set.
This isn't something specific to WPEngine, this is how all the good WP caching plugins work.
I've had the opposite experience of WP Engine, and appreciate leaving caching and performance issues to somebody else, given that my company has a somewhat limited staff for supporting servers and CMS upgrades. I am building a semi-complex Multisite installation with multiple domains. So far, it's been pretty decent.
Your frustration with their caching "secret sauce" is understandable. However, you can get a rough idea of their caching back-end by poking around in the WP Engine config files and plugins bundled with your installation. They also provide a nice little overview here: http://wpengine.com/our-infrastructure/.
If you are staging out anything more complex than a single blog, you will definitely want to have two production instances to test on. For some, this is a raw deal. I personally don't mind.
I am also not a huge fan of their CDN feature, which works by rewriting URLs in your page source to point files and assets to NetDNA servers. However, I am hopeful that they will improve upon it.
That said, I have enjoyed a platform that keeps my Wordpress installations secure with automatic upgrades.
Comparing WPEngine to other hosting providers like Linode or Dreamhost is really unfair. They offer completely different services, and specialize in Wordpress-optimized hosting only.
I also have to say that it was really nice to not be part of Amazon/Heroku's outage this week. WP Engine has a control panel hosted on Heroku, but all their sites live in the Linode network, evading downtime.
Page.ly was awful. When my sites weren't down due to botched infrastructure upgrades, users would get locked out of the admin interface by "FireHost Website Protection" (FireHost was/is their underlying hosting provider). The same sad cycle repeated multiple times:
1. Email FireHost with the event IDs and a request to unblock.
2. FireHost doesn't reply for 24 hours, or if they do it's with "we can't find that event in our logs".
3. I open a Page.ly ticket
4. "El Vaquero Furioso" (yes, that was his name in Zendesk) replies that the issue is fixed and won't happen again.
5. I ask for the root cause and what steps were taken.
6. Crickets, or a snide response from the founder that "we're all on the same team".
Sorry we handled that poorly. At the time Firehost was having some significant issues with their firewall policies. We do constantly walk that line between enforcing security at the edge and annoying customers that may run into WAF blocks. It's a delicate balance.
The problems you had have since been resolved and pagely is running under a new firewall (and new blades) which are much more reliable.
In addressing your frustration my comment of 'we're on the same team' was to demonstrate that we were all working towards a solution for you. As professionals dealing with other professionals we aim to acknowledge and address the problem while steering the conversation to a positive outcome. If our customer views us as an adversary it makes the process more challenging for all involved. By re-framing the conversation as we are all in this together, all working towards a solution, we feel it is more likely the positive solution will be found and implemented sooner. My comment was a gentle reminder to that effect: acknowledging the issue, and we are working with you, and with our partners to remedy it.
Thank you for your past business, and we hope one day to earn it again.
- joshua strebel, founder pagely.
I had the opposite experience as well -- I tried Page.ly first but could not import any of my sites (which all use a variety of custom post types). Their import process seemed more attuned to vanilla installs. Closing out my billing with them was also somewhat arduous.
WPE has been great so far -- very responsive support -- and their caching handles about 8 million pageviews a month with 600-800 simultaneous visitors and very little downtime.
Basically all services have a group of satisfied users and a group of unsatisfied users.
What blows a story like this up (as much as hitting the front-page of HN is 'blowing up') isn't that a service failed to meet the needs of one blogger, but rather that the people running the service respond to criticism (in public) sounding more like petulant children than professionals.
As someone not into blogging, I'm not that familiar with WPEngine, but based on comments here that link to various twitter comments now I suddenly have a negative-leaning opinion of them. Primarily because they couldn't take a negative review without calling the poster a "hater". (FWIW, a lot of my negative opinion is tied up in this stupid word, which IME comes out as a last resort when the person using it knows there's a lot of truth to what the other person is saying but they refuse to acknowledge it for whatever emotional reasons).
To be fair, any WP hosting company is going to run into issues with non-standard installs, and they're not going to be able to accommodate any setup. If you have a customised WP install, it might not work on this sort of provider, they're optimising for speed, not dealing with, and they probably don't want the headache of trying to support any plugins or modifications that you might bring to WP.
It would make sense for them to offer a dev plan with every production plan though, and let you upload your site, and serve it to a limited no of people as a testing ground. Given their 'Power Tools for Power Users' stuff on their website, they really should consider at least offering staging servers for testing.
I guess at present they're just not a good setup for developers trying to modify WP, and are instead trying to capture the market for hardened WP hosting for mostly vanilla installs.
"and are instead trying to capture the market for hardened WP hosting for mostly vanilla installs."
I don't know of any large WP installs that would need the benefit of outsourced WP hosting/experts that use 'vanilla installs'. They may exist, but I can't imagine they're a large portion of the market. Perhaps my scope of experience is too narrow in this respect?
I had an issue where an API call was failing for Googlebot and other crawlers. I was worried about this being seen as "cloaking," so I was trying to figure out what was causing. Couldn't be my code. The API provider said it wasn't them. I finally realized that it was WPEngine when I visited my staging server as Googlebot. The content that was supposed to show up showed up!
Part of their caching setup was causing the API request to fail for a select list of bots. The issue was escalated to a Sys Admin who figured out what the problem was. He tweaked my server and fixed the issue for me within a day. I think that is pretty good.
My point... if something is broken and you think it might be WPEngine's caching. Try it on your staging server.
Of all the things I've ever written on the blog (and ok, most of the earlier stuff is as tedious as watching the guy who watches the grass grow), it's sorta sad that only the one with the sensationalist title has ever made the front page at HN.
So, while I'm here, go look at my very nice bloggers. I regularly submit their stuff.
But now you are posting off-topic comments to that sensationalist post. This does harm to your credibility as a complaining customer, frankly, because now it seems you have an ulterior motive for complaining so loudly.
Personally, I downvoted the comment and suggest you delete it.
Curious why people are downvoting. This is a legitimate question. He refers to these bloggers paternalistically and seems to take some credit for their work, but doesn't actually specify his relationship.
Good for him for sponsoring people, but it's a bit odd to imply he's deserving of any credit for their work simply for hosting a Wordpress instance. If they are paid employees, then that might be more appropriate, though still odd.
I host several sites on WPEngine and we've had some relatively minor issues. But we're getting ready to switch to multisite and now I'm nervous. We're generally happy, but we haven't been blown away.
Jason, if you're not scrambling to deal with this mess, you will be soon. And the main message I have is that you need to back off the marketing side a little and let the technical and customer service side catch up.
Godaddy makes an incredible amount of money, but they're still a terrible company from a technical and customer service perspective. But their marketing is so effective that they can get away with that .
I wonder if Jason Cohen's position in the tech community and general marketing prowess has allowed WPEngine to get ahead of themselves? I know when I read Patrick's blog post about WPEngine, I was really excited and it was a service that I wanted to like. And I do like them, but I'm not excited about them at all anymore.
1. Interestingly, Google is the exact opposite, and their customer service makes me want to take a spoon to my own eyes.
This story and comments have been flogged to death a bit already, but I do have a couple questions/concerns here.
While perhaps the issue ended up mostly being due to a Wordpress core issue, I question why this was an issue for a service that someone is expected to pay $250/month for?
The OP was able to, without a team of engineers/admins get the same content running on a VPS with no issue. I then question exactly what one gets when they pay $250/month for this service? To expect to get any sort of scaling capability out of a $8-$30/month shared hosting account is asinine and anyone running more than a small amount of traffic under such circumstances deserves what they get.
Why then if the end service was not capable of sustaining traffic/memory to the same degree that some shared providers are willing to go, would one pay $250/month? I think this is a question the creators of WP Engine genuinely need to answer...
This is absolutely on the mark. It is well understood that scaling Wordpress correctly requires a couple of core hacks (see this excellent comment: http://news.ycombinator.com/item?id=4693924). Why WPEngine does not incorporate these core hacks is beyond me.
WPEngine uses NFS/NAS which is a show-stopper. Dreamhost has the same issue, and GoDaddy (still?)
You never, ever, use NFS on something with hundreds of files to load for every page (ie. WordPress).
Every file takes 2-2.5ms to stat/load on NFS (from my tests on WPengine) which seems small but adds up fast.
It means you instantly have 700ms or more of overhead on the SERVER side to render a page (not transmit to the browser, just to render).
You can somewhat get around this by using an opcode cache and turning off file stat but there are still some files that have to be checked like static data read from the disk and turning stat off has other issues when editing code like having to flush the cache.
If WPengine is getting 2+ms response times then something is very wrong (are they using linux backed vanilla linux style NFS storage? ....bad, bad, bad). If not, then they need to investigate their load averages on their filers :P.
I don't think you follow that with a local file system, file stat (basically to see if the timestamp has changed or file modified) can be done directly inside the local memory cache.
On NFS/NAS it cannot be done locally, because other devices may have had access to the remote filesystem and changed files. So it has to travel the network to make the query. Even if it doesn't have to touch the disk remotely and can fetch it from a remote cache, it's significantly slower than local. Even if it's only 1ms per file, if there are 300 files, that's 300ms (a third of a second) on the SERVER side, before it can start sending output to the internet.
May as well throw in my own experience with WPEngine...
We run a relatively small member site (<1000 active members) that operates on WordPress. We tried migrating the site to WPEngine, but could never get the membership plugin working as expected. Over a course of about 2 months we tried everything possible to get it working, and talked with their support on a near daily basis.
Overall, their support team tried to be helpful. They weren't always responsive (sometimes we wouldn't get a response for a day, or we would be waiting on hold for a long time), and we ran into the problem of different support people telling us different things about how their system was setup. Eventually one of the co-founders got involved in the support, but we still were not able to resolve the issue.
I managed to find 2 other people using the plugin that had run into the same problem with us on WPEngine. WPEngine claimed that other customers of theirs used the same membership plugin without any issues. I don't feel they lied, but we clearly weren't alone in this problem.
In the end, we moved to another WordPress host and were able to get up and running in no time. Unfortunately, WPEngine refused to refund us fully for the 2 months we wasted trying to get our site setup because we went past the 60 day money-back period. (We had paid for the 3rd month already, and they gave us a partial refund of that month only.) It was a relatively small amount of money, but after all the trouble we had with them it would have left a much better impression if they would have refunded the entire cost we paid.
I'm sure that WPEngine has thousands of happy customers, and our experience was largely unique, but there are many things I felt they could have done a better job of.
As a seasoned WordPress developer and entrepreneur, the OP is making wrong assumptions. If I was the WPEngine owner, I won't be bothered by seeing him go.
WPEngine provides WordPress hosting services. They are professional and they do it for $250/month (which plainly means that their target audience is professional bloggers). For $250/month, they don't provide consultancy service as why your particular setup couldn't install in their server. This is your responsibility.
Setting up a WordPress blog can be easy (5 minutes) and complicated (Professionals charge up to $350/hour). What you are asking for is someone to move your blog. Fine. You are responsible for that. WPEngine offers you a "6 hours coupon" from another service. That is, they are not responsible for that.
Page.ly are probably doing this to get more clients. This is the wrong thing to do, because their support service can't scale this way. I have been doing this to bootstrap my services and it had more negative consequences than positive ones.
tl;dr: The OP is expecting WP consultancy services for signing up to WPEngine.
So you are the OP. I won't argue with you much. I'm in the WordPress business for many years. What I meant Page.ly is providing a generous support service to keep its' clients. This is a bad strategy because it makes wrong assumptions of what their main service is.
There's a section in Guy Kawasaki's Rules for Revolutionaries which talks about exactly this. Rules is still my go-to text for general business management advice. I'll quote the relevant parts here.
"Get Over the Paranoia: The story may be apocryphal, but Nordstrom supposedly allowed a customer to return an automobile tire that he insisted he bought at the store. Of course, Nordstrom doesn't sell tires.
"The paranoid would ask, ``But what if everyone who owns a car returned tires to Nordstrom? Nordstrom would go broke!'' But people won't (and don't), and Nordstrom will never be in danger of going bankrupt for accepting returns of goods it doesn't sell.
"Managers are afraid to implement customer-pleasing, revolution-catalyzing policies because they are afraid that too many people will take advantage of these policies, and they'll end up with the equivalent of a store full of tires."
The entire section goes on to talk about employee empowerment, allocentralism (putting yourself into your customer's shoes), examples from other businesses that either do this or don't, and so on; the relevant summary part from the text would probably be, "However, if your competition is asking people to do something suboptimal, then it's creating an opportunity for you."
So, straight up: it's certainly possible to run a business the way that you're describing. In fact, I'd guess that that's how most businesses run: they do as little as possible to make their money and meet -- but not exceed -- their customers' expectations. So you're not wrong as far as that goes.
But you are wrong to assume that they'll have "a ton of noobs" if they treat one particular customer really well. And, you're equally wrong to assume that having "a ton of noobs" is a bad problem to have; from another part of the same section of the book: "For example, a Whirlpool employee taped a news program's interview with Gail, a woman with several children and a full-time job. Whirlpool employees were challenged to provide appliances that would ``take care of Gail.'' In response, they redesigned the stovetop of Whirlpool's CleanTop to be completely flat, without grease traps or dirt-collecting crevices, and they created the Quiet Partner dishwasher, so the noise of the dishwasher wouldn't distract Gail. Viewing chores through Gail's eyes has helped Whirlpool introduce significant product enhancements."
Having "a ton of noobs" would mean a ton of new customers that are paying you money to show you what is wrong with your product or service. They are a great resource to have, if you take advantage of them. If you can identify one problem that 35% of "a ton of noobs" all have in common, and if you can automate the solution to that problem (or make it significantly easier to solve), then you have just achieved differentiation in a crowded market.
Yes, it can be painful in the short term. You might have to hire some more help -- and you might have to streamline some of your other internal process, like your ticket system, which will also pay dividends in the future.
Avoiding pathological customers is certainly good advice for freelancers, consultants, and really small shops. However, it is not necessarily the only good business strategy.
No, what he means is that Page.ly went above and beyond what is reasonable for the price you're paying them. Yes, it was very nice of them, and most likely they did it because they are a young and hungry company. At the end of if all, I don't really get a sense from what you wrote that WPEngine service was terrible. It sounds like you were a difficult customer with difficult architectural issues that were ultimately the root cause of your performance issues.
It it really an ordinary multisite install though? Do you have 188590 comments? Wouldn't you have trouble with any hosted providers if you are doing operations on that many records in memory (whether that is the fault of WP, or custom plugin code, it doesn't really matter)? If that's the case, it looks like your site just isn't a good fit for this type of service, but millions of others (including multi-sites) with say a few hundred posts and a few hundred or perhaps thousand comments are.
It's a shame that you had to go through all of this trauma to discover this, and there are obvious support failings here, but it does seem that your site is out of the ordinary in some respects, just because of the number of comments on one WP instance.
You realize that puts your credibility in the negatives right? Wordpress is infamous for being one of the worst pieces of software ever created. The fact that your software is terrible does not mean a company whose sole purpose is to be experts at making that terrible software run is doing good by failing to make it run.
Puh-leeze. Everyone thinks some other software is terrible. Wordpress might have some funky bits, but a) it's gotten a lot better over the years, b) it's pretty damn popular, and c) end users don't care if the guts suck if the UI works and it does the job.
I don't understand how you can not understand. You are in part responsible for this terrible software. Which is horribly bloated and inefficient and runs like crap. So it is in your best interests to try to pretend anyone who has a terrible experience with it is "doing something unusual and different". The reality is we're talking about a setup that is about as simple as wordpress can be. The fact that "as simple as wordpress can be" is an insanely complex monstrosity doesn't let people who said "we can handle your complex wordpress installs no problem" off the hook.
I run one of the largest WP installs outside of WP.com, and we've seriously struggled with performance issues. Sounds like a lot of this complaint is a WP design/performance issue. If you're looking to be create a WP serving platform, it will require a huge amount of custom configuration, a huge amount of hardware (even with the best caching), regular DB maintenance and reconfiguration, and core hacks. Automattic redirects the core dev when they need something, generally only enough to add their hooks, then they write a plugin, release it to the community, but always keep it a few versions back (HyperDB support for 3.4 anyone?)
The WP core team doesn't write code with performance, scalability, orthogonal design, good standards or any such thing in mind and you regularly hit gotchas. However... if you sign up to be a WP platform, you're kinda asking for it.
Obligatory performance tips:
1. Cache the heck out of it. CDN, varnish, optcode (APC), query caching on MySQL.
2. As has been noted, file stat'ing is massive. You have to have an exclusive install as close to each serving node as possible, even with optcode caching.
3. Core hack - Modify admin comment searching. Does 5 column wildcard lookups across the entire table. Not scalable. (http://core.trac.wordpress.org/ticket/20487)
4. Core hack - Remove content reallocation on user delete. With large user tables, you lose the ability to delete users (core fix pending: http://core.trac.wordpress.org/ticket/19867)
5. Clean and optimize your DB regularly - Old post revisions are a huge gain
6. Add indexes intelligently - WP doesn't include all the indexes you need
7. Horizontally scale your DB (whether through MySQL clustering, or HyperDB + DBs with roles)
8. Tune everything. I've found that NGINX/Apache and Percona/MySQL make little difference, but setting the right config for each daemon makes huge difference.
I tend to agree, but sometimes 140 characters come across as too emotionally charged, so imagine two words. Hence, "WPEngine sucks", which seems a bit harsh. Several recent articles could be summarized as "Founders dumb", "VC stupid", "SV sucks".
A hosting company without 24 hour support is not a hosting company.
Go with Rackspace. They're expensive compared to budget hosts, but they seem to have an army of sysadmins available 24/7/365 that never say "that's your problem" / "not in scope".
PS: that redirect bug is probably a mismatch of your site_url setting in either wp_blogs, the wp_options table of your primary blog, or the wp-config.php. If the domain set in any of those 3 places doesn't match you get that endless redirect loop. Could also be htaccess sending you to www or no-www, and your config is set the other way. Its an annoyingly common issue when moving MU sites, and they should have been able to fix it.
For sites that are large enough/can afford it, I cannot say enough good things about WordPress VIP - we have 20+ sites on VIP and could not be happier with it.
Since they 'are' WordPress they can make pretty much anything that does not modify core, work. It's not for everyone, but for the right sites (high traffic/high availability/generating revenue) it's fantastic.
WordPress, like most LAMP apps of its era, makes a series of architectural assumptions that turn out to have horrible impact on non-functional qualities … but that’s another rant for another day.
Until I came to this sentence, I was coming to the conclusion that CMSes fall down fairly quickly once you add in large data, large traffic, or complex installs. For the life of me, I can't see why people continue to use CMSes for large projects anymore, when they could potentially code something up faster in Django/Symfony/Rails and, here's the kicker, know the codebase inside and out.
My client has simple marketing (read: 99% static) websites that are built on Drupal. Sounds good at first, and it almost makes sense... and then you get to the non-functional requirement that it needs to withstand a marketing event that can bring 2k+ concurrent users for over 24 hours. Then you watch Drupal meltdown the server.
Valid point. What I keyed on in the article is that it is a complex install, and that tells me that maybe his CMS is working against him. In those cases I like to remove the impediment and simplify the data model (which should simplify the code).
I can't comment on the specifics but he said it was something default in the comments table and modules I think. For a big project like wordpress, I would try to understand why a certain decision was made. Try to understand the consequences of that decision vis-a-vis how I would want/expect it to work.
Doesn't even have to be static to put nginx in front of it! As long as you can set proper cache headers from your script (in this case Drupal, and I imagine it does), you can cache even dynamic-ish sites quite well!
When I applied to do my honours year, I wrote four project proposals. One was to basically turn the architecture of such applications inside out by making the virtual machine the target of deployment. This allows you to get out of a lot of architectural holes that shared hosting forces on you.
Email me if you're interested in reading the proposal, I have it lying around somewhere.
I've had a radically different experience with WP Engine and would happily recommend them to others, and have done so.
I went through 3 or 4 different providers, and the only place that i was really happy with setup, infrastructure and support was WPEngine.
As some of the other guys have stated regards the caching and their suitability to vanilla installs, i think that is most likely the case, and yes mine was a pretty vanilla install in the grand scheme of things, but i'd still choose them over anyone else.
Ben, one of the co-founders of WP Engine here and as someone wrote below - yes, this isn't the most wonderful thing to wake up to in the morning and sort out while still in your PJ's ;)
Ok, so first and foremost I am incredibly sorry, Jacques, for the experience you had. I can say with confidence that your experience with WP Engine is out of the ordinary, but no excuses. I’m writing this as a way of owning up to mistakes that were made on behalf of our company, as well as transparently engage the discussion here. There's a couple of points I'd like to make, if I may...
We're in the process of moving to complete 24/7 support right now (we do have emergency 'site down' 24/7 cover) - as any entrepreneur on HN will tell you, getting your startup to take off is hard and scaling the human aspect is perhaps even harder. With mostly US-based business customers it wasn't as much of an issue to begin with but we're now addressing that.
Obviously if we'd had that 24/7 support in place we could have addressed this more immediately.
However, one of the reasons that your site experienced problems was that there were some aspects of the site development that simply didn't scale. Like any scale-orientated PaaS, we can provide the infrastructure to enable you to scale but your code needs to work in partnership to fully achieve that.
As Sean, my head SysAdmin wrote on a ticket to you, some of your pages were performing a sort on 188590 rows in memory each time. That just doesn't scale. From your post, I'm guessing this is the issue that also occurred when you tried Pagely. While it's awesome that the good folks at Pagely were able to work with you a little to try to address your problem, like most PaaS providers (eg AWS, Heroku) we don't provide consultancy services - there is an amazing WordPress community of consultants and dev shops out there and its just not what we want to get into. Just like Heroku won't fix your Ruby code.
We have many, many clients who are very happy with WP Engine but as a growing business there will always be customers who find that the service isn't right for them, and even occasionally have a bad experience.
So lets try to make this right. I can see that we've already refunded you all the hosting fees you paid for your account, and I'll also make sure we pick up the $500 migration bill that you incurred. If there is anything else we can do please let me know.
I will also take time once I've finished my coffee and out of my PJ's to analyze your case further and see what else we can do internally to ensure this doesn't happen again.
Ben, I really appreciate you taking the time to post this here and demonstrate that you take this seriously, but your cofounder is currently publicly ridiculing customers who have paid good money and been dissatisfied with your service.
As a current customer of WPEngine (who has had issues), nothing makes me want to bail on you guys as much as those two tweets. From the CEO.
Let's give the man the benefit of the doubt, he is obviously sorry about this and said it was "an unfortunate mistake where I didn't know what I was responding too" (see links), let's not crucify someone on a field trial, none of us want to be in his shoes.
I'm neither justifying nor criticizing what he said, just let's be careful not to burn someone or something based on one mistake that we all can one day find repeating.
I don't see what you think is disgusting about these tweets, they express a personal opinion, here they are in case they are deleted:
@billmcneely yup, there's always someone bitter and hateful. Sometimes they even have a point, but choose to make it in a destructive way.
@P_Doubleya @gozmike thanks guys. That's also why I ignore HN. There's always haters.
That's not ridiculing customers, and it's not disgusting or disrespectful, it's just saying that there will always be people critical of your service (not even necessarily customers), particularly when an online mob forms as in this case, furnished with a one-sided story of how things went wrong.
Of course you may disagree with the tone, or the content, but disgusting is a little strong isn't it? Can we dial down the hyperbole a little?
No, he's basically brushing the OP and everyone in this thread off as being hatful and bitter, and not really worth interacting with. It's incredibly dismissive, especially since multiple people in this thread (myself included) have mentioned that they've had issues and are or were paying customers.
I'll grant you that "ridiculing" is probably not the right word...maybe dismissive or condescending would have been more accurate.
And yes, I do find it disgusting that he's publicly behaving this way towards a customer.
I'm not convinced a public forum is the best place for customer support - witness Ben's very reasonable statement above which is still accruing negative responses, in spite of going well beyond the call of duty in refunding money paid to a completely different service. It's quite hard to interact with a crowd baying for blood - emotions run high, and people tend to pick sides and pick holes in any argument. One response to that is to ignore the crowd and try to deal with the particular customers - that's what I take the tweets to mean.
It's fair to criticise those tweets and I don't feel they are appropriate really, but I don't think they should be classified as disgusting - misguided and inappropriate at this particular time, perhaps, but not disgusting. Isn't it better to deal with situations like this without escalating emotions further and degenerating into both sides insulting each other?
I'm not convinced a public forum is the best place for customer support ...
I think a great heuristic here is this: your customers can be publicly negative, but you cannot.
If you publicly flame a customer then, no matter how deserving the customer was, some future potential customers will look at that and say "That could be me." Most businesses, I think, can't afford to risk that.
There are some notable counterexamples, in my opinion they are the exceptions which prove the rule, i.e. when the business wants to fire the customer and make an example of him/her for others.
Sorry, it's just my personal opinion as a WPEngine customer, and I stand by it. I don't want to see the CEO of a service that I pay money to publicly behaving this way towards those customers and others who have had issues. It's the opposite of good customer service, and a great example of biting the hand that feeds you.
He spent weeks trying to get help from them, to no avail. It's more than reasonable for him to post a negative review at this point, explaining the situation, taking blame where he should, and warning others to think twice.
All I want to see from the CEO in a situation like this is: "I'm sorry. We fucked up. Here's how we'll make it right, and here's how we'll make sure it doesn't happen again."
And if you genuinely can't say that because you don't think the customer is right and you're not willing to do anything for them, just keep your mouth shut.
Why do people always insist on being the devil's advocate on HN? Suppose you went to the supermarket to try and return some spoiled food, and they deny you so you give them a bad Yelp review. A harsh one even.
Would you consider it reasonable to be the subject of the manager's tweet, or open letter or any other kind of public message calling you someone who cannot be pleased, who has an agenda against the store, is disposed toward bitterness, and is communicating destructively.
It's totally disrespectful. To act as if a crowd of savvy users who run their own services aren't to be taken seriously and that a single user - who tried for weeks to get help based off of the company's own claims - is this loudmouth minority? That's completely ignorant and rude.
I actively ignore Twitter for the same reasons some might avoid HN, but that doesn't mean that there aren't valid and intelligent conversations to be had there, particularly if you have to deal with customer service and PR.
"Completely disgusting" is taking it a bit far, but it's certainly puzzling, as is the irrational potshot at HN -- a hub of potential customers! "Disgusting"? No. "Bizarre", especially for the man behind A Smart Bear?? Yes.
FYI, the problem is with the Recent Comments widget code that ships in the Wordpress base install. Page.ly had the same database connection dropping issues you did and I presume for the same reason. (edit: By the way, use the db-error.php facility to put up nicer error pages. It's also a really obvious point to insert some monitoring logic -- I think customers would like it if you noticed their problems before they do).
The core issue, as I said elsewhere in this thread, is that unless you partition the underlying wp_comment tables, performance becomes steadily worse as the total number of comments rises.
The only reason that this was never a problem for me on my own servers is that I am loltastically over-provisioned for my requirements. I just never saw it.
You will see it more in future. Like I said: this is not a badly behaved 3rd-party plugin. This is mainline code. Your customers can reasonably expect that you can run Wordpress out of the box without degradation.
None of changes the fact that my original migration experience was a total disaster. I know you're experiencing growing pains.
Well here's one of them. I hope it serves you and your future customers well.
I'm willing to accept fractionally slower page loads, as for the most part they're cached heavily post-generation. I'd feel far more comfortable partitioning in Postgres not least because I'm far more familiar with it.
Hello Ben, I would like to talk to you regarding our bad experience and maybe discuss a refund?, in our case we are a small shop and have been hosting a local newspaper with 500k+ posts and ~60k page views a day for almost 2 years without much trouble.. (nginx cache magic^^)
Since our customer was looking for a more agile development & deployment workflow I personally recommend them switching from us to WPengine based only on Jason reputation and your technical expertise.
Almost a month has passed since they signed in for the service, $500 in fees and ~ $700 on WPmovers afterwards, they keep hosting with us and couldn't arrange a 0 downtime migration at all, neither provide a fixed quote for the migration.
Our case right know is being handle by Sergio, maybe you can talk with him and our customer to find a happy ending?
I've always been an advocate of WPengine and big fan, but after our bad experience and seeing others having similar issues too, I'm probably not recommending WPengine in a while..
While you don't provide consulting services surely this would fall under "troubleshooting WordPress", no? This is taken straight from your website:
The Most WordPress Experts Per 1000 Customers…
Great support? Many hosting companies claim this. But when you’re hosting WordPress, you need specific support. If a plugin breaks, or a theme file doesn’t work, and you’re hosted with a typical hosting company, they’ll throw up their hands and say “Not responsible! We just provide the server and the bandwidth, buddy.”
Not so at WP Engine. Our staff consists entirely of WordPress experts, so we can help you track down and troubleshoot WordPress issues.
While it's awesome that the good folks at Pagely were able to work with you a little to try to address your problem, like most PaaS providers (eg AWS, Heroku) we don't provide consultancy services - there is an amazing WordPress community of consultants and dev shops out there and its just not what we want to get into
To address this point raised by many... you're glazing over the difference between support and consultancy. Many companies provide both but charge extra for the latter.
You quote 'we can help you track down and troubleshoot WordPress issues.' - that's exactly what the support team helps customers with all day long.
But we don't get into consultancy around helping to build custom code, etc.
There's always going to be a point where a hosting provider will not be able to support you further. Many dedicated providers and VPS providers will ensure that the machine is powered up and has connectivity and leave you at that. What we're trying to communicate is we go a lot further into supporting your use and configuration of WordPress - but not into consultancy.
Really? Do you understand just how disingenuous that sounds?
I know Jacques didn't mention that the problem was with the stock installed-with-Wordpress Recent Comments widget until a couple of hour after you posted that comment, but _surely_ someone at WPEngine knew already that this wasn't a "custom code" issue?
From what I'm reading (and please correct me if this is wrong) a stock WP install with no plugins or custom themes, but with several hundred thousand rows in the comments table and the Recent Comments widget running - will crash hard on WPEngine (while running just fine on a "loltastically over provisioned Linode VPS).
In my view - that's a "Wordpress support issue" suitible for diagnosis by a "Wordpress expert", and a long way from "consultancy" for "building custom code".
It's true there needs to be a limit to what customers expect from hosting companies, but like it or not, you've set the expectation bar _very_ high with the marketing copy on your homepage, and you've got a wonderful social media reputation based on those expectations. At some stage, you need to suck it up and dedicate the time/expense to live up to that reputation, or inevitably social media will bring your reputation back down in line with your business behaviour/policies - and the wider the gap between the expectations you've set and the service you deliver, the more ammunition you're handing over to "the haters" to shoot you with.
Not out of the ordinary. It fits with what we experienced with WPE as well. My take: if more people aren't complaining, it's because they're not monitoring their sites well.
Had to move our highest traffic site back to GoDaddy (which makes me cringe, don't get me wrong). Doing the same with our other two sites.
That said, I believe in what you guys are doing, think there's space in the market for someone like WPEngine ... it's just that, in my experience, you guys are executing poorly and dropping the ball quite often.
...you've just built a business which promises things like "infinite scalability" on top of WordPress ...expect 1000x more of situations like this!
...if I were in your place I'd drop the "solving this while yawning in my PJs" relaxed attitude and prepare to bring a sleeping bag to work and stack a fridge full of red-bull (or whatever makes you "fly") ...or try and hire someone like Ledorf ...'cause all the other "PHP gurus" are at Facebook now ...tough luck. I'd hate to be in your place, but no matter how much I hate the "PHP ecosystem" I hope you guys manage to make this work
Ben, I don't know Jacques and don't pretend to speak for him, but I gather from his post that the scaling issue, and you not being able to accommodate him, would have been no big deal if he had not already been through the nightmare of migrating with bad documentation and support not handling that properly.
totally, and support coverage when he needed it (noting he's in Australia so totally different timezone to WP Engine business hours) would have also saved the day.
I'm incredibly proud of the business the team has built, and know we are still an excellent place to host WordPress. But yes, incidents like this give us opportunity to revaluate aspects like our documentation (and support hours) to double down and make things even better one we resolve them.
One of the reasons that your site experienced problems was that there were some aspects of the site development that simply didn't scale. Like any scale-orientated PaaS, we can provide the infrastructure to enable you to scale but your code needs to work in partnership to fully achieve that.
As Sean, my head SysAdmin wrote on a ticket to you, some of your pages were performing a sort on 188590 rows in memory each time. That just doesn't scale.
From a customer service perspective, this comment leaves a really bad taste in my mouth.
You guys say right on your home page that you're "infinitely scalable." And then if you click through to http://wpengine.com/scale-to-millions-of-hits-a-day-or-hour/ you read that WPEngine features "secret sauce" made of "one of the most scalable WordPress architectures on Earth" that makes you "faster and more scalable all the time." The (strong) implication being that any WordPress configuration will perform better on WPEngine than on an alternative hosting platform.
So if someone comes to you with a WordPress configuration that you can't make "faster and more scalable" by running it on your platform, isn't The Right Thing To Do to say "oops, our bad, sorry we misled you" rather than blaming the customer for having a bad WordPress configuration? Because that's what you're doing here -- blaming the customer.
In fact your comment basically boils down to three parts, only one of which is reassuring:
* Paragraphs 1-4 translate to "we're a growing startup and growing customer support is hard," which frankly if I were a customer I wouldn't care about in the least. Scaling your business is your problem, not my problem. We know it's hard, but it's like Tom Hanks said in A League of Their Own (http://www.youtube.com/watch?v=tPqYnC-SW5w) -- there's no crying in business.
* Paragraphs 5-7 translate to "your web site sucks, no wonder we couldn't scale it, nobody could," which is the blaming the customer part. I don't doubt that that is true (WP can be a real hog, especially if it's hacked up), but if you're advertising yourself as The People Who Can Scale WordPress, you shouldn't be surprised when people start coming to you with funky, hacked-up, hard-to-scale WordPress sites. Who else would someone with a site that can't scale go to but the scaling experts? When that happens it's OK to step back and say "we can't scale this," but it's not their fault for taking your marketing materials at their word.
* It's not till Paragraph 8 that we get to the "let's try to make this right" part, which is where this comment should have started.
I don't say this to be a pain or to make what I'm sure is already a difficult day worse. I'm saying it because I like what you guys are trying to do, and responses like this hurt you as much (if not more) as they help.
I've had to deal with situations like this myself, and no matter how tempting it is to blame the customer -- even when the customer is manifestly at fault -- it never, ever, ever improves the situation. It just makes things worse, first by giving the angry customer something new to be angry about, and second by deflecting you from the more important question of what you can do to prevent the situation from happening again. In this case, for instance, better screening of incoming customers to ensure their sites really can be scaled on the WPEngine platform would filter out problems like this before they become problems. (IIRC you've already started doing this, by blacklisting a small set of known-bad plugins. Maybe that list needs to be beefed up?) But time spent sniping at customers is time not spent making those improvements happen.
"The (strong) implication being that any WordPress configuration will perform better on WPEngine than on an alternative hosting platform."
No, there's no inference or implication that any (ie regardless of how its built) WordPress configuration will scale on WP Engine. As I wrote in my prior comment, scalability is a partnership between infrastructure and code. That's the case irregardless of framework and PaaS provider.
We like car analogies at WP Engine - if I sell you a racing car and you're only a moderately good driver then you're going to crash when you approach the corner at 200mph. There's little I can do about that other than teach you to drive, and I'm in the business of building the cars. There are plenty of folks who can help you improve your race-craft out there.
To say that we've missled someone is an unfair assertion.
If I analyze the suggestions you're making in your comment here...
* Jacques complained about lack of 24/7 support, but I shouldn't have tried to reply and explain the reason for that because 'there's no crying in business'. ?????
* The WP Engine website shouldn't mention scaling WordPress because it's not technically possible to scale all WordPress code regardless of how it's written. (What should our site say? What should any scale-orientated PaaS website say given the same issues apply to anyone in this space)
* WP Engine should vet and screen it's customers so that they must prove their code can scale before we sell them an account? ?????
I'm sorry, but if that's the advice you're offering then, with respect, I don't agree.
Well, it's your business, and advice on the Internet is worth as much as you paid for it. So don't worry, I don't take it personally if you disagree.
That being said...
No, there's no inference or implication that any WordPress configuration will scale on WP Engine.
I would humbly suggest that your entire Web site is a refutation of this assertion. Even moving beyond the copy on the home page to the fine-print "Common Questions" section of http://wpengine.com/pricing/ , there's no disclaimer or other "are there WordPress configurations that WPEngine can't scale?" fine print.
My point here isn't that you shouldn't promise scaling, it's that it's not hard to see how a mismatch could arise between your expectations (scaling is hard, customer code can make it harder) and customers' expectations (scaling is easy, WPEngine has figured it out).
Jacques complained about lack of 24/7 support, but I shouldn't have tried to reply and explain the reason for that because 'there's no crying in business'. ?????
Yes. If a customer complains about a lack of 24/7 support, you say "I'm sorry, but we're not in a position to provide that level of support at this time." And nothing else. The reason why you're not in a position to provide it is irrelevant to the customer, and risks sidetracking the conversation into a blind alley about the reasons.
It also feels a little bit like pleading for understanding -- "cut us some slack, this stuff is hard!" -- which is off-putting. Of course it's hard, that's why people pay you to do it for them.
WP Engine should vet and screen it's customers so that they must prove their code can scale before we sell them an account? ?????
I'm not suggesting that you need to give your customers a full CIA background check before bringing them on board, but you've already moved beyond just taking whatever code customers throw at you. If there's other things that need to be disallowed to ensure that WP sites work on your platform, the thing to do is to find out what those things are and add them to the blacklist.
"infinite scaling" means _exactly_ that. If you don't want people to use accepted dictionary definitions of words like "infinite", you should not use them in your marketing.
A few telcos here in Australia have recently lost court cases about their use of the word "infinite" in their cellphone plans/marketing, and then cutting off or charging extra to customers who "used too much". The court, rightly in my opinion, said that the meaning of the word "infinite" is clear and well known, and any claim that it included "limited, of course, by common sense" is bogus. Be careful of letting your marketing team write too much hyperbole - occasionally you'll be held legally accountable for it.
I suspect if the amount in dispute had been an order of magnitude or two larger, Jacques may well have had a strong enough case to find a no-win/no-fee lawyer to go into bat for him... I know I'd prefer not to have to explain to a judge how those first two boxes in the first column on the WPEngine homepage didn't cover all of Jacques complaints.
This comment sounds like one of those BS ISP ads where they promote "Unlimited" serviece but it's not unlimited bandwidth only an "Unlimited" connection. Or how about the cell phone providers offering "Unlimited" data where there's a star at the end and it states that you'll be throttled after 5 GB.
You have to understand the layperson won't know anything about tables etc and stating "infinitely scale" is a HUGE selling point for potential customers. If they wanted to be truthful they wouldn't say that, or they would say we CAN provide infinite scaling IF the following conditions are met.
We like car analogies at WP Engine - if I sell you a racing car and you're only a moderately good driver then you're going to crash when you approach the corner at 200mph. There's little I can do about that other than teach you to drive, and I'm in the business of building the cars. There are plenty of folks who can help you improve your race-craft out there.
The way I see it, you are not selling the car, but providing a driver so that I don't have to worry about how it should be driven.
I should sit in the back and relax while you make sure that my car is driving safely and comfortably.
Maybe your message isn't clear enough if you think that your product is the car. From your website:
To stretch the car analogies, perhaps torturously…
It seems to me wordpress.com is the "we supply the racing car and driver" solution provider. You don't have to worry about how it gets driven, but you also have only limited say in what car you get and which lines the driver takes through corners.
In my version, WPEngine are providing the race track - to which you bring your own car and driver. They go to a bit of effort to improve the safety, mostly by ensuring other track users don't do too much to affect your performance, but you're free to bring your own F1 car, MotoGP motorcycle, Nascar, or go kart; and run at whatever pace you think is appropriate.
Seems to me the OP's problem was that he wanted to run a top fuel drag rail. From his perspective, he made assumptions about what "a race track" meant that didn't include "decreasing radius downhill hairpins". Meanwhile WPEngine's website/marketing didn't anticipate that class of user, and didn't think they needed to point out that their track isn't an ANDRA approved 1440' long straight with a mile or two of slightly uphill runoff at the end.
With my sympathetic hat on, I can easily understand why both parties assumptions were sensible to them when they made them.
With my cynical hat on, it's pretty obvious that you could install a Wordpress theme or plugin on anybodies hosting platform with an O(m^n) or O(n!) function it it somewhere, and no magical cloud autoscaling pixiedust is going to solve your problem. At the same time, the expectation that a company claiming "Insanely Fast. Infinitely Scalable." and "At WP Engine, there’s no “first level” of support–our entire staff are WordPress experts, so you never hear “We don’t know how to do that.”" on their homepage should be able to do a better job keeping a Wordpress site up than some time-limited non-technical guy self-hosting on Linode.
Just a quick note to you and others who seem to be under an incorrect impression: I remember working with Jacques on a university project, and, in my opinion, he did not (and - based on comment history - still doesn't) fall into the "non-technical" bucket.
Good point. By "non-technical", I intended to mean "not interested in being a Wordpress scaling/management expert". If my poor choice of words lead anyone to think I was claiming he was not technically skilled in other areas, my apologies...
"To say that we've missled someone is an unfair assertion."
FWIW, it's clear that you _did_ mislead Jacques.
Whether the assumptions he made were valid or not (and frankly, based on the content of your homepage, it's hard not to have sympathy for his point of view), he clearly expected you guys would do a better job hosting Wordpress that he could do himself on his own Linode VPS. He also clearly expected that if there were problems, your "Expert Wordpress Support" would provide the solutions. Instead he found your support guys effectively saying "we don't know how to do that" and flicking him on to WebSiteMovers, who _also_ end up saying "we don't know how to do that" and charging him $500 for failing.
So I think you did let Jacques down, and I think he has every reason to feel missled.
You should probably acknowledge, at least prominently enough that anyone researching your service with a view to becoming a paying customer will find it, that there are possible pathological cases and configurations which WPEngine's technology/infrastructure/service will not solve. If you don't want to feature this on the homepage (and I can understand you not wanting too), I'd suggest a note near the top of the Disallowed Plugins page - and probably re-wording that page to make clear that avoiding everything on the "complete list" of disallowed plugins doesn't guarantee a successful deployment, and also that it's possible for themes with certain code patterns in them to cause a WPEngine deployment to fail to perform as advertised.
Secondly (and I suspect you've already got this one in hand), you need to train your support staff to recognise more quickly installations with pathological performance things going on, and apologise and extricate yourselves and inform the customer as early in the process as practical. I don't know what the tone/implication in Seans message about the ~188k row sort was, but from reading your defence here I can't help but think there was at least a bit of "you're doing it wrong!" - which even when it's true, it's not the wording you want to feed an unhappy customer.
(Note: this is written as a happy WPE user, with some curiosity about exactly what went wrong, and why some guy with a Linode VPS can, in some cases, keep a Wordpress site up better then WPE. I'm suspecting he's got define('WP_MEMORY_LIMIT', '2048M'); or something similar covering up some quickly hacked together "it works for me" not-very-well-though-out-code - which you can get away with on your own hardware/VPS, but not generally on shared hosting...)
"From a customer service perspective, this comment leaves a really bad taste in my mouth.
You guys say right on your home page that you're "infinitely scalable." [...] The (strong) implication being that any WordPress configuration will perform better on WPEngine than on an alternative hosting platform."
I agree with your general point that, from the standpoint of the entrepreneur, "the customer is always right" should be words to live by. On the other hand, in the paas industry, it should be axiomatic, i.e., understood by all competent parties, that horizontal scaling (the "infinitely scalable" bit) manifestly will NOT address problematic architectures, n+1 queries, full table scans, poorly/non-indexed schemas, etc.
This, to me, is not blaming the customer, it's laying out in reasonable terms assumptions any competent developer or software entrepreneur should understand long before they employ a paas.
Your larger point, though, is valid: even after bracketing the given assumptions of what the responsibility of a paas truly is, better communication up front could temper expectations as to what "infinitely scalable" (a bold phrase, to be sure) really means to the individual customer. It doesn't mean a paas can fix a application that locks the datbase for a full second per query, etc. It does likely mean (I'm assuming here, I have no experience with WPEngine) spinning up new instances and handling increased loads should be Heroku-simple.
In this case, I think WPEngine is taking the hit for what they should (bad communication on the support side) and explaining in real terms what paas's can and can't do. There's a difference between "blaming the customer" and simply being honest about what the bottom-line issue is. NOT addressing this side of it may have long-term implications if another customer comes along with similar expectations. Being clear about what the client's responsibility is doesn't necessarily equate to "blame."
On the other hand, in the paas industry, it should be axiomatic, i.e., understood by all competent parties, that horizontal scaling (the "infinitely scalable" bit) manifestly will NOT address problematic architectures, n+1 queries, full table scans, poorly/non-indexed schemas, etc.
The problem with this is that the original post doesn't say "I have a site that was crashing and locking up all the time, so I took it to WPEngine." It says "I have a site that was running fine on a VPS at Linode, but I got tired of the hassle of managing the VPS, so I took it to WPEngine."
The original post is written by the complainant, of course, so it's possible it's subjective or slanted. But if it's correct on this point, I don't think it's unreasonable for a competent party -- in this case, a person with a WordPress site that runs acceptably on a commodity VPS -- to expect that it would run acceptably on WPEngine as well. Presumably performance killers like the ones you cite would be just as bad on the VPS -- if not worse, since it's unlikely that a VPS' operator will be able to tune MySQL, PHP, Apache, etc. to the degree that an expert at a place like WPEngine would.
> You guys say right on your home page that you're "infinitely scalable."
There's a big difference between being infinitely scalable, and being infinitely foolproof. If your PaaS exposes enough power to the user, they'll find a way to shoot themselves in the foot occasionally.
We are hosting our blog on WPEngine and you guys increased the response speed of our blog a lot compared to other hosting providers! We used to spend hours maintaining the blog and make sure it runs reliably, and deploying a new version used to be scary in the absence of a checkpoint mechanism, but you guys solved this problem well.
I can understand that the customer is frustrated with having to change his code to move from a VPS environment to a cloud environment, but that is the unfortunate truth when scaling. Either you 1) get closer to the real hardware and throw resources at it (more ram, vps, or dedicated machine) or 2) change your code to be horizontally scalable on light cloud instances.
Maybe this is chance to create a help-page that explains to users what resource limits they should stay below (memory, cpu, queries) to handle peaks in traffic and how to identify problem points (bad SQL queries, hungry plugins).
I get what you're doing and why, but you could've avoided some of the pushback that's happening just by not pointing a finger or mentioning Heroku. So instead of "we can only run good code and yours isn't," make it more like "gee whiz, you have it doing an in-memory sort of 189k rows! Is that what you intended? Because that's really an edge case in our world?"
A friend of mine once said, "a customer will defend his bad decision to the death if you point it out to him."
I've been running a site about the same size as the OP's (500k-ish visitors a month) with WPEngine for nearly a year now, and so far I'm very impressed. It's smoother and faster than it was before, we've only had one outage of more than an hour (and they responded by crediting me with a full month's hosting!), and I've found their support to be absolutely top-notch.
I wrote a review of their service after a month - which I don't think I ever submitted to HN, d'oh - and all the points I make in it still stand: http://www.mmomeltingpot.com/2012/03/wpengine-review-after-1... . My key point there, which I stand by, was that moving to WPEngine saved me money over an apparently cheaper self-hosted solution, because I no longer had to spend 2-3 days a month faffing around with server problems. (I believe Patio11 had a similar experience.)
(Note - there's an affiliate link in there, because I find that with money I can buy goods and services, but I added that after writing the review.)
Just recently I've had an extremely helpful dialog with one of their support engineers which ended up troubleshooting a problem I didn't even know my blog had - which subsequently boosted my available ad inventory by about 25%.
Overall, WPEngine aren't perfect - and I'm still checking out other options from time to time - but I've found their support and general solid hosting more than worth the admittedly pretty significant cost.
My experience with WPEngine : Jason Cohen saved my ass.
We went live on a WebbyNode VPS, thought we had configured our system well and suddenly we hit the top spot on HN. The server was on fire and Jason personally showed up and worked on migrating our site over.
Your configuration may have posed problems, you may have had what seems like a shitty support experience, but this public bashing doesn't help anyone in the startup community and reads as if it's motivated by emotions rather than a real problem-focussed attitude.
Give these guys a break and work with them one on one to find a solution to your problems. I'm SURE they'll move mountains to make you happy because that's in the team's DNA.
I can totally understand your negative experience, I have email threads between myself and WPEngine as well as between common customers and WPEngine that prove quite the opposite.
It really sucks that you went through this. I'm just suggesting that you refrain from the public flogging and find another solution. I'm not affiliated with these guys other than as a client and I can tell you that I respect what they bring to the market.
>I'm just suggesting that you refrain from the public flogging and find another solution.
If you've spent more than a day in any online community you should have already learned that this, although it may not be preferable for the company at fault, is the fastest way to get something fixed or something changed.
It'd be irresponsible for Jacques to stand by silently and suck up all the time and money he wasted days after there was a great PR piece for WPEngine on the front-page of this site because it would lead to others having his issue which he could help prevent.
This is a community. It's not an advertisement source. People vote on things they find interesting and discuss them, it's not a place to only post positive things about up and coming companies and hide all sources of negative news.
Are we reading the same thing? I really don't think this post was an attack. It was someone describing a disappointing experience with a service that was supposed to make his side-projects easier to manage and having the exact opposite happen. It's a post about disappointment with a service written on his personal blog, I don't see any pitchfork calling or attacks or jealous hatred. Are you projecting something here?
Give these guys a break and work with them one on one to find a solution to your problems.
In the article it sounds like instead of them trying to work one-on-one to fix the problems, they just outsourced it to a digital removal firm. If that's the response he got, in what way would you expect the OP to be able to work with them one-on-one?
I think it's a matter of how one approaches the issue. With WPEngine, we didn't bounce around e-mail after e-mail.
It took a 10 minute phone call. I knew the names of everyone on my case and they kept me updated through the process.
I'm SURE there was a big fuck up in the support process for the OP. I just feel that there are other ways to skin a cat, particularly in our community where we should be working hard to find constructive solutions to things like this instead of creating a cloud of negativity around young businesses.
I've been hosting a couple WordPress sites on PHPFog (http://phpfog.com) as well as DreamHost. For some reason DreamHost is faster, but the PHPFog UI is much nicer. Both have great support (DreamHosts' is 24/7 if I'm not mistaken) and relatively cheap.
That being said, once you have the sort of requirements that the original poster seems to have, I don't know if any of those companies would be suitable…
I had the exact same experience, but I was paying for their VPS service. I believe it was over $30/month, and it went down all the time with very little traffic. Dreamhost is a tempting host for a new site because it's so easy to set up, but I think you save time and headaches in the long run starting with someone else.
VPS. http://sachagreif.com runs on it and got to the top 5 of HN many times without any trouble (I'm using W3 Total Cache).
The only reason I'm thinking about switching away is that their control panel is a little messy, and they don't support Git deployment out of the box (although you can activate it via ssh, which is what I did).
It also feels faster than PHPFog for some reason, especially the WordPress admin. So overall, I'd say they're a pretty good option for budget hosting.
The typical customer support process is bad for the complicated edge cases, which sometimes end up in threads like these. But it's good for business. You don't want your engineers attending to one person's problem. That's called consulting. You want them building product. Building perfect software is bad from a business point of view. In WPEngine's case, the takeaway from all this is to add a line to the customer requirements list on their website (useful for filtering out the undesirable bottom 5% of customers) that says they will not accept any of the few potential customers that use the native WP commenting system because it's poorly built.
I'm not sure if it's true, but based on the poster's description of WPEngine's customer support experience, it sounds like their customer service is the typical poorly structured (from the customer's point of view) customer service experience I've encountered and/or witnessed many times from companies both large and small.
Some startups don't provide phone support. It's nice to see WPEngine does. Sometimes you just have to talk with someone to beat it into their heads that you have a complicated problem. They can't email email you back a link to some topic-related doc page and move on. They're on the phone with you and they have to make meaningful progress before hanging up.
In the case of email support, 95% of the tickets are answered with a link to a doc page and that's the end of the case. The metrics are excellent and every pats themselves on the back. 95%! And the volume was soooo high! Acknowledging positives is necessary, but measuring failure requires different metrics and 95% success does not necessarily imply 5% failure because each one uses different metrics. You think the the metrics on their dashboard will take into account this thread, for example?
The system breaks down when the customer has a more serious problem. It takes support a few emails back and forth to realize it. The case is handled by random folks based on availability, further degrading the experience. The person who'll pick up your latest reply to the thread isn't going to thoroughly read the whole thread, and by this point you've interacted with five support staff already. Finally, they realize it's outside their control and pass it off to Engineering, where it languishes for a month. Creating a system where the engineer is not allowed to reply directly to the customer but instead has to reply to support who then replies to the customer only removes the stress off the engineer, so they can care less and not solve the problem. Meanwhile, the support staff learn to drop lines like "Sorry for the delay. We appreciate your patience. The engineers are still looking into it." without any second thoughts.
My least favourite part of this type of sharing is when all the competitors come along to now try to pitch their wares. It's not clever, it's not classy, it's just lame.
I don't doubt this was a shit experience, it certainly sounds incredibly infuriating.
Jason & his team work super hard, and have thousands of happy customers, so they're doing lots right. Even the greatest of companies can screw things up now and then, especially in non-straight-forward situations. I'm sure he'll put it right.
> It’s a quiet backwater. Collectively the sites would have maybe 20k unique visitors a day. 50k is a big day hereabouts.
I'm not quite sure why this blog is such a huge problem to host. It appears that the author has some customizations (plugins..) that are really slow. In fact, I would suggest to the author to hire a good developer, fix those core issues (say slow SQL queries, etc.). It might cost him a few thousand dollars upfront, but he can continue running the site on $20-$60/month hosting.
Paying $250/month for that kind of traffic sounds over the top to me.
I know that Wordpress can be slow but it can also be quick. Why not route your traffic through cloudfront ? Why not leverage full page caching for anonymous users ? These are things worth looking at IMHO :)
> It appears that the author has some customizations (plugins..) that are really slow.
The slowest piece of code on the network today is the Recent Comments Widget. That's not a plugin, that's right there in the base install.
I worked this out myself, based on preliminary detective work by Page.ly. And then Page.ly went the rest of the way to ask a Wordpress core contributor for their opinion. The key issue is that this widget performs a query on the wp_comments table which becomes slower and slower as the number of comments rises. The only real fixes are either to move to Disqus (my bloggers said "no") or partition the table (Page.ly politely said that this was outside the scope of anything under their enterprise plans).
> Why not route your traffic through cloudfront ?
> Why not leverage full page caching for anonymous users ?
I also have memcached for the object cache, I use a tuned copy of PerconaDB and nginx is configured to serve static files and supercache-generated whole pages without ever touching PHP.
The blog post you read was served from a blog on that very blog network, on Linode servers in Tokyo that I once again personally managing.
> The key issue is that this widget performs a query on the wp_comments table which becomes slower and slower as the number of comments rises..
I dont' see why a selective read based on primary key indexes is taking so long. How many rows are we talking about here ? I'm sure it's under a billion rows - right ? Partitioning it may help but I think there's something significantly wrong with the way the lookup is going through.
The recent comments widget is probably doing things the wrong way around. Is it caching the returned values ? Does it matter if the recent comments widget is based on data that is.. say 2 minutes old ?
I really don't agree with your performance bottlenecks. I run several sites much larger (at-least 20x traffic per day) and granted that they do not all run Wordpress, I know from experience that your system should not be grinding to a halt with just 50k daily uniques.
Even imagining that the comments lookup is like "get all comments for post ID, 2966 --" - that can't possibly be slow enough to destroy your entire server setup. Maybe you just need more RAM ?
Get a dedicated box and get someone to optimize it. If your entire database can be in RAM (Get a box with 16GB of RAM ?), that should work well. That won't cost you $250/month even.
EDIT: I see from another post:
As Sean, my head SysAdmin wrote on a ticket to you, some of your pages were performing a sort on 188590 rows in memory each time.
Was that the recent comments widget doing that ? Well, that can be fixed with some clever caching or even a clever fetch technique!
Is that necessary for every load ? Do you have a particular blog post with that many comments ? The right way to go about that would be to pick out comments for a particular post and then sort them - query-wise. Again, your WP install probably needs some (much needed) tuning.
I understand your desire to have someone else worry about your servers, but someone also (apparently) has to worry about your code, and as I understand right now - that person is you. Well, you could definitely make some code level improvements and save on the 'server management' department.
I only see one query in there:
$comments = $wpdb->get_results("SELECT $wpdb->comments.* FROM $wpdb->comments JOIN $wpdb->posts ON $wpdb->posts.ID = $wpdb->comments.comment_post_ID WHERE comment_approved = '1' AND post_status = 'publish' ORDER BY comment_date_gmt DESC LIMIT 150");
I'm not sure why they are fetching 150 (why would you use that many?) but this query could be slow if you have many posts and many comments. Also, the goofy thing is that they are wiping out the cache every time and regenerating it instead of (say) incrementally adding to it.
I'd re-write it with an exists, which may help performance instead of a join (I've seen MySQL be pretty dumb with joins when you basically want an exists):
WHERE EXISTS( SELECT * FROM posts WHERE ... )
AND comment_approved = 1
In this case, MySQL may be having a problem where it can't read a multi-column index on post_id and comment_date_gmt DESC (I'm not sure if you can specify ordering in MySQL indexes, been a while), thus it's basically doing an in-memory sort of every approved comment (on a published post, which is likely all of them) in the system, which in his case was about ~200K. This could definitely take several seconds if there isn't enough memory to do it all at once (lots of swapping).
Personally, I'd try and modify the logic to be more conducive to indexing - do you really care if a comment was on a post that you made 10 years ago? MySQL also (IIRC) doesn't have bitmap indexes, so that "comment_approved" thing may or may not invalidate an index usage unless you add it to the index.
I'd probably do something like: the "n" (where n is < 150) most recent comments to a post made within the last n number of days, so you can really cut down on the number of rows you need to sort.
Appears that he has 188590 rows. It definitely sounds like a configuration issue. Maybe the mysql table cache might be bogged down or the query cache is full or malfunctioning. Could be a lot of things.
Sorting that into a temp table, and caching that in memory would be really fast on a server with reasonable RAM and proper indexes.
As my two cents, although I am a Linux engineer, I don't want to manage a Web cluster just for Wordpress, so I use WPEngine in production for about 10 sites and it has worked very well and support has been sufficiently responsive.
Reading about this, I get concerned when people conflate "performance" and "scalability". Performance is about time-based metrics such a requests-per-second. Scalability is about the question of whether I can add more hardware proportionately to my request load and continue to perform. WPEngine's marketing claims are about scalability, not performance.
This guy's WPEngine code was apparently doing a 189k row sort when serving certain requests, which is not going to perform well on any hosting platform, including DIY.
Put simply: cost. Collectively it's a few hundred Gb per month of traffic which is completely covered by my Linode plan.
These days I've put most of the sites behind Cloudflare, which has the effect that a lot of my media is being mirrored in their Sydney DC, which is nice because most of the readership is Australian -- the same reason I went to the Tokyo centre this time.
The Tokyo DC isn't a universal win for Australia. The results are actually pretty bimodal depending on your ISP.
From Perth, iiNet routes directly to Asia, and it's a big win. But Amnet routes AU -> US -> Asia and it's a big loss.
It's a big enough issue Australia-wide that Blizzard sells a dual-region Starcraft 2 in Australia, because (ping to battle.net Singapore for some) > (ping to battle.net US) > (ping to battle.net Singapore for others) to a significant degree.
Not really. WordPress.com's free hosting has limited space, no custom themes, no plugins except a small preapproved set, no ability to run your own ads, and you have to run WordPress's ads for their benefit.
You get a little more storage space, a little bit of CSS control, but you still can't bring your own themes and plugins, edit the code, etc. Plus, there's no way to migrate a multi-gigabyte blog network's data over, AFAIK.
WordPress.com is an SaaS for blogging like Tumblr or Posterous. It happens to be built on a custom, locked down install of the WordPress multi-site software. It's not a website hosting company.
I agree, but the gap exists for "fully" managed WP hosting for someone who isn't quite there yet but needs more power, etc.
We recently chose not to go with WpEngine because of the confusing help solutions we could see, a few reviews we saw. We ultimately went with ServInt and have been really happy with the move and support.
And to your point the pricing didn't feel right, it didn't make sense compared to similar options (or what we thought was similar)-- maybe I am off base though, it just what we discussed at the time.
I am using http://presslabs.com/ to host two of my blogs, one of them is a tech blog that gets a high amount of traffic every day and there has been no migration or downtime problems so far. I couldn't be happier.
The important lesson to be learned here is that even though higher price points tend to shield you from the bulk of toxic customers, you'll still find the occasional toxic customer at even the highest price point from time to time.
As nice a guy as this blogger probably is, he's the bane of services like WPEngine. There is no way to service the needs of his complex edge case of a system for anything like $250/month. They are guaranteed to lose money on him, and the only hope they have is to convince him to leave the service as rapidly as possible to avoid being sucked into the time sink that dealing with him is going to become.
This is the reason you have a "refund" button on the customer page of your admin console. A quick email apologizing for not being able to meet the needs of his unique setup and a refund of his payment, as soon as it became apparent just how impossible things were about to become would have solved this completely.
Or at least one hopes so. The tough thing about dealing with toxic customers is that they grow on you slowly and you don't notice at first. Then one day you realize you've spent four hours just digging into issues and writing emails to this one person who you're probably going to have refund eventually anyway.
I've never met the guys on the WPEngine team. But I feel for them after reading this.
Where did you find this 'complex edge case of a system' in his post? This 'unique setup'?
Why do you think 'impossible things were about to become' when he was able to easily move the sites to WP Engine... after paying a 3rd party to do it?
He didn't describe a single special requirement that I can see. Just plain old WordPress with some existing data to move. Not even a large network or a large amount of traffic. This is supposed to be WP Engine's bread-and-butter customer.
All of WP Engine's plans other than "Personal" say "Hey WordPress MultiSite customer, come here! This plan is for you! Hundreds of thousands of visits a month and 20GB+ of storage!" http://wpengine.com/pricing/
Why are you calling him a 'toxic customer'? He's been running this WordPress install himself on an unmanaged VPS. He wasn't wasting their support staff's time on unnecessary questions -- they were wasting his! That's not a 'toxic customer', that's a 'toxic business'.
I'm having a hard time connecting this comment to the HN submission at all.
He also didn't describe much of any of his technical requirements. I'd say it's a fair assumption his setup was enough different to cause an issue. Whether or not that's a fireable offense is up to the company.
He did, however describe the eventual problem: his comments table was larger than what Wordpress's out-of-the-box feature (Recent Comments) was able to sort. He had very overpowered hardware in his custom situation. It strikes me that this was an extroardinary but still within band problem that WPEngine should have eaten the cost for solving OR tell him MUCH more swiftly they couldn't.
I'm more curious what VPS package you had powering your stuff at Linode and how much you were spending on it each month? Reason being that 250 USD buys a LOT of server these days and it is surprising you ended up with underpowered hardware if your VPS wasn't giving you any problems.
If you're looking to upgrade in the future, you might be much better off throwing a dedicated server at the problem and setting a nightly backup to S3. Also, thank you for posting.
I don't think that was in the post originally. Either way, it's not outside scope, it's not a big deal, and it's not an excuse for weeks of failing to provide support. They didn't fail to even move the site because of the number of comments. Matt Mullenweg, the founder of WordPress, commented himself that they host many blogs with far more comments than that without issue.
I think that's unfair. So if you sign up for a premium service because they say they can help you solve problem X, and then you follow their exact instructions to no avail and get the runaround from customer support, you're a toxic customer?
I don't know what your experience with WPEngine is, but suffice it to say, this post is not the first complaint I've heard, and I've personally had...issues with them as well. And my setup is even more vanilla than the one described in the linked post (which is pretty vanilla, honestly). I love Jason Cohen's contribution to the startup community and I want to love WPEngine, but I am starting to wonder if something is amiss on the technical / customer service side there.