Hacker News new | comments | ask | show | jobs | submit login
My blog can’t keep up: 500 errors all over (lemire.me)
43 points by nkurz 16 days ago | hide | past | web | favorite | 60 comments

30 to 50k uniques per month really isn't that much. At a rough guesstimate, I'd say I've got at least one hosting customer that does that much with Wordpress, gracefully, on a site that's not optimized even a little. I charge them $10/mo.

Static site generators and everything else are lovely and all, but it's hard to beat Wordpress when it comes to being able to just plug in a new feature. Which, of course, also gives some folks the rope they need to hang themselves. As other folks have pointed out, there could be a plugin issue.

But my guess is it's the hosting provider. They can have professional-sounding support but still be running an architecture that hasn't been updated very much in the last 5+ years. mpm-event and php-fpm alone really changed the performance characteristics for bog-standard LAMP hosting.

The error logs should give a number if memory's exhausted. I bet it's super low, because a lot of shared hosting providers still try to cram as many sites as they can onto a beleaguered colo'd box. At a glance, that does look like the csoft.net architecture (they have "Two Dedicated, Redundant Servers (with Private Gigabit LAN)").

There's certainly a broader point to be made about the gross, inexorable bloat of web-based applications, but there are also plenty of people running even more bloated, busy stuff for less money with less trouble.

Oh, and you can install the traditional editor for Wordpress, they've made it available as a free plugin and it works just fine.

There are plenty of people around who can sort out stuff like this. I'm one of them, but I'm heading to bed. Email's in my profile, or just accept help from anybody else that knows what they're doing.

Consider moving away from a CMS like Wordpress. It's great at what it does but I think you've out grown it. Static content generators might help here. They let you edit off line and compile to a static version of the web content you currently serve. I'd suggest looking at Hugo or one of the many specialised for blogging. https://opensource.com/article/18/3/start-blog-30-minutes-hu...

You can then push the content to something Amazon's s3 and put cloudfront in front to speed things up.

You'll have to then offload the comments to a service but that is relatively easy and cheap. A service like discus is only $9 a month. Moving to s3 you'll save that in hosting. (I run two sites like this and pay $0.07 on average per month). Not to mention those services are pretty good at filtering spam.

There are some other options around too. https://darekkay.com/blog/static-site-comments/

Wordpress is nice but it will never out perform static content.

EDIT: I just read this in discus "Free For small, personal, non-commercial sites who do not run any ads. This plan includes everything in Plus except Direct Email Support."

> It's great at what it does but I think you've out grown it.

There is absolutely no justification for him to be running into these sort of issues at such an incredibly low load. Let's pretend he gets 50,000 hits per day (he says 50k per month but I don't know reqs/sec). Than's less than one per second. The $5/mo box over at DigitalOcean can handle 100 times that with the most poorly optimized PHP, Wordpress, and MySQL installation.

The problem is entirely due to a shitty host and/or grossly misconfigured server. No CMS should run into load issues at the microscopic traffic levels he's complaining about. He doesn't need Cloudflare, either.

So, yes, a static site would indeed perform better but to imply that he's actually reaching the limits of a simple CMS is absolutely ridiculous.

>There is absolutely no justification for him to be running into these sort of issues at such an incredibly low load

Yet he is running into these issues. Is it a buggy plugin or a configuration issue?probably, yes.

But that's exactly the problem static sites solve. I don't mean to say he's out grown it as in you can't serve Wordpress at the volume he's currently getting. I mean you have to have everything setup correctly to do so.One option would be to learn the workings of Wordpress and maybe the web server hosting this. If that's what he wants to do with his time great. I suspect he may have other interests, if that's the case static hosting would be a good alternative. It might not be the best, but it's the best I know.

I'd pretty confidently bet this guy has some ridiculous and/or buggy wordpress plugin that loads a bunch of images or is horribly written that is causing the problem. Even a naive and basic wordpress install on crappy hardware should be handle this load.

I developed and maintained the landing page for a top-100 website that was powered by Contentful, ExpressJS and aggressive in-memory caching. The setup was very stable.

I’m guessing your Wordpress install has a bad theme or plugin that is writing errors to the request body and then attempting to send the remaining HTTP headers.

Yeah, you are right. It can be done. But the bulk of the sites I've seen have some cludge that crept in somewhere. Someone installs a broken plugin or writes a custom tweak and things start to fall over. The beauty of static content is this just doesn't happen. You also get some security through simplicity.

I suspect you are very good at what you do. A lot of people are not so good at maintaining an operating site. Especially not a top 100 level.

That said it's a trade off. For every advantage there is a disadvantage. Having to compile to push changes is not without it's headaches. Using external services for comments and other things can run you into all sorts of cors hell.

Can be done doesn't mean it's a good tool for the job. I don't know enough about wordpress to say but I've seen some big sites built off sharepoint and "can be done" involved an extraordinary amount of manpower and working around sharepoint rather than with it. Those details get left out of the case studies.

> Consider moving away from a CMS like Wordpress. It's great at what it does but I think you've out grown it. Static content generators might help here.

This is ridiculous. If he had 100 times the traffic he has, he still wouldn't be anywhere near "outgrowing" Wordpress.

> put cloudfront in front to speed things up.

He already has Cloudfront, and it is completely meaningless snake oil in his situation.

Instead of hosting on S3, aI would recommend google fire add hosting

$50 seems quite a bit of money for a wordpress blog, when you can have wordpress.com host you for $5/mo.

I am running multiple Wordpress customers on digitalocean $20/month plans, and some of them get more than 50k uniques per month, never had any issue. Never did much optimization. I am pretty sure the issue is your hosting. Also check your Apache error log, maybe you are using some plugin or theme that is overloading the memory with some crap code like majority of free Wordpress plugins/themes.

p.s make sure to update to php 7+. The speed difference was very noticeable when I upgraded

I'm just going to say one thing: EasyEngine.

If you need help managing/setting it up, I can do it for $15 a month + DO cost - or AWS/Google/Azure. I'll manage your stack, depending on time to migrate, there might be like $50-100 for migration.

I'm looking to start a WP management service, and hosting company -- mainly to get some MRR during my slow months.

I'm currently a Full stack laravel/vue/react freelancer.

For reference, I ran a political blog during 2016 w/ 100k hits an hour for a few weeks (traffic died down a ton), but my site wasn't functional at all on shared hosting.

After easy engine on DO's lowest tier, everything just worked. Nginx + redis cache is amazing.

Just drop me an email : patrick @ zvive.com -- linkedin: https://linkedin.com/in/patrickcurl

I recommended the static approach but if someone is willing to manage it for $15 a month and charge under $100 for migration, I'd go with that. You wont have to learn a new tool or figure out how to migrate.

Yeah static works good... in fact I'm working on a SaaS app w/ blog built in that will be open source (w/ upgrades), the frontend will have theme-able blog that's built statically via nuxt generate, the backend will be vue SPA - not using Nuxt, since SEO really isn't required. The stack is Laravel for the backend...

It's 'SaaS' in that the actual app will have easy subscription plans, analytics, single/teams, etc...

I was working on my own SaaS and realized other than some potential code examples from dead github repo's and Laravel's Spark, there isn't much on the market or github -- even in other frameworks like express or rails that covers this niche.

Definitely nothing open source, I'm hoping to get some sponsors early on to offset some of the costs of development till I get other streams of revenue...

I got off on a tangent lol, but I don't think $15/month + $99 migration is super sustainable...esp if I scale, but it would be nice to have even $1000/month just on autopilot via hosting.

Plus you can upsell stuff. I've got a brother who's a stock boy (he's 28, I'm almost 40), who graduated with a Bachelor's in Digital Design, I could probably kick him in the ass w/ some motivation and teach him to manage everything and maybe get him to work in his actual career field for once...He'd work for min wage right now just to get some beef on his resume.

How does EasyEngine compare to dokku-wordpress?

Here's a good overview of different possible scenarios though when choosing a container : http://www.jancarloviray.com/blog/paas-comparison-2017-dokku...

Can't say, haven't used dokku-wordpress. I know EasyEngine recently moved to docker images, which I'm not totally a fan of -- simply because you can only have X number of images per instance, iirc -- and that limits how many sites you could potentially host on a single server.

Do you mean it uses more memory? or you are hitting the docker limitation per host?

https://easyengine.io/handbook/v3-to-v4-migration "As v4 depends on docker, it is affected by Docker’s limitation of 31 network per machine.

v4 creates one network per site. Then v4 also needs few Docker networks for global services. So roughly we limit the number of sites to 27."

I'm also wondering what all those docker instances do to performance, seems to me there might be a better way.

That’s just a bad hosting service.. I manage somewhere in the neighborhood the OP gets with a basic Wordpress deployment. It’s extremely stable, and only ~7/month. I use limited plugins and have optimized for speed.

My guess is a plug-in or theme is bugging out here. Easy solution is to remove all and slowly add until the problem exists again.

In 2015 I had “Crayon Syntax Highlighter was Killing my WordPress” for instance:


Ignore all the comments telling you to switch to static content generation, that's generally a large time endeavour and for a blog that is getting 50k uniques a month should be pretty easy to serve all requests with a little CloudFlare tuning.

First, let CloudFlare do 99% of the work. Don't worry about optimizing the linux box, PHP, MySQL, PHP Opcode Cache, WP-Cache at first.

First, let's check that CloudFlare is caching and serving all static assets such as js, css, images, .xml, .ico. I don't see a cache-control header being returned on js, css, images, etc. Make sure this is setup in CloudFlare with Caching Level => Standard and Browser Cache Control => 1 month. Also, consider enabling Auto Minify => Javascript and CSS. Finally enable Brotli and make sure HTTP/2 is enabled under network.

Now, here is where you can really make things fly. It is possible to have CloudFlare cache the result of generated html pages, so requests don't even hit your shared hosting provider. This is done with Page Rules. The trick is going to be crafting a rule that you can match posts and posts only. With my site, I leave .html extension on my files, so my rule is simply:


Your challenge will be creating a matching rule while excluding admin backend pages. Also, most certainly you probably going to want a much lower cache time than 8 hours. My site content does not change often. Perhaps something like 30 or 60 minutes to start.

Good luck, let me know any questions, my contact info is in my profile.

Although a lot of the recommendations are great, and to me it also looks like there are specific hosting problems at play, possibly serious overcommit by the provider or some plugin problems, I do think there's a deeper issue:

> What am I missing? How can I be in so much trouble in 2019 with such a relatively modest task?

Something is not quite right with our web-platform when we have these sorts of problems, and the solutions also require quite a bit of sophistication.

As an experiment, I recently switched my http://objective.st site to run on Objective-Smalltalk itself on a Digital Ocean $5/month droplet (their smallest). Surprisingly made it to the HN front-page and didn't hiccup, despite an egregious memory leak on my part (fixed now).

This was reported as "blazing fast"[1], despite being a single binary hosted in Frankfurt, having no CDN support and no static files, with everything served from the embedded web-server. Heck, I haven't even hooked up gzip-compression

I don't log much, so I don't really know how much traffic it's getting. I do know that hitting it with wrk remotely I got around 1000 requests/s, locally on the same box it was around 8K. So I think there's plenty of headroom. Oh, and the logs do show continuous attempts to break in POSTing to what I presume are php vulnerabilities.

I am emphatically not saying I did something amazing or even particularly good, everything was somewhere between straightforward and egregiously simple-minded, and there were other (mostly easy) ways of accomplishing the same thing. Previously the same site was served using static files running the same code as a static site generator.

However, it does confirm my suspicion that we've path-depended ourselves into a corner.

[1] https://news.ycombinator.com/item?id=18965054

Which PHP version is being used? If it is still 5.* I would consider upgrading to 7.*. Just estimating now: 45000 users would be like about 1 user/minute, so maybe 10 requests per user per minute, could equal on peak max. 1-2 requests per second.

Some shared hosting shouldnt run into trouble.

For 50$ (Canadian, I guess), will not someone like wordpress.com do better? Their business package is USD25 per month.

Their plugin selection isnt great though and you cant install your own.

The WordPress.com Business Plan[1] actually allows you to upload your own plugins and themes.

[1] https://en.support.wordpress.com/business-plan/

Which is probably a good thing, given that shitty plugins are exactly what cause problems like this.

I come across colleagues in the same boat as the OP, but the truth of the matter is that a $5/mo VPS hosting WordPress is really not going to handle a surge of high traffic. Been there, done that, when I had a $20/mo Digital Ocean droplet with my blog on it. Anytime I posted a link to it on HackerNews or Reddit, it used to die under the 'hug of death'.

Since moving my blog to SquareSpace, I've never had that issue. In years. Note: Not an advert and not an affiliate for SS, just a happy user.

We still have a WP blog on a VPS for our SaaS, but I am keen to move that over to another platform, because I have enough on my plate to develop code for my SaaS itself. For my blog, I just want to post my thoughts without having to wade through a WordPress Admin console and see dozens of flags, warnings and popups about a myriad of plugins that need attention. Seriously - our WP admin console causes a lot of stress just logging in, which sucks all the creativity out of me.

SquareSpace (and even Medium, for all the negativity surrounding it) makes it so easy to start with a blank slate and just write and let the ideas flow.

To help in case of a surge of traffic you can enable some sort of microcaching on the web server. For example with a cache TTL of as little of a second you are sure that Wordpress will never see more than 1 req/sec (per page, but usually surges are specific to one or a few pages), and at the same time you will never be serving stale information.

The downside is that your Wordpress installation will not see the requests served from cache, which means so if you rely on a Wordpress plugin for statistics and/or analytics the data will not be correct.

Anyone else use Ghost? I've got Ghost running in a Docker container on a DO droplet. It's the smallest droplet there is. I don't have a ton of traffic to my blog (at least, I don't think—I haven't found a privacy-respecting analysis system that I like (maybe roll my own... just count page request)) but it does the job.

We generally serve several of the front page stories on HN every day (some we host, some we don't) - so very used to dealing with high traffic loads

@OP - If you're interested, drop me an email on john@ghost.org and I'll set you up with a 2 month free trial on our Pro service where you can deploy a new instance in a few seconds and kick the tires. No obligation.

+1 for Ghost. We run our development blog on it for our SaaS and it runs on the smallest Digital Ocean droplet just fine. Not a lot of traffic, but even so, the CPU and memory footprint on a small VPS is negligible.

If I recollect correctly (haven't used WP in years), the bulk of error 500-related problems on WordPress are due to poorly written plugins -- of which there are many. One of the better ways to track down a problem is to disable plugins one by one until the problem goes away.

Further to that, delete the folders. Plugins are often targeted directly if written poorly, so if you don't need 'em, delete 'em.

Disabling them is good enough, actually. The php code in them won't run at all unless they're enabled.

WordPress might be slow, but it's probably just the host or not enough PHP memory assigned to it. There is nowhere near enough traffic to bring down a WP site, unless you have a ton of bots pinging your site that never appear in your actual google analytics (because they are boths) .

* look at an actual server access log. cPanel offers one if your host is offering cPanel. * try and increase your WordPress memory limit: https://docs.woocommerce.com/document/increasing-the-wordpre... * check your server's PHP error log and see if there is an error * set WP_DEBUG to true in wp_config.php

Hugo. It's completely static and great out of the box. You can add new posts in Markdown.

I looked into Hugo but stuck with WordPress because the transition of 2M posts with 1,000+ daily new posts seemed too much for the build process:


Hugo is indeed a great tool. I like it because I can write all my content in Markdown and have full control over how the site looks. Editing a Hugo template, or creating my own, is also much easier than it is in WordPress.

Oh, good old traditional LAMP shared hosting and WP. Honestly, I doubt that combo will ever work reliably. WP hosting is bit of a dark art in the best of times, and operating under shared hosting does not help. That being said, self-hosting on something like AWS probably would end up much more expensive, and would take also some effort to set up and maintain.

Personally I would either start looking for fully managed WP service (wordpress.com), or switching to a different, easier to host, platform

I use wp-stateless to offload images, and cache-control to serve most of the content through CDN. (I also put a serverless reverse proxy infront to mixin other stuff on a common domain). I get great performace off shared hosting. https://corepox.net/devlog/simple-extensible-performant-low-...

Caching 'anonymous' requests at edge with Cloudflare workers - no need for the $200 p/m business plan


50 CAD a month (33€) for such a small site and then it doesn't even work properly. It boggles the mind. I'm sure you can get a working setup for 5€/month without breaking a sweat.

Having a company that deals with your Wordpress installation is only beneficial if they get it right.

You can export to static pages (one for every blog) with a plugin. You'll lose the comments functionality, but views will work.

An easier/quicker solution is to just stick a cdn in front, that way you can cache pages and still have to back-end work.

No offense intended at all but I can't help but wonder how a technologist/computer scientist has ended up asking for help resolving 500 errors on his blog rather than hypothesizing what the problem could be and testing his theories.

A quick Google search on the error he's getting says it's an Apache error either from invalid file permissions or running out of memory. His provider refuses to transparently increase memory so the hypothesis seems stronger here. Now just test it by spinning up a simple load balancer, clone your website on a different server, split the traffic 50/50, and see if the errors continue. Repeat this process until you've isolated the issue. Trying to solve the problem by adding a cache layer isn't going to help before the problem is understood.

We don’t know that the author didn’t google and try a few things. And it’s perfectly possible to be a computer scientist and not know how to tune a web server. There are lots of specialisms and paths. Personally I admire someone who, despite being an expert, isn’t afraid to ask questions and admit when they don’t know something.

That's a great point. My comment above seems to be coming from a rather limited mindset

It’s getting more funny if you look at his GitHub account.

https://github.com/lemire “His research is focused on software performance and indexing.”

I think it’s just a poor advertising. :)

Looking at one of the papers he has published

"Many applications use sequences of n consecutive symbols (n-grams). Hashing these n-grams can be a performance bottleneck. For more speed, recursive hash families compute hash values by updating previous values. We prove that recursive hash families cannot be more than pairwise independent. While hashing by irreducible polynomials is pairwise independent, our implementations either run in time O(n) or use an exponential amount of memory. As a more scalable alternative, we make hashing by cyclic polynomials pairwise independent by ignoring n-1 bits. Experimentally, we show that hashing by cyclic polynomials is is twice as fast as hashing by irreducible polynomials. We also show that randomized Karp-Rabin hash families are not pairwise independent."

I think he is working on the fundamental science of computer science. People like him are the reason we have nice things like web servers. That doesn't mean they know the specifics of Apache or Wordpres.

Seems like Wordpress has outlived its usefulness. It's like the Windows XP of the internet. It was great and simple, until the security problems and feature creep plagued it to death.

Most of the time the problem is not Wordpress but poorly written themes/plugins (especially free ones). Also $5/month shared hosting running on a decade old hardware are not helping either.

This won't end well for csoft.net

Reading this, I feel they should hand this job on to someone else. There comes a point where if you're not experienced enough to do this, no amount of Googling and investing more time will help you.

tl;dr I get the feeling that this person has been sold the idea that throwing money at things will help (why is he using argo for a smallish blog, it's not worth the money?)

I've had similar issues with several wordpress installs.

Some of the less known issues I found that can add to this problem:

Spiders/Indexing Bots Brute Force Login Attempts Hidden Hacked Pages

Helps to look at logs sometimes, if you get hit with all three of the above at the same time, it's a tricky mess to clean.

You may find relief making an aggressive noindex robots.txt file (there are tons of bots to block which are worthless and use up your resources. - it may be best to say * disallow all, then specifically allow google, bing, etc (would love to find a way to tell archive.org it's okay in that situation))

Even when doing the robots block, I've found some aggressive spiders or bots that use some of their user agents need to be blocked via ip subnets and cidrs with htacess for example.

Brute force login has many options for slowing down access. I love a combo of ipgeo block, sucuri, graphic captcha, shield security

plugin for disable xml-rpc or use the setting for such with the shield security plugin (this blocks massive login attempts) to block all xml-rpc use.

Hidden Hacked Pages If you find bots spidering a bunch of product pages for things in other countries in your logs for example, you may have been hit with a bunch of pages being created that you may not see on your may front end, or even in the backend admin area of your site.

Removing those pages is a start, and then a plugin like 404 to 301 redirect may be beneficial - but you will likely find a bunch of bots coming to index them, and other hackers who have been sold access to your site trying to login and make more of them.

For me it was more log checks and looking up ips to add to the server block lists to slow it down. Just make sure you don't add google's or bing's ips to the list.

With a few sites I had that got some of these issues I decided to pull down wp-login.php - and in an effort to lighten up the load on the shared server, I decided to use one of the plugins that make wordpress into a static site (there are two good ones in the repo now)

Once I generated the static html and css, I created a new wp-login.php that would log all ips and user agents / date to a separate text file, and display a message on the screen warning the visitor hacker that I was casting a spell on those who access the login page.

The amount of bots trying to login, and trying to access the old hacker-created-pages that were made over a year ago is still crazy. Their bots are still coming and trying the same login page and trying to access the same pages that have been removed over a year ago.

If I had left those pages available as php pages, each one of those visits would of been taxing my sql, even if they were showing the wp 404 page, or failed login.

I hope your issue is simpler, like a bad plugin. Certainly I've run into that issue with other problems not related to these as well.

Pssh, all you need to do is install docker, convert static files and blog post into lambdas, then route each unique http request to a serverless IPv6 address located on the nearest continent.

I, at least, thought this was funny.

Can you elaborate a little? I'm not sure I understand what you're suggesting but it sounds interesting.

Here is the architecture: https://serverless.com/blog/serverless-wordpress-hosting-wit...

https://www.getshifter.io/ seems to be a relatively turnkey solution for existing Wordpress sites to migrate to serverless hosting. It spins up a Wordpress docker image when somebody is in the admin panel, then builds and deploys a new static site to S3 when a change is made. Each build is immutable and easy to switch between.

There are limitations with static site hosting. Their approach to comments is: Use Disqus.

I can recommend you Publii CMS (https://getpublii.com), it's excellent static blog tool I use from over one year and manage four blogs.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact