Daniel Lemire's blog

, 42 min read

My blog can’t keep up: 500 errors all over

53 thoughts on “My blog can’t keep up: 500 errors all over”

  1. Travis Downs says:

    I don’t know if this comment will show up, but one thing you could consider is to take one or two years worth of the $50 WP hosting fee (so $600 or $1200) and offer it on contract to someone who knows what they are doing with WP, for them to:

    1) Move to a hosting provider that costs say only $10 or $20 per month maximum.
    2) Fix the performance problems through caching and/or identifying the offending code.

    Since step one means you save $30 to $40 a month, this would have a very high return on investment. Of course, you face the challenge of verifying what they’ve done, etc.

    1. Your comment did make it through.

      1) Move to a hosting provider that costs say only $10 or $20 per month maximum.

      I think that the $10 to $20 price range is for unmanaged instances. I would need to handle security, SSL, updates, configuration, backups. I like that someone other than me is doing backups and securing Apache/PHP.

      When I write that csoft.net has been professional, I mean that I haven’t lost any data in 15 years. They also locked things down for me when I was being hacked.

      2) Fix the performance problems through caching and/or identifying the offending code.

      I’d be willing to pay someone for that, but given that I have a managed instance right now, this person could only do so much. You can’t reconfigure Apache, for example, or switch to another server. It seems to come down to installing and configure wordpress plugins, but I am skeptical that it is much help when things go to hell, because at that point, anything php will randomly fail. The .htaccess things is more interesting, of course. There might be some magic hacking there. But by the time you hit php, it is all over.

      Moving to an unmanaged instance and paying someone to help me manage is certainly an option. I think it will cost more in the end, but that might be the solution.

      1. Travis Downs says:

        I’d be willing to pay someone for that, but given that I have a
        managed instance right now, this person could only do so much. You
        can’t reconfigure Apache, for example, or switch to another server

        To be clear, I meant that (1) and (2) have to come together. Basically get someone who knows what they are doing with WP, and that comes with a recommended hosting solution in that price range (yes, I suspect they start at way under $50). This person would migrate your existing WP setup and ensure it all works great in a “pushbutton” fashion before handing it off.

      2. Paul says:

        … based on that response this whole post seems like an exercise for venting.

        You don’t seem to want change or improvement or lower costs.

        There are definitely better & at the same less expensive hosting options.

        There are definitely worthwhile plugins (WP Super Cache, S3 integrations, etc).

        15 years is a lot of posts, fortunately WordPress it self makes it very easy to export all that data, you could very very easily do that and set up a second website where you could validate data integrity and start exploring ways to save money, harrden WordPress (for security), and take measures necessary to improve performance. You could also offload Commenting to a 3rd party like Disqus that exists solely to make commenting easy from a spam & performance perspective.

        But it seems like with all these options you’re content to be malcontent.

        1. Paul: If you read the next post after this one, you will learn that I have moved to a different host (SiteGround). This happened within a week of writing this post. It is a managed instance, so I get server security, backups and all that. It is also quite fast. It is not, generally speaking, cheaper, but “high cost” was not my concern. (SiteGround is “affordable”, I’d say.)

          The mistake I made was to underestimate how bad csoft.net hosting was. Simply switching host solved my problems.

          Some specific comments:

          • Once your PHP scripts start failing more or less randomly, using plugins like WP Super Cache is not going to save you. These plugins rely on PHP working properly. I had used WP Super Cache for many years, but it did not help the stability of the setup in my experience. In a follow-up blog post I describe how WP Super Cache relies on 3 seconds caching… So it is not going to do a good job, even when it works well, at preventing your wordpress scripts from being hit (hard?).

          • Disqus is not appealing to me, as an independent blogger. I want to own and control my platform. I don’t want to offload comment data and the corresponding user information to a third party that may then possibly resell this data. Reading about disqus (which has, effectively, a monopoly on this type of service) brings many complains having to do with bloat and low performance, sudden changes as to how things work and as to how they are displayed. And, of course, the service may cost money (reasonably enough): I see $9 a month.

          • Moving software to a different setup is not free. The move was much less painful that I expected (took 1h to 2h), but weeks later I am still solving problems related to the move. Some tiny problems that came up took hours to solve, as I needed to research the problems. Avoiding an unnecessary move is rational.

          • Several commenters pointed out that I could just drop WordPress and use something else. I fear that they greatly underestimate how hard this would be. Yes, I know about things like Hugo. My relatively simple home page is built using Hugo… and it took me nearly took weeks of hacking to get it to be how I want. Porting my blog to something like Hugo would be a major disruption, might imply moving to disqus (see point above) and so forth.

          • In my comment above I was answering Travis who was basically recommending that “I pay someone” and go to an unmanaged instance (if I understand correctly). Given that it took me less than one hour to get a clue that performance would be better at SiteGround… It is unclear whether paying someone would have worked. There are transaction costs: you need to setup a job description, you need to agree on the terms, you need to select the person. Then if you go to an unmanaged instance, there are going to be ongoing support costs to make sure things keep on working. It is possible that Travis was right and that I have been wrong, but I think that my own approach to stick with managed instances and mostly do it myself is probably reasonable.

          As someone said, geeks like change, as long as it is change that they cause, otherwise they hate it. Reasonably, I try to maintain stability in my tools and workflow. In this instance, I admit that I was too conservative, I should have moved off csoft.net years go. However, I think that wordpress is probably going to serve me well for years to come. It may be old tech, but the benefits that come from its maturity are probably underrated.

          1. Travis Downs says:

            To be clear I was never recommending to go unmanaged. I was recommending that you pay someone who was familiar with the ecosystem to solve your WordPress headache. Since csoft gives you almost zero knobs to turn, this would most likely involve the selection of and migration to another managed instance.

            If you are willing to do it yourself, that could be a good option and in this case it turned out to be a great one since the effort migrate was small (a fraction of what I would have guessed – but that’s why I’m not a WP expert) and you were willing to attempt it yourself.

            I doubt paying someone permanently to handle an unmanaged WP instance makes much economic sense except in unusual scenarios so I wasn’t suggesting that.

  2. Daniel Walmsley says:

    Consider moving away from a CMS like WordPress. It’s great at what it does but I think you’ve out grown it. Static content generators might help here. They let you edit off line and compile to a static version of the web content you currently serve. I’d suggest looking at Hugo or one of the many specialised for blogging.
    https://opensource.com/article/18/3/start-blog-30-minutes-hugo

    You can then push the content to something Amazon’s s3 and put cloudfront in front to speed things up.

    You’ll have to then offload the comments to a service but that is relatively easy and cheap. A service like discus is only $9 a month. Moving to s3 you’ll save that in hosting. (I run two sites like this and pay $0.07 on average per month). Not to mention those services are pretty good at filtering spam.

    There are some other options around too. https://darekkay.com/blog/static-site-comments/

    WordPress is nice but it will never out perform static content.

  3. Laurent says:

    You would get a free hosting with google app engine. And never any 500.
    I host appblit.com on it since 2010 and never worry about a thing. Python, NodeJS or Java. I use python.

  4. Sumit Gupta says:

    With just 30K user, and $50 hosting I don’t think your blog should fail. It means your hosting is way too expensive for quality they are providing. I am using VPS from digital ocean and any $40 server from them if done write will give much better performance to your blog. You need to find another host for sure and possibly with VPS. wordpress only hosting are good too, but they get too complex at time.

  5. Sagar says:

    Host a static site on GitHub or Amazon S3 (+Cloudfront/flare). Comments can be with Disqus or just link to a post in your own sub reddit / HN.

    1. mebas says:

      +++ This is the way to go

    2. dodgyb says:

      I second Sagar. There are some straightforward guides on migrating from WordPress to Jekyll, e.g.

      Unlimited Scale and Free Web Hosting with GitHub Pages and Cloudflare

      4 Steps To Migrate From WordPress To Jekyll

      You could use the savings to commission a poor student to do it for you

      1. I am skeptical that I can port over my blog to Jekyll and its thousands of comments, posts and everything without break a lot of content.

        1. Zoran says:

          you can try with Publii static CMS at getpublii.com
          it’s a great piece of software, it’s FREE, runs on Windows, Mac and Linux as desktop app, so you don’t need to login in order to post, you can write while offline and when online just push the update. also has a very nice WordPress importer option, so you can import your WP posts etc…
          and finally you can upload directly from the app to Github pages, Gitlab, Netlify, Amazon S3, or where ever you want thru built-in FTP, SFTP

        2. Timo says:

          As someone who’s been slowly trying to move his much smaller blog (lot fewer articles, couple of orders of magnitude less traffic) from WordPress to Jekyll, I can confirm that it’s not as easy as it’s sometimes made out.

          Transferring the blog posts is relatively easy, migrating the comments is a little harder – especially if you want to self host them like I do. Getting the details right like RSS feeds working transparently so your subscribers only notice minimal, if any, disruption is a surprising amount of work.

          That said, I personally still see benefit in moving away from WordPress. WP has a big target on its back security-wise as it’s running so many sites, and I’ve had to recreate a few too many blog posts because the online editor ate them.

    3. Yes. I would do this if I were starting anew today. When I started this blog, GitHub and AWS did not exist.

  6. rich says:

    Regarding:

    Cloudfare reports 45,000 uncached requests for the day, and most of them are in the last couple of hours.

    I believe the issue is that when CloudFlare caches your page(s), it does so “locally” – whether that means only at that particular CloudFlare datacenter or, more likely, narrowly than that. At least I am pretty sure that Cloudflare does not take the cached response and widely distribute it among its servers.

  7. I would say move to managed WordPress hosting like WPEngine or Kinsta. They will take care of all the heavy lifting of scaling so you can focus on writing.

    1. Yes, but I have a pre-existing, large and complex blog. I believe that these wordpress hosting services work better for people starting new blogs.

  8. Sasa says:

    Move the blog from wordpress to static site, then you can serve the blog on a free server to that number of users easily.

  9. There are WordPress plugins to convert your existing posts to static site generators (to name a few: Jekyll and Hugo). Jekyll can be hosted by GitHub Pages.
    As said above, you then have to convert every comment to Disqus.

  10. FWIW my blog gets roughly the same amount of traffic, and I don’t think it’s ever been down. I pay $0 for this, it’s hosted using GitHub Pages – they use Cloudflare internally I believe.

    I’ve migrated away from WordPress after realizing I really hate using their editor and this makes me hate the process of writing blog posts. Unfortunately it took some time to migrate and preserve all contents, but I’ve been very happy with the setup since. Some people I know use Hugo, I use vanilla Jekyll that GitHub Pages generate for me.

    1. Thanks. When I started my blog, GitHub did not exist. Facebook did not exist. So there is path dependence involved. I now have to deal with things as they are. I can start anew, of course… but that’s a choice that would cost me.

  11. John says:
  12. Stefano Costa says:

    One of the sites I monitor has an active user base greater than yours and hasn’t experienced your problems (the setup is pretty pretty powerful though).

    He too uses WP, and this is the list of plugin in site deploy:

    https://css-tricks.com/wordpress-plugins-we-use-3rd-edition/

    hope it gives you some insights.

    1. Thanks. In my case, the php scripts themselves fail, so I am doubtful that adding more plugins can help. Plugins tend to hurt more than they help when they can’t be expected to run without failure.

      Over the years, I tried dozens and dozens of plugins, and I have been consistently disappointed.

  13. Bonjour Daniel,

    I write on Medium and the editor is wonderful. Some articles get featured for extra distribution.

    If you stick with WordPress install the classic editor plugin to remove Gutenberg.

    GitHub pages is also a great free option as mentioned.

    Bonne chance,

    —Martin

    1. I write on Medium and the editor is wonderful.

      A blog on Medium is certainly nice, but it is no longer an independent blog. Someone else decides how it looks and how it is organized, and monetized.

  14. Catherine says:

    First, Gutenberg is terrible. Install the WordPress Classic Editor (https://wordpress.org/plugins/classic-editor/) as soon as possible and disable Gutenberg.

    For your hosting problems, it seems you’re paying an awful lot just to host a WP blog. Normally a WP blog, with that kind of traffic should be fine
    on a 5-10$ month VPS without any special tinkering. There’s something wrong here.

    If you don’t want to waste your time by fixing your WordPress install (have you tried to export your data and do a clean reinstall ?), you could go with https://wordpress.com/pricing/ (note the .com) the Premium plan should work fine. You’ll still need to migrate your data.

    1. I tried installing the classic-editor plugin, and then enabling it, but the Gutenberg editor sticks and won’t go away. It is frustratingly hard to debug such issues.

      1. I should point out that I get frequent 500 errors, so I am not sure whether I should be expecting any given plugin to work properly.

      2. Catherine says:

        Have you disabled the Gunteberg editor? It’s listed among your plugins.

        1. I can’t find a gutenberg plugin and there seems to be nothing like it in wp-content/plugins so I think it is intrinsic to wordpress (not a plugin).

      3. The issue was finally resolved after my blog stopped crashing.

    2. If you don’t want to waste your time by fixing your WordPress install

      I don’t think that the problem is with my installation.

      you could go with wordpress.com

      I expect I would not be able to bring my blog over without breaking things, and I would do away with my independence.

  15. Martin Drapeau says:

    AWS Lightsail offers bitnami LAMPs for 10$/mo. WordPress ready with caching and all. I migrated many WordPress blogs there and its fast. I get about 50,000 monthly visits and no performance issues at all. Check it out.

    1. Thanks. I am aware than unmanaged instances are cheaper, but I am not sure I want to handle configuration, update, security and all that.

  16. Francois Saint-Jacques says:

    Give any static site generator a try, blog posts are markdown document which you commit. Statically generated, you’ll never get 500 if you front it with CloudFlare. As Arseny said, hosting it on github pages will be almost free. The true downside is maybe comments, but then you can move to Discuss and never worry about spam again.

    My personnal preference would be Hugo because the installation process is trivial compared to the other.

    1. Here the issue is that there is path dependence. My home page is built using Hugo (https://lemire.me/en/), but it was redone entirely (I threw away my old page). It was a major undertaking to make it look the way I want. I fear it would be weeks of work to port my blog over. Certainly, it would be doable and more efficient, but the cost might be substantial.

  17. Chris Ryland says:

    From my notes:

    Changed all MaxClients settings (prefork MPM, worker MPM, event MPM) in /etc/apache2/apache2.conf to 40 from 150, on the theory that, with 2GB of physical RAM, and WP_MAX_MEMORY_LIMIT (default) 40M, it’ll use a max 1.6G of RAM which will stop the thrashing “fall over” behavior we’re seeing. (free -m gives 1.7G free when apache’s stopped.)

    The main thing to be sure is that Apache (if you have any control over that at your ISP) isn’t forking more processes than will fit in memory, or else you’ll start thrashing.

    1. I don’t think I have control over Apache, as I am using a managed instance.

  18. Have you considered moving to a WordPress.com hosted blog?

    1. I have created new blogs there, for side projects, but I am concerned about my existing, self-hosted blog.

    2. I have two concerns with that. One is that moving to a blogging platform breaks my independence. The second one is that I don’t think I can realistically port over my existing blog. I will unavoidably break things in the process.

      1. You’re probably right. I’ve used WordPress.com throughout for my blogs and have not moved to a self-hosted site as yet.
        I’ve exported posts to new blogs so that each site has its own focused niche. But obviously I couldn’t take the existing number of impressions to the new blog. Since the blogs aren’t commercial, it didn’t matter to me. Comments were easily ported as well. This, of course, is only for the free blogs WP.com offers. I haven’t tried any of the professional packages WP.com offers. Ads are displayed on free blogs irrespective if you’ve signed up for WordAds or not.

  19. Travis Downs says:

    Just a note on cloudflare: it didn’t save your blog from the “hug of death” because by default it only caches static resources. In the case of this blog that’s only a few mostly inconsequential files. The main page, rendered by WordPress (which is the thing that falls over) isn’t cached cloudflare more or less does nothing.

    1. Yes. It seems that you are entirely correct. To make matters worse, wordpress deliberately defeats caching by appending a query string to static ressources like css or js files. And even when WordPress does not do it, sites like Facebook append their own query strings, again defeating caching. Thankfully, you can tell Cloudflare to ignore that.

  20. Fazal Majid says:

    I would also advise moving to a static site generator like Hugo but the traffic we’re talking about is not something that should cause even a RaspberryPi to break a sweat.

    Turn PHP’s opcache on, or APC if it’s an older version of PHP. The MySQL configuration probably needs tuning as well. Some plugins may need to be disabled.

    A good tool to figure out what is going on is the Xdebug profiler https://xdebug.org/index.php

    1. Thanks. I think that to turn PHP opcache, I need to tweak the server configuration. I am on a managed instance and I cannot configure the server.

  21. mmaton says:

    I’m edge caching anonymous (cookieless) requests using cloudflare’s workers & not having to pay for a business account: https://www.mmaton.com/2018/07/02/cloudflare-cache-anonymous-requests/

    1. That’s an interesting hack.

  22. Stefanos says:

    Daniel, I could help you in any way possible if you like. Feel free to email me so we could start a thorough investigation. I don’t plan to charge you or anything, so don’t you worry about it.

    1. Thanks Stefanos.