WordPress Database Constantly Crashing? XML-RPC Attacks Likely to Blame

1 TL;DR

If your WordPress installation has been running fine for a long time and suddenly seeing DB errors such as “Error connecting to database”, your website might be a victim of XML-RPC attacks. Usually assailants can just utilize brute force attacks and bring down the website. You can simply disable XML-RPC features to stop such attacks.

I thought it would never happen to me – after all this is a personal blog with <100 monthly views (usually). But today I noticed something unusual: early in the morning I received an email saying my blog is down. Further inspection showed it’s an “Error connecting to database” error. Well, life lessons taught me that power cycling a machine can be the best way to fix an issue – and that’s what I initially did.

docker-compose down
docker-compose up -d

(Something irrelevant: one advantage that docker-compose offers over conventional LAMP stack is the ease of management – in this case two commands were suffice to restart the whole stack.)

Did it work? Ye…. well no! After ~10 mins of uptime the blog went down again, giving that same db error. I verified that db was indeed down according to docker ps listings.

2 Some attempts to fix

Usually WordPress is quite stable once I have set it up – unless something basic got messed up. I checked the following things just to make sure everything’s in order

• docker-compose.yml. It appeared to be normal though, as environment variables were correctly set and matched up on both mysql and wp side, and all necessary requirements were present.

• mysql installation. If for some reason mysql image got tempered (unlikely), it could start doing all sorts of weird stuff. To make absolutely sure that my docker infrastructure is good, I went ahead and pruned all images and made docker download everything again.

  docker system prune -a
  docker-compose up -d
  

  The result wasn’t that great though, the blog quickly went down yet again over that very db problem.

• I also restarted the whole VPS. I hate to do so, but that’s the sacrifice one has to make sometimes. Except that it didn’t help with the problem at all.

3 More info needed

At this point, I decided not to grope in the dark anymore since all the “quick and easy” fixes were not working. I need some insight into what is really happening. Like breakpoints, logs are your friend in troubleshooting.

• First I checked db container’s log:

  docker logs db_kazumiblog
  

  but unfortunately it didn’t really tell me much about the whole disaster. All I saw were generic status messages when mysql booted up and initialized.

• After some research I started to speculate that there was a memory leak in mysql which caused it to be involuntarily shut down by the system. If this were true this would also explain why the mysql shut down without any warnings in the log. I checked the system log while highlighting “memory” and “oom” (out of memory): 

  journalctl -k | grep -i -e memory -e oom 
  

  The result confirmed exactly my speculation.

  

  It was apparent that the mysql instance was being too hungry of memory and got itself killed by the system (possibly repeatedly). The question was, why would it do that? It has been stable for 8 months and I never noticed such problem before.

• Then I came across a post that described almost exactly the same issue. The top reply suggested that XML-RPC were to blame. To verify this, I checked the nginx access log (/var/log/nginx/access.log), aaand…

  

  Wowzers, there were thousands of lines of XML-RPC calls from that Russian IP! I have no doubt this is what overloaded the mysql instance. I remember being suggested to turn off XML-RPC feature, but I thought the blog was too small to attract any attackers, and left it as it was. The incident today proved me totally wrong.

4 Remedies

Once the root cause became clear, it was easy to fix – I simply disabled XML-RPC through a plugin. This webpage also lists some alternative methods to turn it off (not tested).

I will also work on improving the backend security of the blog, which I used to overlook. But even a slight oversight could bring me much headaches.

Last but not least I will start backing up the whole VPS – just in case.