No announcement yet.

Mantic Forum downtime: what happened and the future

This is a sticky topic.
  • Filter
  • Time
  • Show
Clear All
new posts

  • Mantic Forum downtime: what happened and the future

    First of all: sorry about the downtime. It wasn't supposed to take anywhere near that long, and now we're back I'd like to talk a little about what happened.


    The Mantic Forum has a team of moderators, and an administrator (me) -- Mantic pays for software licensing and for the server, but basically leaves me to it. Mantic's choice of software was vBulletin, which predates me being involved in the running of the forum.

    vBulletin version 5.9 was running on our site for sometime. On November 19th, I received an email from Bytemark (our hosting provider) informing me that the forum server was sending out spam. Here's that email:

    I've received a few reports that our server 'statler' has been sending out spam. We've tracked down the origin of the spam to your VM - a PHP script running on it called functions_editor.php. Below I've included the full headers of an example email.
    I'd suggest you check over all your PHP scripts for evals to prevent this from happening, and tighten up security, change passwords, etc.
    Also, using statler for outbound mail is a deprecated service that was intended for our customers who had ADSL lines with us - could you please configure your VM to send mail directly?
    I immediately began to investigate the issue, and stopped the server from being able to send any mail. I tried to log into the vBulletin site to download a fresh copy of the software, but was locked out. It seems the problem had been big enough that vBulletin had reset their customer passwords, so I emailed Chris at Mantic and finally got access on December 7th. Chris and I agreed that the vBulletin cloud would be a better option, as it'll keep the forum up to date. On December 10th, Chris purchased the vBulletin Cloud license, and I began the process of moving things over.

    What happened next?

    On December 12th, I took the server offline, backed up the database and the attachments and sent them over to vBulletin later that evening.

    I kept a public log, which you can see below. Basically, vBulletin had difficulty importing our database several times. Every time they'd do an import, they get an error which looked this like:

    After repeatedly trying to import the database, they eventually made progress and a few hours ago let me know that the site was good to go.

    2015-12-17 21:09 CST  Support have followed up with the developer. We should hear from them tomorrow morning (PST)
    2015-12-17 17:37 CST  More database errors are causing the upgrade from 5.9 to 5.10 to fail. vBulletin is continuing to look at the problem.
    2015-12-16 20:50 CST  A new import happened, the Mantic Forum logo appears now, but the same error.
    2015-12-16 15:14 CST  "I just got a note from the dev that is working on this and he wants to have the migration redone because of the issues."
    2015-12-16 10:52 CST  Followed up with vBulletin, waiting for an answer.
    2015-12-14 22:03 CST  vBulletin support confirms there's a problem, and we continue to await a fix.
    2015-12-14 15:45 CST  Pinged the vBulletin folks. Will post when there's news.
    2015-12-13 15:00 CST  All files have been imported by vBulletin, but there's an error on the site when logged in. vBulletin escalating the problem to their developers.
    2015-12-13 11:02 CST  All files delivered to vBulletin last night, now waiting for their support team to import the database (1.1GB) and attachments (831MB) to our new vBulletin Cloud site.
    2015-12-12 16:17 CST  okay, migration has begun!

    What could we have done differently?

    We could have built a new server, installed the software ourselves as before and moved to a new email host in order to avoid using Bytemark's depreciated service. If we'd done this, the site would have been back in some form on the 13th, but we'd still be in a situation where we'd have moderators with lots of problems accessing certain features of the software, or the unapproved posts problem. We may have those problems today, but moving to the hosted cloud provider means we're getting the latest and greatest vBulletin updates. It also means we're paying them for support now, and I can talk to them directly.
    I'm Matt, the administrator of the forum. If you have a problem, tweet me mattl or try and send an email to mattl at cnuk dot org