An explanation for Mortal Kombat Online’s downtime…

Well, Mortal Kombat Online is facing an extended downtime. Apparently we are suffering from such a catastrophic hardware failure, so to speak, that it’s going to take a few days and a LOT of work to get everything back up and running the way it was. I’ve left the standard disclaimer on the site itself, but I figured I would use this post on my own site to give an in-depth explanation as to what happened and allow others to give feedback if necessary.


In any event, here’s what’s happening.

Back in Christmas of 2005, the hard drive on the main MK Online server suffered a crash. We were forced to get our hosting provider to replace the drive; when we had them replace it, we demanded that they put in mirrored drives like we had when we first got our dedicated server with them. (At the time, we were actually on a new dedicated server.) They did so, but instead of using two identical drives they used a single SCSI (heavy-duty) drive and a much larger IDE (consumer-level) drive. We had meant to get with our hosting provider to fix that, as such a configuration is much less than ideal. However, we never got around to it as bigger issues kept coming up.

Well, recently we’d been noticing system crashes, caused by memory errors. We soon realized that it was because of virtual memory (which is on the IDE drive), and it appears that the IDE drive is close to failure. The problem has gotten much worse over the past week, to the point where the filesystems are being corrupted. The last straw came an hour ago, when the forum database table that held the actual posts was lost. Obviously, we can’t continue like this.

So what are we doing to correct this? I’m working on restoring a backup copy of the database from this morning. With luck, it will be in a decent state but there is the very real possibility of data loss. In the meantime, our hosting provider is building a newer, faster server for us. Once it’s ready, we’ll rush to copy the data from the old server onto the new one and try and get us back online as quickly as possible. Still, we expect to be offline until the weekend at least.

We apologize for the delays and inconvenience this has caused. Trust me, I’m as frustrated as anyone else about this, if not more so. (After all, I’m the one trying to pick up the pieces here….) Hopefully once we have everything corrected it’ll be smooth sailing. :-)

EDIT – 7:00 PM CDT – It looks like the latest backup of the database is intact. Everything up to this morning at around 9 AM EDT appears to have been saved. What a relief… heh. Now we just need to wait for the new server to come online, and we’ll work from there. I’ll provide more information as it becomes available.

EDIT – 7:50 PM CDT – By the way, the Java chat is still available for those of you who come onto #mortalkombat (aka, the #MortalKombat Online IRC Network). You can reach it by clicking here.

EDIT – 7:20 AM CST 10 May – Thanks for the input, everyone. I just wanted to point out a few things based on comments I am seeing…

  • For those asking, when the site comes back up it will come back up with no changes in layout, software, etc. CCShadow hasn’t had time to work on version 8 of CDS (the backend of the site), and to expect us to change software packages at a drop of the hat is unrealistic at best.
  • Jonin01 is right in that this will take time. I’ve done everything I can do on this end; now it’s just up to our hosting provider to get a machine ready for us to migrate to. Once I hear the new machine is up and running, I can copy all of our old data to it and bring it online. Until then, it’s simply a matter of waiting.
  • Finally, for those offering assistance, it is much appreciated. Truth be told, we have everything under control; I suppose it helps a great deal when I do this stuff (system administration and support) as my paying job as well as for MK Online. It’s really not as catastrophic as it sounds; even if we were to lose the main server filesystem today I have a site backup elsewhere that can be used to get MK Online back up and running. We’re in good shape, all things considered; it’s more annoyances than anything else right now.

As before, once I get more information I’ll be sure to post it here. Thanks for your support and patience while we work to get these problems taken care of. :-)

EDIT – 2:20 PM CDT 11 May – The new server is ready for us to start working on it. However, being out of town and without a direct path to the internet, I can’t access the server properly to begin working on it. I should be home by 10 PM CDT, at which point I’ll begin working on it. If all goes well, we should be back up and running tomorrow evening. :-)

EDIT – 10:50 PM CDT 11 May – There’s a problem with the operating system’s configuration that’s preventing us from using it. Rather than going through hours of trying to fix it manually, I’ve had CCShadow contact our hosting provider with precise instructions for an OS reinstall. Hopefully sometime tomorrow it’ll be ready for the actual site reimplementation.

EDIT – 9:40 PM CDT 12 May – No, our hosting provider hasn’t rebuilt the server yet. I’m going to get with CCShadow as soon as he gets online about contacting them directly and getting them to hurry it up. I know everyone’s getting annoyed; I’m probably more annoyed than anyone at this point. Thanks for your patience.

EDIT – 3:35 PM CDT 13 May – Our hosting provider went ahead and made the needed changes without requiring a reinstallation of the operating system. I’m in the process of configuring the server and restoring the data. This will take a while, so please be patient.

EDIT – 7:50 PM CDT 13 May – All of the data is restored. We’re working now to get the new server onto the old server’s IP address and do some final testing. Once that’s done we’ll reopen the site.

EDIT – 8:50 PM CDT 13 May – It’s going to be a while, as their automated phone system is backed up and I can’t get a call through. CCShadow will call them when he gets home and get it online. I don’t know if we’ll be open tonight, but we’ll do our very best. Otherwise, we’re hoping tomorrow at the very latest.

EDIT – 8:25 AM CDT 14 May – The good news is that the new server is on the necessary IP address. The bad news is that I’m encountering some serious issues getting the site to work properly on the new server. I’ll keep working on it throughout the day to see if I can get the problems fixed.

EDIT – 11:30 AM CDT 14 May – CCShadow and I have taken a preliminary look at the server. Right now we think there’s some incompatibility or software conflict that’s causing CDS not to work properly. I’ll continue looking at it when I have time (which isn’t often as there are upgrade issues here at work that need resolving; as that’s my paying job it has to take priority), and CCShadow will do heavy-duty debugging when he gets home tonight.

EDIT – 3:10 PM CDT 14 May – If the testing we’ve been doing is anything to go by, we’ve knocked out all but one minor annoying bug. Right now we’re evaluating whether it’s safe to reopen the site now or not. If CCShadow gives the go-ahead, we’ll go ahead and reopen.

EDIT – 3:55 PM CDT 14 May – Never mind. It turned out that one bug is affecting more site functions than we thought. Reopening will have to wait until CCShadow’s had a chance to stamp out the bug. Sorry, folks… this is frustrating us as much as it is you. Hopefully we can be back up and running tonight…

EDIT – 10:35 PM CDT 14 May – We’re back up!

286 thoughts on “An explanation for Mortal Kombat Online’s downtime…

  1. My god the WITHDRAWL is killing me….Please get back up. I know it’s not for lack off effort. Thank’s for all the hard work.

  2. this shit is taking time, maybe they are just making excuses to close the site dont you agreee

  3. “this shit is taking time, maybe they are just making excuses to close the site dont you agreee”

    That’s the biggest load of bullshit I’ve heard in a very long time.

  4. LOL@KingMob, wouldn’t be surprised if it’s that loser JTF….sounds like something he’d say.Anyway, to all you guys bored as hell you can ummm play some Mortal Kombat to heal your MK craving! ; )

    For the general people, just wait it out, watch porn, grab a movie, go out, play MK or something….

    P.S. SCREW the Boston Red Sox!!! >:-/

  5. I’m so bored, ive been waiting aLL DAY! Anyways glad your testing it out, TodaY IT WILL DEFIANTLY BE up and running, i hope. Well hope it come’s up in like 2 hour’s, or so. Good luck.

  6. Word on the street is that Mods are already able to log into MKO. I wont be suprised if MKO will be opened to rest of the folks today. Be patient, the blackout is almost over.

  7. thank you so much D’arque for working hard on getting MKO back up and running; you’re making it sound as if we’ll be defintely be able to get on sometime today, hopefully that will work out.

  8. Word on the street is that Mods are already able to log into MKO. I wont be suprised if MKO will be opened to rest of the folks today. Be patient, the blackout is almost over.

    The site was unlocked to a couple of staff and mods for testing purposes, which is how we found the bugs I mentioned in my latest edits. At this point I’ve done about all I can; CCShadow needs to look at it and debug from here. He’ll be able to do sometime late this evening.

  9. From what I’ve read so far.D’Arque has done all be can and now it’s over to CCShadow. Straighten those bugs out buddy and hopefully MKO can be back tonight. Thanks for keep us all informed, D’Arque. Nice project management skill you’ve got there. Are you skilled at MKA?

  10. Hopefully you guys will have everything up again soon, and no worries, your loyal Mk Online regulars are behind you!

  11. I hate being lied to, You said it was gonna be back up today, WTF, its STILL not up! I’m trying to be patient but this is getting annoying.

  12. Shyriou, if it’s you posting as me again, you are going to banned. Was it really worth it?

  13. i am getting a little annoyed having to wait another day every day that its been stated we’ll be able to get on, but i’m not mad at D’arque; he’s doing the best he can, and i can see that. It’s just i’m annoyed at the situation of not being able to get on MKO, but maybe after i come back after TKD i’ll be able to get on,hopefully. That’s what i wanted to get off my chest.

  14. Contingency planning…where is it? DR procedures? BCP?
    You’re an IT Administrator…what is happening here?
    48 hrs is a reasonable time, not a week. Some serious discussions with hostrocket should take place i imagine. Your service level agreement has been blown out of the water…
    # 1 hour Hardware Replacement SLA
    # 99.5% Network Uptime SLA

  15. BTW, Shyriu (or whoever he is) has been added to the ban list on the site, Congrats, mate. That’s what you get for spoofing kingjolly.

    As for AussieNinja’s remarks… I don’t know what our SLA is. I don’t deal directly with the hosting provider. That’s CCShadow’s job. HR has done as much as they can on their end. Right now the remaining problems are ones we have to fix. We’re hoping to have them fixed ASAP, though.

  16. I don’t mean to butt in to much, but I know Shyriu and he usualy doesn’t act like that. He can be a jerk, but he doesn’t break any rules as far as I know. Are you sure that it is even him, and not someone using that name just to get him in trouble? I am not making acusations and I could be wrong but I know there are people that would like to get rid of him….look at the recent comment if you don’t know already. Doesn’t the fact that he misspelled his own name seem rather suspicious to you? Not my buisness I know, but I would hate to see someone get banned if they are innocent.

  17. I just took a nap, and it’s still not up damn… Well i hope it’s up by atleast 9:30 tonight, i hope, it’s up. Sincerly,-redman

  18. Good Lord! What the fuck happen here? Sigh….

    Well, It’s been nice arguing with you on MKO, Shyriu.

  19. Yoo, MK2007 thanks for that update man.They soo look like a couple lol.I still say that Chameleon looks more badass then Khameleon but don’t get me wrong, she looks hot for a transparent chick lol.

  20. Hey DAB, have you ever thought about using a Myspace page or something to the liking as maybe nothing more than a backup for when stuff like this happens?? We’ve had a couple of really cool MKA Wii Videos and pictures come up and it’s just a real shame that a lot of these people who depend on MKO for all their MK news and info don’t have access to it.

  21. SHAME ON YOU! To think that dumbass was actually me! Pfff… At least ONE person was actually smart enough to notice I wouldn’t post stupid stuff like that. I write maybe “mean” or “condescending” stuff, but not “retard”.

Comments are closed.