Server issues and site downtime

Discuss things related to this website and ask questions about the forum. New members can introduce themselves here too. Announcements will also be posted here. Note topics are auto pruned if not viewed within 30 days.
Post Reply
User avatar
Arokhs Twin
Site Admin
Posts: 1295
Joined: Wed Jul 18, 2001 9:36 pm
Location: United Kingdom
Contact:

Server issues and site downtime

Post by Arokhs Twin »

There has been a few issues with the server crashing when under heavy load such as backing up etc. Today seemingly several files were deleted and / or corrupt and permissions were set to defaults.

I restored yesterday's backup and it seems OK but there are performance issues and some things such as avatars don't load or take a long time to load. Sometimes the site takes a while to respond and / or load. I suspect by looking at the log files the server has a faulty Hard Disk or corrupt filesystem. In other words the site may disappear for a while whilst it's fixed.

AT
By fire and by blood I join with thee in the Order of the Flame!
Webmaster of Arokh's Lair

User avatar
Blackbeard
Dragon
Posts: 162
Joined: Wed Jun 24, 2009 7:54 am
Location: Sweden

Re: Server issues and site downtime

Post by Blackbeard »

Phew, i thought you went "fuck it"-mode and shut the site down. Good to see it back up again.

User avatar
Arokhs Twin
Site Admin
Posts: 1295
Joined: Wed Jul 18, 2001 9:36 pm
Location: United Kingdom
Contact:

Re: Server issues and site downtime

Post by Arokhs Twin »

BTW data should not be lost as the database is stored on another server and that one is OK. It's the main hosting server that has the problem so hopefully the only thing we will have is downtime. Even if the server needs a rebuild it will not take long as I guess the host will switch out the hardware and put the virtual machine images back onto it.

It all runs under VMware so backing up and restoring an entire server isn't as hard as it sounds.
By fire and by blood I join with thee in the Order of the Flame!
Webmaster of Arokh's Lair

User avatar
Arokhs Twin
Site Admin
Posts: 1295
Joined: Wed Jul 18, 2001 9:36 pm
Location: United Kingdom
Contact:

Re: Server issues and site downtime

Post by Arokhs Twin »

Seems to be back to normal now. No idea what went wrong but the server uses a bank of SSD's as a cache which had failed on my other website a while back and that had similar symptoms. The server hardware hosts multiple virtual servers not just mine so if the hardware fails it tends to get noticed pretty quickly.

The fix (on my other site) was to replace the hard drives with SSD's and dump the cache especially now that large SSD's have come down in price.
By fire and by blood I join with thee in the Order of the Flame!
Webmaster of Arokh's Lair

Post Reply