« previous next »
Pages: [1]   Go Down

Author Topic: *** November 10th downtime ***  (Read 1487 times)

0 Members and 1 Guest are viewing this topic.

Offline miskie

  • Mailman for the VRWC
  • Administrator
  • Hero Member
  • *****
  • Posts: 10461
  • Reputation: +1035/-54
  • Make America Great Again. Deport some DUmmies.
*** November 10th downtime ***
« on: November 11, 2014, 09:32:39 AM »
As I'm sure most of you are aware, The Cave was offline for several hours yesterday.

The short version is this -

One of the drives that power the server began to fail, which sent the server into an FSCK loop. FSCK is Linux (and current generation Apple) speak for "File System Consistency Check" - Where the system did what was necessary to preserve the integrity of itself. Several hours later the server brought itself back online for long enough for me to crawl around inside and discover the main system drive was failing.

At that moment, I tossed $50 bucks at a tech at the datacenter The Cave is housed at to clone the freshly cleaned data onto a new, error free drive.

I did a quick peek at several hosed sites, and all of them seem to be behaving normally, without any data loss - So, it seems the mission was successful.

-My apologies for the downtime, but sometimes crap like this happens.


PS - I do not think CC getting hammered during the midterms made this happen, but it could have been partially responsible for exposing an unknown precondition of that particular drive. 
Logged

Offline EagleKeeper

  • Hero Member
  • *****
  • Posts: 2585
  • Reputation: +134/-100
  • ΜΟΛΩΝ ΛΑΒΕ
Re: *** November 10th downtime ***
« Reply #1 on: November 11, 2014, 10:27:31 AM »
Ok, who am I kidding, I'd like to ask you some questions about server hardware and software architecture.
Logged
Never interrupt your enemy when he is making a mistake.
- Napoleon Bonaparte

If you wait by the river long enough the bodies of your enemies will float by.
-Sun Tzu

Offline EagleKeeper

  • Hero Member
  • *****
  • Posts: 2585
  • Reputation: +134/-100
  • ΜΟΛΩΝ ΛΑΒΕ
Re: *** November 10th downtime ***
« Reply #2 on: November 11, 2014, 03:35:42 PM »
Ok, you're pixy booting a linux phone OS running as a virtual machine?
Logged
Never interrupt your enemy when he is making a mistake.
- Napoleon Bonaparte

If you wait by the river long enough the bodies of your enemies will float by.
-Sun Tzu

Offline miskie

  • Mailman for the VRWC
  • Administrator
  • Hero Member
  • *****
  • Posts: 10461
  • Reputation: +1035/-54
  • Make America Great Again. Deport some DUmmies.
Re: *** November 10th downtime ***
« Reply #3 on: November 11, 2014, 04:29:56 PM »
Quote from: EagleKeeper on November 11, 2014, 10:27:31 AM
Ok, who am I kidding, I'd like to ask you some questions about server hardware and software architecture.

Shoot away..
Logged

Offline miskie

  • Mailman for the VRWC
  • Administrator
  • Hero Member
  • *****
  • Posts: 10461
  • Reputation: +1035/-54
  • Make America Great Again. Deport some DUmmies.
Re: *** November 10th downtime ***
« Reply #4 on: November 11, 2014, 04:34:12 PM »
Quote from: EagleKeeper on November 11, 2014, 03:35:42 PM
Ok, you're pixy booting a linux phone OS running as a virtual machine?

I do, yes. But not with The Cave.. If you are looking for a virtual Android solution for home computer use, look into Genymotion.  Be warned though. You will need to do a bit of work with Genymotion to be able to run multimedia properly, but on a well equipped PC, the experience it will provide is as good as any actual tablet.
Logged

Offline EagleKeeper

  • Hero Member
  • *****
  • Posts: 2585
  • Reputation: +134/-100
  • ΜΟΛΩΝ ΛΑΒΕ
Re: *** November 10th downtime ***
« Reply #5 on: November 11, 2014, 04:53:11 PM »
Specifically about the cave server.

Is it a VM?
And why would you have to pay them $50.00 to do their job?

Don't they have a service level agreement?
Logged
Never interrupt your enemy when he is making a mistake.
- Napoleon Bonaparte

If you wait by the river long enough the bodies of your enemies will float by.
-Sun Tzu

Offline miskie

  • Mailman for the VRWC
  • Administrator
  • Hero Member
  • *****
  • Posts: 10461
  • Reputation: +1035/-54
  • Make America Great Again. Deport some DUmmies.
Re: *** November 10th downtime ***
« Reply #6 on: November 11, 2014, 05:02:46 PM »
Quote from: EagleKeeper on November 11, 2014, 04:53:11 PM
Specifically about the cave server.

Is it a VM?
And why would you have to pay them $50.00 to do their job?

Don't they have a service level agreement?

Okay - Answers - No, Its not a VM - its actual physical hardware. Its a 2.83 ghz quad core Intel running 64 bit CentOS 6.5 with 8GB of ram. as for the SLA,  it included the diagnostic and repair/replacement of a failing drive free of cost - however, since the datacenter is in Texas and I in Massachusetts, I opted to have them clone the data to the new drive for me instead of doing it myself. There are some things that really can't be done safely via an SSH prompt. That's what I paid for. 
Logged
Pages: [1]   Go Up
« previous next »