« previous next »
Pages: [1] 2   Go Down

Author Topic: explanation of the outages earlier today  (Read 8517 times)

0 Members and 1 Guest are viewing this topic.

Offline Wretched Excess

  • Hero Member
  • *****
  • Posts: 15284
  • Reputation: +485/-84
  • Life, Liberty and the pursuit of Happy Hour
explanation of the outages earlier today
« on: May 13, 2008, 01:07:35 PM »
There were actually two problems. (well, three, before I was done typing this)

first I believe server pronto had a networking issue at about 10:30 eastern time.  they resolved that
after an hour, and we came back up very briefly.  but shortly thereafter, they either dumped the server,
or it lost power for some other reason.

I do know that all process running on the server right now have a 12:20 start time, and I got an
email from them saying one of their techs was "looking at the server, not aware that the problem
was resolved". :thatsright:

when we came back up the second time, we had a corrupted table.  that just took a little while
to get resolved.

I was in the middle of writing this when the whole thing crapped out again.  (for th third time) this
one was another networking issue at our colocation provider.  they are having a really bad
day.

sorry, folks.  we were really rocking when the first problem hit this morning.



« Last Edit: May 13, 2008, 03:42:13 PM by Wretched Excess »
Logged

Offline Chris_

  • Little Lebowski Urban Achiever
  • Hero Member
  • *****
  • Posts: 46845
  • Reputation: +2028/-266
Re: explanation of the outage earlier today
« Reply #1 on: May 13, 2008, 01:09:38 PM »
I thought it was because I was posting the greatest post in Internet History.

Future generations will cry and rend their hair when they find out it was stillborn.
Logged
If you want to worship an orange pile of garbage with a reckless disregard for everything, get on down to Arbys & try our loaded curly fries.

Offline DixieBelle

  • Administrator
  • Hero Member
  • *****
  • Posts: 12143
  • Reputation: +512/-49
  • Still looking for my pony.....
Re: explanation of the outage earlier today
« Reply #2 on: May 13, 2008, 01:17:37 PM »
Tee hee FD. I had a screed to rival the Pittster going. Lost to the intertubes forever. Sigh. :-)
Logged
I can see November 2 from my house!!!

Spread my work ethic, not my wealth.

Forget change, bring back common sense.
-------------------------------------------------

No, my friends, there’s only one really progressive idea. And that is the idea of legally limiting the power of the government. That one genuinely liberal, genuinely progressive idea — the Why in 1776, the How in 1787 — is what needs to be conserved. We need to conserve that fundamentally liberal idea. That is why we are conservatives. --Bill Whittle

Offline Lord Undies

  • Hero Member
  • *****
  • Posts: 11388
  • Reputation: +639/-250
Re: explanation of the outage earlier today
« Reply #3 on: May 13, 2008, 01:25:25 PM »
Oh, you amateurs!  Always type your screeds on Word or Notepad and then copy & paste.

Must I do all the thinking around here?   :yawn: 
Logged

Offline dutch508

  • Hero Member
  • *****
  • Posts: 12528
  • Reputation: +1660/-1068
  • Remember
Re: explanation of the outage earlier today
« Reply #4 on: May 13, 2008, 01:34:53 PM »
Quote from: Wretched Excess on May 13, 2008, 01:07:35 PM

There were actually two problems. (well, three, before I was done typing this)

first I believe server pronto had a networking issue at about 9:30 eastern time.  they resolved that
after an hour, and we came back up very briefly.  but shortly thereafter, they either dumped the server,
or it lost power for some other reason.

I do know that all process running on the server right now have a 12:20 start time, and I got an
email from them saying one of their techs was "looking at the server, not aware that the problem
was resolved". :thatsright:

when we came back up the second time, we had a corrupted table.  that just took a little while
to get resolved.

I was in the middle of writing this when the whole thing crapped out again.  (for th third time) this
one was another networking issue at our colocation provider.  they are having a really bad
day.

sorry, folks.  we were really rocking when the first problem hit this morning.





THOSE ****ERS FROM CU CORRRUPTED THE DATA STREAM!!!! THEY ARE JUST PISSED BECAUSE THEIR SHITHOLE OF A SITE IS STILL DOWN!!!!! THEN THEY BROKE IN AND BLEW UP THE SERVERS!!!!! AFTER THAT, THEY MAULED W.E. WITH A TRAINED BEAR!!!!!

 :tampon: :tampon: :tampon: :tampon: :tampon:
Logged
The torch of moral clarity since 12/18/07

2016 DOTY: 06 Omaha Steve - Is dying for ****'s face! How could you not vote for him, you heartless bastards!?!

Offline Randy

  • Resident Grouch with a
  • Hero Member
  • *****
  • Posts: 4244
  • Reputation: +202/-39
  • Odd
Re: explanation of the outage earlier today
« Reply #5 on: May 13, 2008, 01:40:40 PM »
Quote from: Wretched Excess on May 13, 2008, 01:07:35 PM

There were actually two problems. (well, three, before I was done typing this)

first I believe server pronto had a networking issue at about 9:30 eastern time.  they resolved that
after an hour, and we came back up very briefly.  but shortly thereafter, they either dumped the server,
or it lost power for some other reason.

I do know that all process running on the server right now have a 12:20 start time, and I got an
email from them saying one of their techs was "looking at the server, not aware that the problem
was resolved". :thatsright:

when we came back up the second time, we had a corrupted table.  that just took a little while
to get resolved.

I was in the middle of writing this when the whole thing crapped out again.  (for th third time) this
one was another networking issue at our colocation provider.  they are having a really bad
day.

sorry, folks.  we were really rocking when the first problem hit this morning.





Well it's either Glowball Warming or that damn Bush that caused it. Pick one. :-)
Logged

Offline Wretched Excess

  • Hero Member
  • *****
  • Posts: 15284
  • Reputation: +485/-84
  • Life, Liberty and the pursuit of Happy Hour
Re: explanation of the outage earlier today
« Reply #6 on: May 13, 2008, 02:25:04 PM »
Just heard from serverpronto. They confirmed mu suspicions that there was a power problem. They had a "circuit overload" in the middle of everything else.

That what put us on our a$$es so badly.

Logged

Offline jtyangel

  • Hero Member
  • *****
  • Posts: 9116
  • Reputation: +497/-110
Re: explanation of the outage earlier today
« Reply #7 on: May 13, 2008, 03:03:49 PM »
Thx a lot for making me attend to other things today :banghead: :bird: :-)
Logged

Offline Dixie*Darling

  • Hero Member
  • *****
  • Posts: 1117
  • Reputation: +133/-25
Re: explanation of the outage earlier today
« Reply #8 on: May 13, 2008, 03:14:01 PM »
Quote from: dutch508 on May 13, 2008, 01:34:53 PM

 :tampon: :tampon: :tampon: :tampon: :tampon:

All hail the might and power of our army of angry tampons!   :lmao:
Logged

Offline Ralph Wiggum

  • It's unpossible that I'm a
  • Hero Member
  • *****
  • Posts: 18629
  • Reputation: +2073/-49
Re: explanation of the outage earlier today
« Reply #9 on: May 13, 2008, 03:19:56 PM »
I was afraid the site went down because of the threads I added to the new forum last night. :bawl: :-)
Logged
Voted hottest "chick" at CU - My hotness transcends gender

Offline Wretched Excess

  • Hero Member
  • *****
  • Posts: 15284
  • Reputation: +485/-84
  • Life, Liberty and the pursuit of Happy Hour
Re: explanation of the outage earlier today
« Reply #10 on: May 13, 2008, 03:29:31 PM »
yikes.  I just had time to really look at the server logs.  the first problem was indeed a network problem, but the
server lost power not once, not TWICE, but THREE times, at 11:40, 12:20, and 13:38 while we had
intermittent or no network access.  the original network problem was more than likely caused by a power
failure to some piece of network gear that provided our network connectivity, but didn't immediately cause
a power problem for the server.  but the power problem obviously spread.

no wonder we had a hopelessly scrambled table;  it damaged the ubunto filesystems on the second power
bump.  the filesystems rebuilt automatically on a reboot, but that totally explains the corrupt table.  in fact, it's
a wonder that it wasn't worse.

I'm glad I didn't know it was this bad until it was all over.  we'll need to reboot late one night this week, because
it didn't exactly come up clean the last time, either.




« Last Edit: May 13, 2008, 03:43:18 PM by Wretched Excess »
Logged

Offline Wretched Excess

  • Hero Member
  • *****
  • Posts: 15284
  • Reputation: +485/-84
  • Life, Liberty and the pursuit of Happy Hour
Re: explanation of the outages earlier today
« Reply #11 on: May 13, 2008, 09:23:06 PM »

response times into and out of serverpronto aren't so hot right now. :censored:
Logged

Offline rich_t

  • Hero Member
  • *****
  • Posts: 7942
  • Reputation: +386/-429
  • TANSTAAFL
Re: explanation of the outages earlier today
« Reply #12 on: May 13, 2008, 09:59:46 PM »
Power problems at a server farm?  That don't sound too good.
Logged
"The American people will never knowingly adopt socialism. But, under the name of 'liberalism,' they will adopt every fragment of the socialist program, until one day America will be a socialist nation, without knowing how it happened." --Norman Thomas, 1944

Offline Chris_

  • Little Lebowski Urban Achiever
  • Hero Member
  • *****
  • Posts: 46845
  • Reputation: +2028/-266
Re: explanation of the outages earlier today
« Reply #13 on: May 13, 2008, 10:45:52 PM »
Quote from: rich_t on May 13, 2008, 09:59:46 PM
Power problems at a server farm?  That don't sound too good.

I should say not.
Logged
If you want to worship an orange pile of garbage with a reckless disregard for everything, get on down to Arbys & try our loaded curly fries.

Offline Miss Mia

  • Hero Member
  • *****
  • Posts: 8052
  • Reputation: +353/-137
Re: explanation of the outages earlier today
« Reply #14 on: May 13, 2008, 10:52:13 PM »
This is unacceptable WE.  I'm going back to C...










oh wait, nevermind I'll stay.   :-)
Logged
Stink Eye
"Bloodninja: It doesn't get any more serious than a Rhinocerus about to charge your ass."

Offline Wretched Excess

  • Hero Member
  • *****
  • Posts: 15284
  • Reputation: +485/-84
  • Life, Liberty and the pursuit of Happy Hour
Re: explanation of the outages earlier today
« Reply #15 on: May 13, 2008, 10:52:58 PM »
Quote from: Schadenfreude on May 13, 2008, 10:45:52 PM
Quote from: rich_t on May 13, 2008, 09:59:46 PM
Power problems at a server farm?  That don't sound too good.

I should say not.

oh, it's not.  we had no idea at the time that it was dumping over and over because we couldn't get into to see what the hell was going on.  I suspected it had lost power at least once, since I noticed (during one of the short intervals it was up) that everything had a very recent process start time. once I saw the logs later that afternoon, I nearly passed out.  we could be in the adjoining gravesite from TOS right now.


Logged

Offline rich_t

  • Hero Member
  • *****
  • Posts: 7942
  • Reputation: +386/-429
  • TANSTAAFL
Re: explanation of the outages earlier today
« Reply #16 on: May 13, 2008, 10:53:33 PM »
Quote from: Schadenfreude on May 13, 2008, 10:45:52 PM
Quote from: rich_t on May 13, 2008, 09:59:46 PM
Power problems at a server farm?  That don't sound too good.

I should say not.

This server farm needs to check into what went wrong with their UPS system.  Blown circuit breakers aka circuit overloads are indicative of possible poor power planning.
Logged
"The American people will never knowingly adopt socialism. But, under the name of 'liberalism,' they will adopt every fragment of the socialist program, until one day America will be a socialist nation, without knowing how it happened." --Norman Thomas, 1944

Offline Wretched Excess

  • Hero Member
  • *****
  • Posts: 15284
  • Reputation: +485/-84
  • Life, Liberty and the pursuit of Happy Hour
Re: explanation of the outages earlier today
« Reply #17 on: May 13, 2008, 10:53:49 PM »
Quote from: Miss Mia on May 13, 2008, 10:52:13 PM
This is unacceptable WE.  I'm going back to C...










oh wait, nevermind I'll stay.   :-)

you actually had me for a second.  :o :-)
Logged

Offline Miss Mia

  • Hero Member
  • *****
  • Posts: 8052
  • Reputation: +353/-137
Re: explanation of the outages earlier today
« Reply #18 on: May 13, 2008, 10:55:15 PM »
Quote from: Wretched Excess on May 13, 2008, 10:53:49 PM
Quote from: Miss Mia on May 13, 2008, 10:52:13 PM
This is unacceptable WE.  I'm going back to C...










oh wait, nevermind I'll stay.   :-)

you actually had me for a second.  :o :-)

It's the "unacceptable" part, I always use that word when I'm trying to sound "serious."
Logged
Stink Eye
"Bloodninja: It doesn't get any more serious than a Rhinocerus about to charge your ass."

Offline Wretched Excess

  • Hero Member
  • *****
  • Posts: 15284
  • Reputation: +485/-84
  • Life, Liberty and the pursuit of Happy Hour
Re: explanation of the outages earlier today
« Reply #19 on: May 13, 2008, 10:56:03 PM »
Quote from: rich_t on May 13, 2008, 10:53:33 PM
Quote from: Schadenfreude on May 13, 2008, 10:45:52 PM
Quote from: rich_t on May 13, 2008, 09:59:46 PM
Power problems at a server farm?  That don't sound too good.

I should say not.

This server farm needs to check into what went wrong with their UPS system.  Blown circuit breakers aka circuit overloads are indicative of possible poor power planning.

yeah, we agree.  and we're on it.  they are a very well recommended colocation provider, but today sucked. :censored:
Logged

Offline Rebel

  • Stick a fork in us. We're done.
  • Administrator
  • Hero Member
  • *****
  • Posts: 16808
  • Reputation: +1259/-215
Re: explanation of the outages earlier today
« Reply #20 on: May 13, 2008, 10:57:12 PM »
Outages are going to occur, people. 2 week outages, however, NEVER occur unless there is piss-poor planning. I can assure you, WE and I are on top of every outage we ever had/have.
Logged
NAMBLA is a left-wing organization.

Quote
There's a reason why patriotism is considered a conservative value. Watch a Tea Party rally and you'll see people proudly raising the American flag and showing pride in U.S. heroes such as Thomas Jefferson. Watch an OWS rally and you'll see people burning the American flag while showing pride in communist heroes such as Che Guevera. --Bob, from some news site

Offline rich_t

  • Hero Member
  • *****
  • Posts: 7942
  • Reputation: +386/-429
  • TANSTAAFL
Re: explanation of the outages earlier today
« Reply #21 on: May 13, 2008, 10:59:52 PM »
Quote from: Wretched Excess on May 13, 2008, 10:52:58 PM
Quote from: Schadenfreude on May 13, 2008, 10:45:52 PM
Quote from: rich_t on May 13, 2008, 09:59:46 PM
Power problems at a server farm?  That don't sound too good.

I should say not.

oh, it's not.  we had no idea at the time that it was dumping over and over because we couldn't get into to see what the hell was going on.  I suspected it had lost power at least once, since I noticed (during one of the short intervals it was up) that everything had a very recent process start time. once I saw the logs later that afternoon, I nearly passed out.  we could be in the adjoining gravesite from TOS right now.




Have the owners of this board considered setting up a "mirror" server with another server farm in case of a catastrophic failure with the current server farm?

It might not be logistically feasible, but might at least be worth looking into.
Logged
"The American people will never knowingly adopt socialism. But, under the name of 'liberalism,' they will adopt every fragment of the socialist program, until one day America will be a socialist nation, without knowing how it happened." --Norman Thomas, 1944

Offline Rebel

  • Stick a fork in us. We're done.
  • Administrator
  • Hero Member
  • *****
  • Posts: 16808
  • Reputation: +1259/-215
Re: explanation of the outages earlier today
« Reply #22 on: May 13, 2008, 11:01:52 PM »
Quote from: rich_t on May 13, 2008, 10:59:52 PM
Have the owners of this board considered setting up a "mirror" server with another server farm in case of a catastrophic failure with the current server farm?

It might not be logistically feasible, but might at least be worth looking into.

What's your real name? I don't remember getting a donation from you, is why I'm asking. This site isn't a corporation. We go by donations and the money we two Admins put into it.
« Last Edit: May 13, 2008, 11:04:10 PM by Rebel »
Logged
NAMBLA is a left-wing organization.

Quote
There's a reason why patriotism is considered a conservative value. Watch a Tea Party rally and you'll see people proudly raising the American flag and showing pride in U.S. heroes such as Thomas Jefferson. Watch an OWS rally and you'll see people burning the American flag while showing pride in communist heroes such as Che Guevera. --Bob, from some news site

Offline Rebel

  • Stick a fork in us. We're done.
  • Administrator
  • Hero Member
  • *****
  • Posts: 16808
  • Reputation: +1259/-215
Re: explanation of the outages earlier today
« Reply #23 on: May 13, 2008, 11:03:46 PM »
...I.e., a mirrored site would cost double what it does now...JUST for hosting. Then you have to add the costs for the server....which we own as it is ours. We built it. We're not "hosted" by some other site. This site is ours. This server is ours. We own everything but the connection.
Logged
NAMBLA is a left-wing organization.

Quote
There's a reason why patriotism is considered a conservative value. Watch a Tea Party rally and you'll see people proudly raising the American flag and showing pride in U.S. heroes such as Thomas Jefferson. Watch an OWS rally and you'll see people burning the American flag while showing pride in communist heroes such as Che Guevera. --Bob, from some news site

Offline rich_t

  • Hero Member
  • *****
  • Posts: 7942
  • Reputation: +386/-429
  • TANSTAAFL
Re: explanation of the outages earlier today
« Reply #24 on: May 13, 2008, 11:12:13 PM »
Quote from: Rebel on May 13, 2008, 11:01:52 PM
Quote from: rich_t on May 13, 2008, 10:59:52 PM
Have the owners of this board considered setting up a "mirror" server with another server farm in case of a catastrophic failure with the current server farm?

It might not be logistically feasible, but might at least be worth looking into.

What's your real name? I don't remember getting a donation from you, is why I'm asking. This site isn't a corporation. We go by donations and the money we two Admins put into it.

Hey...  I was just asking a question or two.  I had no idea how this board was funded.  I'll contact you via private message if you really want to know my real name.
Logged
"The American people will never knowingly adopt socialism. But, under the name of 'liberalism,' they will adopt every fragment of the socialist program, until one day America will be a socialist nation, without knowing how it happened." --Norman Thomas, 1944
Pages: [1] 2   Go Up
« previous next »