What happened there? (Downtime) - Page 7 - Alfa Romeo Forum
You are currently unregistered, register for more features.    
Announcements & Comments A lounge for staff announcements.
If we have ideas, requests or something to tell you, it will go in here.

Reply
 
Thread Tools Search this Thread
PD.
Status: -
Unregistered
 
Join Date: Jan 2009
Location: Czech Republic
County: -
Posts: 1,840
Images: 18
Re: What happened there? (Downtime)

Bit long in the tooth this one and I had to edit a few bits out for certain reasons, but could this be AO's problem?

Quote:
The server keeps crashing



Background

We were going live in a couple of weeks, and now put the production server into the server room. This was not really a "server room" in the true sense, more an IT office. With servers and other equipment. No racks and stuff in those days, just towers next to one another.

The server was a 386 OS/2 machine running 3.04.



The problem

The basic network (Novell base) had been working reliably for some time, but once we installed the server, the Novel server was failing regularly every second night. Basically the Novel server would shut down around 9:30 pm, and come up again 20-30 minutes later. Unfortunately the interruption was affecting the backup system, and we were not getting a backup on these nights.

Really this was critical. Although we had to go live, we could not afford to loose the entire network and backups, and more importantly we just HAD to know what was going on.



Analysis of the issue

clearly the problem happened 2 days after we installed and regularly. So we looked at what could cause it. The only connection we could see was the network which was an ARC net star configuration, and the power. The UPS on the Novel server, was a big one and had one spare power outlet that we used. We checked the batteries etc. but no problem. We tried everything, but could not resolve the issue.

We made the decision to remove the server, and took it out of the server room, so we could check it for problems. Sure enough, that night the system worked fine, no system crash. It was now clear that the server was causing the Novel server to shut down, but how and why?

We had now worked out that the server was failing Monday, Wednesday and Friday nights, and it was now Thursday, so we decided to come back on Friday and just monitor everything to see if we could find the failure. We knew it happened between 9 and 10, so at worst case it should take one hour to see something.



Getting Ready

We setup early. The server is plugged in and back on line. We set up some monitoring tools. We had a client running a repeating report to screen, to see activity. The Novel guys had monitoring software on everything. The server console was open, so we could see if the server stopped.


Testing Time

It was getting to be a boring uneventful time, but we were pretty sure the system would soon fail, and we were all a bit on edge it was about 9:30, so it should be happening any moment now.

At this point, the door to the office opens. A cleaner sticks her head in, and asks if she can clean the office. We discuss this, and Since its Friday, we decide to let her do her job, otherwise the office will be a mess on Monday.

So the cleaner comes in with her trolley bucket etc. and a vacuum cleaner. She casually walks over to the Novel Server, unplugs the server from the UPS, plugs her vacuum into the UPS and starts vacuuming.

She did look at us all extremely oddly the way we were all laughing, and couldn't understand why it was so important to know if she vacuumed this room every Monday, Wednesday and Friday at 9:30.
PD. is offline  
Sponsored Links
Advertisement
 
Status: No more guiletta... Auctioned off
Founder of AO
 
alfaowner's Avatar
 
Join Date: Apr 2001
Location: United Kingdom
County: Essex
Posts: 8,044
Images: 83

Member car:

Maserati GTS

Re: What happened there? (Downtime)

OK, patience required, we will try for Sunday but it may be Wednesday before new memory arrives, we are installing another 4 GIG of server memory and a new RAID hard drive, therefore expect some down time in the next few days to allow the hardware to be fitted. Fingers crossed this will indeed help us out, those non club members that want to sign up, now would be the time!

Regards,

Ian
alfaowner is offline  
Status: Meh.
AO Platinum Member
 
NineOneSix's Avatar
 
Join Date: Dec 2006
Location: United Kingdom
County: Hertfordshire
Posts: 30,067
Images: 9
Garage

Member car:

GTV V6 3.0

Re: What happened there? (Downtime)

wow, the database out grown the last raid already
NineOneSix is offline  
Status: No more guiletta... Auctioned off
Founder of AO
 
alfaowner's Avatar
 
Join Date: Apr 2001
Location: United Kingdom
County: Essex
Posts: 8,044
Images: 83

Member car:

Maserati GTS

Re: What happened there? (Downtime)

Quote:
Originally Posted by explosive alfa View Post
wow, the database out grown the last raid already
No I think it has a problem, we are going to swap it anyway to try an eliminate the issues.
alfaowner is offline  
Status: Meh.
AO Platinum Member
 
NineOneSix's Avatar
 
Join Date: Dec 2006
Location: United Kingdom
County: Hertfordshire
Posts: 30,067
Images: 9
Garage

Member car:

GTV V6 3.0

Re: What happened there? (Downtime)

ahhh right, perhaps the disks were on the way out.

whats the brand of the raid server and main server if its ok to ask
NineOneSix is offline  
Status: No more guiletta... Auctioned off
Founder of AO
 
alfaowner's Avatar
 
Join Date: Apr 2001
Location: United Kingdom
County: Essex
Posts: 8,044
Images: 83

Member car:

Maserati GTS

Re: What happened there? (Downtime)

The problem is self induced, our software isnt the quickest because it has an awful lot of queiries, some that we have created by making the forum more like a web site than a forum, you could say we are a victim of our own success, but then no one said this was ever going to be easy!
alfaowner is offline  
Status: Broken again...
Club Member
Membro Premio
 
GhostyDog's Avatar
 
Club Member Number: 17
Join Date: Sep 2005
Location: Right Side O The Pennines
County: West Yorkshire
Posts: 25,663
Images: 6
Garage
Re: What happened there? (Downtime)

Quote:
Originally Posted by alfaowner View Post
The problem is self induced, our software isnt the quickest because it has an awful lot of queiries, some that we have created by making the forum more like a web site than a forum, you could say we are a victim of our own success, but then no one said this was ever going to be easy!
Think yourself lucky your not developing on Sharepoint, more to the point though good to know it's going to resolve the lack of search functionality, it's arse only having the last hours threads in the new posts search.
GhostyDog is offline  
Status: BORED
 
David P's Avatar
 
Join Date: Aug 2003
Location: Ireland
County: Cork
Posts: 74,296
Images: 19
Garage
Re: What happened there? (Downtime)

More whitescreens & crashs
David P is offline  
Status: i don't use fuelly
AO Platinum Member
 
nigelread's Avatar
 
Join Date: Sep 2006
Location: United Kingdom
County: Essex
Posts: 22,304
Images: 5
Re: What happened there? (Downtime)

Same here
nigelread is offline  
Status: Zoe FTW
AO Rosso Alfa Member
 
baby278278's Avatar
 
Join Date: Jul 2007
Location: United Kingdom
County: Essex
Posts: 46,602
Images: 49
Re: What happened there? (Downtime)

And again
baby278278 is offline  
Status: off down the shed.
AO Gold Member
 
alfacool's Avatar
 
Join Date: Mar 2007
Location: County Durham
County: Durham
Posts: 6,070

Member car:

159 TI 2.2

Re: What happened there? (Downtime)

Everything is so sloooooooooooooooooooow!
alfacool is offline  
Status: Meh.
AO Platinum Member
 
NineOneSix's Avatar
 
Join Date: Dec 2006
Location: United Kingdom
County: Hertfordshire
Posts: 30,067
Images: 9
Garage

Member car:

GTV V6 3.0

Re: What happened there? (Downtime)

most of tonight have been offline
NineOneSix is offline  
Status: Spider's ready to go!
Global Mod Team
 
classicgal93's Avatar
 
Join Date: Sep 2006
Location: Canada
County: British Columbia
Posts: 59,016
Images: 2
Garage
Re: What happened there? (Downtime)

Site kicked me off but I'm back on
classicgal93 is offline  
Status: -
AO Platinum Member
 
GTA Jim's Avatar
 
Join Date: Apr 2007
Posts: 14,575
Images: 3

Member car:

147 GTA

Re: What happened there? (Downtime)

It amazing how many of the long standing members with a shed load of posts are still free members! Maybe if they put their hand in their pocket it would help pay for a better server and put an end to the problem.
GTA Jim is offline  
Status: The BSD says konnichiwa
Global Mod Team
 
BigFoot's Avatar
 
Join Date: Jul 2003
Location: Italy
County: Milano
Posts: 113,594
Images: 14
Garage
Re: What happened there? (Downtime)

There you go Jim, took the words right off my mouth
BigFoot is offline  
Status: -
AO Platinum Member
 
GTA Jim's Avatar
 
Join Date: Apr 2007
Posts: 14,575
Images: 3

Member car:

147 GTA

Re: What happened there? (Downtime)

Quote:
Originally Posted by BigFoot View Post
There you go Jim, took the words right off my mouth
It just surprises me that these people who are on here day in day out and still aren't paid members! I dont understand it!
GTA Jim is offline  
Smallville
Status: - Update
Guest
 
Posts: n/a
Re: What happened there? (Downtime)

Been reading this with interest, is there nothing you can do to make them join BF??

And how are you GTAJim?
 
Status: -
AO Platinum Member
 
GTA Jim's Avatar
 
Join Date: Apr 2007
Posts: 14,575
Images: 3

Member car:

147 GTA

Re: What happened there? (Downtime)

Quote:
Originally Posted by Cunn 147 View Post
Been reading this with interest, is there nothing you can do to make them join BF??

And how are you GTAJim?
Something needs to be done about the situation, us fully paid members are suffering and newbies arent going to want to pay to join a slow site that keeps crashing!

Glad I'm home mate! Been up Derbyshire in heavy snow its been sh!te!!! If I had 1 for every pikey idiot in a 4x4 thinking they were invincible I would buy AO a faster server!
GTA Jim is offline  
Status: The BSD says konnichiwa
Global Mod Team
 
BigFoot's Avatar
 
Join Date: Jul 2003
Location: Italy
County: Milano
Posts: 113,594
Images: 14
Garage
Re: What happened there? (Downtime)

I respect people's decision not to pay as many internet forums don't charge, but if we want our cake and eat it, a Medio membership works out at just over £1 a month. Not a lot of cash huh?!
BigFoot is offline  
Smallville
Status: - Update
Guest
 
Posts: n/a
Re: What happened there? (Downtime)

Quote:
Originally Posted by BigFoot View Post
I respect people's decision not to pay as many internet forums don't charge, but if we want our cake and eat it, a Medio membership works out at just over 1 a month. Not a lot of cash huh?!
That's my point there BF!!!

Its like when I 1st used the site I was on once a week hence why I didnt pay, then I discoverd the friendly side and made my 1st post and been hooked since, that's when I paid top doller membership also

It is annoying that the site does keep crashing but I know you can't make people pay for somethiing that is free everywhere else.
 
Status: -
AO Platinum Member
 
GTA Jim's Avatar
 
Join Date: Apr 2007
Posts: 14,575
Images: 3

Member car:

147 GTA

Re: What happened there? (Downtime)

Quote:
Originally Posted by BigFoot View Post
I respect people's decision not to pay as many internet forums don't charge, but if we want our cake and eat it, a Medio membership works out at just over 1 a month. Not a lot of cash huh?!
Oh yep I respect that too as not everyone can afford it but as you say its just over a squid a month! Do people just expect this site to just be here and run itself?
GTA Jim is offline  
Smallville
Status: - Update
Guest
 
Posts: n/a
Re: What happened there? (Downtime)

Quote:
Originally Posted by GTA Jim View Post
Something needs to be done about the situation, us fully paid members are suffering and newbies arent going to want to pay to join a slow site that keeps crashing!

Glad I'm home mate! Been up Derbyshire in heavy snow its been sh!te!!! If I had 1 for every pikey idiot in a 4x4 thinking they were invincible I would buy AO a faster server!
Funny but not funny

I went on the Surrey run today and think I have a serious problem with the old girl
 
Smallville
Status: - Update
Guest
 
Posts: n/a
Re: What happened there? (Downtime)

Quote:
Originally Posted by GTA Jim View Post
Oh yep I respect that too as not everyone can afford it but as you say its just over a squid a month! Do people just expect this site to just be here and run itself?
Everyone wants something for nothing these days James sadly
 
Status: -
AO Rosso Alfa Member
 
kurzon's Avatar
 
Join Date: Jul 2004
Location: London, UK
Posts: 45,472
Images: 1

Member car:

156 2.0 JTS

Re: What happened there? (Downtime)

Guys, let's be careful how we speak here. I believe it's been pointed out that perhaps a dodgy raid disk or something else is to blame, which is going to be swapped. I don't know the in's and out's but it has been said many times that the site is growing in membership numbers which might be contributing to the problems. Simply throwing cash at the forum isn't the solution. Better to work out the long term objectives of the forum and work out revenue generating intiatives, be it club membership or other to support it and it's growth. Only the AO Admin/Owners etc know the real score. I am sure they have strong ideas were this place should go.

Do not alienate people buy saying most who use it should pay up. We would lose alot of people if this stance was taken and remember alot just join up for their tech needs and don't return afterwards.

Then there are people out there who would say that there is enough money to support the membership level here, but perhaps they do not appreciate the look and feel/development that has gone in to making it "snazzy"/bespoke as it is. Besides, a profit here or there is not entirely unwarranted, is it?

So let's keep being welcoming and not shoving the need to pay down peoples throats. It's worked so far and many like ourselves do dig into our pockets when "we" think the time is right. The odd little friendly nudge is fine but some of the words used in this thread thus far seem a littly too pushy. IMHO.



[There was a time that drivel was blamed for all the slowness... shall we have those return, any current members of the same mindset and elect for certain theads for closure ]

"I believe in coincidences. Coincidences happen every day. But I don't
trust coincidences." - Garak, DS9
So true
The BORED Collective
WE are SpartaBORED! I was DarthBORED! I am FearBORED! :elaugh: I was PredatorBORED! I was PhantomBORED!...of the AO-Theatre I have been iBORED! TerminatorBORED! He'll be back!
kurzon is offline  
Smallville
Status: - Update
Guest
 
Posts: n/a
Re: What happened there? (Downtime)

Im sorry, please forgive me.
 
Reply

Go Back   Alfa Romeo Forum > The Alfaowner Experience > Announcements & Comments

Tags
downtime , happened

Thread Tools Search this Thread
Show Printable Version Show Printable Version
Email this Page Email this Page
Search this Thread:

Advanced Search



Posting Rules  
You may not post new threads
You may post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off

 
For the best viewing experience please update your browser to Google Chrome