03/26/2015 7th Alarm / Site outage

tbendick

Administrator
Staff member
Joined
Dec 20, 2006
Messages
1,757
Just want to say sorry to all for today's outage. Due to the high demand from the 7th alarm, the site crashed.  When we rebooted the server to try and bring it back online it crashed each time.

What happened was, in a way a denial of service attack. However this was not done to cause harm just too much traffic for our server.

 

manhattan

Well-known member
Joined
Dec 7, 2007
Messages
1,883
Well, I guess Mr. Murphy had to practice his Law somewhere.  Thanks for getting the site up again so quickly and for all the work you do to keep things going every day.
 

Bulldog

Bulldog
Joined
Apr 16, 2008
Messages
1,937
norshor said:
This has happened more than once. What can be done about it?
Just asking.
There's an easy solution, just add more bandwidth to a higher powered server!  The only problem is that costs $$$
 

STAjo

Well-known member
Joined
Jun 15, 2012
Messages
3,483
Again, As Always ... We Cannot Thank You Enough Tom, For All Your Great Work Here !

You Sir, Rock !  8)
 

tbendick

Administrator
Staff member
Joined
Dec 20, 2006
Messages
1,757
norshor:  The problem is that to pay $$$$$ like bulldog said for an event that happens so rarely is very hard.  Think about as if you went out and purchased a moving van that you only needed once a year. 

There are some cloud solutions that can grow with demand but at last check the money that the site makes vs the cost was not there. I will check again and see if pricing has become any better.

We do pay a good amount for our dedicated server with specs that are very good.
Last outage was from a log that was corrupt. Not all issues can be fixed with more money some are just computer bugs and hiccups.
 

nfd2004

Well-known member
Joined
Jun 22, 2007
Messages
6,287
Just want to say THANKS Lt Bendick for what you do for us. Without you, NONE OF THIS would be possible. No online radio transmissions. No rundowns of the fires. No history of the FDNY saved. Many solid friendships made.

  At one point during the fire, I noticed Radio Reference had OVER 18,600 listeners. I have never seen that before. But people follow the FDNY from all over the world.

  I know a lot of guys, including myself, are very thankful for what you have given us. I've seen the numerous radios/scanners you have purchased to provide us with the online listening. I was so impressed that I took a picture of it. Other than maybe a dispatch center, I have never seen any thing like it. I had no idea of what it takes and the work and expense you put into it. You have a family of two young children to support with todays cost of living and you never asked any of us for one dime.

  Thank you for what you do for us. I greatly appreciate it.
 

Joes1051

Member
Joined
Apr 18, 2010
Messages
46
10-4  Tom. Like I said, I was just curious. And as NFD added, it obviously takes more than I thought! Keep up the great work.
 

tbendick

Administrator
Staff member
Joined
Dec 20, 2006
Messages
1,757
just to give an idea of volume, total people per audio feed during the 7th.

Scanner feed:6846
Manhattan Fire: 20573 
EMS Citywide:566
Fire DARS:  6242 
 

manhattan

Well-known member
Joined
Dec 7, 2007
Messages
1,883
How I wish we had a pool going on that.  Those are exactly the numbers I would have picked!

Seriously, Tom, congratulations and thanks for handling all this, day-to-day.
 

grumpy grizzly

Well-known member
Joined
Jun 27, 2007
Messages
3,158
The site is fantastic, keep up the good work. Things happen, after a few minutes I was able to back door it through the radio reference site. :) :) :)
 

jmag228

Active member
Joined
Jan 2, 2012
Messages
219
Thanks for all you do to keep us informed about something we all find so important, the greatest fire department in the world and the people we meet . Stay Safe Lieutenant .
 

tbendick

Administrator
Staff member
Joined
Dec 20, 2006
Messages
1,757
Did some research into updating the server. 

Cloud based server is still out of reach. It would cost 5 times more then what we are paying now.  Issue is that on a daily basis we run fine its the abnormal peak load that's the issue. The money is just not there.

Updated dedicated server.
Our host has some sales/deals going on currently. Our current config is almost 50% off if we were to purchase today.  With that if we were to use the money we spend each month on a new server at current prices we should be able to double our specs.

So going to talk with sales people and see if there is a way to "Upgrade" our server. Most times you  have to purchase a new server and transfer all the data. 

Will keep everyone informed.
 
Top