Author Topic: 03/26/2015 7th Alarm / Site outage  (Read 3738 times)

Offline tbendick

  • Webmaster
  • Administrator
  • Hero Member
  • *****
  • Posts: 1725
  • Gender: Male
    • Nycfire.net
03/26/2015 7th Alarm / Site outage
« on: March 26, 2015, 08:10:43 PM »
Just want to say sorry to all for today's outage. Due to the high demand from the 7th alarm, the site crashed.  When we rebooted the server to try and bring it back online it crashed each time.

What happened was, in a way a denial of service attack. However this was not done to cause harm just too much traffic for our server.


Nycfire.net

03/26/2015 7th Alarm / Site outage
« on: March 26, 2015, 08:10:43 PM »

Offline manhattan

  • Hero Member
  • *****
  • Posts: 1126
Re: 03/26/2015 7th Alarm / Site outage
« Reply #1 on: March 26, 2015, 08:15:21 PM »
Well, I guess Mr. Murphy had to practice his Law somewhere.  Thanks for getting the site up again so quickly and for all the work you do to keep things going every day.

Offline norshor

  • Newbie
  • *
  • Posts: 46
Re: 03/26/2015 7th Alarm / Site outage
« Reply #2 on: March 26, 2015, 08:51:55 PM »
This has happened more than once. What can be done about it?
Just asking.

Offline Bulldog

  • Hero Member
  • *****
  • Posts: 1535
Re: 03/26/2015 7th Alarm / Site outage
« Reply #3 on: March 26, 2015, 09:08:02 PM »
This has happened more than once. What can be done about it?
Just asking.
There's an easy solution, just add more bandwidth to a higher powered server!  The only problem is that costs $$$

Offline STAjo

  • Hero Member
  • *****
  • Posts: 2454
  • Gender: Male
  • Former NY State EMT, NYC Security, Buff
Re: 03/26/2015 7th Alarm / Site outage
« Reply #4 on: March 26, 2015, 09:10:13 PM »
Again, As Always ... We Cannot Thank You Enough Tom, For All Your Great Work Here !

You Sir, Rock !   8)

Offline tbendick

  • Webmaster
  • Administrator
  • Hero Member
  • *****
  • Posts: 1725
  • Gender: Male
    • Nycfire.net
Re: 03/26/2015 7th Alarm / Site outage
« Reply #5 on: March 26, 2015, 10:01:45 PM »
norshor:  The problem is that to pay $$$$$ like bulldog said for an event that happens so rarely is very hard.  Think about as if you went out and purchased a moving van that you only needed once a year. 

There are some cloud solutions that can grow with demand but at last check the money that the site makes vs the cost was not there. I will check again and see if pricing has become any better.

We do pay a good amount for our dedicated server with specs that are very good.
Last outage was from a log that was corrupt. Not all issues can be fixed with more money some are just computer bugs and hiccups.

Online nfd2004

  • Hero Member
  • *****
  • Posts: 4264
  • Gender: Male
Re: 03/26/2015 7th Alarm / Site outage
« Reply #6 on: March 26, 2015, 10:28:10 PM »
 Just want to say THANKS Lt Bendick for what you do for us. Without you, NONE OF THIS would be possible. No online radio transmissions. No rundowns of the fires. No history of the FDNY saved. Many solid friendships made.

  At one point during the fire, I noticed Radio Reference had OVER 18,600 listeners. I have never seen that before. But people follow the FDNY from all over the world.

  I know a lot of guys, including myself, are very thankful for what you have given us. I've seen the numerous radios/scanners you have purchased to provide us with the online listening. I was so impressed that I took a picture of it. Other than maybe a dispatch center, I have never seen any thing like it. I had no idea of what it takes and the work and expense you put into it. You have a family of two young children to support with todays cost of living and you never asked any of us for one dime.

  Thank you for what you do for us. I greatly appreciate it.

Offline norshor

  • Newbie
  • *
  • Posts: 46
Re: 03/26/2015 7th Alarm / Site outage
« Reply #7 on: March 26, 2015, 10:42:31 PM »
10-4  Tom. Like I said, I was just curious. And as NFD added, it obviously takes more than I thought! Keep up the great work.

Offline tbendick

  • Webmaster
  • Administrator
  • Hero Member
  • *****
  • Posts: 1725
  • Gender: Male
    • Nycfire.net
Re: 03/26/2015 7th Alarm / Site outage
« Reply #8 on: March 26, 2015, 10:55:51 PM »
just to give an idea of volume, total people per audio feed during the 7th.

Scanner feed:6846
Manhattan Fire: 20573   
EMS Citywide:566
Fire DARS:  6242   

Offline manhattan

  • Hero Member
  • *****
  • Posts: 1126
Re: 03/26/2015 7th Alarm / Site outage
« Reply #9 on: March 26, 2015, 11:06:05 PM »
How I wish we had a pool going on that.  Those are exactly the numbers I would have picked!

Seriously, Tom, congratulations and thanks for handling all this, day-to-day.

Offline grumpy grizzly

  • Hero Member
  • *****
  • Posts: 2589
  • Gender: Male
Re: 03/26/2015 7th Alarm / Site outage
« Reply #10 on: March 27, 2015, 08:49:52 AM »
The site is fantastic, keep up the good work. Things happen, after a few minutes I was able to back door it through the radio reference site. :) :) :)
FAC 20 TASS 68-69 SVN. Hue/PhuBai , Boston Spark from 71-79, Chicago 79-15, Bloomington/Normal 2015- present

Offline NYBravest82

  • Sr. Member
  • ****
  • Posts: 407
    • LCPhotowerx
Re: 03/26/2015 7th Alarm / Site outage
« Reply #11 on: March 27, 2015, 09:33:36 PM »
Sometimes things go wrong, it happens. No worries!! Keep up the great job!

Offline jmag228

  • Jr. Member
  • **
  • Posts: 97
Re: 03/26/2015 7th Alarm / Site outage
« Reply #12 on: March 27, 2015, 10:24:46 PM »
Thanks for all you do to keep us informed about something we all find so important, the greatest fire department in the world and the people we meet . Stay Safe Lieutenant .

Offline tbendick

  • Webmaster
  • Administrator
  • Hero Member
  • *****
  • Posts: 1725
  • Gender: Male
    • Nycfire.net
Re: 03/26/2015 7th Alarm / Site outage
« Reply #13 on: April 02, 2015, 09:55:25 AM »
Did some research into updating the server. 

Cloud based server is still out of reach. It would cost 5 times more then what we are paying now.  Issue is that on a daily basis we run fine its the abnormal peak load that's the issue. The money is just not there.

Updated dedicated server.
Our host has some sales/deals going on currently. Our current config is almost 50% off if we were to purchase today.  With that if we were to use the money we spend each month on a new server at current prices we should be able to double our specs.

So going to talk with sales people and see if there is a way to "Upgrade" our server. Most times you  have to purchase a new server and transfer all the data. 

Will keep everyone informed.

 

anything