Anyone in the United States or much of the world for that matter, is very aware of that giant of sporting spectaculars, the fall classic known as the World Series. The first game was an absolute thriller, starting off with a leadoff first pitch inside-the-park homerun (last accomplished in a World Series in 1929 and first leadoff since 1903), continuing for 14 innings over 5 hours and 9 minutes, tying the record for the most innings in World Series history. Pretty heady stuff, but that wasn’t even the big story.
“We are experiencing technical difficulties”
What stole the show was the inexplicable 4 to 5 minute blackout that literally shutdown the game and made millions of viewers blast four letter tirades at their televisions. The problem - the Fox broadcast lost power, which would never have been noticed had the first generator not failed, but the backup generator also failed. What are the chances of that?! Both backup generators ‘failed’. I’m sure someone at Fox will become the ceremonial fall guy for that one.
With Fox paying around $500 million for the broadcast rights to the World Series this is a no laughing matter and a very expensive “technical glitch”. It is very obvious what the immediate impact is in a situation like this – millions of pissed off viewers, panic stricken Fox technicians, irked broadcasters (Joe Buck was NOT happy) and mortified Fox executives. But what about the aftermath? Is there any lingering negative effect on the Fox brand, trust, and goodwill? Only time will tell. The point is, it was the worst possible thing that could happen at the worst possible time on the world’s largest stage.
But they did everything right, didn’t they?
Technically speaking, Fox, if they did as they say, had the right setup – a backup generator with a second backup generator in the event of a power failure and unlikely failure of the first generator. This is what we call N+1 in technology speak. It means for critical systems, always have one more than you need to failover to if there is an issue. This certainly met the criteria for a ‘critical system’ and they had N+1, but it still wasn’t enough. You can bet Fox is opening their wallet today to spend whatever it takes to avoid issue again, even if it means having four generators on standby with a person physically watching each one throughout the broadcast. By the way, having more redundancy than one more backup than you need is typically called 2N+1, where you have twice the redundancy, or in this case 3-4 generators. We may see just that outside the stadium on Wednesday night.
It’s not if it happens, but when
The situation was bad enough, but had it happened during one of the 162 regular season games back in April it would have registered as a blip on the radar. Happening during the first game of the World Series blew up the radar and made every front page – it become the story. If you are a business owner or operator, you should take notice. I’m sure Fox thought this could ‘never happen’ but now we know otherwise. What if this happened to your business at the absolute most critical time? What would it cost you? What would you be willing to spend to prevent it from happening again?
Sadly, most small to mid-sized companies do not have redundant systems simply due to the cost and the minimal risk of a critical failure actually happening. The cost vs. risk just won’t justify it… unless it happens to you. However, this very public SNAFU happening in the most unlikely of situations to one of the world’s largest companies (number 97 on the Fortune 500) shows it can happen to anyone at any time.
Is your business aware of the risks?
Forget about backup generators. Do you even know where the weak points are in your business? Do you know every single-point-of-failure that could bring your company down? If you don’t, you should get busy and find out. Not sure where to begin? Contact us and we can help you start the process. After all, being in the dark is the worst place to be, especially when watching the World Series.