When Things Go Wrong...

This year started with a bang and then a bust. On Thursday, January 8 at 9:00am at the OM Office we had a scheduled maintenance. We needed to relocate our firewall device, (like a DSL modem but 100x more powerful) and were expecting 10 minutes of downtime… Until the unexpected happened. 

The firewall would not start. It had become what we call a brick in the IT world—completely useless. By 10:30am we were in a war room to evaluate the situation. Myself and my colleagues were racking our brains to see what IT services would be effected. There were a few, but thankfully most of our high profile IT services had moved to a new data centre a few days before Christmas. I presented 3 recovery options to the team, and we decided to try using an older firewall. We pulled it out of the cupboard, blew the dust off of it, and I began restoring it to previous settings. Thankfully, God had put it on our hearts to backup the configuration prior to the move. What a relief! Even so, it was like asking an old-model Mini Cooper to perform a job that required a semi-truck (lorry). 

By 2:30pm we were up again. The old firewall wasn't as powerful as the broken firewall, but by God’s grace we limped along. Because of the limited capabilities of the replacement firewall, we had to cut off all but essential services. Five hundred workers weren't able to get into their email during the outage. Our remote connections to 11 different locations were down, so some people weren't able to do their work. Most of our workers in Carlisle also were sent home.

We hoped we could get a replacement firewall by Friday, but by the time all the paperwork was completed, it wasn't going to arrive until Monday. We were desperate to get a replacement firewall even sooner because the old firewall felt like it was going to burst into flames from working beyond its means. After scouring the internet for several hours, my colleague found a replacement model we could get next day! This meant that our simple maintenance task was completed by 2:30pm on Friday and hundreds of people had been effected. 

Normally my work is done behind the scenes and people don't know when things go wrong. This time things went spectacularly wrong and missionaries all around the world were effected adversely. It reinforces how important IT is in the OM world.