What do you do when your Company is constantly having Outages?

Its been a while for me, but there was a period of my life where I was working for a company that was in a constant state of outage.  They had a mix of services, and over the course of 2 years, I was flown across the country and around the world apologizing for the (lack) of services that my company provided.  While I love traveling and accumulating Air Miles, this was not my idea of a trip as you can imagine.

So what did I do right and wrong?  Well, I got very good at apologizing and groveling and it helped me write my policy on dealing with Irate Customers.  While not exactly ideal I definitely learned a lot from this experience and I definitely made a positive impact on my companies bottom line.  How?

Well, simply put, the Customers stayed! As you can well imagine, when a business and its service is being impacted by a 3rd party the natural inclination of anyone is to pull the service and move to another vendor.  When SLAs are constantly being missed and month on month, services are not improving this is even more likely.

Now – it’s easy to say that I “saved” the customers … but how?

Communication 

Its easier to say than to do – especially when you don’t have any news or even worse when you have bad news (you expected it to be fixed in 1 day and it’s going to take 1 week!).  As I mentioned earlier, I quickly became skilled at speaking to Customer’s face-to-face which happened with quite a few of our Tier1 customers.  I also became skilled at sending out mass emails, posts on message boards and forums and phone calls.  Setting a timeline for an update by any/all of these methods and then ensuring that I met that criteria were key.

Now communication is actually a two-way thing.  Speaking to the customers is great, but what if you don’t have anything to tell them?  Support and Helpdesk teams and Management are frequently on the “short end” of the stick without any updates from Engineering and Programming teams.  More often than not, these internal teams have no concept of the impact that the service interruption are causing to the customers.  It’s your job to persuade them that the customers MATTER and the reason you & they are in a job – working for your company is the money that your customers are paying!!  They will take their business away eventually if you don’t tell them what is going on.

OK, assuming that you’re talking to your customers and your other internal teams are talking to you … what’s next?  Well, you need to ensure that your company is actually doing something to fix the problem!!  The company that I mentioned with constant outages?  Well, they were all with different services … each time one thing was fixed another in a different product was impacted.

From my point of view, it was 2 years of hell, but no one single customer was impacted for that total amount of time.  How do you fix this though, because it is extremely draining on your staff regardless … well, Quality Control is useful.  Make sure that any new product launches are properly tested and tested and tested again before being released into a live environment.  Try to get your staff to break it if possible while it’s in the testing phase.  Make sure your documentation, release notes, and training material are complete and accurate.

Ensure that Senior Management gets involved at the appropriate intervals based on your Escalation Matrix so that they are aware of the impact to the Customers … DO NOT be afraid of escalating.  If you are ON CALL 24/7 so are they!  The money will be released when the phone rings at 2am!