Recovering from a Disaster

I love to play a simple, yet addictive game called Bejeweled on my Palm Treo.  Recently, Pop Cap Games – maker of Bejeweled – released it on Facebook, too.  It’s free to play and hey, they even award prizes based on collective team scores earned every week.  So not only would I normally play because I love the game, I play to perhaps win a prize, too.  What’s important to note, though, is that I don’t pay anything to play.  I don’t pay to access Facebook (at least not yet) and I don’t pay to play Bejeweled.

Yesterday, there was a fire at the Seattle-based hosting facility which Pop Cap uses to host their games.  It apparently brought several organizations to their collective knees – hopefully no one was hurt in the blaze.  Bejeweled, though, was down.  Pop Cap owed no explanation to anyone – there were no paid users, no SLAs, no force majeure.  But their response to the user community was exactly what I would expect from large corporate vendors, yet never receive.

First, Pop Cap posted a notice on the Bejeweled page stating: 1) an apology for the outage! (remember, they didn’t cause it); 2) what happened to cause the outage; 3) reassurance that each users’ data was safely backed up and would be restored when the servers were back online; and 4) an estimated time when the game would be available again.  Second, today the game is back online.  You still see a notice about what happened and why things will look a little funny with your friends’ scores for a little while.  And they apologized again.

Generally speaking, discussion of force majeure in the commercial contracting world (the suspension of contractual obligations as a result of an unforeseeable event) is usually fairly quick and pretty painless once you know the basics.  Both sides usually only spend a maximum of 30 seconds discussing this section of the contract and thus if anything actually does happen that would require invocation of the provision, many are flummoxed about how to handle it.  In fact, some even forget to invoke – they simply look to Service Level Agreements to see if there’s anything that can be done to recover from the other side.

Part of this could be easily solved if enterprise software vendors would follow Pop Cap’s lead in terms of how they handled downtime.  But again, generally speaking, they don’t.  They want the customer to have to report a problem; the customer to have to call in for status updates on fixes; and almost never explain why there was a problem in the first place.  Oh, and I’ve never heard a vendor apologize, either – which, as any psychologist will tell you, is really easy and goes a long way to assuaging feelings.

So, take it from Pop Cap – who with millions of probable users (I don’t have an exact count, of course) and none of them paying a cent, managed to just make me want to go buy something from them simply because of how they handled a problem with a free game and without having any contract tell them how to behave, either.

Advertisement

Service Levels and Remedies

We’ve been discussing Service Levels for the last few weeks.  We covered construction, setup, drafting and even some recommendations on how the Service Levels might look.  But what happens in the event that the Service Levels aren’t met (they’re “blown”)?

We should start with a review of why we sought Service Levels in the first place – which is usually because you really just want what you’ve been promised in terms of service or performance.  In this case, remedies for blown Service Levels should be two-fold:  1) to get service restored as quickly as possible, and 2) to recompense for any damages caused as a result of the blown service level.  So you’ve probably even drafted the Service Level metrics so as to put a lot of importance on fewer outages or less downtime – with significant financial penalties.  But you don’t really want the money, you want great service.  The net result is that in the event of a blown Service Level, you probably won’t enforce the financial penalty… it’s more bark than bite.  But you are a stickler for responsiveness and attentiveness, and you will enforce the remedies if you feel that not only have the Service Levels been blown, but that you’re getting ignored, too.  This is justifiable.

But maybe you’ve got a little more jaded view and you already know that you’re not going to get exactly what was promised.  You drafted the Service Levels with really tight controls – trying to count every little thing (without regard to what’s really important) in the particular deal.  You instead feel that software or services are exact sciences and that if the vendor is silly enough to make a promise, you should get everything you’re “entitled” to.  The result is that at the first hint of a blown Service Level, you’re on the phone with the vendor asking for service credits (never mind restoration of service).

Personally, I hope you’re in the first group of folks and not the second.  Software and Services, by their nature, are going to have hiccups.  This is just how it goes.  Even older, established products can have difficulties (environments are constantly changing).  So instead of trying to beat the vendor with your contract, you use the contract strategically to make sure that you’re getting what is really important to the deal … and not some sort of small financial benefit for each blown level.

However, there are times where it is absolutely justifiable to use the contract as a sword and a shield.  One of these cases is what’s known as “death by ducks” – the situation where you have many small Service Level issues.  None of them, on their own, would be worth enforcing as a true blown event.  But together, you get pecked to death by the small things because they cumulatively add up to a significant performance issue.  Here, you should have anticipated this issue and have a small extra section in your Service Level language detailing the remedies available in the event of x number of small issues that add up to a certain Severity Level.  Heck, you can even define it as a certain number of Sev3’s = a Sev2.  And a certain number of Sev2’s = a Sev1.  How would you handle it?

Zen and Art of Service Levels (with apologies to Robert Pirsig and Eugen Herrigel)

“The aim of Zen practice is to discover [this] Buddha-nature within each person, through meditation and mindfulness of daily experiences. Zen practitioners believe that this provides new perspectives and insights on existence, which ultimately lead to enlightenment.” —Wikipedia

As silly as it sounds, the way to master service levels is to draft them over and over.  Yeah, this is the same way to get better at anything, contracts especially.  But service levels are a little special.  I think it’s because they’re going the way of the Dodo.  As few people ask for them, even fewer know to even think about them.  It’s the same cycle that increases the quality of service levels – just in reverse.  Pirsig’s book was focused on trying to define “quality” and in the end, he settled upon a mix of rationality and romanticism.

I said before that service levels have to be SMART: Specific, Measurable, Attainable, Relevant, Time-Bound.  We’ll blend the rationality and romanticism as we go.

Specific – Service levels start with an understanding of the exact quantities of some metric.  This could really be anything, but tempered with the next quality, you have to be able to count it.  Typically, we start with things that are time-related: uptimes and downtimes, repair times and fix times.  Rationality wins here almost every day (the truly romantic notion is that service levels aren’t needed at all because everything’s going to work out as planned) – these things are really easy to measure… and frankly, ease of measurement is necessary because the folks who will be monitoring the service levels aren’t really interested in tracking them.  But why not be a little romantic, too?  Pick something unique about the particular situation.  Maybe you’re licensing software that processes transactions (so you’d count the transactions processed), or maybe you’ve hired an outsourcer to answer your support calls and service levels could be managed based on the number of successful versus unsuccessful customer service resolutions.

Measurable – This might seem obvious, but you’ve got to be able to measure what it is you’re going to base any metrics upon.  Just counting isn’t necessarily enough.  Rather, you might need to be able to track start/stop times/days (and then do the math to calculate the difference).  If the calculation is manual, you also need people who can keep track.  This, perhaps, is the most problematic part of any service level management… as the folks who want the benefits of the service level (usually managers) are not the people watching the clock or experiencing the outages first-hand (the staff).  So unless the staff has some sort of reason to monitor the metric accordingly, none of this is going to matter.

Attainable – I promised you before that the Myth of the Nines would come back into your life, and here it is.  The simple truth is that Five-9 availability is a pipe dream.  5.26 minutes of downtime a year.  Just think about how long your average PC takes to power-cycle.  Servers are typically a little longer.  Even with redundant systems, backups, high-availability resources and every other techincal resource… it’s just not reasonable.  Notice I didn’t say that it was impossible.  It’s 100% possible.  You can have 100% availability.  The issue is cost.  No one ever wants to PAY for that kind of availability.  Not even your most demanding customers.  Wanna’ test this theory?  Price it out from your vendor(s) (as it’ll take more than one to keep even a single service up 24/7/365) and ask your most demanding customer if they’ll pay for the ENTIRE service themselves (since that’s the real cost to get it).  Let me know if they’re willing to do it, because I have a bridge or two to sell them.  Seriously, I’m not trying to be facetious.  I’m a pretty demanding customer myself, but even I know and understand financial limits.

Relevant – Tied to measurable and specific is that each of your service level metrics be relevant to whatever service you’re receiving/providing.  So if you’ve chosen to measure successful versus unsuccessful customer service resolutions, but it’s not tied to the behavior of the service provider, that’s not a relevant metric.  The provider doesn’t have any control over what is being measured, even with perfect behavior.  So where is their incentive to work towards meeting the metrics (or agreeing to them in the first place)?

Time-Bound – Service levels are limited to time.  At first, this sounds quite limiting, but we’re not talking about time in terms of the length of the relationship (service levels should extend for the entire length of the relationship).  Rather, the time we’re talking about here is the time frame in which each metric will be measured.  So, perhaps you’re watching uptime on a daily basis… or the number of widgets produced in a week… or the number of successful service calls completed in a year… or the average length of time it takes to fix a problem of a given severity level over the span of a quarter.

OK, so now that you’ve considered all five requirements, you should have one or more appropriate service levels.  If you still need some ideas, check back with me for the next installment.  Meanwhile, if you have some ideas for inclusion in the next installment, send them along!

Service Level Basics

I eat out a lot – exempting breakfast (I don’t eat it), I would say that I’m at a restaurant for about 10 of every 14 available meals.  Never mind what this does to my budget, let’s focus on the food.  Now, I’m a pretty simple eater – in fact, I love things plain.  When I go to McDonalds or Burger King, I get the burgers with nothing on them – just meat and bread.  Add in some fries and a drink, and I’m a happy man.

So in most situations, I’ve got three components to my meal: an entree, a side and a drink.  Statistically speaking, there are eight possible combinations of quality (assume that each item can only be good or bad):

bad burger, bad fries, bad drink
bad burger, bad fries, good drink
bad burger, good fries, good drink
bad burger, good fries, bad drink
good burger, bad fries, bad drink
good burger, bad fries, good drink
good burger, good fries, bad drink
good burger, good fries, good drink

Thus, for any given purchase of just these three components to my meal, I have a 1 in 8 chance of getting all three “good” items.  That’s a .12512.5% [thanks to John O. for correcting my math] chance – WAY less than 1% that I’m going to enjoy all three items.  On the other hand, there’s also only a 1 in 8 chance of having all three be “bad”.  There’s a 3 in 8 chance that one will be “good” and a 3 in 8 chance that two will be “good”.  So what do I do?

I set my expectations accordingly and know that there’s a 50% chance that I’ll enjoy at least two of the items (the 3 in 8 that two will be good plus the 1 in 8 that all three will be good).  Yes, I know that there’s a 50% chance for the reverse – but remember also that there are some other variables that we need to account for.  In all name-brand fast-food joints, there are quality standards set by the franchisor.  McDonald, Burger King, Chick-Fil-A, Wendy’s, Arby’s, KFC, Taco Bell, etc… they all have: food that is pre-packaged and sent to the stores (reducing the likelihood of differentiation by store); cooking standards (look behind the counter some time and see if you can find the poster showing the correct “doneness levels”); even standard equipment (fryers, etc) to reduce variations.

So in actuality, there’s a better than 50% chance that my food will be “good” (meeting the corporate standard) because of these outside variables.

OK, so what does this all have to do with software, services and service levels?

Well, it’s 100% the same.  Service levels are quality-based promises a customer seeks from a vendor.  There are a lot of variables (such as the software), a few standardized items (usually the hardware), and you try to pick a few key metrics that you think will be able to give you a quality rating on the meal (the service itself).  The question is whether you can appropriately gauge how often you’re going to be satisfied with what you’ve purchased and cope with it when you’re not.

In the software and services world, service levels are typically measured in response time or uptime, used to enforce the vendor’s sales-pitches that the particular good or service will be as incredible as it was during the demo.  Vendors, of course, don’t like service levels, and customer’s predictably, love them.  However, in all of the years I’ve been playing this game, I very rarely see service levels that benefit either party.

To be effective, service levels have to be SMART (as made popular by Peter Drucker):  Simple, Measurable, Attainable, Relevant and Time-Bound (we discussed these earlier when talking about writing SOWs, too).  So while you might have a service-level grid in your template agreement, for any particular product or service, you have to evaluate those pre-defined levels and see if they make sense for whatever it is that is being purchased.  This is no easy task and requires a lot of input from your colleagues down in IT support, architecture, engineering and management.  You have to look first at the product or service’s use (Is it customer facing?  Is it mission critical (yes, be honest on this one)?  Is it internal-use-only?  Is it backend-use-only?)  Then you have to look at WHEN the product is going to be used (day, night, weekends, random).

But most important, you have to look at the actual impact of being without the product or service and for how long you can be without it before a real negative impact sets in.  So, for example, how long could you be without your word processing application company-wide before productivity takes a significant hit?  Can you actually calculate the damages that would result if noone had access to e-mail for an hour or two?  Probably not.  So you’re left with guess work.  Which makes a vendor (and many customers) pretty squeemish about putting hard dollars to soft numbers.

Over the next few posts, I’m going to talk about a few specifics and we’re going to re-visit the Myth of the Nines.  Get out your red pens and engage track-changes… we’re going to alter your service level perceptions.

Oh, and because I was talking about the 1 in 8 chance of getting three good food items earlier, well, it happened yesterday.  The Burger King in the Charlotte Airport nailed a plain Whopper, fries and a coke.  And it was at a time when I was REALLY desperate for good food, too.  Less than 113% chance, perhaps, but still possible.

Contract Metrics

Lemme’ guess.  Your management asks you to measure your performance.  This is a push from management textbooks everywhere.  Quality programs, Six Sigma… they’re all about measuring what you do and finding ways to improve efficiency.

With contracts, measurement is both easy and tricky at the same time.  It’s easy because there are a lot of numbers to keep track of.  The number of contracts, the number of renewals, the sum of all money spent in a given time period, etc.  It’s hard because the things that really matter: quality of the deal, risk allocation measures, actual cost savings, etc, are more difficult to put a number on.  Or, perhaps you’re trying to measure contract professional performance – how many of a particular term they got included in their contracts, the number of times they’ve started from your template forms (remember, you’re ALWAYS starting from YOUR templates, right?).

But are simple counts of these things a true measure of success, top quality performance, etc?  Or are they simply the easy things to measure to placate your management?

I’m not going to tell you what to measure, at least not yet (Symetrisk will do it for you soon).  But I want you to think about what you’re measuring and how your going to do the measurement itself.  Do the metrics matter?  Do you have the ability to make real inferences (not correlations – you won’t have that kind of data) from what you’ve gathered?

If the answer is no, it’s time to re-think your metrics.

Value of a Contracts Person

I’ve been asked, at every job I’ve been on, to justify my existence.  Some places just want to know what I do.  Others really care about the dollar value of the role to the company.  E-mail did it once.  Another was a full-blown business case.  But the end result is always the same – somehow, in some way, I have to convince someone who doesn’t know what I do that I matter.  So I figured that perhaps some of the lessons I learned could work for you, too.

Format:  Find what your organization wants, but typically, I suggest finding your organization’s business case document and using it.  Like being overdressed is better than being underdressed, so it is with the format.

Length:  If you don’t have a business case template to follow, you should be able to justify yourself in under 2 pages.

Purpose:  Don’t be super blunt.  Explain the reason you’re there:  you bridge a gap between lawyers and technologists/executives.  You have specialized skills.  You have experience and training.  You have the ability to protect the company.  You have the capacity to complete deals more efficiently.

Cost:  You won’t normally have access to other people’s salaries.  But you can make mild assumptions about counsel and executive salaries compared to yours.  More specifically, you show that the time that a specialist such as yourself requires to close a deal is much less than those who are not specialists.. and as a result, you can get more done for your annual salary.  This then allows counsel to work on more “important” things and prevents non-trained individuals from attempting to do the job (you have to come up with a smooth way of saying this within your company so as not to offend anyone).

Cost Savings/Avoidance:  If you’ve been in the game long enough, you’ll have an idea of how much you can save your firm on an annual basis.  But remember, this usually only works if you have access to all of the deals (not just those people want to hand you).  So be careful about promising a given level of savings, but sometimes, you’ll need to do so.

New Edition of the Software Licensing Handbook Released!

FOR IMMEDIATE RELEASE:  September 12, 2008

New Book Helps Anyone Negotiate Software Licenses

RALEIGH, NC – The second edition of the popular Software Licensing Handbook, offers negotiators of all skill-levels an opportunity for the upper hand when in the often complex world of software licensing.

Written by expert negotiator Jeffrey I. Gordon who specializes in IT and telecommunications procurement, this publication is full of comprehensive instruction and advice presented in straightforward language that can be applied by the novice or seasoned negotiator. From language nuances to tips and tricks that will give you an edge, the Software Licensing Handbook is designed to help both buyers and sellers navigate the complex waters of negotiating software licenses.

This latest edition includes more than 100 additional pages of licensing and negotiation advice and comes with a free first-look at Symetrisk™, a first-of-its-kind deal evaluation tool. “After working in this industry for years, it has been hard to ignore that the intricacies of software licensing and the related negotiation process rarely receive more than a mention in business books or how-to manuals,” stated Gordon.  “I wrote this book to fill the void and provide a one-stop-shop for everything related to the software licensing process.”

99.999

Back in the day, we used to fight for five-nine availability.  This meant that you would have access to the product or service 99.999% of the time.  (If you’ve not read Wikipedia’s article on the Myth of the Nines, now’s your chance.)  Mathematically, this equates to only having 5.26 minutes of downtime a year.  Pretty impressive, when you consider how long it takes the average computer to simply reboot.

We would tie the availability requirement to a penalty (aka “financial incentive”).  This is a kissing cousin to service and support response/repair times and similar penalties… and it heavily applies to situations where you need steady and consistent access to a particular product.

For ASP/SaaS products, uptime is extremely important, as it not only defines the amount of time you have access to use the product but also is a measure of the service provider’s value/responsibility (you don’t have access to the product to fix it yourself).  Until recently, though, network bandwidth wasn’t really available to sustain the offerings and was the scapegoat for much of the argument to reduce or eliminate uptime requirements, even if it was the vendor’s back-end server performance causing the problem(s).

As a result, compromises were made to allow uptime calculations only based on daytime work hours (so downtime at night wouldn’t be counted) for those businesses which didn’t have 24/7 operations.  Compromises were also made to allow vendors the ability to challenge a request for credits based on downtime.  Of course, compromises are fine so long as you’re not compromising your business model.

Flash-forward almost a decade.  Gigoam commented recently about the fact that the current movement towards more cloud-based computing is going to require more attention towards availability.  I agree.  And I’m a little disappointed that I haven’t yet seen a likewise return of significant uptime commitments.

The bandwidth is here (to understand the difference between then and now, check out this table and compare 10Base-T versus Gigabit Ethernet, almost a 1000x speed difference) and applications are designed to be a little more efficient, too.  The result is a resurgence of ASP/SaaS products.  There are several that I love, actually.  They’re incredibly good products, with good support and they don’t typically have downtime problems.  But I’m not seeing uptime guarantees of any significant amount.

To get decent uptime availability commitments from your providers, you need to be in-the-know regarding your own proposed usage and the product offering:  Know what the service’s value is to your organization.  Know when it must be available and when it can be down.  Know about server-loads and peak-usage periods.  Know the difference between hot/warm/cold backups and have an understanding of what failover might require.

99.999% availability might never be offered again… but something should be.  Make sure you’re asking for an uptime guaranty from your providers. Never forget that the middle name of ASP and the last name of SaaS is service.

Microsoft trying to convert you from perpetual to SaaS

Well, as I predicted years before I started writing this blog, Microsoft is now trying to convert the average home user from a perpetual software license model to “software as a service” (Saas).

My knee-jerk reaction is that this isn’t going to be good for the average (any) user – business or consumer.  But let’s play it out and see what happens:

In the current, perpetual model, the average cost of Microsoft Office 2007 is $119 (per Amazon.com).  This is a one-time expense and allows you to install Office on two machines (desktop and laptop) so long as you only use it on one machine at any given moment in time.  The average person never buys any kind of support for this product unless it’s a pay-per-incident issue that is SO complex that they can’t get help with it from friends or strangers via the internet.  But you do get all of the updates to the current version of the product (ie: if you’re on version 2004, you’d get all updates to 2004, but not get version 2007).

Because it’s a perpetual license, you can use this product FOR EVER, without ever having to pay another fee to Microsoft unless you want to upgrade to their latest version (which, at the time I’m writing this, happens about every 3 years per platform, alternating between PC and Macintosh).  From a depreciation perspective, if you were going to buy the latest and greatest version of the product every three years, you would divide the purchase price by 3 to find out your annual cost of ownership:  $39.67, which works out to $0.108/day.  Not too bad for the product that supports all of your e-mail, writing, spreadsheet and presentation tasks.

We don’t yet have pricing available for Microsoft’s new online offering, called Albany, but we do know that they’re going to bundle in a few already-available-for-free services.

We also know that Google already offers something quite similar (GoogleDocs) for free.  If you’re already a GoogleDocs user versus a Microsoft Office user, you have made a choice to go with one or the other for a reason (most would say that they choose Microsoft for “guaranteed compatibility” and “support if needed” … and Google users say that they want “openness”, “freedom” and “collaboration ability”).  I highly doubt that Microsoft is going to offer their product/service for free… but I’ve been wrong before.

However, this really isn’t about Microsoft versus Google – it’s about a bigger issue of whether a conversion from Perpetual Licensing to SaaS is really a benefit to either the vendor or the consumer.  Perpetual software users like not having to upgrade every time the vendor releases a “fix.”  They like knowing that they don’t have to keep paying for maintenance when the product hasn’t really changed much over time.  They like having a one-time depreciable expense (if they’re business users).  Oh, and they like knowing that if the vendor ever goes out of business, it doesn’t matter too much, since the software is installed locally.

SaaS offers a level of convenience not found with perpetual products.  You are always on the latest version, always covered by support and you have less of an administrative headache since the product isn’t installed locally.  Sure, you have to have greater bandwidth (I’m guessing Microsoft will actually have you download a full version of the product which will simply “phone home” every time you double-click on the product to use it).  But you give up the ability to sever your ties with the vendor yet continue using the product.

I like the SaaS model for some situations – I use one for my contract management system, for example.  But for everyday, standard use products?  Especially those in millions of homes world-wide?  I’m not sure we’re there yet.  I’m REALLY concerned about the quality of service – and the constant communication connection (from a privacy perspective) of all of these phone-home events.

What do you think?

Quick Survey

I was talking with some folks today about Purchasing versus Contracting. I have worked in a lot of places and only one had a consolidated group (which, even then, was technically segregated). So, I’m interested in your feedback. Thank you!
[memedex: pollid#477228]

(You must have flash installed to see this survey.)