Out of service
Written by Brett Farrell
I woke to a curious problem on 28 February 2011: my mobile service was absent. It was unlikely I had not paid my bill as Vodafone had never missed removing money from my account each month. It took a WIFI connection and Twitter to confirm the truth: Vodafone could not provide service. During the course of the morning the real story emerged that thieves had broken into a Vodafone datacentre in Basingstoke and had removed or damaged equipment that was mission critical to providing mobile telephony to most of Southern England.
It took time to process what had really happened. Thieves. Robbery. Datacentre. Damage. No Service. It seemed impossible that such an important piece of technology infrastructure could be breached so easily, and if the story were true that the redundancy had either failed or did not exist. Security and redundancy are the most important things that datacentre customers want to know about and it's worth attempting to overlay the legal issues on top of them. The issues faced by Vodafone are no different to those faced by companies in FS.
Datacentre suppliers' reputations rise and fall on how secure their sites are. The sites are heavily alarmed and monitored by CCTV. It is also usual to have a datacentre monitored with on-site staff 24/7. It's not usually possible to enter one unless you are pre-approved by your company, have provided your passport for vetting purposes and signed a non-disclosure agreement about what else you may see. Could sledgehammers get past this?
The whole point of redundancy is to ensure that if one thing fails another takes over. Datacentres are designed on principles of multiple failovers and ensuring that there are not any single points of failure. N+1 is bandied about like a religion that you dare not question. I have no intention to second guess the datacentre design used by Vodafone, but the outcome was that it failed and did not failover. I know this because my phone did not have service for about six hours.
The security and redundancy specifications are codified into a contract where a datacentre provider promises to provide the required levels of security and redundancy. It's often that these obligations are not absolute and are watered down with legal language like 'reasonable' or 'best endeavours', which are designed to reduce the burden that the datacentre supplier has on itself to adhere to the promises. In the Vodafone example I suspect the first thing that Vodafone's legal team would have done is to parse the contract with the datacentre provider to see whether or not the contract was in any way breached.
In a case like this, once the contract is breached, the normal remedy is damages payable by the breaching party to restore the loss suffered. Datacentre suppliers tend to try to reduce this risk by offering service credits for certain levels of breach based on seriousness. Liability for these breaches is usually capped to some pre-agreed amount and that is also recorded into the contract. No matter what the remedy available, it's not difficult to imagine that there was a breach of this contract either for not having provided the requisite level of security, or more worryingly, adequate redundancy. The practical question simply becomes what is payable to restore the loss, and how much legal argument there is going to be about that in principle.
Incidentally for me to claim from Vodafone, service would need to be unavailable for seven days. But I shall refrain from a discussion about the Unfair Contract Terms Act.
What bothers me is this: the contract may protect you and ensure that you receive monetary compensation if a disaster does occur, but the contract is impotent in stopping the disaster in the first place. If you ever need to consult the contract chances are it is probably too late. The commercial terms are usually found at the back of the contract and are reviewed by the 'business' and lawyers encouraged to 'just stick to the legal bits at the front'. Nothing can replace proper due diligence at the outset and asking your legal advisors their views early on to shape the protections. We learn from the Vodafone debacle that their contract did not help to prevent the failure. However my point is this: involving the legal team with the commercial negotiations will reduce risk. I know this from personal experience, having pointed out to the 'business' that the promises they were expecting are not in the commercial part of the contract. That then focused the business onto deeper due diligence and questioning just how +1 was this datacentre supplier’s N.