Enterprise Computing: Barclays Bank Services Down Due to Storage Array Problems

Chris M Evans — Wed, 17 Jun 2009 13:11:05 +0000

It’s been reported in a few places that yesterday Barclays (UK bank) suffered an issue with a “disc array” (presumably they mean disk array) that took out their ATM and online banking systems. See the comments here and here.

Allegedly, Barclays now use USP-V arrays as their back-end storage devices, so presumably HDS USP-Vs were involved in yesterday’s problems. Systems seemed to have been down for a number of hours before normal service was resumed.

The first thing to say is that “stuff” happens. Hardware fails – arrays fail and it’s the same for all vendors. No vendor can ever claim that their hardware doesn’t fail once in a while. We all know that RAID is not infallible; in fact, it isn’t even necessary to have a hardware failure to experience service outage as many problems are caused by human error.

What surprises me with this story is the time Barclays appeared to take to recover from the original incident. If a storage array is supporting a number of critical applications including online banking and ATMs, then surely a high degree of resilience has been built in that caters for more than just simple hardware failures? Surely the data and servers supporting ATMs and the web are replicated (in real time) with automated clustered failover or similar technology?

We shouldn’t be focusing here on the technology that failed. We should be focusing on the process, design and support of the environment that wasn’t able to manage the hardware failure and “re-route” around the problem.

One other thought. I wonder if this problem would have been avoided with a bit of Hitachi HAM?

The cuts are happening again

Chris M Evans — Tue, 29 Jul 2008 12:57:00 +0000

The Register has reported that some UK banks are forcing their contractors to take a 10% pay cut or take a hike. It is usual for contractors to take the hit first as they are an easy target compared to terminating permanent staff and to be fair, as a contractor/consultant myself, if the chop comes then so be it, it’s part of the nature of the business.

However, the downside is that the most able contractors are likely and will surely leave first. Many of those who remain will have a lethargic attitude to work and overall, the company suffers.

In addition, Barclays are offshoring more jobs, presumably to further save costs. In my experience, moving jobs offshore doesn’t work. It simply moves skills away from the business and causes the customers to lose control over their IT operations. This isn’t necessarily a reflection on poor quality of work from the outsourcing suppliers; far from it, I’m sure most are equally as competent or skilled as their UK counterparts. The issue is more to do with operating remotely from the customer; not understanding the “local” issues; not understanding local culture and not being part of the team who see and chat with each other every day. As someone who currently works with multiple clients for 1-2 days a week, I experience this problem even though I know my customers well. Imagine if you’ve never met most of the people you are providing complex services for.

One final note; Barclays claim they worked out they were paying “over the market rate”. That’s a phrase I’ve never understood. It assumes all contractor’s skills are equally good (or bad) and so they can be paid, or have their pay cut to the same degree. It also assumes that all contractors were taken on and overpaid by an equal amount. I wonder if they did the same market comparision for permanent staff and considered cutting their salaries by an equivalent amount? Somehow I think not…

The Storage Architect » Barclays

Enterprise Computing: Barclays Bank Services Down Due to Storage Array Problems

The cuts are happening again