Some years ago, my wife and I stayed at an elegant hotel in Manchester. It turned out to be an interesting week – for all the wrong reasons.
The first night the bed tried to eat me. I was awoken by a sharp pain in the upper aspect of my left foot: on turning over I experienced a similar pain on the sole. One of the bedsprings had broken across its top and the resulting two sharp ends were acting like jaws.
The next day the London riots happened, so we switched on the news. The sound came from BBC1 whilst the picture was BBC2 – which was showing the Muppets. I can’t begin to describe the surrealistic effect of hearing about tragic events, apparently narrated by Fozzie Bear.
Then, our room was suddenly plunged into blackness: the door-card switch to our room’s electricity supply had failed completely and needed to be bypassed.
After which, on our last night, the room air conditioning disintegrated, releasing a cascade of water and forcing us to migrate to another room in the middle of the night.
Later that day, somewhat bleary-eyed, I went to pay the bill. There was a pause, a swift consultation of the receptionist with her on-duty manager, followed by a heartfelt apology together with a generous discount spontaneously applied by the hotel before I’d even asked.
Honour was satisfied. The net result? When next I go to Manchester, I will want to stay at that hotel. (It was the Radisson Blu Edwardian, on the old Free Trade Hall site, and I’m glad to give them the PR. Great place; excellent staff.)
Why will I go back? I have always maintained that the difference between a good and a bad organisation isn’t that a good organisation never makes mistakes, but that when it does, it accepts responsibility and then quickly and reliably sorts out the problem.
Segueing neatly on…
And so to the e-Referral Service. It will not have escaped your attention that shortly after the move to the new e-Referral Service, the whole thing bit the dust. It didn’t actually crash: it just ground to a halt. Referrers couldn’t acquire UBRNs (the all-important reference number for each referral) before the system timed out; and providers’ worklists often went AWOL.
The e-RS ‘war room’ was immediately alerted, quickly analysed the situation and over the next few hours and days put in a series of workarounds and fixes, prior to getting in a definitive fix at the beginning of July.
Despite all this, on day one of the release the papers were full of carping criticism, with frequent comments such as “Why didn’t they test it properly?”, “Yet another government IT project that’s been performed incompetently”, and so on, and so forth.
From the horse’s mouth
As it happens, I’m connected to the project – though relatively peripherally – so I know something of what has gone on behind the scenes, and of the decision-making processes.
Right from the beginning everyone on the national programme board knew that there was no possibility that the new e-RS could be proved completely before launch.
Yes, individual parts could be tested (and they were, repeatedly, and intensively); and yes, it was stress-tested. Individual checks are all very well – but how do you test the system as a whole when it (suddenly) has to be used by 40,000 people simultaneously, each using it in their own inimitable fashion?
In addition, the new system had to take across all the data from the previous ten years of Choose and Book activity, including all the hidden idiosyncrasies in its use by providers and referrers over this time.
It was clear that it was always going to be a gamble as to whether the release on 15 June would go off without a hitch.
There were any number of high level debates beforehand about whether the e-RS service had closed off all possible points of failure. But each time the inexorable conclusion came round that there was no way that anyone, anywhere, could ever test the system completely before going live – and that the next step was into the unknown.
It’s like Donald Rumsfeld’s ‘known unknowns’ and ‘unknown unknowns’ — it is impossible to cater for ‘unknown unknowns’ because, by definition, you don’t know what they are.
I had always believed that the best way to plan here would be to assume that the launch was going to hit unexpected difficulties, and that the appropriate solution was to set up an ultra-fast process to identify problems, analyse them and then rectify them: and I’m glad to say that this approach was ultimately adopted by the programme board.
And on go-live (or should that be ‘go-dead’?)…
In the end, it transpired that the initial system slowdown wasn’t a bug: therefore no amount of further isolated testing would ever have spotted it.
Instead, this problem occurred because of the way in which certain providers have used their work lists historically. This put a huge and unexpected load on particular aspects of the system once it was being used by lots of people simultaneously. It was a bottleneck, rather than a bug.
If you want an analogy, it’s as if the e-RS had been asked to build nine new motorways, all of which worked pretty well.
However, in the new configuration, motorway number seven ended up carrying fifteen times the amount of traffic than the rest, which meant that the interchange between it and the other motorways got blocked and the rest of the road system consequently sludged up. All it needed was a re-phasing of the traffic lights at the interchange, and everything would work smoothly again. (The only practical problem was – try and work that out beforehand!)
Fair criticism?
Back to the launch, and to the criticism. I was pretty annoyed (aka blazing mad) with many of the careless and ignorant comments made in the press immediately following the first problems of the e-RS. (Isn’t it easy to carp? Everything is easy to the man who doesn’t have to do it himself.)
There’s no way the system wasn’t tested; those running it were certainly not lazy, inexperienced or thick – indeed, they are some of the best, most industrious and most competent minds I’ve ever had the privilege to work with.
The failure, when it came, was exactly where we said it might – in just the area that couldn’t be tested beforehand, however much we might have wanted to test it.
And finally, all those who wrote in to say “It was working OK beforehand, why change it?” seem to have forgotten the huge amounts of criticism and complaints about lack of functionality that had previously been levelled at its original incarnation as Choose and Book.
How do I think the launch went? My hope was that the whole system would be working properly within a fortnight of its release.
In the event, that’s more or less what happened, though in the meantime other problems have surfaced — which (again) would have been extremely difficult to spot before go-live.
My two week aspiration has therefore had to be extended somewhat – but everything possible is being thrown at the problem: the e-RS organisation won’t be resting until the whole thing is fully stable, totally on-track and working exactly as intended (and at full speed, too).
It goes back to my experience with the hotel: the measure of a class act isn’t whether it gets everything right first time, but on how effectively and comprehensively it handles any problems that arise. Viewed in this light, the launch of the e-Referral Service has succeeded spectacularly.
 | Dr John LockleyDr John Lockley is clinical lead for informatics at Bedfordshire Clinical Commissioning Group and a part-time GP. |  |