Dido Harding questioned on safety standards of using Microsoft Excel
- 15 October 2020
The head of NHS Test and Trace has been asked to explain the safety standards considered when using Microsoft Excel to hold vital Covid-19 test data.
Greg Clark, chair of parliament’s science and technology committee, has written to Baroness Dido Harding for clarification on whether the decision to use Excel to hold test data breached statutory or regulatory requirements for safety.
“What certification took place of the Microsoft Excel-based system, what consideration was given to the standards required for a safety critical piece of software, and what safeguards are in place to prevent this problem from happening again,” Clark wrote.
It comes as nearly 16,000 Covid-19 cases went unreported between 25 September and 2 October, reportedly due to an error with a Microsoft Excel spreadsheet.
It is believed Public Health England (PHE) developers used an old file format known as XLS, which can only handle about 65,000 rows of data. When the spreadsheet reached its maximum size it prevent new cases from being added to the file.
The glitch resulted in efforts to trace thousands of contacts of those who tested positive being delayed.
Harding has faced calls to “consider her position” as head of NHS Test and Trace due to the missed cases alongside some of the lowest Test and Trace figures reported.
Figures released on 8 October revealed just 68.6% of close contacts of those who had tested positive were able to be reached. At the same time Covid-19 cases in England have increased 56%.
The GMB Union said the Test and Trace programme had seen “blunder after blunder” under Harding’s leadership, calling for it to be “led by experts and public authorities”.
Harding is responsible for overseeing the new National Institute for Health Protection announced in August, which merges Public Health England with NHS Test and Trace as well as the Joint Biosecurity Centre under a single leadership team.
Experts told Digital Health News that “Excel-gate” highlighted the need for “good quality technical capability” within organisations.
‘A systematic failing, not an IT glitch’
Harding is not alone in facing questions over the decision to use Excel to store coronavirus testing data.
The Association of Professional Healthcare Analysts has written to health secretary Matt Hancock and NHS chief executive Simon Stevens saying it was “only a matter of time” before thousands of infected people we missed by PHE due to the use of Excel.
“This is not an IT glitch and to describe it as such trivialises the issue, this is the symptom of a more serious, chronic failure of the systems and processes,” the letter states.
“This is a systematic failure that led to an individual making a poor decision on the most appropriate method of collecting data. This can only be improved by long term investment in infrastructure and training.”
They added that the risks and limitations of software systems must be evaluated to ensure they are “fit for purpose”.
“An out of date version of Excel should not be used to transfer vital data sets in the real world,” the letter added.
The association made 11 recommendations including investment in a “suitably skilled analytical workforce” to demonstrate the importance of data to drive decision making; application of appropriate analytical methods and provision of suitable tools to analysts and data scientists; and investment in analytical capabilities within the health and care system.
“We should look at this event not as a catalyst to blame or shame, but as an opportunity for learning and improvement,” they added.
14 Comments
NHS Digital is a profoundly dysfunctional organisation obsessed with data security and data control to no clear purpose. PHE has manifestly failed in protecting the public’s health, leading to the death rate from COVID in England fighting it out for bottom place with countries such as USA and Peru. The relationship between PHE and Directors of Public Health is somewhat unclear. It also remains unclear who is actually in charge on the frontline of public health or what these various organisations are actually doing to support the frontline. It is easier to blame government and individual politicians. Dido Harding had hardly been in post when she was blamed for the failures of these organisations which urgently require root and branch reorganisation, starting from the top. Excel is simply one of multiple small examples that are the responsibility of NHS Digital and PHE to put right and have not done so. It is quite possible that the expertise is simply not there.
The problem relates to the modeler or Data Base administrator. If used properly Excel works just fine. To the comment about “March1” being treated as a date — that is correct. But, if the entry is preceded by a space it doesn’t happen. Also there can be ways to check the data base if there is concern such as using Data Filters.
Index(……, Match(xx,xxx,0)) works great to manage a data base.
The Excel file is easily transferable or viewable by most whereas other forms of a Database may not be accessible to others, or they may not know how to use.
Spreadsheets are old skool, use BI
CSV format should have been used.
Have to say the data quality and format that this data is provided to local authorities is dire and inconsistent.. how they expect local authorities to confidently use this data to carry out track and trace is mind boggling
But ‘older’ as in 17 years ago…
My understanding is that the file format was being used to transfer data between systems, not that Excel itself was being used to store data. As I commented elsewhere, the data collection mechanisms in use are a Health Robinson affair that have been tolerated by PHE and probably the HPA before it for many years.
As usual it’s ‘follow the money’: why hadn’t PHE invested to improve its clunky data collection? Who was holding the purse strings?
Sorry that should have been Heath Robinson!
As an aside, this is not the first time PHE has drawn criticism for failing to invest in IT.
Where was the CSO in all of this? Is there Clinical Risk Management Plan for this project? Has anybody at PHE even heard of DCB0129 and DCB0160?
Excel in any flavour is not the right type of program. A proper database should have been used.
Don’t now if you were replying to me or not William but I totally agree with your comment.
The point I was making was that the authors of the letter blamed Excel when it was the file format that specifically caused the error. Of course, ultimately, even the xlsx format would have run out of available rows.
If you’re stupid enough to use Excel in the first place then that is an entirely different matter.
Neil. Excel has more problems than the file format though. It’s too easy to corrupt data, and indeed its autocorrect function for data entry can corrupt it automatically. One example is this:
“Excel is a behemoth in the spreadsheet world and is regularly used by scientists to track their work and even conduct clinical trials. But its default settings were designed with more mundane applications in mind, so when a user inputs a gene’s alphanumeric symbol into a spreadsheet, like MARCH1 — short for “Membrane Associated Ring-CH-Type Finger 1″ — Excel converts that into a date: 1-Mar. This is extremely frustrating, even dangerous, corrupting data that scientists have to sort through by hand to restore. It’s also surprisingly widespread and affects even peer-reviewed scientific work. One study from 2016 examined genetic data shared alongside 3,597 published papers and found that roughly one-fifth had been affected by Excel errors.”
Surely the NHS paying massive fees to its consultants can come up with a decent database programmer to do a proper job!
“An out of date version of Excel should not be used to transfer vital data sets in the real world,” the letter added.
If you’re claiming to be experts in your field then you should get your terminology right. It wasn’t an out of date version of Excel it was an older version of the Excel File Type (xls instead of xlsx) that was used. The current version of Excel can still create, open and save xls files.
William, did I say anywhere that I agreed with the use of Excel?
Comments are closed.