The head of NHS Test and Trace has assured there was no risk to patient data during this month’s Excel error.
Responding to parliament’s science and technology committee, Baroness Dido Harding said there was “at no time any risk to patient data”, adding that it is kept securely “in line with data protection regulations”.
The committee wrote to Harding on 8 October asking for clarification on whether the decision to use Microsoft Excel to hold Covid-19 test data breached statutory or regulatory requirements for safety.
In her response published on 16 October, Harding said a new system had been commissioned to replace the “legacy system” but failed to address whether the use of Excel had breached regulatory requirements.
“This new system is currently being built and will be assessed as part of our winter readiness plans,” she wrote.
It comes as nearly 16,000 Covid-19 cases went unreported between 25 September and 2 October, reportedly due to an error with a Microsoft Excel spreadsheet.
Public Health England (PHE) developers reportedly used an old file format known as XLS, which can only handle about 65,000 rows of data. When the spreadsheet reached its maximum size, it prevent new cases from being added to the file.
The glitch resulted in efforts to trace thousands of contacts of those who tested positive being delayed.
Health secretary Matt Hancock told MPs on 5 October that the incident was being “investigated fully”, confirming that a decision to replace the “legacy system” had been made in July.
“The technical issue was not caused by the use of an old format. Rather, PHE have confirmed that there was a failure in the process for preparing some testing data for uploading onto the central system that stores all test results,” Harding said in her response.
“A rapid mitigation has been put in place by PHE working together with NHS Test and Trace that splits large files and a full end to end review of all systems was immediately instigated to mitigate the risk of this happening again.”
She confirmed that increased testing meant some Excel files used to transfer data exceeded their file size, preventing further data from being uploading.
Harding has faced calls to “consider her position” as head of NHS Test and Trace due to the missed cases, coupled with some of the lowest Test and Trace figures reported.
21 October 2020 @ 21:55
Interesting pair of statements:
“The technical issue was not caused by the use of an old format.[…] She confirmed that increased testing meant some Excel files used to transfer data exceeded their file size, preventing further data from being uploading.”
My analysis:
https://sysmod.wordpress.com/2020/10/13/uk-covid-19-track-trace-excel-snafu-uncontrolled-spreadsheets-lead-to-data-loss/
22 October 2020 @ 07:17
The 90’s solution to transferring results was to use hl7v2. Csv has limitations, it struggles with report structures and data formats are too variable.
Most (all?) lab systems support hl7v2 and wouldn’t support phe csv format.
We don’t have a UK standard for hl7v2 results but the format is workable. So no central push for standardisation but nhs/industry moved that way because reports are difficult to send as csv.
Using csv and excel may appear to simplfy the transfers but it appears to have made the process more complex and has led to errors.
22 October 2020 @ 09:49
Thanks Kevin. I had wondered about a multi-line-record “CSV” all right, perhaps it was simply that they picked for a press release the most commonly understandable word to use for text files (although with a pipe field delimiter) rather than “the HL7 V2 API”. I’ve seen text instrument output but not combining comma-separated values as well, except as perhaps multi-valued fields. Are they likely to move on from V3 XML to JSON, to add to the alphabet soup?
May I ask, do you think that it is really likely that they were using Excel 2003 or earlier? It’s hard to believe that the public sector would be so starved of technology resources, especially in a pandemic with billions being spent on it. But it’s the only explanation I can think of for a 65K limit. Unless that too was a misreporting.
23 October 2020 @ 19:45
i work for (y)our nhs and am not managerial, not clinical, i have seen this issue (max excel rows), with my own eyes, my heart goes out to nhs technical who, like me, have no choice but to work with old skool vers of s/w, would nhs clinical have to put up with clinical tools that were 15 years out of date ? ? ? some nhs orgs waste a fortune on tech, fortunately i am not currently working at one of those, there is a leadership problem and i no longer think it is the party politicans that are the problem ! i repeat my question … would nhs clinical have to put up with clinical tools that were 15 years out of date ? ? ?
22 October 2020 @ 13:47
Work is underway on Json/XML HL7 FHIR believe this is for primary care (and is probably compatible with existing v2)