Dr Foster has called for the quality of NHS data to be given the same priority as hitting targets.
In a new report on the ‘Uses and Abuses of Performance Data in Healthcare’, the benchmarking organisation says performance measurement is essential so politicians, managers and clinicians know what is working and can spot problems and “so patients and taxpayers can know what is being delivered, in their name, and with their money”.
However, it says this throws up a dilemma: in order for performance management to work, there must be “open and honest reporting”; but there must be “real consequences for underperformance” that might discourage openness and honesty.
It says those designing and working in accountability regimes should be aware of specific problems, such as the danger of distraction – or “challenging, obfuscating or denying data that suggests under-performance, instead of fixing performance problems” – as well as other, well known pitfalls of targets, such as gaming, tunnel vision, and bullying.
The report, authored by former NHS Direct chair and National Audit Office board member Joanne Shaw, Dr Foster co-founder Roger Taylor, and consultant Katy Dix, says more attention should be paid to designing out data problems and the unintended consequences of targets.
“Although multiple audit reports over a period of years have highlighted problems with data quality, long-term solutions remain elusive,” the report says.
“The failure of the NHS to adequately address this issue calls into question its commitment to fair and effective performance management.
“This can be seen most clearly in the imbalance between the level of seriousness with which poor performance on targets is treated, compared to the effort expended on identifying gaming and inaccurate recording of information.
“In practice, the unglamorous, back office work of managing, coding, checking and cleaning information tends to attract attention only when miscoding can be blamed for apparently poor performance, while the quality of data underpinning average or strong performance tends not to be examined.
“Now that the National Audit Office has taken over responsibility for overseeing the audit of local services, including the NHS, there is an opportunity to revisit the emphasis and resources devoted to auditing data quality… recognising that constant attention to data quality will be needed over the longer term.”
The report also recommends that “counter balancing metrics” should be drawn up for targets that might be subject to gaming; suggesting, as an example, that ambulance queuing times and patient outcome measures should be run alongside the four hour A&E waiting time target.
It says policy makers should avoid setting “thresholds” that can be manipulated, that they should be more open about the data used to assess targets, and that they should apply performance measures fairly, so organisations have more incentive to work with them.
However, the report stresses that there are “impressive examples of targets, measurement and reporting leading to significant improvement."
These include the significant reductions in English patient waiting times associated with the 18-week referral to treatment time target and lower mortality for cardiac surgery in Great Britain and Ireland, where heart surgeons have led the way on publishing their own data in response to the Bristol Royal Infirmary scandal.