The UK government and the health service must engage with GPs and the public if the programme is to go ahead, the NHS national data guardian has told MPs.

Dame Fiona Caldicott, who gave evidence to a Science and Technology Committee evidence session on big data yesterday, said that one of the main reasons that the project failed on its first attempt was that those leading it did too little work with primary care clinicians.

The programme was first approved in 2013 as a project to expand the Hospital Episode Statistics and to link them to other data sets, starting with GP data, and to make this available to researchers and others.

However, the project was paused after it came under heavy criticism for confusion over what the data would be used for and how patients could opt out.

Explaining the decision to put the project on hold, Dame Fiona told MPs at the evidence session that there had been “loud and extensive protest” from several groups, including GPs.

She said they were not content with the purpose for the programme and not clear about what would happen to a patient’s data once it was sent to the ‘safe haven’ of the Health and Social Care Information Centre.

She said: “That reflected the fact there had not been enough work done with primary care physicians and their support teams, and with the public, about the purpose of – the explanation about benefits was not clear enough.

“In the end, with the clamour about those problems, there really had to be a pause.”

The programme was resurrected this year with the launch of four pathfinder areas to trial ways to communicate the risks and benefits of the programme and how to opt out.

However, these have also now been put on hold until Dame Fiona gives final approval on the communications plan. A decision is expected in January 2016.

Speaking at the evidence session, she emphasised the importance of making sure that doctors and patients understood the benefits of having a national dataset for anonymised patient information, as well as the risks inherent in allowing personal data to be used for non-direct care.

“Were it to be restarted, it would be along the lines of much improved communications to both GPs and patients and a much wider communications programme to the public at large.”

Professor John Williams, director of the Royal College of Physicians’ Health Informatics Unit, was also speaking at the evidence session.

He said the debate had been confused between the risks of patient data at a local level and the risks when part of a large dataset, and that more work needed to be done to communicate the risk/benefit balance of the project.

“It would be possible to identify a patient and enthusiastic and malevolent hacker will be able to do that. And we will probably never completely eradicate those risks.

“We have just got to get the balance right so the safeguards are there and we need to explain to the public there are great benefits but there are some risks.”

Dame Fiona added that she was disappointed that since the 2013 publication of the Caldicott Principles on the use or transfer of patient identifiable information in the NHS that the culture of sharing information in the NHS had “not moved on in the way we’d hoped”.

She suggested that one of the reasons was that the NHS and the government had not done enough to continue to educate and training NHS staff on the value of sharing information beyond a “limited induction”.

She also mentioned that if information governance was part of the clinical governance framework that the NHS adheres to then the organisation would not be facing the problems it has when it comes to attitude to sharing of data.