Publication bias means that null results do not make it into the public domain. Assessing publication bias is straightforward in subjects where all studies have to be registered in advance – clinical trials for example. But there is little evidence on publication bias in service delivery / health services research. The CLAHRC WM Director suspects that this lack of evidence arises because much social science literature is observational rather than experimental, and it is so hard to collect convincing evidence on publication bias among such studies. There is no registry of studies; the original hypothesis may not correspond to comparisons reported; many studies might not be written up; and the investigators may evaluate a large number of associations so that results do not neatly dichotomise into significant or null. In addition, the famous funnel plot may be less likely to signal bias than is the case for much clinical research. This is because the association between sample size and risk of publication bias is less likely to hold when the size of the sample is limited more by the size of the database than the cost of recruiting individual participants. These problems were overcome in an interesting article that studied the destiny of 249 grant-funded (peer review) studies conducted within a single ongoing data collection survey over a ten year period.[1] Most of the studies consisted of an evaluation of modifications of the survey instrument (questionnaire) used to populate the survey database. The results show a massive effect. Studies with a positive result (as judged by the author) were much more likely to be written up and, if written up, much more likely to be published. The fact that the source studies were all based on a single database removes (or at least strongly mitigates) bias due to interaction between study topic and probability of a positive result.

These results reinforce the CLAHRC WM Director’s weariness to accept positive results of association studies, such as those that relate patient perception of care to standardised morality rates. Such results feed into the prevailing meta-narrative, in this case that organisational culture determines the quality of the full range of front line services. A null result is less likely to survive peer review under such circumstances. The paper cited here interviewed holders of grants based in the database, and found that they were disheartened by null results and often did not bother to submit them, anticipating that they would be rejected. They are right to be pessimistic since null results were less likely to be accepted when submitted, in keeping with the natural human tendency to reject studies that do not fit with prevailing or preconceived ideas.[2] [3]

What do we recommend? Only studies where the protocol has been published should be considered for publication, and they should all be published provided the protocol was adhered to. The clinical research world has tightened up its act. It is high time for the service delivery world to stop claiming scientific exceptionalism and adhere to the standard tenets of good scientific practice that hark back to Francis Bacon.[4]

-- Richard Lilford, CLAHRC WM Director

Leave a comment

This recent article in Science [1] gives five evidence-based reasons for greater emergence of, and more devastating consequences from, antibiotic resistance in low-income countries (LICs) compared to higher income countries:

1. LICs harbour a higher rate of extremely virulent organisms, such as those responsible for typhoid fever and tuberculosis.
2. Antimicrobials are (even) less carefully regulated and are frequently available ‘over the counter’.
3. Treatment for established infection is more often delayed and access to supportive care for life-threatening infections is less widely available.
4. Laboratory testing for the infectious agent is less widely available and therefore ‘syndromal’ treatment based on suspicion, rather than confirmation of a bacterial cause for ill health, is understandably widespread.
5. Routine dosing of animals to improve yields is more widespread.

Of course, increasing resistance to antibiotics in LICs impacts on high-income countries, and the article shows how molecular tracking of microbial genetics has enabled the dissemination of resistance genes to be mapped across the world.
The CLAHRC WM Director thinks that new molecular genetic techniques will help target antibiotics at the correct diagnosis, and also help in controlling the spread of resistant organisms. The technology is becoming ever cheaper, as discussed in a previous blog.

-- Richard Lilford, CLAHRC WM Director

We are currently in the process of updating our website and we plan to have a new look ‘patients and public’ section to improve the way we communicate information about the things we are doing that relate to PPI co-ordination and to PPI research activities.

The ‘patient and public’ section of the website will include three main headers, ‘Involvement’; ‘Engagement’ and ‘Participation’ to align to the NIHR reporting and classification structures – more information about what is meant by these ‘headers’ can be found at the NIHR INVOLVE website. We also plan to add examples of impact to demonstrate how and where PPI is influencing research and implementation activities being carried out by the CLAHRC West Midlands initiative, among other things. We hope we are capturing how and where our PPI Advisors are influencing activities through our interim theme reporting mechanisms, but please do let us know if you have any examples to share and we will add stories to our website and to this News Blog from time to time.

Shortly, we will be looking to recruit a part-time PPI Officer to help us with some of the aspects relating to PPI co-ordination, so watch this space for the advertisement. The next PPI Supervisory Committee meets in early July and shortly after this meeting, we plan to let you know how we will be taking forward the recommendations of the recently published ‘Going the Extra Mile’ document, following a strategic review of public involvement in the NIHR.

In other news, we are also supporting the NIHR INVOLVE campaign Make it Clear to include lay summaries of research projects being undertaking by Themes. We are in the process of updating our iterative ‘Projects Document’ to include plain English summaries for each planned research study, this information will also be included on the revamped website. We also now provide (where possible) lay summaries of the presentations made at Programme Steering Committee meetings, which the attending PPI Advisors say are very valuable to aid in the scientific understanding of work.

Many changes are ahead… so please keep reading our regular updates. Feeling inspired… if you would like to write story on your experience of PPI – either from a professional or from a lay perspective, please let us know and we can include in a forthcoming issue of this Blog.

-- Nathalie Maillard, CLAHRC WM Head of Programme Delivery

This is a really good point, about the absence of strategic human resource management in wider quality/safety debates. I think there have been some good additions to the knowledge base recently on the academic HR side, some funded by the NIHR HS&DR programme, like a review on staff engagement by Katie Truss (about to publish) and recently published analysis by Martin Powell of staff satisfaction/cohesion and performance.  Would be good to bring together the learning about health-related evidence on HR issues...

-- Tara Lamont

Thank you Richard for a thought-provoking piece, which was amplified by an excellent visual presentation. I had not considered a lottery as an incentive for patients. In high-income countries millions play the lotto every week and TV shows like 'BGT' and 'I Am Celebrity...' survive (even flourish) on our hopes of winning a luxury car by entering a virtual lottery. Is a lottery to incentivise health care any different to change the behaviour of patients? It might sit uncomfortably with many people, but the data would suggest otherwise.
-- Max Feltham