|
|
|
|
Machine Learning and the Demise of the Standard Clinical Trial!
An increasing proportion of evaluations are based on database studies. There are many good reasons for this. First, there simply is not enough capacity to do randomised comparisons of all possible treatment variables.[1] Second, some treatment variables, such as ovarian removal during hysterectomy, are directed by patient choice rather than experimental imperative.[2] Third, certain outcomes, especially those contingent on diagnostic tests,[3] are simply too rare to evaluate by randomised trial methodology. In such cases, it is appropriate to turn to database studies. And when conducting database studies it is becoming increasingly common to use machine learning rather than standard statistical methods, such as logistic regression. This article is concerned with strengths and limitations of machine learning when databases are used to look for evidence of effectiveness.
When conducting database studies, it is right and proper to adjust for confounders and look for interaction effects. However, there is always a risk that unknown or unmeasured confounders will result in residual selection bias. Note that two types of selection are in play:
- Selection into the study.
- Once in the study, selection into one arm of the study or another.
Here we argue that while machine learning has advantages over RCTs with respect to the former type of bias, it cannot (completely) solve the problem of selection to one type of treatment vs. another.
Selection into a Study and Induction Across Place and Time (External Validity)
A machine learning system based on accumulating data across a health system has advantages with respect to the representativeness of the sample and generalisations across time and space.
First, there are no exclusions by potential participant or clinician choice that can make the sample non-representative of the population as a whole. It is true that the selection is limited to people who have reached the point where their data become available (it cannot include people who did not seek care, for example), but this caveat aside, the problem of selection into the study is strongly mitigated. (There is also the problem of ‘survivor bias’, where people are ‘missing’ from the control group because they have died, become ineligible or withdrawn from care. We shall return to this issue.)
Second, the machine can track (any) change in treatment effect over time, thereby providing further information to aid induction. For example, as a higher proportion of patients/ clinicians adopt a new treatment, so intervention effect can be examined. Of course, the problem is not totally solved, because the possibility of different effects in other health systems (not included in the database) still exists.
Selection Once in a Study (Internal Validity)
However, the machine cannot do much about selection to intervention vs. control conditions (beyond, perhaps, enabling more confounding variables to be taken into account). This is because it cannot get around the cause-effect problem that randomisation neatly solves by ensuring that unknown variables are distributed at random (leaving only lack of precision to worry about). Thus, machine learning might create the impression that a new intervention is beneficial when it is not. If the new intervention has nasty side-effects or high costs, then many patients could end up getting treatment that does more harm than good, or which fails to maximise value for money. Stability of results across strata does not vitiate the concern.
It could be argued, however, that selection effects are likely to attenuate as the intervention is rolled out over an increasing proportion of the population. Let us try a thought experiment. Consider the finding that accident victims who receive a transfusion have worse outcomes than those who do not, even after risk-adjustment. Is this because transfusion is harmful, or because clinicians can spot those who need transfusion, net of variables captured in statistical models? Let us now suppose that, in response to the findings, clinicians subsequently reduce use of transfusion. It is then possible that changes in the control rate and in the treatment effect can provide evidence for or against cause and effect explanations. The problem here is that bias may change as the proportions receiving one treatment or the other changes. There are thus two possible explanations for any set of results – a change in bias or a change in effectiveness, as a wider range of patients/ clinicians receive the experimental intervention. It is difficult to come up with a convincing way to resolve the cause and effect problem. I must leave it to someone cleverer than myself to devise a theorem that might shed at least some light on the plausibility of the competing explanations – bias vs. cause and effect. But I am pessimistic for this general reason. As a treatment is rolled out (because it seems effective) or withdrawn (because it seems ineffective or harmful), so the beneficial or harmful effect (even in relative risk ratio terms) is likely to attenuate. But the bias is also likely to attenuate because less selection is taking place. Thus the two competing explanations may be confounded.
There is also the question of whether database studies can mitigate ‘survivor bias’. When the process (of machine learning) starts, then survivor bias may exist. But, by tracking estimated treatment effect over time, the machine can recognise all subsequent ‘eligible’ cases as they arise. This means that the problem of survivor bias should be progressively mitigated over time?
So what do I recommend? Three suggestions:
- Use machine learning to provide a clue to things that you might not have suspected or thought of as high priority for a trial.
- Nest RCTs within database studies, so that cause and effect can be established at least under specified circumstances, and then compare the results with what you would have concluded by machine learning alone.
- Use machine learning on an open-ended basis with no fixed stopping point or stopping rule, and make data available regularly to mitigate the risk of over-interpreting a random high. This approach is very different to the standard ‘trial’ with a fixed starting and end data, data-monitoring committees,[4] ‘data-lock’, and all manner of highly standardised procedures. Likewise, it is different to resource heavy statistical analysis, which must be done sparingly. Perhaps that is the real point - machine learning is inexpensive (has low marginal costs) once an ongoing database has been established, and so we can take a 'working approach', rather than a 'fixed time point' approach to analysis.
|
|
|
Which snake in India causes the greatest number of deaths?
Email CLAHRC WM your answer.
|
|
|
Answer to our previous quiz: The brain cells that control appetite are tanycytes. The research was led by Prof Nicholas Dale of the University of Warwick, and more information can be found here.
Congratulations to Sian Scogings who was first to answer correctly.
|
|
|
Learning from Excellence
Learning from Excellence (LfE) is a programme that was started at Birmingham Children's Hospital NHS Foundation Trust and has been running for just over three years. In contrast to traditional approaches to patient safety, which have sought to prevent future harm by learning from past errors, LfE seeks to capture, and in turn learn from, the many incidences of excellent practice that also occur alongside this. Learning from error tends to put preventative mechanisms in place, whereas LfE seeks to celebrate and promote new and innovate ways of working.
The programme uses the theory of Appreciative Inquiry to underpin engaging individuals and enacting change within the organisations. LfE is supported by Appreciating People and the West Midlands Patient Safety Collaborative (WM PSC) and has seen huge interest across the UK and abroad. LfE picked up a prestigious award at the Health Service Journal Patient Safety awards earlier in the summer in the Education and Training category. On the back of this interest and success, the WMPSC is holding a national conference in Birmingham on Thursday 16 November to showcase the work and discuss future evolution and roll*out of the programme, with over 400 people registered for the event to date.
NIHR CLAHRC West Midlands is delighted to be undertaking the evaluation of the programme, with Maartje Kletter and Celia Taylor from the University of Warwick working with the teams from Birmingham Children's Hospital, University Hospital Birmingham, and other centres around the country who have implemented LfE. Maartje will be presenting a poster of the evaluation at the conference, with a literature review completed and the fieldwork of surveying participants who have been trained in Appreciative Inquiry and who are implementing, or have already implemented, LfE about to be issued. The data will be analysed using the Kirkpatrick model, and a logic model is being developed to assist with this. The research team and wider CLAHRC representatives are looking forward to the event and will feedback progress on the project in a later News Blog.
-- Paul Bird, Head of Programme Delivery, CLAHRC WM
|
|
|
Director's Choice - From the Journals
A Very Interesting Paper Using Mendelian Randomisation to Determine the Effect of Extra Years of Education on Heart Disease
It turns out that there are a number of genes, all associated with aspects of neurodevelopment, that predict how many years a person will spend in formal education.[1] It is already very well established that more years of education are associated with large reductions in coronary heart disease (CHD) (mediated by behaviour such as lower calorie intake, less smoking, more exercise).[2] So the authors of a recent well-written and most interesting BMJ paper did the obvious thing.[3] [4] They related the (random) presence or absence of educational propensity genes to CHD. Bingo, they measured a large effect (the genes that predispose to larger durations of formal education associate with reduced CHD). Now, the thing with Mendelian randomisation is that the genotype must not be linked to the outcome (CHD in this case), other than through the putative explanatory variable (duration of education in this case). The authors are aware that it is quite possible that education genes are linked to the outcome (CHD), net of (any) effect on education. To deal with this possibility they perform sensitivity analyses. They examine the association of genetic variates associated with education and the behaviours that lead to CHD. If the effects on education and on CHD behaviours are similar across the genetic variates this suggests that the effect on CHD is through education and not through another variable. And so it was. They also looked to see whether genetic variants already known to be associated with CHD (genes for high cholesterol, etc.) were also associated with education. If the genes associated with education do not associate with these other risk factors, then that favours a cause and effect explanation. There was no association. However, such an association would only be expected if there was a ‘massive’ effect of ‘education genes’ that bypassed education.
This all falls short of proof. Since the educational genes lead to education through mental processes, it is reasonable to suppose that almost all genetic variates that affect education also affect behaviour. Thus, they would affect CHD, even if there was no extra education. The authors say that their conclusion is strongly supported by identical twin studies where one twin stayed longer in education than the other, but this too ignores the fact that these twins are different, for all that their inherited genotype is the same, and so these differences could be the cause of both increased education and decrease in the behaviours that lead to heart disease.
One more point – even if years of education really are causative, this might well apply only to people genetically predisposed to more education and may not apply among those not so predisposed – there may be an interaction between the genes that predisposes to education and response to that education. After all, why would one persist in the classroom if you were not predisposed to benefit from the experience? People not predisposed would find being coerced to do so most unpalatable, and such an approach could even have a perverse effect. This is an excellent article and is beautifully presented. But I am a little more sceptical than the authors. I would like to see a debate on the issues.
-- Richard Lilford, CLAHRC WM Director
Leave a comment
References
Is it Possible to Teach Empathy?
News blog readers will know that I am fascinated by the question of whether it is possible to teach people to be kinder, more patient-centered, and to show more empathy. A recent meta-analysis of RCTs sheds important light on the critical issue of empathy training.[1] Unlike previous systematic reviews, this study included only experimental studies. Overall, 19 studies met the inclusion criteria for the meta-analysis.
One important issue concerns how the endpoint was measured. In 11 of the 19 included studies the outcome was an objective measure, while in the remainder the outcome was self-reported.
Overall, educational interventions produced a positive benefit that was statistically significant. When the authors made an adjustment for possible publication bias, the effect size was only slightly reduced, remaining highly significant statistically.
I expected to find that the effect size was greater for the self-reported outcomes than for objective outcomes. In fact, the effect size was larger and more highly significant for the objective measures of effect.
Some people classify empathy training in two forms: cognitive and effective, to cover the intellectual and emotional aspects of empathy. Others have questioned this dichotomy, arguing that the emotional and the cognitive parts have to interact to produce empathetic behaviour. As it turned out, all studies included a cognitive component.
This is a very interesting and important study. My main problem with the study is that they do not give a breakdown according to whether the objective measure was self-reported or objective. Also, the results do not tell us how enduring the effects were. I have argued before that one of the main criteria of good communication and compassionate care is the desire to achieve these projectors. The most important thing to instil is a deep-seated desire to do a better job. It would seem that training has a part to play in achieving this objective. However, sustained exposure to excellent role models is also critically important and a crucial part of the education of health professionals.
-- Richard Lilford, CLAHRC WM Director
Leave a comment
Reference
Importance of Cohort Rather Than Cross-Sectional Studies
to Determine the Heritability of Conditions
In a recent News Blog we showed how longitudinal studies could improve on cross-sectional epidemiological studies using alcohol-induced effects on lexical cognition, as our example.[1]
Cross-sectional twin studies may also under-estimate inheritability of disorders, such as Autism Spectrum Disorder (ASD), because the control twin may be in a ‘yet to be diagnosed’ state. An interesting article in JAMA shows that the inheritability of ASD is much higher if the correct (longitudinal) method is used.[2] All studies agree that familial environment has hardly any effect on the probability of ASD.
-- Richard Lilford, CLAHRC WM Director
I have long argued for greater use of Bayesian interpretation of clinical trial results – I suggested this approach with respect to trials of treatment for rare diseases back in 1994.[1] This approach is now advocated in a recent report in JAMA.[2] The approach advocated is the use of a neutral, an enthusiastic, and a sceptical prior. The authors also outline an approach to be used when the data is not compatible with the prior; in such a scenario they advocate a ‘data wins’ rule. I would wish to ensure that the trial was of impeccable design with complete follow up before accepting such an approach. For instance, I would allow my sceptical prior to dominate positive results from a potentially biased trial of homeopathy.
-- Richard Lilford, CLAHRC WM Director
Leave a comment
References
Breastfeeding and SIDS
Over the years many studies have shown an association between breastfeeding and decreased risk of sudden infant death syndrome (SIDS), with a previous meta-analysis showing an adjusted odds ratio of 0.55 (95% CI 0.44-0.69), which increased to 0.27 (95% CI 0.24-0.31) with exclusive breastfeeding.[1] However, it has been difficult to identify just how long breastfeeding needs to continue to realise this benefit. This is because duration of breastfeeding has not been correlated with reduction in risk. As a follow-up to their original meta-analysis, Thompson and colleagues worked in cooperation with the authors of the included studies to obtain individual-level data.[2] They were able to glean information on duration of breastfeeding so that the association between duration and effect could be examined. In total 9,104 infants were analysed from eight case-control studies. Although analysis showed some protection against SIDS associated with any breastfeeding up to 2 months, this was not statistically significant after controlling for potential confounders. When confounders were controlled for, analysis found that any breastfeeding for at least 2 months, compared to no breastfeeding, had an adjusted odds ratio (aOR) of 0.60 (95% CI 0.44-0.82), while it was a similar aOR of 0.61 (95% CI 0.42-0.87) for exclusive breastfeeding. The aOR for any amount of breastfeeding compared to none improved with increased duration – an aOR of 0.40 (95% CI 0.26-0.63) with 4-6 months breastfeeding, and 0.36 (95% CI 0.22-0.61) with at least 6 months breastfeeding. A similar improvement was seen with at least 4 months of exclusive breastfeeding (aOR 0.46, 95% CI 0.29-0.74).
In order to lower the incidence of SIDS it is important that new mothers are encouraged to breastfeed and to continue for at least 2 months, even if they are unable to do so exclusively, as any amount of breastfeeding seems to confer more protection than none.
-- Peter Chilton, Research Fellow
Leave a comment
References
A Debt of Gratitude
For a while now I have been working closely with the African Population and Health Research Center (APHRC), which is based in Nairobi, Kenya. Last week’s issue of the Lancet carried an article on the APHRC, where they paid tribute to the outgoing director Alex Ezeh.[1] Alex had the wisdom to identify the enormous challenge posed by the rapidly expanding slums in African cities. He and his colleagues have produced ground-breaking work on health dynamics in urban slums. He was my inspiration, and I followed where he led. Together we compiled a Lancet Series summarising the state of the literature regarding the health of people living in the slums and proposing models to inform future research and policy making.[2] [3] These studies were recently summarised in the African version of 'The Conversation'.[4] Our work has resulted in the award of a NIHR unit to study the provision of healthcare in slums in Africa and Asia. We have also secured funds from the Rockefeller foundation to run a Bellagio conference on statistical aspects of slum health. Currently, we are pursuing research into water and sanitation in slums, as this is one of the biggest problems leading to diarrhoea, stunting and death, especially in children under the age of five.
I have an enormous debt of gratitude to APHRC in general and Alex Ezeh in particular. I look forward to my ongoing association with Alex and to working very closely with his outstanding successor, Catherine Kyobutungi, who was also profiled in last week’s Lancet.[5]
-- Richard Lilford, CLAHRC WM Director
Leave a comment
References
|
|
|
National NIHR CLAHRC Multi-Morbidity Research Event
Save the date - CLAHRC East Midlands are organising a national multi-morbidity research event to take place on Thursday 18 January 2018 at Stamford Court, Leicester. More information will be available closer to the date.
UK Evaluation Society Midlands Regional Network
The UK Evaluation Society (UKES) exists to promote and improve the theory, practice, understanding and utilisation of evaluation and its contribution to public knowledge and to promote cross-sector and cross-disciplinary dialogue and debate. The Society is launching a Midlands Regional Network that aims to link and support people in the region working in evaluation or with evaluation evidence by providing:
- A forum to debate and discuss evaluation theory and practice and disseminate evaluation findings.
- Specialist training and professional development opportunities for evaluation practitioners.
- Activities that raise the awareness and understanding of evaluation throughout the region.
- Opportunities for evaluators to network and share ideas and experiences.
The network will be strongly committed to supporting economic and social development of the region through working in collaboration with other analytical communities and public, private and third sectors in the generation of evidence by evaluation and its use.
A launch event is being planned for December or early new year. To register interest, please complete our joining survey or email Karl King. Other members of the steering group include Rebecca Riley (City REDI, University of Birmingham), Michelle Hollier (Winning Moves), Tracey Wond (University of Derby) and George Bramley (Institute of Applied Health Research, University of Birmingham).
The networks’ events will be open to all members and non-members of UKES interested in evaluation both within and outside the region. While the network is affiliated to the UK Evaluation Society there is no requirement to be a member of the Society. However, being a member of the Society would secure you significant discounts on training that the network plans to run in the region.
Grant Awarded to Study Virginia Mason Lean System
Nicola Burgess, Associate Professor at Warwick Business School has been awarded a £400,000 grant from the Health Foundation to evaluate the implementation of the Virginia Mason lean system. Nicola will be working in collaboration with WBS academics, including CLAHRC WM Deputy Director Prof Graeme Currie. For more information, please click here.
Job Opportunity
University College London are advertising for a Teaching Fellow in Applied Health Research, working within NIHR CLAHRC North Thames. This is a full-time position until 30 September 2019 in the first instance. The closing date for applications is 12 November 2017. For details and to apply, please click here.
Return to top
|
|
|
NIHR Fellowship Programme
Round 11 of the NIHR Fellowship Programme is now available, with five levels of awards: Doctoral, Post-Doctoral, Transitional Research, Career Development, and Senior Research. The NIHR Fellowships Panel are looking for individuals with expertise in clinical trials and statistics to join the panel. The deadline for applications to all post-doctoral levels is 13:00 on 5 December 2017. The deadline for applications to the doctoral level is 13:00 on 19 December 2017. For more information, and to apply, please click here.
Return to top
|
|
|
|
Janet Jones
Ms Janet Jones is a Research Fellow working for the CLAHRC WM Chronic Disease team, theme 4.
Janet started her working life in the telecoms industry before moving to the NHS. Whilst at the NHS Janet studied part-time at the Open University for an undergraduate degree in the History of Science, Medicine and Technology. This ultimately led to a job working at the University of Birmingham as a Trial coordinator. As a Trial coordinator Janet worked on several research studies including: the herbal medicine study –investigating the use of herbal medicines by cancer patients; the diet and physical activity study – assessing the practicality of altering the behaviour of patients recently diagnosed with high-risk colorectal adenomas; and the RECEPTS study (Receptionists recognition and referral of patients with stroke) – using a simulated patient methodology to document how GP receptionists behave when patients call the GP practice with symptoms of stroke.
In 2014 Janet commenced studying for a PhD and has just moved into the writing-up stage. The title of her PhD is: Core outcome set development: Understanding how qualitative research approaches can help to accommodate patient preferred outcomes in trial research.
In her current role Janet is working on an evaluation of the Walsall and Wolverhampton Care Home Improvement Programme. Interviews and focus groups with care home managers and staff will be carried out to capture information on changes to safety practices at care home and individual level.
Return to top
|
|
|
|
Re: Patient and Public Involvement: Direct Involvement of
Patient Representatives in Data Collection
It still amazes me that, compared to almost any other industry, the healthcare industry makes so very little use of my ability as a patient to enter data about myself directly onto a computer system.
What I might enter about myself may not always be as accurate as that carefully documented by the all knowing, highly trained professional, but if so why, everywhere I go, do I have to fill in yet another form, for someone, often funded by the taxpayer, so that, if the secretary can read my writing, he/she can enter then the medical and social information I have supplied on paper into yet another incompatible electronic database.
-- Rupert Fawdry
Re: Another Study on the Hazards of American Football
I was wondering this ["some of the head trauma is consequent of the... celebrations"] the other day – after every play they seem to forcibly crack heads. There have been numerous celebration-related injuries. A particularly difficult subset of head trauma to study though...
-- Kim Sein
Re: Fair is Fair: Preventing the Misuse of Visiting Hours to Reduce Inequities
I was shocked by the comment in your otherwise excellent piece on ITU visiting that “Ethnic minority or migrant families bring with them different cultural norms and behaviours which may impact adversely on the family members of indigenous patients.”
Isn’t this an example of the very thing you commented on earlier – i.e. bias (or even ‘institutional’ racism)?
Why did you not say “Dominant culture families (whether by class/ education/ race) bring with them cultural norms and behaviours that may impact adversely on the family members of ethnic minority or migrant patients.” – maybe we need to think about mechanisms that explain the inverse care law?
-- Susan Bewley
--Director's Reply-- Thanks Susan. I think the point that was being made was that one family may behave in a way that, without violating its own cultural norms, may nevertheless offend another family. Sure, this kind of interaction is bi-directional and I am sure the authors do not mean to imply otherwise.
-- Richard Lilford
Re: Calling All Men - Screening for Prostate Cancer...
Why is ’cause-specific death’ rather than ‘all-cause’ death as used in screening considered sufficient? It isn’t. All medical interventions cause harm (whether anxiety/ opportunity costs or mortality and morbidity, let alone the special harms of ‘overdiagnosis’). Some cause adequate benefit (ie the total benefit minus the harm). When it comes to our bank accounts we know we have to count money both in and out. Did I miss something or are we letting screening proponents get away with this sleight of hand?
-- Susan Bewley
--Author's Reply-- Thank you. There are many reasons to prefer all-cause death, as discussed in a previous News Blog. One is that when a person dies with disseminated cancer, this is more likely to be attributed to a specific organ when an early cancer has been detected in that organ.
-- Richard Lilford
Return to top
|
|
|
|
Brett J, Staniszewska S, Simera I, Seers K, Mockford C, Goodlad S, Altman D, Moher D, Barber R, Denegri S, Entwistle AR, Littlejohns P, Morris C, Suleman R, Thomas V, Tysall C. Reaching consensus on reporting patient and public involvement (PPI) in research: methods and lessons learned from the development of reporting guidelines. BMJ Open. 2017; 7(10): e016948.
Butterworth S, Singh SP, Birchwood M, Islam Z, Munro ER, VOstanis P, Paul M, Khan A, Simkiss D. Transitioning care-leavers with mental health needs: ‘they set you up to fail!’ Child Adolesc Ment Health. 2017; 22(3): 138-47.
Cooper JA, Parsons N, Stinton C, Matthews C, Smith S, Halloran SP, Moss S, Taylor-Phillips S. Risk-adjusted colorectal cancer screening using the FIT and routine screening data: development of a risk prediction model. Br J Cancer. 2017.
The Epidural and Position Trial Collaborative Group. Upright versus lying down position in second stage of labour in nulliparous women with low dose epidural: BUMPES randomised controlled trial. BMJ. 2017; 359: j4471.
Farre A, McDonagh JE. Helping Health Services to Meet the Needs of Young People with Chronic Conditions: Towards a Developmental Model for Transition. Healthcare. 2017; 5(4): 77.
Farre A, Shaw K, Health G, Cummins C. On doing ‘risk work’ in the context of successful outcomes: exploring how medication safety is brought into action through health professionals’ everyday working practices. Health Risk Soc. 2017; 19(3-4): 209-25.
Finnikin S, Ryan R, Marshall T. Statin initiations and QRISK2 scoring in UK general practice: a THIN database study. Br J Gen Pract. 2017
Kidney E, Greenfield S, Berkman L, Dowswell G, Hamilton W, Wood S, Marshall T. Cancer suspicion in general practice, urgent referral, and time to diagnosis: a population-based GP survey nested within a feasibility study using information technology to flag-up patients with symptoms of colorectal cancer. BJGP Open. 2017; 1(3).
Taylor B, Mathers J, Parry J. Who are community health workers and what do they do? Development of an empirically derived reporting taxonomy. J Public Health. 2017.
Return to top
|
|
|
|
|
|
|
|
|
