RCTs in criminology, education, and international development

Criminology

A 2005 review found 83 randomized experiments in criminology published in 1982-2004, compared with only 35 published in 1957-1981. [ 80 ] The authors classified the studies they found into five categories: "policing", "prevention", "corrections", "court", and "community". [ 80 ] Focusing only on offending behavior programs, Hollin (2008) argued that RCTs may be difficult to implement (e. g. if an RCT required "passing sentences that would randomly assign offenders to programmes") and therefore that experiments with quasi-experimental design are still necessary. [ 81 ]

Education

RCTs have been used in evaluating a number of educational interventions. For example, a 2009 study randomized 260 elementary school teachers' classrooms to receive or not receive a program of behavioral screening, classroom intervention, and parent training, and then measured the behavioral and academic performance of their students. [ 82 ] Another 2009 study randomized classrooms for 678 first-grade children to receive a classroom-centered intervention, a parent-centered intervention, or no intervention, and then followed their academic outcomes through age 19. [ 83 ]

International development

RCTs are currently being used by a number of international development experts to measure the impact of development interventions worldwide. Development economists at research organizations including Abdul Latif Jameel Poverty Action Lab [ 84 ] [ 85 ] and Innovations for Poverty Action [ 86 ] have used RCTs to measure the effectiveness of poverty, health, and education programs in the developing world. While RCTs can be useful in policy evaluation, it is necessary to exercise care in interpreting the results in social science settings. For example, interventions can inadvertently induce socioeconomic and behavioral changes that can confound the relationships (Bhargava, 2008).

For some development economists, the main benefit to using RCTs compared to other research methods is that randomization guards against selection bias, a problem present in many current studies of development policy. In one notable example of a cluster RCT in the field of development economics, Olken (2007) randomized 608 villages in Indonesia in which roads were about to be built into six groups (no audit vs. audit, and no invitations to accountability meetings vs. invitations to accountability meetings vs. invitations to accountability meetings along with anonymous comment forms). [ 87 ] After estimating "missing expenditures" (a measure of corruption ), Olken concluded that government audits were more effective than "increasing grassroots participation in monitoring" in reducing corruption. [ 87 ] However, similar conclusions can also be reached by suitable modeling of the data from longitudinal studies. Overall, it is important in social sciences to account for the intended as well as the unintended consequences of interventions for policy evaluations.

Disadvantages

Many papers discuss the disadvantages of RCTs. [ 59 ] [ 60 ] Among the most frequently-cited drawbacks are:

Limitations of external validity

The extent to which RCTs' results are applicable outside the RCTs varies; that is, RCTs' external validity may be limited. [ 59 ] [ 61 ] Factors that can affect RCTs' external validity include: [ 61 ]

Where the RCT was performed (e. g. what works in one country may not work in another) Characteristics of the patients (e. g. an RCT may include patients whose prognosis is better than average, or may exclude "women, children, the elderly, and those with common medical conditions" [ 62 ] ) Study procedures (e. g. in an RCT patients may receive intensive diagnostic procedures and follow-up care difficult to achieve in the "real world") Outcome measures (e. g. RCTs may use composite measures infrequently used in clinical practice) Incomplete reporting of adverse effects of interventions

Costs

RCTs can be expensive; [ 60 ] one study found 28 Phase III RCTs funded by the National Institute of Neurological Disorders and Stroke prior to 2000 with a total cost of US$335 million, [ 63 ] for a mean cost of US$12 million per RCT. Nevertheless, the return on investment of RCTs may be high, in that the same study projected that the 28 RCTs produced a "net benefit to society at 10-years" of 46 times the cost of the trials program, based on evaluating a quality-adjusted life year as equal to the prevailing mean per capita gross domestic product. [ 63 ]

Time

The conduction of a RCT takes several years until being published, thus data is restricted from the medical community for long years and may be of less relevance at time of publication. [ 64 ]

Relative importance of RCTs and observational studies

Two studies published in The New England Journal of Medicine in 2000 found that observational studies and RCTs overall produced similar results. [ 65 ] [ 66 ] The authors of the 2000 findings cast doubt on the ideas that "observational studies should not be used for defining evidence-based medical care" and that RCTs' results are "evidence of the highest grade." [ 65 ] [ 66 ] However, a 2001 study published in Journal of the American Medical Association concluded that "discrepancies beyond chance do occur and differences in estimated magnitude of treatment effect are very common" between observational studies and RCTs. [ 67 ]

Two other lines of reasoning question RCTs' contribution to scientific knowledge beyond other types of studies:

If study designs are ranked by their potential for new discoveries, then anecdotal evidence would be at the top of the list, followed by observational studies, followed by RCTs. [ 68 ] RCTs may be unnecessary for treatments that have dramatic and rapid effects relative to the expected stable or progressively worse natural course of the condition treated. [ 59 ] [ 69 ] One example is combination chemotherapy including cisplatin for metastatic testicular cancer. which increased the cure rate from 5% to 60% in a 1977 non-randomized study. [ 69 ] [ 70 ]

Difficulty in studying rare events

Interventions to prevent events that occur only infrequently (e. g. sudden infant death syndrome ) and uncommon adverse outcomes (e. g. a rare side effect of a drug) would require RCTs with extremely large sample sizes and may therefore best be assessed by observational studies. [ 59 ]

Difficulty in studying outcomes in distant future

It is costly to maintain RCTs for the years or decades that would be ideal for evaluating some interventions. [ 59 ] [ 60 ]

Pro-industry findings in industry-funded RCTs

Some RCTs are fully or partly funded by the health care industry (e. g. the pharmaceutical industry ) as opposed to government, nonprofit, or other sources. A systematic review published in 2003 found four 1986-2002 articles comparing industry-sponsored and nonindustry-sponsored RCTs, and in all the articles there was a correlation of industry sponsorship and positive study outcome. [ 71 ] A 2004 study of 1999-2001 RCTs published in leading medical and surgical journals determined that industry-funded RCTs "are more likely to be associated with statistically significant pro-industry findings." [ 72 ] One possible reason for the pro-industry results in industry-funded published RCTs is publication bias. [ 72 ]

Therapeutic misconception

Although subjects almost always provide informed consent for their participation in an RCT, studies since 1982 have documented that many RCT subjects believe that they are certain to receive treatment that is best for them personally; that is, they do not understand the difference between research and treatment. [ 73 ] [ 74 ] Further research is necessary to determine the prevalence of and ways to address this "therapeutic misconception ". [ 74 ]

Narrowing of the studied question

Randomized clinical trials are usually only inspect one variable or very few variables, rarely reflecting the full picture of a complicated medical situation; whereas the case report. for example, can detail many different aspects of the patient’s medical situation (e. g. patient history, physical examination. diagnosis. psychosocial aspects, follow up). [ 64 ]

Statistical error

RCTs are subject to both type I ("false positive") and type II ("false negative") statistical errors. Regarding Type I errors, a typical RCT will use 0.05 (i. e. 1 in 20) as the probability that the RCT will falsely find two equally effective treatments significantly different. [ 75 ] Regarding Type II errors, despite the publication of a 1978 paper noting that the sample sizes of many "negative" RCTs were too small to make definitive conclusions about the negative results, [ 76 ] by 2005-2006 a sizeable proportion of RCTs still had inaccurate or incompletely-reported sample size calculations. [ 77 ]

Cultural effects

The RCT method creates cultural effects that have not been well understood. [ 78 ] For example, patients with terminal illness may attempt to join trials as a last ditch attempt at treatment, even when treatments are unlikely to be successful.

Conflict of interest dangers

A 2011 study done to disclose possible conflicts of interests in underlying research studies used for medical meta-analyses reviewed 29 meta-analyses and found that conflicts of interests in the studies underlying the meta-analyses were rarely disclosed. The 29 meta-analyses included 11 from general medicine journals; 15 from specialty medicine journals, and 3 from the Cochrane Database of Systematic Reviews. The 29 meta-analyses reviewed an aggregate of 509 randomized controlled trials (RCTs). Of these, 318 RCTs reported funding sources with 219 (69%) industry funded. 132 of the 509 RCTs reported author conflict of interest disclosures, with 91 studies (69%) disclosing industry financial ties with one or more authors. The information was, however, seldom reflected in the meta-analyses. Only two (7%) reported RCT funding sources and none reported RCT author-industry ties. The authors concluded “without acknowledgment of COI due to industry funding or author industry financial ties from RCTs included in meta-analyses, readers’ understanding and appraisal of the evidence from the meta-analysis may be compromised.” [ 79 ]

Advantages

RCTs are considered by most to be the most reliable form of scientific evidence in the hierarchy of evidence that influences healthcare policy and practice because RCTs reduce spurious causality and bias. Results of RCTs may be combined in systematic reviews which are increasingly being used in the conduct of evidence-based medicine. Some examples of scientific organizations' considering RCTs or systematic reviews of RCTs to be the highest-quality evidence available are:

As of 1998, the National Health and Medical Research Council of Australia designated "Level I" evidence as that "obtained from a systematic review of all relevant randomised controlled trials" and "Level II" evidence as that "obtained from at least one properly designed randomised controlled trial." [ 48 ] Since at least 2001, in making clinical practice guideline recommendations the United States Preventive Services Task Force has considered both a study's design and its internal validity as indicators of its quality. [ 49 ] It has recognized "evidence obtained from at least one properly randomized controlled trial" with good internal validity (i. e. a rating of "I-good") as the highest quality evidence available to it. [ 49 ] The GRADE Working Group concluded in 2008 that "randomised trials without important limitations constitute high quality evidence." [ 50 ] For issues involving "Therapy/Prevention, Aetiology/Harm," the Oxford Centre for Evidence-based Medicine]] as of 2009 defined "Level 1a" evidence as a systematic review of RCTs that are consistent with each other, and "Level 1b" evidence as an "individual RCT (with narrow Confidence Interval )." [ 51 ]

Notable RCTs with unexpected results that contributed to changes in clinical practice include:

After Food and Drug Administration approval, the antiarrhythmic agents flecainide and encainide came to market in 1986 and 1987 respectively. [ 52 ] The non-randomized studies concerning the drugs were characterized as "glowing", [ 53 ] and their sales increased to a combined total of approximately 165,000 prescriptions per month in early 1989. [ 52 ] In that year, however, a preliminary report of a RCT concluded that the two drugs increased mortality. [ 54 ] Sales of the drugs then decreased. [ 52 ] Prior to 2002, based on observational studies, it was routine for physicians to prescribe hormone replacement therapy for post-menopausal women to prevent myocardial infarction. [ 53 ] In 2002 and 2004, however, published RCTs from the Women's Health Initiative claimed that women taking hormone replacement therapy with estrogen plus progestin had a higher rate of myocardial infarctions than women on a placebo, and that estrogen-only hormone replacement therapy caused no reduction in the incidence of coronary heart disease. [ 45 ] [ 55 ] Possible explanations for the discrepancy between the observational studies and the RCTs involved differences in methodology, in the hormone regimens used, and in the populations studied. [ 56 ] [ 57 ] The use of hormone replacement therapy decreased after publication of the RCTs. [ 58 ]

Reporting of RCT results

The CONSORT 2010 Statement is "an evidence-based, minimum set of recommendations for reporting RCTs." [ 47 ] The CONSORT 2010 checklist contains 25 items (many with sub-items) focusing on "individually randomised, two group, parallel trials" which are the most common type of RCT. [ 1 ] For other RCT study designs, "CONSORT extensions" have been published. [ 1 ]