Trends in 21st Century Epidemiology: From Scientific Discoveries to Population Health Impact

John P.A. Ioannidis: The Role of Epidemiology in Knowledge Integration and Meta-Research

Slide 1 of 36: The Role of Epidemiology in Knowledge Integration and Meta-Research

John P.A. Ioannidis
C.F. Rehnborg Professor in Disease Prevention
Professor of Medicine and Professor of Health Research and Policy
Director, Stanford Prevention Research Center
Stanford University School of Medicine
Professor of Statistics (by courtesy)
Stanford University School of Humanities and Sciences


Slide 2 of 36

[Image] showing the continuum of translational research from discovery to reducing the burden of disease in a population. The innermost circle is "Knowledge Integration." Three circles surround the inner circle: KS, KM, and KT, which represent "Knowledge Synthesis," "Knowledge Management," and "Knowledge Translation," respectively. The outermost layer contains five figures: "Discoveries," "Clinical Applications," "Evidence-Based Recommendations," "Use in Practice," and "Population Health Impact." Two double-sided arrows connect each box to the boxes beside it and form the outermost ring.

Source: Ioannidis JP, Schully SD, Lam TK, and Khoury MJ. Cancer Epidemiol Biomarkers Prev. 2013; 22(1):3-10. Copyright © 2013, American Association for Cancer Research.


Slide 3 of 36

[Image] showing the breakdown of published articles in cancer literature by type. There are roughly 2.7 million articles total, and the table shows that there is overlap between the literature on associations and the literature on treatments and interventions. Articles with relevance to treatment (approx. 1.4 million total) far outnumber articles with relevance to prevention (approx. 200,000 total) across all subsets.

Source: Ioannidis JP, Schully SD, Lam TK, and Khoury MJ. Cancer Epidemiol Biomarkers Prev. 2013; 22(1):3-10. Copyright © 2013, American Association for Cancer Research.


Slide 4 of 36

[Image] showing different types of approaches to knowledge management and knowledge synthesis, as well as methods of both that use multiple levels of information.

Source: Ioannidis JP, Schully SD, Lam TK, and Khoury MJ. Cancer Epidemiol Biomarkers Prev. 2013; 22(1):3-10. Copyright © 2013, American Association for Cancer Research.


Slide 5 of 36

[Image] showing number of systematic reviews and meta-analyses in different subfields of cancer (excluding trials and treatment). Within each subfield, it also shows the number of systematic reviews (SR) per meta-analysis (MA) and the total number (n) of articles per meta-analysis. Although subsets with the greatest number of articles published are "gene/genome/genetic" and "epigenetic/methylation/mutation, the subsets with the highest ratios of SR/MA and n/MA were "occupation/occupational" and "smoking/smoke/tobacco."

Source: Ioannidis JP, Schully SD, Lam TK, and Khoury MJ. Cancer Epidemiol Biomarkers Prev. 2013; 22(1):3-10. Copyright © 2013, American Association for Cancer Research.


Slide 6 of 36: Examples of Knowledge Integration at the Meta-Research Level

  • Associations
  • Predictions
  • Treatments

Slide 7 of 36: Nutrients and Cancer Risk

  • Open a popular cookbook
  • Randomly check 50 ingredients
  • How many of those are associated with significantly increased or significantly decreased cancer risk in the scientific literature?

Slide 8 of 36: Associated with Cancer Risk

  • veal, salt, pepper spice, flour, egg, bread, pork, butter, tomato, lemon, duck, onion, celery, carrot, parsley, mace, sherry, olive, mushroom, tripe, milk, cheese, coffee, bacon, sugar, lobster, potato, beef, lamb, mustard, nuts, wine, peas, corn, cinnamon, cayenne, orange, tea, rum, raisin

Slide 9 of 36

[Image] showing the effect estimates reported in the literature on associations between specific foods and cancer risk by malignancy type (top) or ingredient (bottom). Only ingredients with ≥10 studies are shown. Gastrointestinal malignancies were the most commonly studied (45%), followed by genitourinary (14%), breast (14%), head and neck (9%), lung (5%), and gynecologic (5%) malignancies.

Source: Schoenfeld JD and Ioannidis JPA, Am J Clin Nutr. 2013; 97(1):127-134.


Slide 10 of 36

[Image] showing the distribution of standardized z-scores among single studies and meta-analyses of associations between specific foods and cancer risk. Scores available from article abstracts are shown in black, whereas those found in the full text are in gray; 62% of the nominally statistically significant effect estimates were reported in abstracts, whereas most (70%) of the nonsignificant results appeared only in the full text and not in the abstracts.

Source: Schoenfeld JD and Ioannidis JPA, Am J Clin Nutr. 2013; 97(1):127-134.


Slide 11 of 36: Prognostic Tumor Markers

[Image] showing two pie charts which represent the percentage of negative vs. positive articles from two publication databases. Positive articles were publications where the abstract reported any statistically significant prognostic effect for any marker and any outcome. Negative articles reported no such effects. The vast majority of articles (91% in Database 1 and 96% in Database 2 ) reported significant effects for the tumor markers they studied.

Source: Adapted from Kyzas PA, Denaxa-Kyza D, and Ioannidis JPA. Eur J Cancer. 2007; 43(7): 2559-2578.


Slide 12 of 36: Further analysis of claims in "negative" prognostic studies

[Image] showing an analysis of the negative prognostic studies represented in the previous chart. Of the 32 negative prognostic articles in Database 1, 27 of them used one of the 3 mechanisms (claiming significance for other analyses, expanding on non-significant trends, and offering some other apologies) and 10 of them used more than one of the three mechanisms. Of the 66 negative prognostic articles in Database 2, 45 of them used one of the three mechanisms and 16 of them used more than one of the three mechanisms.

Source: Adapted from Kyzas PA, Denaxa-Kyza D, and Ioannidis JPA. Eur J Cancer. 2007; 43(7): 2559-2578.


Slide 13 of 36: Effect sizes for the top-cited biomarkers in the biomedical literature

[Image] showing an analysis of the most highly-cited papers on biomarkers. The Y axis shows the relative risk in the largest study and the X axis shows the relative risk in the highly-cited study. Because most of the points in this scatter plot fall below the diagonal, this chart shows that in studies with large populations, the estimates of the effect are much smaller than in the similar, highly-cited studies that preceded them.

Source: Adapted from Ioannidis JPA, Panaigotou OA. JAMA. 2011; 305(21):2200-2210.


Slide 14 of 36

[Image] showing unadjusted hazard ratios and confidence intervals for individual studies that examined clinical outcome prediction by MicroRNAs in human cancer.

Source: Nair VS, Maeda LS, and Ioannidis JPA. Clinical outcome prediction by microRNAs in Human Cancer: A systematic review. J Natl Cancer Inst. 2012; 104(7): 528-540. Used by permission of Oxford University Press.


Slide 15 of 36: Field Synopses

Chatzinasiou F, Lill CM, Kypreou K, et al. Comprehensive Field Synopsis and Systematic Meta-analyses of Genetic Association Studies in Cutaneous Melanoma. J Natl Cancer Inst. 2011; 103(16): 1227-1235.


Slide 16 of 36

Published Genetic Meta-Analyses by Country, Year

[Image] showing the number of genetic meta-analyses published per year for 10 different countries for the years 2001-2012. During the years 2009-2012, China produced 2-4 times more of these meta-analyses than the United States.

Source: Ioannidis JPA, Chang C, Lam TK, Schully SD, and Khoury MJ. Submitted to Cancer Epidemiol Biomarkers Prev, 2013.


Slide 17 of 36: Replicated: only 6 of 53 landmark studies for Amgen oncology drug target projects

"The failure to win "the war on cancer" has been blamed on many factors, ...But recently a new culprit has emerged: too many basic scientific discoveries... are wrong."

Source: Begley S. Reuters. March 25, 2012.


Slide 18 of 36

700 randomized trials on advanced breast cancer:
it all works (more or less)

[Image] showing a network of all available randomized trial studies and the comparisons of methods that they examine for treating advanced breast cancer. The figure shows a fragmented literature where methods are tested in isolation, with few comparisons to determine the relative efficacy across numerous methods.

Source: Adapted from Mauri D, Polyzos NP, Salanti G, et al. J Natl Cancer Inst. 2008; 100(24):1780-1791.


Slide 19 of 36: 1200 (and counting) clinical trials of bevacizumab

[Image] showing a forest plot of hazard ratios and confidence intervals for the largest phase-3 trials that looked at survival rates with bevacizumab among cancer patients.

Source: Ioannidis JPA and Karassa FB. BMJ. 2010; 340:c4875.


Slide 20 of 36

Pereira TV, Horwitz RI, and Ioannidis JPA. Empirical Evaluation of Very Large Treatment Effects of Medical Interventions. JAMA. 2012; 308(16):1676-1684.


Slide 21 of 36

[Image] showing a scatter plot of the odds ratios of the initial trial in the medical literature (X axis) vs the odds ratios of subsequent random-effects meta-analyses (Y axis). Because most of the points in this scatter plot fall below the diagonal, this chart shows that most large treatment effects emerge from small studies, and when additional trials are performed, the effect sizes become typically much smaller.

Source: Pereira TV, Horwitz RI, and Ioannidis JPA. JAMA. 2012; 308(16):1676-1684.


Slide 22 of 36: Learning to live with small/tiny effects

Source: Siontis GCM and Ioannidis JPA. Risk factors and interventions with statistically significant tiny effects. Int J Epidemiol. 2011; 40(5):1292-1307.


Slide 23 of 36: Large-scale collaboration

[Image] of Manhattan plots showing significance of association of all SNPs in the TAG (Tobacco and Genetics) Consortium meta-analyses for four smoking phenotypes.

Reprinted by permission from Macmillan Publishers Ltd: Tobacco and Genetics Consortium. Nat Genet. 2010; 42(5):441-447.


Slide 24 of 36: Taking Multiplicity into Account

[Image] of a Manhatten plot showing the significance of association for each of 188 environmental toxic exposures in association to triglycerides (first chart), LDL-C (second chart), and HDL-C (third chart).

Adapted from Tzoulaki I, Patel CJ, Okamura T, et al. Circulation. 2012; 126(21):2456-2464.


Slide 25 of 36: Nutrient-Wide Association Study

[Image] showing the nutrient-wide associations with systolic blood pressure levels for nutrients received from foods and urine excretion markers, and for nutrients received from foods and supplements.

Source: Tzoulaki I, Patel CJ, Okamura T, et al. Circulation. 2012; 126(21):2456-2464.


Slide 26 of 36: Validated heatmaps for nutrients

[Image] of two square heatmaps showing the correlations of Pearson coefficients between nutrients. The left heatmap represents data from the International Study of Macro/Micronutrients and Blood Pressure (INTERMAP), and the right represents data from the National Health and Nutrition Examination Survey (NHANES). The heatmaps show a dense correlation pattern among validated nutrients, particularly among fiber, copper, magnesium, and folacin.

Source: Tzoulaki I, Patel CJ, Okamura T, et al. Circulation. 2012; 126(21):2456-2464.


Slide 27 of 36: Improving research reporting standards: STROBE-ME, 2011

Source: Gallo V, Egger M, McCormack V, et al. STrengthening the Reporting of OBservational studies in Epidemiology - Molecular Epidemiology (STROBE-ME): An Extension of the STROBE Statement. PLoS Med. 2011; 8(10).


Slide 28 of 36: Improving research reporting standards: GRIPS, 2011

Source: Janssens AC, Ioannidis JPA, van Duijn CM, et al. Strengthening the reporting of Genetic Risk Prediction Studies: the GRIPS statement. PLoS Med. 2011; 8(3).


Slide 29 of 36: Registration

Source: Ioannidis JPA. The Importance of Potential Studies That Have Not Existed and Registration of Observational Data Sets. JAMA. 2012; 308(6): 575-576.


Slide 30 of 36: Levels of Registration

  • Level 0: no registration
  • Level 1: registration of dataset
  • Level 2: registration of protocol
  • Level 3: registration of analysis plan
  • Level 4: registration of analysis plan and raw data
  • Level 5: open live streaming

Slide 31 of 36

[Image] showing a breakdown of policies for individual journals for public deposition of certain data types, sharing of materials and/or protocols, and whether this is a condition for publication and percentage of papers with fully deposited data. In general, journals with a higher impact factor have more policies that encourage data-sharing.

Source: Alsheikh-Ali AA, Qureshi W, Al-Mallah MH, et al. Public Availability of Published Research Data in High-Impact Journals. PLoS One. 2011; 6(9).


Slide 32 of 36: Repeatability

Source: Ioannidis JPA, Allison DB, Ball CA, et al. Repeatability of published microarray gene expression analyses. Nat Genet. 2009; 41(2):149-155.


Slide 33 of 36

[Image] of an exploded pie chart that shows the results of various efforts to replicate published microarray gene expression analyses. Of the 18 articles evaluated, 10 could not be reproduced, and several of the articles that could be reproduced included discrepancies.

Reprinted by permission from Macmillan Publishers Ltd.: Ioannidis JPA, Allison DB, Ball CA, et al. Nat Genet. 2009; 41(2):149-155.


Slide 34 of 36

ref Improving Validation Practices in "Omics" Research

Source: Ioannidis JPA and Khoury MJ. Improving Validation Practices in "Omics" Research. Science. 2011; 334(6060):1230-1232.


Slide 35 of 36

[Image] showing six steps for validating "omics" researcj for use in medicine and public health: analytic validity, repeatability, replication, external validation, clinical validity, and clinical utility.

Source: Ioannidis JPA and Khoury MJ. Science. 2011; 334(6060):1230-1232.


Slide 36 of 36

[Image] showing suggestions for the future of knowledge integration, separated into three subgroups: knowledge management, knowledge synthesis, and knowledge translation.

Source: Ioannidis JP, Schully SD, Lam TK, and Khoury MJ. Cancer Epidemiol Biomarkers Prev. 2013; 22(1):3-10. Copyright © 2013, American Association for Cancer Research.

Return to Top