Meta-analysis steps

This page walks through the steps of conducting a meta-analysis. For an introduction to what a systematic review and a meta-analysis are, see What is a systematic review / meta-analysis?

Defining the research area

The first step in conducting a meta-analysis implies defining very clearly the focus of the meta-analysis. What is the research question addressed by the meta-analytic review? For instance, we may want to study the efficacy of an intervention aimed at reducing hypertension, or gender differences in depression, or associations between physical exercise and body weight. More generally, in the meta-analysis we can compare differences between two groups or examine associations between two variables.

Defining inclusion and exclusion criteria

Inclusion and exclusion criteria define which studies are eligible or not for being included in the meta-analysis. They refer to:

• characteristics of the study (e.g., population, design)
• characteristics of the publication (e.g., language, type, year of publication)

Searching and selecting primary studies

In order to retrieve all the relevant literature it is necessary to use multiple search strategies. Main search strategies include:

• Search in reference databases (e.g., PsycINFO, ERIC, MEDLINE, EMBASE, Scopus, Web of Science, Dissertation abstract, etc.)
• Search in the reference list of reviews available on the same topic
• Search in the reference list of pertinent primary studies
• Search in indexes of journals that publish most papers on the topic of the meta-analysis (especially useful to find articles in press)
• Contacts with experts in the field

After having conducted the search, it is necessary to check each retrieved reference to see if it matches inclusion criteria. In this way, it is possible to identify the primary studies to be included in the meta-analysis.

Coding primary studies

Coding is the process by which primary studies are examined in order to extract relevant data to perform the meta-analysis. The coding protocol serves as a guide to the coding procedure.

Assessing study quality

Before pooling results, it is good practice to assess the methodological quality (risk of bias) of each primary study, using a tool appropriate to the study design (e.g., Cochrane RoB 2 for randomized trials, the Newcastle-Ottawa Scale for observational studies). See Assessing study quality and risk of bias for the main tools and how quality feeds into the analysis.

Computing effect sizes

For each study, it is necessary to compute an effect size, its variance, standard error, and confidence interval. The effect size is a measure of the magnitude of a relationship between two variables or a difference between groups. Main types of effect sizes are based on:

• means (Cohen's d, Hedges' g, raw unstandardized difference)
• binary data (risk ratio, odds ratio, risk difference)
• correlations (Pearson's correlations, Fisher's Z)
• survival data (hazard ratio)

The variance, standard error, and confidence interval provide an estimate of the precision of an effect size. The best way of reporting these results is through a forest plot — a plot of effect sizes (with confidence intervals) of all the studies included in the meta-analysis. See Methods and formulas for the exact effect-size formulas.

Aggregating effect sizes

After having computed an effect size for each study, it is possible to compute an overall effect size by combining effect sizes by means of:

• Fixed-effect model: assumes there is a true effect size common to all studies. In assigning a weight to each study, it takes into account only one source of variance: the within-study variance.
• Random-effects model: assumes the true effect size is normally distributed. In assigning a weight to each study, it takes into account two sources of variance: within-study variance and between-studies variance.

See Statistical models for how each model computes weights and the pooled effect.

Assessing heterogeneity

Heterogeneity across study effect sizes can be assessed through two statistics:

• Q statistic: used for establishing if there is significant heterogeneity across studies.
• I²: used to quantify heterogeneity; it estimates the proportion of observed variance that reflects real differences in effect sizes.

See Methods and formulas for the Q and I² formulas.

Testing moderators

Moderators (or predictors) are factors assumed to affect the magnitude of the effect sizes across the studies in which they are present. If the moderator is categorical, its effect is tested by a subgroup analysis; if the moderator is continuous, its effect is tested by a meta-regression.

Evaluating publication bias

Publication bias exists when published studies (those that can be easily retrieved) differ systematically from unpublished studies (gray literature). In meta-analysis, the potential impact of publication bias can be assessed through different methods:

• funnel plot
• Egger's linear regression method
• Begg and Mazumdar's rank correlation method
• Duval and Tweedie's Trim and Fill method
• Rosenthal's Fail-safe N — see Methods and formulas for its formula

Publishing a meta-analysis

In order to publish a high-quality meta-analysis it is useful to refer to meta-analysis reporting standards. In particular, various guidelines are currently available in different research fields:

• MARS (Meta-Analysis Reporting Standards), included in the Publication Manual of the American Psychological Association, 6th ed. (2010)
• PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses): a Statement, Explanation, Checklist, and Flow diagram — see PRISMA flow diagram and reporting guideline for how to read it
• MOOSE (Meta-analysis Of Observational Studies in Epidemiology; Stroup et al., JAMA 2000)