Assessing study quality and risk of bias

Not all primary studies are equally trustworthy. Before (or alongside) computing effect sizes, a meta-analysis should assess how much each study's design and conduct could have biased its results — a step distinct from publication bias (see Methods and formulas), which concerns which studies exist in the literature at all, not how well any single one was run.

Risk-of-bias tools

Different study designs need different tools:

Cochrane RoB 2 (Sterne et al., 2019) — the standard tool for randomized controlled trials. It walks through five places bias can creep in: how participants were randomized and allocated, whether the intervention was actually delivered as planned, how missing outcome data were handled, how the outcome was measured, and whether the reported result was selectively chosen from multiple analyses.
Newcastle-Ottawa Scale (NOS) — for non-randomized and observational studies (cohort, case-control), scoring selection of groups, comparability, and ascertainment of exposure/outcome.
PEDro scale — widely used in physiotherapy and rehabilitation trials.
JBI critical appraisal checklists — a family of tools covering many study designs (case reports, cross-sectional studies, qualitative research, etc.).

Each tool typically rates a study per domain (e.g., low / some concerns / high risk) and then reaches an overall judgment for that study.

Risk-of-bias assessment across five studies and five domains, rated low, some concerns, or high risk, with an overall judgment per study

Using quality in the analysis

A risk-of-bias assessment is only useful if it feeds back into the meta-analysis. Common ways to do this:

Sensitivity analysis: re-run the pooled analysis excluding high-risk studies, and check whether the overall effect changes meaningfully.
Quality as a moderator: treat the risk-of-bias judgment as a categorical moderator (see Meta-analysis steps) to test whether lower-quality studies report systematically different effects.
Reporting, not filtering: some meta-analyses deliberately include all eligible studies regardless of quality, but report the risk-of-bias distribution transparently (e.g., in a PRISMA-style summary figure) so readers can judge the evidence themselves.

See Meta-analysis steps for where this fits in the overall process and for the reporting guidelines that require it, and the full reference list for RoB 2 and the other tools cited above.