Doing a Systematic Review: A Student’s Guide

Download

What is Systematic Review?

A systematic review is a comprehensive, structured analysis of existing research on a specific topic. It uses predefined criteria to identify, evaluate, and synthesize relevant studies, aiming to provide an unbiased summary of the current evidence.

The explicit and systematic approach of a systematic review distinguishes it from traditional reviews and commentaries.

How systematic reviews differ from narrative reviews:

• Goals: Narrative reviews provide a summary or overview of a topic, while systematic reviews answer a focused review question.
• Sources of Literature: Narrative reviews often use a non-exhaustive and unstated body of literature, which can lead to publication bias. Systematic reviews consider a list of databases, grey literature (research outside conventional journal publishing, such as theses, conference proceedings and unpublished reports), and other sources.
• Selection Criteria: Narrative reviews usually use subjective or no selection criteria, which can lead to selection bias. Systematic reviews have a clear and explicit selection process.
• Appraisal of Study Quality: Narrative reviews vary in their evaluation of study quality. Systematic reviews use standard checklists for a rigorous appraisal of study quality.

Systematic reviews are time-intensive. They need a research team with multiple skills and contributions. In some cases, a systematic review cannot meet the objectives of the review question.

In these cases, scoping reviews (which are sometimes called scoping exercises/scoping studies) may be more useful to consider.

Scoping reviews are different from systematic reviews because they may not include a mandatory critical appraisal of the included studies or synthesize the findings from individual studies.

Systematic Review vs Meta-Analysis and Umbrella Review

Review types differ mainly in their purpose and in how they treat study quality.

Review type	Main purpose	Critical appraisal
Narrative review	Summarizes or overviews a topic	Varies from review to review
Scoping review	Maps the breadth of a field	Not mandatory
Systematic review	Answers a focused review question	Standard checklists for each included study
Umbrella review	Compares existing systematic reviews and meta-analyses	Appraises the methodological quality of the reviews

A meta-analysis statistically combines the results of several quantitative studies into one estimate. Glass (1976) coined the term. It is an optional companion to a systematic review, not a rival.

A systematic review supplies what a meta-analysis needs: a declared search, explicit eligibility criteria and an appraisal of each study. In return, a meta-analysis gives the review a more precise answer when the studies are similar enough to pool.

A systematic review without one is still complete. A meta-analysis of a haphazardly assembled set of studies is not a systematic review at all, however sophisticated its statistics.

An umbrella review sits one level higher. It systematically searches for and appraises existing systematic reviews and meta-analyses on a broad topic, comparing their findings, methodological quality and inconsistencies (Ioannidis, 2009).

Narrative reviews also come in different forms. A critical review selects and evaluates the literature to build and defend a particular argument. An expert review, sometimes commissioned from a senior figure, rests on one authority’s judgment rather than a repeatable process (Grant & Booth, 2009).

Assessing The Need For A Systematic Review

Before starting, check whether an existing or ongoing review already answers the question. Duplicating one is research waste. Then decide whether a new review is justified.

Scoping reviews frequently serve as preliminary steps before conducting full systematic reviews. They help assess the available literature’s breadth, identify key concepts, and determine the feasibility of a more comprehensive review.

This initial exploration guides researchers in refining their approach for subsequent in-depth analyses.

This process should begin by searching relevant databases.

Resources to consider searching include:

• NICE: National Institute for Health and Clinical Excellence
• Campbell Library of Systematic Reviews for reviews in education, crime and justice, and social welfare
• EPPI: Evidence for Policy and Practice Information Centre, particularly their database of systematic and non-systematic reviews of public health interventions (DoPHER)
• MEDLINE: Primarily covers the medical domain, making it a primary resource for systematic reviews concerning healthcare interventions
• PsycINFO: For research in psychology, psychiatry, behavioral sciences, and social sciences
• Cochrane Library (specifically CDSR): Focuses on systematic reviews of health care interventions, providing regularly updated and critically appraised reviews

If an existing review addressing the question of interest is found, its quality should be assessed to determine its suitability for guiding policy and practice.

If a high-quality, relevant review is located, but its completion date is some time ago, updating the review might be warranted.

Relevance matters most in rapidly evolving fields. Collaborating with the original research team can help during an update, as they may provide access to their data.

If the review is deemed to be of adequate quality and remains relevant, undertaking another systematic review may not be necessary.

Sometimes a new review or an update is justified. The next step is to assemble a review team and, possibly, an advisory group. They then develop the review protocol, the detailed written plan for every stage of the review.

How To Conduct A Systematic Review

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) is a reporting guideline designed to improve the transparency and completeness of systematic review reporting.

PRISMA was created to tackle the issue of inadequate reporting often found in systematic reviews:

• Checklist: PRISMA features a 27-item checklist covering all aspects of a systematic review, from the rationale and objectives to the synthesis of findings and discussion of limitations. Each checklist item is accompanied by detailed reporting recommendations in an Explanation and Elaboration document.
• Flow Diagram: PRISMA also includes a flow diagram to visually represent the study selection process, offering a clear, standardized way to illustrate how researchers arrived at the final set of included studies.

PRISMA is a reporting guideline: it governs how a review is described, not how well it was conducted. Two extensions apply the same logic to specific stages.

• PRISMA-P: A 17-item checklist for the protocol itself. Before screening starts, it asks authors to specify their sources, eligibility criteria and a repeatable draft search strategy (Moher et al., 2015).
• PRISMA-S: A 16-item checklist for reporting the literature search. It asks for each database’s full search strategy, the date each source was last searched and any supplementary methods (Rethlefsen et al., 2021).

Step 1: write a research protocol

A protocol in the context of systematic reviews is a detailed plan that outlines the methodology to be employed throughout the review process.

The protocol serves as a roadmap. It guides researchers through each stage of the review in a transparent, replicable manner.

This document should provide specific details about every stage of the research process, including the methodology for identifying, selecting, and analyzing relevant studies.

For example, the protocol should specify search strategies for relevant studies, including whether the search will encompass unpublished works.

The protocol must be created before the research begins. This ensures transparency and reproducibility.

This pre-determined plan ensures that decisions made during the review are objective and free from bias, as they are based on pre-established criteria.

Protocol modifications are sometimes necessary during systematic reviews. While adhering to the protocol is crucial for minimizing bias, there are instances where modifications are justified. For instance, a deeper understanding of the research question that emerges from examining primary research might necessitate changes to the protocol.

A 2026 review of acceptance and commitment therapy (ACT) shows what a documented change looks like. Gower et al. (2026) had planned to exclude low-quality trials. That rule left a restrictive sample in which methodological problems were common.

So the team revised the protocol before extracting any data and included trials of all quality levels. They reported the change and the reason for it.

Systematic reviews should be registered at inception (at the protocol stage) for these reasons:

• To help avoid unplanned duplication
• To enable the comparison of reported review methods with what was planned in the protocol

This registration prevents duplication (research waste) and makes the process easy when the full systematic review is sent for publication.

PROSPERO is an international database of prospectively registered systematic reviews in health and social care. Non-Cochrane protocols should be registered on PROSPERO.

Step 2: formulate a research question 

A focused research question is crucial for a systematic review. It underpins every stage of the review process.

The question defines the review’s nature and scope, guides the identification of relevant studies, and shapes the data extraction and synthesis processes.

It’s essential that the research question is answerable and clearly stated in the review protocol, ensuring that the review’s boundaries are well-defined.

A narrow question may limit the number of relevant studies and generalizability, while a broad question can make it challenging to reach specific conclusions.

PICO Framework

The PICO framework is a model for creating focused clinical research questions. The acronym PICO stands for:

• Population/Patient/Problem: This element defines the specific group of people the research question pertains to.
• Intervention: This is the treatment, test, or exposure being considered for the population.
• Comparison: This is the alternative intervention or control group against which the intervention is being compared.
• Outcome: This element specifies the results or effects of the interventions being investigated

Using the PICO format when designing research helps to minimize bias because the questions and methods of the review are formulated before reviewing any literature.

The PICO elements are also helpful in defining the inclusion criteria used to select sources for the systematic review.

The PICO framework is commonly employed in systematic reviews that primarily analyze data from randomized controlled trials.

Not every PICO element is always required. For instance, a research question does not always need a comparison.

Types of questions that can be answered using PICO:

Therapy

“In patients with a recent acute stroke (less than 6 weeks) with reduced mobility (P), is any specific physiotherapy approach (I) more beneficial than no physiotherapy (C) at improving independence in activities of daily living and gait speed (O)?”

Prevention

“For women who have experienced domestic violence (P), how effective are advocacy programmes (I) compared to other treatments (C) on improving the quality of life (O)?”

Etiology/Harm

Are women with a history of pelvic inflammatory disease (PID) (P) at higher risk for gynecological cancers (O) than women with no history of PID (C)?

Diagnosis

Among asymptomatic adults at low risk of colon cancer (P), is fecal immunochemical testing (FIT) (I) as sensitive and specific for diagnosing colon cancer (O) as colonoscopy (C)?

Prognosis

Among adults with pneumonia (P), do those with chronic kidney disease (CKD) (I) have a higher mortality rate (O) than those without CKD (C)?

Alternative Frameworks

The selection of an appropriate framework for a research question hinges significantly on aligning the framework with the type of evidence under review.

• PICOCS: This framework, used in public health research, adds a “Context” element to the PICO framework. This is useful for examining how the environment or setting in which an intervention is delivered might influence its effectiveness.
• PICOC: This framework expands on PICO by incorporating “Costs” as an element of the research question. It is particularly relevant to research questions involving economic evaluations of interventions.
• ECLIPSE: Expectations, Client group, Location, Impact, Professionals involved, Service, and Evaluation. It is a mnemonic device designed to aid in searching for health policy and management information.
• PEO: This acronym, standing for Patient, Exposure, and Outcome, is a variation of PICO used when the research question focuses on the relationship between exposure to a risk factor and a specific outcome.
• PIRD: This acronym stands for Population, Index Test, Reference Test, and Diagnosis of Interest, guiding research questions that focus on evaluating the diagnostic accuracy of a particular test.
• PFO: This acronym, representing Population, Prognostic Factors, and Outcome, is tailored for research questions that aim to investigate the relationship between specific prognostic factors and a particular health outcome.
• SDMO: This framework, which stands for Studies, Data, Methods, and Outcomes, assists in structuring research questions focused on methodological aspects of research, examining the impact of different research methods or designs on the quality of research findings.
• PICOS: This framework adds Setting to PICO. It suits reviews where the delivery environment, such as a community setting versus a specialist clinic, is a genuine review parameter.
• SPIDER: Sample, Phenomenon of Interest, Design, Evaluation and Research type. It was built for qualitative and mixed-methods questions, where “population” and “intervention” fit awkwardly (Cooke et al., 2012).

Step 3: Search Strategy

PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) provide appropriate guidance for reporting quantitative literature searches.

Present the full search strategies for all databases, registers and websites, including any filters and limits used.

PRISMA 2020 Checklist

A search strategy is a comprehensive and reproducible plan for identifying all relevant research studies that address a specific research question.

This systematic approach to searching helps minimize bias and distinguishes systematic reviews from other types of literature reviews.

It’s important to be transparent about the search strategy and document all decisions for auditability. The goal is to identify all potentially relevant studies for consideration.

Here’s a breakdown of a search strategy:

Search String Construction

It is recommended to consult topic experts on the review team and advisory board in order to create as complete a list of search terms as possible for each concept.

To retrieve the most relevant results, a search string is used. This string is made up of:

• Keywords: Search terms should be relevant to the subject areas of the research question and should be identified for all components of the research question (e.g., Population, Intervention, Comparator, and Outcomes – PICO). Using relevant keywords helps minimize irrelevant search returns. Sources such as dictionaries, textbooks, and published articles can help identify appropriate keywords.
• Synonyms: These are words or phrases with similar meanings to the keywords, as authors may use different terms to describe the same concepts. Including synonyms helps cover variations in terminology and increases the chances of finding all relevant studies. For example, a drug intervention may be referred to by its generic name or by one of its several proprietary names.
• Truncation symbols: These broaden the search by capturing variations of a keyword. They function by locating every word that begins with a specific root. For example, if a user was researching interventions for smoking, they might use a truncation symbol to search for “smok*” to retrieve records with the words “smoke,” “smoker,” “smoking,” or “smokes.” This can save time and effort by eliminating the need to input every variation of a word into a database.
• Boolean operators: The use of Boolean operators (AND/OR/NEAR/NOT) helps to combine these terms effectively, ensuring that the search strategy is both sensitive and specific. For instance, using “AND” narrows the search to include only results containing both terms, while “OR” expands it to include results containing either term.

Information Sources

The primary goal is to find all published and unpublished studies that meet the predefined criteria of the research question. This includes considering various sources beyond typical databases.

Information sources for systematic reviews can include a wide range of resources like scholarly databases, unpublished literature, conference papers, books, and even expert consultations.

Specify all databases, registers, websites, organisations, reference lists and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted.

PRISMA 2020 Checklist

An exhaustive, systematic search strategy is developed with the assistance of an expert librarian.

• Electronic Databases: Searches should include seven key databases: CINAHL, Medline, APA PsycArticles, Psychology and Behavioral Sciences Collection, APA PsycInfo, SocINDEX with Full Text, and Web of Science: Core Collections.
• Grey Literature: In addition to databases, forensic or ‘expansive’ searches can be conducted. This includes: grey literature database searches (e.g. OpenGrey,WorldCat, Ethos),  conference proceedings, unpublished reports, theses , clinical trial databases, searches by names of authors of relevant publications. Independent research bodies may also be good sources of material, e.g. Centre for Research in Ethnic Relations, Joseph Rowntree Foundation, Carers UK.
• Citation Searching: Reference lists often lead to highly cited and influential papers in the field, providing valuable context and background information for the review.
• Handsearching: Manually searching through specific journals or conference proceedings page-by-page is another way to ensure all relevant studies are captured, particularly those not yet indexed in databases.
• Contacting Experts: Reaching out to researchers or experts in the field can provide access to unpublished data or ongoing research not yet publicly available.

This is not an exhaustive list of every potential database.

Example: A Published Search Strategy

A systematic computerized search was performed for publications that appeared between 1974 and 2018 in English language journals. Four databases were searched including PsychINFO, Embase, OVOID MEDLINE, and AMED. The databases were searched with combinations of search terms relating to attachment (“attachment” OR “working model” OR “safe haven” OR “secure base” OR “felt security”) AND romantic couples (“dyad” OR “couple” OR “spous” OR “partner” OR “romantic” OR “wife” OR “husband” OR “close relationship” OR “interpersonal” OR “intimate” OR “mari”) AND social support (“support prov” OR “caregiving” OR “support giv” OR “social support” OR “enacted support” OR “support received” OR “receiv* support” OR “prov support” OR “dyadic coping” OR “interpersonal coping” OR “collaborative coping” OR “help‐seeking” OR “emotional support” OR “tangible support” OR “instrumental support” OR “perceived support” OR “responsive” OR “buffer” OR “partner support” OR “Support avail*” OR “available support”). The reference lists of the retrieved studies were checked to find other relevant publications, which were not identified in the computerized database searches.

Source: McLeod, S., Berry, K., Hodgson, C., & Wearden, A. (2020). Attachment and social support in romantic dyads: A systematic review. Journal of clinical psychology, 76(1), 59-101.

Inclusion Criteria

Specify the inclusion and exclusion criteria for the review.

PRISMA 2020 Checklist

Researchers should set clear eligibility criteria before searching.

Inclusion criteria are used to select studies for a systematic review and should be based on the study’s research method and PICO elements.

Set eligibility criteria a priori, before searching. This maintains transparency and minimizes bias. Ideally, researchers should aim to include only high-quality randomized controlled trials that adhere to the intention-to-treat principle.

Selection should never be arbitrary. The rationale for every inclusion and exclusion criterion should be clearly articulated in the research protocol.

Inclusion criteria should be piloted by applying them to a sample of papers to check if they can be reliably interpreted and appropriately classify studies. The pilot phase can refine and clarify inclusion criteria and ensure that multiple reviewers can apply the criteria consistently.

When specifying the inclusion and exclusion criteria, consider the following aspects:

• Intervention Characteristics: Researchers might decide that, in order to be included in the review, an intervention must have specific characteristics. They might require the intervention to last for a certain length of time, or they might determine that only interventions with a specific theoretical basis are appropriate for their review.
• Population Characteristics: A systematic review might focus on the effects of an intervention for a specific population. For instance, researchers might choose to focus on studies that included only nurses or physicians.
• Outcome Measures: Researchers might choose to include only studies that used outcome measures that met a specific standard.
• Age of Participants: If a systematic review is examining the effects of a treatment or intervention for children, the authors of the review will likely choose to exclude any studies that did not include children in the target age range.
• Diagnostic Status of Participants: Researchers conducting a systematic review of treatments for anxiety will likely exclude any studies where the participants were not diagnosed with an anxiety disorder.
• Study Design: Researchers might determine that only studies that used a particular research design, such as a randomized controlled trial, will be included in the review.
• Control Group: In a systematic review of an intervention, researchers might choose to include only studies that included certain types of control groups, such as a waiting list control or another type of intervention.
• Publication status: Decide whether only published studies will be included or if unpublished works, such as dissertations or conference proceedings, will also be considered.

Example: Published Inclusion Criteria

Studies that met the following criteria were included: (a) empirical studies of couples (of any gender) who are in a committed romantic relationship, whether married or not; (b) measurement of the association between adult attachment and support in the context of this relationship; (c) the article was a full report published in English; and (d) the articles were reports of empirical studies published in peer‐reviewed journals, dissertations, review papers, and conference presentations.

Source: McLeod, S., Berry, K., Hodgson, C., & Wearden, A. (2020). Attachment and social support in romantic dyads: A systematic review. Journal of clinical psychology, 76(1), 59-101.

Iterative Process

The iterative nature of developing a search strategy for systematic reviews stems from the need to refine and adapt the search process based on the information encountered at each stage.

A single attempt rarely yields the perfect final strategy. Instead, it is an evolving process involving a series of test searches, analysis of results, and discussions among the review team.

Here’s how the iterative process unfolds:

1. Initial Strategy Formulation: Based on the research question, the team develops a preliminary search strategy, including identifying relevant keywords, synonyms, databases, and search limits.
2. Test Searches and Refinement: The initial search strategy is then tested on chosen databases. The results are reviewed for relevance, and the search strategy is refined accordingly. This might involve adding or modifying keywords, adjusting Boolean operators, or reconsidering the databases used.
3. Discussions and Iteration: The search results and proposed refinements are discussed within the review team. The team collaboratively decides on the best modifications to improve the search’s comprehensiveness and relevance.
4. Repeating the Cycle: This cycle of test searches, analysis, discussions, and refinements is repeated until the team is satisfied with the strategy’s ability to capture all relevant studies while minimizing irrelevant results.

Iteration keeps the search comprehensive and unbiased. Each round of refinement makes researchers more confident that they have found every relevant study.

It also keeps the strategy sensitive enough to capture every relevant study while the scope stays manageable.

Document the strategy and every modification meticulously. This transparency allows future replication of the review.

Searching is a pragmatic activity. Suppose a search identifies 15,000 items. If the review team is two people working in their free time, it may be necessary to narrow the review’s scope and inclusion criteria.

The team could revise the publication date (for example, the last ten years rather than 20), the population, or the study designs.

Step 4: Search the Literature

Conduct a systematic search of the literature using clearly defined search terms and databases.

Applying the search strategy involves entering the constructed search strings into the respective databases’ search interfaces. These search strings, crafted using Boolean operators, truncation symbols, wildcards, and database-specific syntax, aim to retrieve all potentially relevant studies addressing the research question.

At this stage, the researcher uses each database’s features. These refine the search and manage the retrieved results.

This might involve employing search filters provided by the database to focus on specific study designs, publication types, or other relevant parameters.

Applying the strategy is not merely mechanical. It demands a thorough understanding of database functionality and a discerning eye for adjusting the search to the results retrieved.

Step 5: screening and selecting research articles

Once the search strategy is finalized, it is applied to the selected databases, yielding a set of search results.

These search results are then screened against pre-defined inclusion criteria to determine their eligibility for inclusion in the review.

The goal is to identify studies that are both relevant to the research question and of sufficient quality to contribute to a meaningful synthesis.

Eligible studies are saved in Endnote or Mendeley. Each record holds the title, authors, date, journal and abstract if available.

Study Selection

Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process.

PRISMA 2020 Checklist

Study selection involves multiple reviewers to ensure rigor and reliability.

To minimize bias and enhance the reliability of the study selection process, it is recommended that at least two reviewers independently assess the eligibility of each study. This independent assessment helps reduce the impact of individual biases or errors in judgment.

Two reviewers should independently screen titles and abstracts, removing duplicates and irrelevant studies based on predefined inclusion and exclusion criteria.

1. Initial screening of titles and abstracts: Screen titles and abstracts against the predefined criteria. Exclude studies clearly outside the scope. When relevance is uncertain, keep the study: wrongly keeping one costs a retrieval, wrongly discarding one loses evidence.
2. Retrieving and assessing full texts: Where the title and abstract cannot settle eligibility, obtain the full text. Assess it meticulously against the criteria.
3. Resolution of disagreements: Agree a predefined strategy in advance. Consensus discussion or arbitration by a third reviewer keeps selection fair and impartial.

PRISMA 2020 asks reviewers to disclose automation tools. A 2026 scoping review of AI resources for screening titles and abstracts found 174 relevant studies published since 2019 (Barragán et al., 2026).

Studies most often judged tools by sensitivity, specificity or the reviewer time saved. Validation metrics varied widely between studies. The most frequent recommendation was to keep a human reviewer in the screening loop.

Example: Screening in a Published Review

First, the search results from separate databases were combined, and any duplicates were removed. The lead author (S. M.) and a postgraduate researcher (F. N.) applied the described inclusion criteria in a standardized manner. First, both the titles and abstracts of the articles were evaluated for relevance. If, on the basis of the title and/or abstract, the study looked likely to meet inclusion criteria hard copies of the manuscripts were obtained. If there was doubt about the suitability of an article, then the manuscript was included in the next step. The remaining articles were obtained for full‐text review, and the method and results sections were read to examine whether the article fitted the inclusion criteria. If there was doubt about the suitability of the manuscripts during this phase, then this article was discussed with another author (C. H.). Finally, the reference lists of the eligible articles were checked for additional relevant articles not identified during the computerized search. For the selected articles (n = 43), the results regarding the relationship between attachment and support were included in this review (see Figure 1, for PRISMA flowchart).

Source: McLeod, S., Berry, K., Hodgson, C., & Wearden, A. (2020). Attachment and social support in romantic dyads: A systematic review. Journal of clinical psychology, 76(1), 59-101.

PRISMA Flowchart

The PRISMA flowchart is a visual representation of the study selection process within a systematic review.

The flowchart illustrates the step-by-step process of screening, filtering, and selecting studies based on predefined inclusion and exclusion criteria.

The flowchart visually depicts the following stages:

• Identification: The initial number of titles and abstracts identified through database searches.
• Screening: The screening process, based on titles and abstracts.
• Eligibility: Full-text copies of the remaining records are retrieved and assessed for eligibility.
• Inclusion: Applying the predefined inclusion criteria resulted in the inclusion of publications that met all the criteria for the review.
• Exclusion: The flowchart details the reasons for excluding the remaining records.

This systematic and transparent approach, as visualized in the PRISMA flowchart, ensures a robust and unbiased selection process, enhancing the reliability of the systematic review’s findings.

The flowchart serves as a visual record of the decisions made during the study selection process, allowing readers to assess the rigor and comprehensiveness of the review.

• How to fill a PRISMA flow diagram

Step 6: Critically Appraising the Quality of Included Studies

Quality assessment provides a measure of the strength of the evidence presented in a review.

High-quality studies with rigorous methodologies contribute to a more robust and reliable evidence base, increasing confidence in the review’s conclusions.

Conversely, including low-quality studies with methodological weaknesses can undermine the review’s findings and potentially lead to inaccurate recommendations.

Standardized checklists and scales are commonly used to judge quality. These tools make the assessment transparent and reproducible. They assess risk of bias, meaning how far a study’s design or conduct may have produced a misleading estimate of its true effect.

The choice of tool should be justified and aligned with the study design and the level of detail required. Using quality scores alone is discouraged; instead, individual aspects of methodological quality should be considered.

Common Appraisal Tools by Study Design

• For RCTs:
  • Jadad score
  • Cochrane Risk of Bias tool
• For non-RCTs:
  • Cochrane Effective Practice and Organisation of Care (EPOC) Group Risk of Bias Tool
  • Quality Assessment of Diagnostic Accuracy Studies (QUADAS)
  • Newcastle – Ottawa Quality Assessment Scale for case-control and cohort studies
• For Public Health studies:
  • EPHPP Assessment Tool
  • Critical Appraisal Skills Programme (CASP) Appraisal Checklist
  • Cochrane Public Health Group (CPHG)

The Joanna Briggs Institute (JBI) offers a further widely used suite of appraisal tools. It covers RCTs, quasi-experimental studies, cohort studies, case series and more.

Example: Quality Appraisal in a Published Review

The quality of the study was not an inclusion criterion; however, a study quality check was carried out. Two independent reviewers (S. M. and C. H.) rated studies that met the inclusion criteria to determine the strength of the evidence. The Effective Public Health Practice Project Quality Assessment Tool for Quantitative Studies was adapted to assess the methodological quality of each study (Thomas, Ciliska, Dobbins, & Micucci, 2004). The tool was adjusted to include domains relevant to the method of each study. For example, blinding was removed for nonexperimental studies. Following recommendations by Thomas et al. (2004) each domain was rated as either weak (3 points), moderate (2 points), or strong (1 point). The mean score across questions was used as an indicator of overall quality, and studies were assigned an overall quality rating of strong (1.00–1.50), moderate (1.51–2.50),

Source: McLeod, S., Berry, K., Hodgson, C., & Wearden, A. (2020). Attachment and social support in romantic dyads: A systematic review. Journal of clinical psychology, 76(1), 59-101.

Evidence Tables

Record aspects of the appraisal of included studies as evidence tables (NICE 2009). These are simple text tables that summarise the design and scope of each study. Each table records a study’s design, population, methods and quality.

The reader of the review can use the evidence tables to check the details, and assess the credibility and generalisability of findings, of particular studies.

Appraisal may be combined with data extraction tables.

Step 7: extracting data from studies

To effectively extract data from studies that meet your systematic review’s inclusion criteria, you should follow a structured process that ensures accuracy, consistency, and minimizes bias.

1. Develop a data extraction form:

• Standardized form: Design a form, on paper or electronically, tailored to your review question and the types of studies included.
• Pilot test the form: Test the form on a small sample of included studies (e.g., 3-5). Assess for clarity, completeness, and usability. Refine the form based on feedback and initial experiences.
• Reliability: Ensure all team members understand how to use the form consistently.

2. Extract the data:

• General Information: This includes basic bibliographic details (journal, title, author, volume, page numbers), study objective as stated by the authors, study design, and funding source.
• Study Characteristics: Capture details about the study population (demographics, inclusion/exclusion criteria, recruitment procedures), interventions (description, delivery methods), and comparators (description if applicable).
• Outcome Data: Record the results of the intervention and how they were measured, including specific statistics used. Clearly define all outcomes for which data are being extracted.
• Risk of Bias Assessment: Document the methods used to assess the quality of the included studies and any potential sources of bias. This might involve using standardized checklists or scales.
• Additional Information: Depending on your review, you may need to extract data on other variables like adverse effects, economic evaluations, or specific methodological details.

3. Dual independent review:

• Dual extraction: Have at least two reviewers independently extract data from each study using the standardized form. Cross-check the data to minimize bias and catch discrepancies.
• Disagreements: Agree a predefined strategy for resolving them, such as discussion, consensus or arbitration by a third reviewer.
• Exclusion reasons: Record why any study is excluded during data extraction. This enhances the transparency and reproducibility of your review.
• Author contact: If necessary, contact study authors to obtain missing information or clarify unclear information. This matters most for data critical to your review’s outcomes.
• Documentation: Clearly document the entire extraction process, including challenges encountered and decisions made. This enhances the transparency and rigor of your systematic review.

Generative AI is now being tested for this step. Jansen et al. (2025) compared eight large language models with human-extracted data from 22 psychology review databases (312,329 data points, 2,179 studies).

Accuracy was unacceptable on every metric for 20% of the 186 variables. Only about a fifth reached high or excellent accuracy. Context and moderator variables did better than effect-size variables.

The authors suggest using AI to accelerate extraction while keeping meaningful human control.

By following these steps, you can effectively extract data from studies that meet your inclusion criteria, forming a solid foundation for the analysis and synthesis phases of your systematic review.

Step 8: synthesize the extracted data

Synthesis is the key element of a systematic review. It brings together the findings of the included studies to draw conclusions from the body of evidence as a whole.

This means collating, combining and summarizing the findings. The aim is a reliable, comprehensive answer to the review question.

Reviewers weigh the strength of the evidence, examine the consistency of the observed effects, and investigate any inconsistencies.

The data synthesis will be presented in the results section of the systematic review.

• Develop a clear text narrative that explains the key findings
• Use a logical heading structure to guide readers through your results synthesis
• Ensure your text narrative addresses the review’s research questions
• Use tables to summarise findings (can be same table as data extraction)

Where the included data can be pooled, the synthesis may culminate in a meta-analysis, which combines the studies’ results into a single statistical estimate. A meta-analysis is optional. Many rigorous systematic reviews synthesize narratively instead.

A single review can land on both sides of that choice. Alkærsig et al. (2026) narratively synthesized 127 studies of 12 treatments for trauma-related symptoms in young children.

They pooled results in a meta-analysis only for the five treatments with enough controlled studies. Five treatments returned no evidence at all. A documented gap is a finding in itself. It differs from a null result.

Without a pooled estimate, reviewers must still report their synthesis method transparently. The SWiM guideline (Synthesis Without Meta-analysis) has nine reporting items (Campbell et al., 2020). They cover how studies were grouped, the standardized metric used, the synthesis method applied and how results were presented.

Identifying patterns, trends, and differences across studies

Narrative synthesis is the textual, rather than purely statistical, method of drawing the findings of the included studies into an integrated account. It analyzes relationships within and between studies to assess how robust the evidence is.

All systematic reviews should include elements of narrative synthesis. Tables and text are two examples.

The goal is to go beyond summarizing individual studies. Aim for a new understanding that integrates and interprets the evidence systematically and transparently.

Organize your data:

• Group studies by themes, interventions, or outcomes
• Create summary tables to display key information across studies
• Use visual aids like concept maps to show relationships between studies

Describe the studies:

• Summarize the characteristics of included studies (e.g., designs, sample sizes, settings)
• Highlight similarities and differences across studies
• Discuss the overall quality of the evidence

Develop a preliminary synthesis:

• Start by describing the results of individual studies
• Group similar findings together
• Identify overarching themes or trends

Explore relationships:

• Look for patterns in the data
• Identify factors that might explain differences in results across studies
• Consider how study characteristics relate to outcomes

Address contradictions:

• Consider differences in study populations, interventions, or contexts
• Look at methodological differences that might explain discrepancies
• Consider the implications of inconsistent results
• Don’t ignore conflicting findings
• Discuss possible reasons for contradictions

Avoid vote counting:

• Don’t simply tally positive versus negative results: a tally treats a small, weak study as equal to a large, rigorous one
• Instead, consider the strength and quality of evidence for each finding

Assess the robustness of the synthesis:

• Reflect on the strength of evidence for each finding
• Consider how gaps or limitations in the primary studies affect your conclusions
• Discuss any potential biases in the synthesis process

Step 9: discussion section and conclusion

The discussion turns the synthesis into something readers can act on. It has five jobs.

Summarize key findings:

• Summarize key findings in relation to your research questions
• Highlight main themes or patterns across studies
• Explain the nuances and complexities in the evidence
• Discuss the overall strength and consistency of the evidence
• This provides a clear takeaway message for readers

Consider study quality and context:

• Assess whether higher quality studies tend to show different results
• Examine if findings differ based on study setting or participant characteristics
• This helps readers weigh the relative importance of conflicting findings

Discuss implications:

• For practice: How might professionals apply these findings?
• For policy: What policy changes might be supported by the evidence?
• Consider both positive and negative implications
• This helps translate your findings into real-world applications

Identify gaps and future research:

• Point out areas where evidence is lacking or inconsistent
• Suggest specific research questions or study designs to address these gaps
• This helps guide future research efforts in the field

State strengths and limitations:

• Discuss the strengths of your review (e.g., comprehensive search, rigorous methodology)
• Acknowledge limitations (e.g., language restrictions, potential for publication bias)
• This balanced approach demonstrates critical thinking and helps readers interpret your findings

Minimizing Bias

No single step eliminates bias in a systematic review. A set of interlocking safeguards holds it down at every stage, which is why the process is so procedural.

1. Protocol development and publication: Developing a comprehensive protocol before starting the review is essential. Publishing the protocol in repositories like PROSPERO or Cochrane Library promotes transparency and helps avoid deviations from the planned approach, thereby minimizing the risk of bias.
2. Transparent reporting: Adhering to reporting guidelines, such as PRISMA, ensures that all essential aspects of the review are adequately documented, increasing the reader’s confidence in the transparency and completeness of systematic review reporting.
3. Dual independent review: Employing two or more reviewers independently at multiple stages of the review process (study selection, data extraction, quality assessment) minimizes bias. Any disagreements between reviewers should be resolved through discussion or by consulting a third reviewer. This approach reduces the impact of individual reviewers’ subjective interpretations or errors.
4. Rigorous quality assessment: Assessing the methodological quality of included studies is crucial for minimizing bias in the review findings. Using standardized critical appraisal tools and checklists helps identify potential biases within individual studies, such as selection bias, performance bias, attrition bias, and detection bias.
5. Mitigate publication bias: Systematic reviews should aim to include all relevant studies, regardless of their publication status.
   • Searching beyond published literature: Explore sources of “grey literature” such as conference proceedings, unpublished reports, theses, and ongoing clinical trial databases.
   • Contacting experts in the field: Researchers can reach out to authors and investigators to inquire about unpublished or ongoing studies
   • Considering language bias: Expanding the search to include studies published in languages other than English can help reduce language bias, although this may increase the complexity and cost of the review.

Critical Evaluation of Systematic Reviews

A systematic review is the strongest way to synthesize evidence, but it has real limits. Six documented problems follow, then recent work that addresses some of them.

• Publication Bias: Studies with null results are less likely to be published, so even a well-run search over-represents positive findings.
• Heterogeneity: Studies that meet the eligibility criteria can still differ too much to be pooled safely.
• Weak Studies: Appraisal tools identify weak studies but cannot strengthen them, so the weakness passes into the synthesis.
• Appraising Reviews: Many reviews skip formal quality assessment, and tools such as AMSTAR 2 and ROBIS judge the reviews themselves.
• Reporting vs Conduct: PRISMA improves how a review is described, not how well it was done.
• Resource Burden: Full reviews are slow and costly, so rapid and living reviews trade rigor against speed and currency.

Publication Bias and the File-Drawer Problem

Dual independent screening and grey-literature searching reduce publication bias, but they cannot remove it.

The distortion happens upstream of the review, when individual studies are written up and accepted for publication. Searching harder cannot recover studies that were never published.

Rosenthal (1979) named this the file-drawer problem. Studies that find no significant effect are less likely to be submitted, accepted or published. The reviewable literature therefore over-represents positive findings, however exhaustively a review searches it.

The problem is not merely historical. Franco et al. (2014) audited registered social-science studies and found that a substantial share of those with null results were never written up.

A review built only from what was published will overstate an effect, however carefully it searches.

Heterogeneity Between Included Studies

Meeting a review’s eligibility criteria does not make studies interchangeable. Studies asking superficially the same question can differ in their populations, interventions, outcome measures and designs. This variation between studies is called heterogeneity.

Pooling such studies risks averaging away genuinely different effects rather than estimating one true effect more precisely. This is the “apples and oranges” objection to synthesis.

Higgins et al. (2003) developed the now-standard statistics for detecting and quantifying heterogeneity. Informal claims of consistency are not enough.

Ioannidis (2016) adds that heterogeneity is often masked rather than resolved. Reviewers under pressure to produce a clean synthesis can select or subgroup studies in ways that manufacture apparent consistency.

A homogeneous literature may then be an artifact of how the review was conducted.

Weak Studies Cannot Be Rescued by Appraisal

Critical appraisal tools make quality assessment transparent and reproducible. They identify weak studies, but they do not strengthen them.

A synthesis of ten flawed trials is not rescued by a rigorous protocol. The review inherits their bias.

Careful methods applied to weak evidence still give weak conclusions.

Ioannidis (2016) argues that the problem has grown as published reviews have multiplied. Many are produced quickly from the same overlapping pool of primary studies.

Some are of poor or unclear methodological quality, and some reach directly conflicting conclusions about the same question. Readers end up little better informed.

Chalmers and Glasziou (2009) trace part of this back to research waste at the primary-study stage. Poorly designed, unpublished or incompletely reported trials all propagate silently into any later synthesis.

Appraising the Reviews Themselves

The gap also appears one level up. An umbrella review of 75 systematic reviews and meta-analyses on dissociation checked how many appraised their own included studies (Garofalo et al., 2026). Only 54.7% (41 of 75) had done so.

An umbrella review cannot compensate for that gap. It is only as trustworthy as the appraisal practices of the reviews it synthesizes.

The primary-study tools in Step 6 judge bias in individual studies, not the conduct of a review. Two instruments were built for reviews themselves.

Tool	What it judges	How it works
AMSTAR 2	A review’s overall conduct	A 16-item methodological checklist
ROBIS	Risk of bias in the review itself	Signalling questions in four domains, then an overall judgment

AMSTAR 2 rates a review’s protocol registration, search comprehensiveness, justification of excluded studies, risk-of-bias assessment and conflict-of-interest reporting (Shea et al., 2017).

ROBIS asks signalling questions across four domains: study eligibility criteria, identification and selection of studies, data collection and appraisal, and synthesis and findings. Reviewers then give an overall risk-of-bias judgment (Whiting et al., 2016).

Can chatbots do this work?

Forero et al. (2025) compared four large language model chatbots with human experts’ published assessments of 20 systematic reviews. The best-performing model matched human raters only 58% of the time on ROBIS and 70% on AMSTAR 2.

Reporting Is Not Conduct

A reporting guideline cannot fix everything. Both versions of PRISMA (Moher et al., 2009; Page et al., 2021) govern how a review is described, not how well it was conducted.

A review can follow the checklist item by item and still rest on an unsystematic search, a biased selection process or an uncritical appraisal.

The checklist gives readers the information needed to notice a weak review. It does not stop one being produced or published. Spotting a weak review remains the reader’s job.

The timing of Ioannidis’s (2016) critique of redundant and conflicted reviews is telling. It appeared seven years into the PRISMA era. Transparent reporting and genuine methodological rigor are separate achievements, not two names for the same thing.

The Resource Burden of Rigorous Review

A rigorous review is expensive and slow to produce. Dual independent screening, piloted extraction forms, standardized appraisal and certainty-of-evidence ratings all take time. That sits uneasily with a fast-moving evidence base.

Two responses trade rigor against speed and currency in different directions.

Approach	How it works	Trade-off
Rapid review	Compresses or omits steps, such as fewer reviewers, narrower searches and less formal appraisal	Delivers a synthesis in weeks rather than a year or more, but shortens or skips safeguards against bias
Living review	Keeps the full method but repeats it continually as new evidence appears	Stays current, but needs additional, ongoing resourcing

Ganann et al. (2010) frame rapid reviews as a trade-off with implications, not a free efficiency gain. Neither response removes the tension. Faster reviews give up safeguards against bias, and living reviews demand sustained resources.

Contemporary Research

Recent research on the method itself clusters around three themes. They are reporting quality, shelf life and the kinds of evidence a review can handle.

PRISMA 2020: Updating the Reporting Standard

The 2009 PRISMA statement (Moher et al., 2009) responded to evidence that reviews were often reported too incompletely for readers to judge their conclusions. Its adoption measurably improved reporting through the 2010s. Methods then moved on, so the guideline needed updating.

The new certainty-of-evidence item usually points to GRADE. It rates confidence in each outcome as high, moderate, low or very low. Reviewers start from the study designs. They then downgrade, or sometimes upgrade, the rating for factors such as risk of bias, inconsistency and imprecision (Guyatt et al., 2008).

Living Systematic Reviews

Reviews age while evidence keeps accumulating.

A living systematic review is updated continually as new relevant evidence appears, rather than published once and left to date (Elliott et al., 2014).

A mixed-methods evaluation found living reviews feasible and acceptable to the teams and end users who piloted them. It also found they demand extra infrastructure and ongoing resourcing compared with a conventional review (Millard et al., 2019).

Together, PRISMA 2020 and living reviews answer one worry. A review is only as trustworthy as the completeness of its account, and only as useful as its shelf life.

Qualitative and Mixed-Methods Reviews

The same procedural skeleton works when the eligible evidence is qualitative. A declared search, PRISMA-guided screening and standardized appraisal still apply. Only meta-analysis falls away.

Davidar et al. (2026) searched four databases for qualitative studies of attachment injuries in romantic relationships. They appraised the 12 included studies with the CASP checklist.

They then used framework synthesis, mapping each study’s findings onto a complex-PTSD model, to argue that attachment injuries form a distinct relational trauma. They also flagged the evidence base’s limits: 73.8% of participants were female and 83% of samples came from North America.

McGill et al. (2026) applied the same scaffold elsewhere. A keyword search of six databases returned 10,357 records, which screening narrowed to 21 qualitative studies.

Thematic synthesis produced three cross-cutting themes on how receiving an ADHD diagnosis in adulthood reshapes a person’s relationships with themselves, others and support services. Neither review pools an effect size. Neither is any less systematic for it.

Ali et al. (2025) went further. They refused to sort evidence into qualitative or quantitative.

Their review of autistic burnout synthesized 48 studies: 30 qualitative, 7 quantitative and 11 mixed-methods.

They converted quantitative findings into narrative form so they could be integrated thematically with the qualitative themes and participant quotations. Every study was appraised with the Mixed Methods Appraisal Tool. Each criterion was reported individually, not as one score.

GRADE was built to rate confidence in a pooled quantitative estimate, so it transfers poorly to thematic findings. Lewin et al. (2015) developed GRADE-CERQual, which rates confidence in each individual review finding. It weighs four components.

• Methodological limitations: the quality of the studies contributing to the finding.
• Relevance: how well those studies’ contexts match the review question.
• Coherence: how consistent the finding is across the contributing data.
• Adequacy: how rich and sufficient the underlying data are.

Real-World Applications of Systematic Reviews

Systematic reviews turn scattered research into evidence that decision-makers can use. Three settings show how, followed by examples from psychology.

Clinical Guidelines and Evidence-Based Medicine

Systematic reviews underpin evidence-based medicine. Sackett et al. (1996) formalized the idea that clinical decisions should combine the best research evidence with clinical expertise and patient values.

The Cochrane Collaboration was founded to make this possible at scale. Its purpose was to prepare, maintain and disseminate systematic reviews of the effects of health care (Chalmers, 1993).

The founding case was itself a systematic review. An overview of controlled trials of corticosteroids for women in preterm labor reached a striking result (Crowley et al., 1990). The treatment reduced the risk of the baby dying from complications of immaturity by around 30 to 50%.

Each trial was too small to show it. Together they were scattered across three decades, beyond any one clinician’s view.

Bodies such as NICE in the UK now routinely commission systematic reviews as part of the evidence base for a clinical guideline.

No single trial is normally decisive on its own.

Education, Psychological Practice and Public Policy

School systems face the same problem. Evaluation evidence is scattered and of uneven quality. Systematic (or best-evidence) synthesis has become the standard way to turn decades of program evaluations into guidance on what works in the classroom (Slavin, 2008).

It weighs the size and quality of a program’s evidence base, rather than counting how many studies exist. A well-conducted synthesis can tell a school district which literacy intervention has the strongest track record. A single evaluation cannot.

Psychology uses systematic reviews the same way.

Evidence-based practice combines the best available research with clinical expertise. It also takes account of each patient’s characteristics, culture and preferences (APA Presidential Task Force on Evidence-Based Practice, 2006).

Systematic reviews and meta-analyses of treatment outcome research are the main route to that best available research. Roberts et al. (2017) reviewed personality trait change through intervention. That is the kind of synthesis a clinician or guideline committee draws on.

Systematic reviews also inform decisions in criminal justice, social welfare and public health, the fields the Campbell Library and EPPI databases serve.

The stakes are real. Chalmers and Glasziou (2009) estimate that a large share of research investment is avoidably wasted. The waste arises when studies are poorly designed, never published, or never synthesized with earlier work. Decision-makers then reinvent evidence that already exists.

Examples of Systematic Reviews in Psychology

Published reviews show the range of questions the method can answer.

Review	Question addressed
Galante et al. (2014)	Effect of kindness-based meditation on health and well-being
Schneider & Preckel (2017)	Variables associated with achievement in higher education, reviewed across meta-analyses
Murray et al. (2012)	Children’s antisocial behavior, mental health, drug use and educational performance after parental incarceration
Roberts et al. (2017)	Personality trait change through intervention
Chu et al. (2017)	The interpersonal theory of suicide across a decade of cross-national research
Purgato et al. (2026)	Comparative efficacy of task-sharing psychosocial interventions for preventing common mental disorders in low- and middle-income countries
Rosenek et al. (2026)	Childhood trauma and adult neuroticism, split by trauma subtype across 136 studies
Liu et al. (2026)	Help-seeking in early psychosis, synthesized from qualitative studies alone into a testable model
McLeod et al. (2020)	Attachment and social support in romantic couples, the review used for the worked examples in this guide

Bibliography

• Ali, D., Bougoure, M., Cooper, B., Quinton, A. M. G., Tan, D., Brett, J., Mandy, W., Maybery, M., Magiati, I., & Happé, F. (2025). Burnout as experienced by autistic people: A systematic review. Clinical Psychology Review, 122, Article 102669. https://doi.org/10.1016/j.cpr.2025.102669
• Alkærsig, M., Banzon, T., Roest, J., Elklit, A., Stein, A. R., Vang, M. L., & Løkkegaard, S. S. (2026). Psychological treatments for young children suffering from trauma-related symptomatology: Systematic review and partial meta-analyses of the current evidence-base for 12 methods. Clinical Psychology Review, 125, Article 102725. https://doi.org/10.1016/j.cpr.2026.102725
• APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285. https://doi.org/10.1037/0003-066X.61.4.271
• Barragán, A. M., Ortiz Bonett, S. E., Rodríguez-Grande, E.-I., Orjuela-Cañón, A. D., Perdomo, O. J., & Sánchez-Vanegas, G. (2026). Artificial intelligence resources for the screening of titles and abstracts in systematic reviews: A scoping review. Cochrane Evidence Synthesis and Methods, 4(5), Article e70098. https://doi.org/10.1002/cesm.70098
• Campbell, M., McKenzie, J. E., Sowden, A., Katikireddi, S. V., Brennan, S. E., Ellis, S., Hartmann-Boyce, J., Ryan, R., Shepperd, S., Thomas, J., Welch, V., & Thomson, H. (2020). Synthesis without meta-analysis (SWiM) in systematic reviews: Reporting guideline. BMJ, 368, Article l6890. https://doi.org/10.1136/bmj.l6890
• Chalmers, I. (1993). The Cochrane Collaboration: Preparing, maintaining, and disseminating systematic reviews of the effects of health care. Annals of the New York Academy of Sciences, 703, 156–165. https://doi.org/10.1111/j.1749-6632.1993.tb26345.x
• Chalmers, I., & Glasziou, P. (2009). Avoidable waste in the production and reporting of research evidence. The Lancet, 374(9683), 86–89. https://doi.org/10.1016/S0140-6736(09)60329-9
• Chu, C., Buchman-Schmitt, J. M., Stanley, I. H., Hom, M. A., Tucker, R. P., Hagan, C. R., Rogers, M. L., Podlogar, M. C., Chiurliza, B., Ringer, F. B., Michaels, M. S., Patros, C. H. G., & Joiner, T. E., Jr. (2017). The interpersonal theory of suicide: A systematic review and meta-analysis of a decade of cross-national research. Psychological Bulletin, 143(12), 1313–1345. https://doi.org/10.1037/bul0000123
• Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative Health Research, 22(10), 1435–1443. https://doi.org/10.1177/1049732312452938
• Crowley, P., Chalmers, I., & Keirse, M. J. N. C. (1990). The effects of corticosteroid administration before preterm delivery: An overview of the evidence from controlled trials. British Journal of Obstetrics and Gynaecology, 97(1), 11–25. https://doi.org/10.1111/j.1471-0528.1990.tb01711.x
• Davidar, R. S., Ballal, D., & Premlal, D. (2026). Systematic review and framework synthesis of attachment injuries among romantic couples. Journal of Social and Personal Relationships, 43(9), 2682–2706. https://doi.org/10.1177/02654075261436570
• Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014). Living systematic reviews: An emerging opportunity to narrow the evidence-practice gap. PLOS Medicine, 11(2), Article e1001603. https://doi.org/10.1371/journal.pmed.1001603
• Forero, D. A., Abreu, S. E., Tovar, B. E., & Oermann, M. H. (2025). Automated analyses of risk of bias and critical appraisal of systematic reviews (ROBIS and AMSTAR 2): A comparison of the performance of 4 large language models. Journal of the American Medical Informatics Association, 32(9), 1471–1476. https://doi.org/10.1093/jamia/ocaf117
• Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484
• Galante, J., Galante, I., Bekkers, M. J., & Gallacher, J. (2014). Effect of kindness-based meditation on health and well-being: A systematic review and meta-analysis. Journal of Consulting and Clinical Psychology, 82(6), 1101–1114. https://doi.org/10.1037/a0037249
• Ganann, R., Ciliska, D., & Thomas, H. (2010). Expediting systematic reviews: Methods and implications of rapid reviews. Implementation Science, 5, Article 56. https://doi.org/10.1186/1748-5908-5-56
• Garofalo, C., Severi, I., Bruno, G., Delvecchio, E., Schimmenti, A., & Mazzeschi, C. (2026). The conceptualization and assessment of dissociation: An umbrella review of synthesis studies. Clinical Psychology Review, 124, Article 102702. https://doi.org/10.1016/j.cpr.2026.102702
• Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3–8. https://doi.org/10.3102/0013189X005010003
• Gower, T., Styler, Z. R., & Litz, B. T. (2026). Randomized controlled evidence of the efficacy of acceptance and commitment therapy: A systematic review and meta-analysis. Clinical Psychology Review, 128, Article 102786. https://doi.org/10.1016/j.cpr.2026.102786
• Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
• Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 336(7650), 924–926. https://doi.org/10.1136/bmj.39489.470347.AD
• Head, B. A., Schapmire, T. J., & Zheng, Y. (2017). Telehealth in palliative care: A systematic review of patient-reported outcomes. Journal of Hospice & Palliative Nursing, 19(2), 130–139. https://doi.org/10.1097/njh.0000000000000319
• Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ, 327(7414), 557–560. https://doi.org/10.1136/bmj.327.7414.557
• Ioannidis, J. P. A. (2009). Integration of evidence from multiple meta-analyses: A primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. CMAJ, 181(8), 488–493. https://doi.org/10.1503/cmaj.081086
• Ioannidis, J. P. A. (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. The Milbank Quarterly, 94(3), 485–514. https://doi.org/10.1111/1468-0009.12210
• Jansen, T., Liebenow, L. W., Mertens, U., Schmidt, F. T. C., Lohmann, J. F., Fleckenstein, J., & Meyer, J. (2025). Data extraction by generative artificial intelligence: Assessing determinants of accuracy using human-extracted data from systematic review databases. Psychological Bulletin, 151(10), 1280–1306. https://doi.org/10.1037/bul0000501
• Lewin, S., Glenton, C., Munthe-Kaas, H., Carlsen, B., Colvin, C. J., Gülmezoglu, M., Noyes, J., Booth, A., Garside, R., & Rashidian, A. (2015). Using qualitative evidence in decision making for health and social interventions: An approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLOS Medicine, 12(10), Article e1001895. https://doi.org/10.1371/journal.pmed.1001895
• Liu, W., Carnelley, K. B., & Newman-Taylor, K. (2026). Help-seeking in people with early psychosis: A systematic review of the qualitative literature and novel model of facilitators and barriers. Clinical Psychology Review, 127, Article 102753. https://doi.org/10.1016/j.cpr.2026.102753
• McGill, L., Jardim-Lalor, I., & O’Connor, C. (2026). A systematic review of lived experiences of receiving a diagnosis of ADHD in adulthood. Journal of Attention Disorders, 30(10), 1274–1289. https://doi.org/10.1177/10870547261455946
• McLeod, S., Berry, K., Hodgson, C., & Wearden, A. (2020). Attachment and social support in romantic dyads: A systematic review. Journal of Clinical Psychology, 76(1), 59–101. https://doi.org/10.1111/jclp.22868
• Millard, T., Synnot, A., Elliott, J., & Green, S. (2019). Feasibility and acceptability of living systematic reviews: Results from a mixed-methods evaluation. Systematic Reviews, 8, Article 325. https://doi.org/10.1186/s13643-019-1248-5
• Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ, 339, Article b2535. https://doi.org/10.1136/bmj.b2535
• Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4, Article 1. https://doi.org/10.1186/2046-4053-4-1
• Murray, J., Farrington, D. P., & Sekol, I. (2012). Children’s antisocial behavior, mental health, drug use, and educational performance after parental incarceration: A systematic review and meta-analysis. Psychological Bulletin, 138(2), 175–210. https://doi.org/10.1037/a0026407
• Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71
• Purgato, M., Prina, E., Cadorin, C., Tedeschi, F., Albanese, E., Abdulmalik, J., Bano, B., Betancourt, T. S., Jordans, M., Lund, C., Ostuzzi, G., Papola, D., Piccoli, M., Turrini, G., Tol, W. A., & Barbui, C. (2026). Comparative efficacy of task-sharing psychosocial interventions for preventing common mental disorders in low- and middle-income countries: A systematic review and network meta-analysis with 10,747 participants. Clinical Psychology Review, 129, Article 102785. https://doi.org/10.1016/j.cpr.2026.102785
• Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., Blunt, H., Brigham, T., Chang, S., Clark, J., Conway, A., Couban, R., De Kock, S., Farrah, K., Fehrmann, P., Foster, M., Fowler, S. A., Glanville, J., … Young, S. (2021). PRISMA-S: An extension to the PRISMA Statement for reporting literature searches in systematic reviews. Systematic Reviews, 10, Article 39. https://doi.org/10.1186/s13643-020-01542-z
• Roberts, B. W., Luo, J., Briley, D. A., Chow, P. I., Su, R., & Hill, P. L. (2017). A systematic review of personality trait change through intervention. Psychological Bulletin, 143(2), 117–141. https://doi.org/10.1037/bul0000088
• Rosenek, N., Wake, S., Runton, R., Davies, A., Albaroudi, N., Pollen, A., Ellett, L., & Morriss, J. (2026). The relationship between childhood trauma and adult neuroticism: A systematic review and meta-analysis. Clinical Psychology Review, 124, Article 102700. https://doi.org/10.1016/j.cpr.2026.102700
• Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
• Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: What it is and what it isn’t. BMJ, 312(7023), 71–72. https://doi.org/10.1136/bmj.312.7023.71
• Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher education: A systematic review of meta-analyses. Psychological Bulletin, 143(6), 565–600. https://doi.org/10.1037/bul0000098
• Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, Article j4008. https://doi.org/10.1136/bmj.j4008
• Slavin, R. E. (2008). Perspectives on evidence-based research in education—What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(1), 5–14. https://doi.org/10.3102/0013189X08314117
• Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., & Churchill, R. (2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69, 225–234. https://doi.org/10.1016/j.jclinepi.2015.06.005