Experimental design refers to how participants are allocated to different groups in an experiment. Types of design include repeated measures, independent groups, and matched pairs designs.
Researchers must decide how to allocate participants to the different conditions of the independent variable (IV) before running a study. This choice affects how many participants are needed and which statistical test analyzes the results.
For example, a study with 10 participants could test everyone in both conditions. This is a repeated measures design. Alternatively, the researcher could split the group in half, testing each half in only one condition.
Types
Three types of experimental designs are commonly used:
1. Independent Measures
Independent measures design, also known as between-groups, is an experimental design where different participants are used in each condition of the independent variable. Participants should be randomly allocated to conditions, giving each person an equal chance of being assigned to either group.
For example, in a study with two conditions, one group of participants completes each condition:
- Con: More people are needed than with the repeated measures design (i.e., more time-consuming).
- Pro: Avoids order effects, such as practice or fatigue, because each participant takes part in one condition only.
- Pro: Reduces demand characteristics, since participants who see only one condition find it harder to guess what the study is testing.
- Con: Differences between participants, such as age, gender, or social background, may affect results. These are called participant variables, a type of extraneous variable.
- Control: After the participants have been recruited, they should be randomly assigned to their groups. This should ensure the groups are similar, on average (reducing participant variables).
2. Repeated Measures Design
Repeated measures design is an experimental design where the same participants take part in every condition of the independent variable. It is also known as within-groups or within-subjects design.
- Pro: As the same participants are used in each condition, participant variables (i.e., individual differences) are reduced.
- Con: Order effects can occur, where the order of the conditions affects participants’ behavior rather than the IV itself.
- Con: A practice effect can improve performance in a later condition as participants grow familiar with the task, while a fatigue effect can worsen performance as participants tire.
- Pro: Fewer people are needed as they participate in all conditions (i.e., saves time).
- Control: Researchers counterbalance the order of conditions across participants to spread any order effect evenly, as explained below.
Counterbalancing
Suppose we used a repeated measures design in which all of the participants first learned words in “loud noise” and then learned them in “no noise.”
We expect the participants to learn better in “no noise” because of order effects, such as practice. However, a researcher can control for order effects using counterbalancing.
The sample would be split into two groups: experimental (A) and control (B). For example, group 1 does ‘A’ then ‘B,’ and group 2 does ‘B’ then ‘A.’ This is to eliminate order effects.
Although order effects occur for each participant, they balance each other out in the results because they occur equally in both groups.
This example splits participants into two groups, but counterbalancing can also work within a single participant. In ABBA counterbalancing, each person completes the sequence A, B, B, A, so any steady practice effect is balanced across both conditions.
Counterbalancing controls practice and fatigue effects well because they build up gradually. It is less effective against a carryover effect, where one treatment’s impact lingers into the next condition.
A drug that has not fully left the body is one example. When carryover is likely, researchers space out the conditions or switch to an independent measures design.
3. Matched Pairs Design
A matched pairs design is an experimental design where pairs of participants are matched in terms of key variables, such as age or socioeconomic status.
One member of each matched pair is randomly assigned to the experimental group, and the other to the control group.
- Con: If one participant drops out, the researcher loses both members’ data.
- Pro: Reduces participant variables, since each condition contains people with similar abilities and characteristics.
- Con: Finding closely matched pairs is very time-consuming.
- Pro: Avoids order effects, so counterbalancing is not necessary.
- Con: Matching people exactly is impossible unless they are identical twins.
- Control: Members of each pair are randomly assigned to conditions, though this does not solve every problem.
Summary
Experimental design refers to how participants are allocated to an experiment’s different conditions, or IV levels. There are three types:
- Independent measures / between-groups: Different participants are used in each condition of the independent variable.
- Repeated measures / within groups: The same participants take part in each condition of the independent variable.
- Matched pairs: Each condition uses different participants, but they are matched in terms of important characteristics, e.g., gender, age, intelligence, etc.
Learning Check
Read about each of the experiments below. For each experiment, identify (1) which experimental design was used; and (2) why the researcher might have used that design.
1 . To compare the effectiveness of two different types of therapy for depression, depressed patients were assigned to receive either cognitive therapy or behavior therapy for a 12-week period.
The researchers gave each participant a standardized depression test to assess symptom severity. They then paired participants with similar scores across the two therapy groups.
2 . To assess the difference in reading comprehension between 7 and 9-year-olds, a researcher recruited each group from a local primary school. They were given the same passage of text to read and then asked a series of questions to assess their understanding.
3 . To assess the effectiveness of two different ways of teaching reading, a group of 5-year-olds was recruited from a primary school. Their level of reading ability was assessed, and then they were taught using scheme one for 20 weeks.
At the end of this period, their reading was reassessed, and a reading improvement score was calculated. They were then taught using scheme two for a further 20 weeks, and another reading improvement score for this period was calculated. The reading improvement scores for each child were then compared.
4 . To assess the effect of the organization on recall, a researcher randomly assigned student volunteers to two conditions.
Condition one attempted to recall a list of words organized into meaningful categories. Condition two attempted to recall the same words, randomly grouped on the page.
Experiment Terminology
Ecological validity
The degree to which an investigation represents real-life experiences.
Experimenter effects
These are the ways that the experimenter can accidentally influence the participant through their appearance or behavior.
The clues in an experiment lead the participants to think they know what the researcher is looking for (e.g., the experimenter’s body language).
Independent variable (IV)
The variable the experimenter manipulates (i.e., changes) is assumed to have a direct effect on the dependent variable.
Dependent variable (DV)
Variable the experimenter measures. This is the outcome (i.e., the result) of a study.
Extraneous variables (EV)
All variables which are not independent variables but could affect the results (DV) of the experiment. Extraneous variables should be controlled where possible.
Confounding variables
Variable(s) that have affected the results (DV), apart from the IV. A confounding variable could be an extraneous variable that has not been controlled.
Random Allocation
Randomly allocating participants to independent variable conditions means that all participants should have an equal chance of taking part in each condition.
The principle of random allocation is to avoid bias in how the experiment is carried out and limit the effects of participant variables.
Order effects
Changes in participants’ performance due to their repeating the same or similar test more than once. Examples of order effects include:
(i) practice effect: an improvement in performance on a task due to repetition, for example, because of familiarity with the task;
(ii) fatigue effect: a decrease in performance of a task due to repetition, for example, because of boredom or tiredness.
(iii) carryover effect: a lasting change from one condition that affects performance in a later condition, for example when a drug has not fully left the body or a strategy learned in one task carries over to the next.
Critical Evaluation
Each experimental design controls one threat to a study’s validity while leaving another active. Recognizing these trade-offs helps researchers choose the right design and judge how much to trust a study’s results.
Internal and External Validity
Independent groups designs control order effects but leave participant variables uncontrolled, since different people sit in each condition. Repeated measures designs control participant variables but introduce order effects, because the same people complete every condition.
Matched pairs designs sit in between. They control participant variables on the matched characteristics only, and, like independent groups, they avoid order effects entirely. Choosing a design is a bet about which uncontrolled source of variance is most likely in a specific study.
Within-subjects designs also create an unusual testing situation, since few real-world settings expose someone to every condition in sequence. This trade-off between internal validity and external validity is a design-level property, not a flaw in any single study.
Randomised controlled trials illustrate the internal-validity end of this trade-off. Because a drug’s effect can persist in the body, giving the same patient every condition would let one treatment carry over into the next, so independent groups is the default design in clinical trials.
Reaction-time and psychophysics research sits at the opposite end. Individual differences in speed are large and stable, and most laboratory manipulations are brief and reversible, so repeated measures with counterbalancing is standard there, trading a less natural testing sequence for statistical power.
Contemporary Research
The most-cited test of how well psychology findings hold up is the Reproducibility Project, a large-scale replication effort published in 2015.
Aim: To estimate what proportion of published psychology findings would replicate when the original methods were repeated with adequate statistical power (Open Science Collaboration, 2015).
Method: A team of 270 researchers repeated 100 studies from three major psychology journals, using a sample size large enough to detect the original effect with high power.
Results: Only 35 of the 97 original significant findings replicated with a significant result in the same direction, and the average replication effect was about half the original size.
Conclusion: Many published findings do not survive a well-powered repeat attempt, and small samples and flexible analysis choices are more likely causes than fraud. The result reshaped how psychologists judge evidence, driving wider use of preregistration, larger samples, and multi-site replication.
Large multi-site projects have since tested how widely results generalize. Many Labs 2 (Klein et al., 2018) reran 28 classic effects across 125 samples in 36 countries. Most effects that replicated at all did so in most samples.
Many journals now expect researchers to preregister their design and analysis plan before data collection (Munafò et al., 2017). This practice limits undisclosed changes that can inflate false-positive results.
Key Takeaways
- Three Allocation Methods: Independent groups, repeated measures, and matched pairs are the three ways researchers assign participants to the levels of the independent variable.
- Independent Groups Trade-off: Using different people in each condition avoids order effects, but random allocation is needed to spread out participant variables like age or ability.
- Repeated Measures Trade-off: Testing the same people in every condition controls participant variables and needs fewer participants, but it risks order effects such as practice, fatigue, and carryover.
- Counterbalancing Has Limits: Reversing the order of conditions balances out practice and fatigue effects well, but it does not fully remove an asymmetric carryover effect.
- Matched Pairs Compromise: Pairing different participants on relevant characteristics avoids order effects while recovering some of the sensitivity lost by not testing the same person twice.
- Design Shapes the Statistical Test: Independent-groups data use unrelated-samples tests, while repeated-measures and matched-pairs data use related-samples tests.
- Replication Matters: A well-powered 2015 replication project found that only about a third of significant psychology findings replicated, pushing the field toward preregistration and larger samples.
References
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., Batra, R., Berkics, M., Bernstein, M. J., Berry, D. R., Bialobrzeska, O., Binan, E. D., Bocian, K., Brandt, M. J., Busching, R., … Nosek, B. A. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), Article 0021. https://doi.org/10.1038/s41562-016-0021
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716
