An ANOVA (analysis of variance) is a statistical test that checks whether the means of two or more groups differ by more than chance would produce. It does this by comparing how much the group means vary from one another with how much the scores vary inside each group.
Another key part of ANOVA is that it splits the independent variable into two or more groups.
For example, one or more groups might be expected to influence the dependent variable. The other group serves as a control group, which does not receive the treatment and is not expected to influence the outcome.
Assumptions of ANOVA
The assumptions of the ANOVA test are the same as the general assumptions for any parametric test:
- Independence: No one can appear in two groups, and one person’s score cannot influence another’s. This applies to independent-samples (between-subjects) designs.
- Normality: The dependent variable should be normally distributed within each group, so the middle scores are the most frequent and the extreme scores the least frequent.
- Equal Variances: Population variances must be equal (homoscedastic): the spread of scores, such as the standard deviation, is similar across populations. ANOVA is most sensitive to this one.
- Group Sizes: Equal sample sizes are desirable but not required. Balanced groups simply tolerate unequal variances far better (Box, 1954).
How Robust Is ANOVA to Violated Assumptions?
ANOVA is not equally sensitive to every assumption. Simulation studies suggest that non-normality is the smaller worry and unequal variances the bigger one.
A Monte Carlo study across a wide range of distributions found the F test robust, in terms of Type I error, in every condition examined. This held whether group sizes were equal or unequal (Blanca et al., 2017).
Unequal variances are the genuine threat. When group sizes also differ, a variance ratio above 1.5 was a rule-of-thumb warning for the F test’s robustness (Blanca et al., 2018).
- Aim: To work out mathematically how far unequal variances distort the F test in a one-way ANOVA.
- Method: A mathematical analysis, with no participants. Box worked out the distribution of the quantities behind the F statistic for normal data, then checked his approximations numerically.
- Results: With equal group sizes, moderate inequality of variance did not seriously affect the test. With unequal group sizes, much larger discrepancies appeared (Box, 1954).
- Conclusion: The F test tolerates unequal variances only when group sizes are balanced.
Box’s result is still the theoretical basis for modern advice. It covers only unequal variances with normal data, and later simulations have mapped the problem more precisely.
- Aim: To argue, and test by simulation, that psychologists should use Welch’s F-test instead of the classical F test as the default for one-way ANOVA.
- Method: A methodological article combining a review of researchers’ practices with Monte Carlo simulations of three tests (classical F, Welch’s, Brown-Forsythe). No human participants were involved.
- Results: Under realistic departures from equal variances, the classical F test could give severely biased results and invalid inferences. Welch’s test was the better alternative across realistic scenarios.
- Conclusion: Welch’s F-test should be the default for one-way designs, because it protects against unequal variances at little cost (Delacre et al., 2019).
Welch’s test weights each group by its precision and adjusts the degrees of freedom downward. When variances are equal, it gives almost the same answer as the classic F test. The switch costs little.
Simulations have limits: the results hold only for the scenarios chosen. Welch’s correction also does not help with non-independent scores or strong non-normality in small samples.
Types of ANOVA Tests
There are different types of ANOVA tests. The two most common are a “One-Way” and a “Two-Way.”
The difference between these two types depends on the number of independent variables in your test.
The ANOVA family is larger than these two tests. The table shows when each variant applies.
| Type | Use it when |
|---|---|
| One-way ANOVA | One between-subjects factor has three or more levels, such as three therapies. |
| Factorial (two-way, three-way) ANOVA | Two or more factors are crossed, and you want main effects and interactions. A two-way ANOVA has exactly two factors. |
| Repeated-measures ANOVA | The same participants are measured under every level. |
| Mixed-design ANOVA | One factor is between-subjects and another is within-subjects. |
| ANCOVA | A continuous covariate must be statistically controlled. |
| MANOVA | Several dependent variables are analysed together. |
One-way ANOVA
A one-way ANOVA (analysis of variance) has one categorical independent variable, known as a factor, which sorts people into groups such as type of therapy. Its dependent variable must be continuous (interval or ratio level) and normally distributed.
The factor divides cases into two or more mutually exclusive levels, meaning the categories or groups of the factor. Each person sits in one level only.
The one-way ANOVA test for differences in the means of the dependent variable is broken down by the levels of the independent variable.
An example of a one-way ANOVA is testing a therapeutic intervention (CBT, medication, placebo) on depression scores in a clinical sample. One factor, three levels.
Note: Both the One-Way ANOVA and the Independent Samples t-Test can compare the means for two groups. However, only the One-Way ANOVA can compare the means across three or more groups.
P Value Calculator From F Ratio (ANOVA)
Two-way (factorial) ANOVA
A two-way ANOVA (analysis of variance) has two categorical independent variables, or factors, and a normally distributed continuous (i.e., interval or ratio level) dependent variable.
The independent variables divide cases into two or more mutually exclusive levels, categories, or groups. A two-way ANOVA is a factorial ANOVA with exactly two factors. Factorial designs can have three or more.
A factorial ANOVA can test several influences on depression scores at once. Imagine three factors: social contact (high, medium, low), job status (employed, self-employed, unemployed, retired), and family history (none, some). That makes it a three-way ANOVA.
What are “Groups” or “Levels”?
In ANOVA, “groups” or “levels” refer to the different categories of the independent variable being compared.
For example, if the independent variable is “eggs,” the levels might be Non-Organic, Organic, and Free Range Organic. The dependent variable could then be the price per dozen eggs.
The egg example maps onto ANOVA’s vocabulary like this.
| Term | In the egg example | Meaning |
|---|---|---|
| Factor (independent variable) | Egg type | The categorical variable that defines the groups |
| Levels (groups or conditions) | Non-organic, organic, free-range organic | The categories of the factor |
| Dependent variable | Price per dozen | The continuous outcome being compared |
Most psychology experiments use fixed levels: the specific conditions the researcher chose, such as these three egg types. Conclusions then apply to those levels only. Random levels, such as twelve schools drawn from all schools, aim to generalise to the wider population.
ANOVA F -value
The test statistic for an ANOVA is denoted as F. It divides the variance between the group means by the variance within the groups: F = mean square between / mean square within.
How the F Ratio Works
The within-groups estimate reflects random error alone, because everyone in a group received the same treatment. The between-groups estimate reflects that error plus any real treatment effect.
If the treatment does nothing, F hovers near one. If it matters, F grows.
The ANOVA F value can tell you if there is a significant difference between the levels of the independent variable, when p < .05.
A higher F is stronger evidence against equal means, but it must be read against its degrees of freedom. The same F can be significant in one study and not in another.
The null hypothesis is that all population means are equal. The alternative is that at least one mean differs, not that every mean does.
Three quantities build F from the raw data:
- Sum of squares (SS): The sum of squared distances from a mean. The total splits into a between-groups part and a within-groups part.
- Degrees of freedom (df): k − 1 between groups and N − k within groups, where k is the number of groups and N the number of scores.
- Mean square (MS): A sum of squares divided by its degrees of freedom, which turns it into a variance estimate.
Post Hoc Tests: Which Means Differ?
ANOVA alone does not tell us which means differ. That is why a significant F is called an omnibus result: it says some difference exists, not where.
When the initial F test indicates that significant differences exist between group means, post hoc tests locate them. They suit cases with no specific hypotheses to test.
Post hoc tests compare each pair of means (like t-tests), but unlike t-tests, they correct the significance estimate to account for the multiple comparisons.
The follow-up you choose depends on whether the comparison was planned:
- Tukey’s HSD (post hoc): The usual choice for all pairs (Tukey, 1949). It sets one critical difference, so any two means further apart than that differ significantly.
- Planned contrasts: Comparisons specified from theory before seeing the data, such as CBT versus the other two groups. Few are made, so no heavy correction is needed.
Two cautions apply. Running post hoc tests after a non-significant F invites the false positives the omnibus test exists to prevent. And a significant F with one non-significant pair is not a contradiction.
What Does “Replication” Mean?
Replication requires a study to be repeated with different subjects and experimenters. This would enable a statistical analyzer to confirm a prior study by testing the same hypothesis with a new sample.
How to run an ANOVA?
For large datasets, it is best to run an ANOVA in statistical software such as R or Stata. Let’s refer to our Egg example above.
Code the three egg types as numbers. Then declare the variable a factor. Otherwise the software treats the codes 1, 2 and 3 as a scale and tests a straight-line trend instead of differences among three means.
Egg type is the independent variable, and price per dozen is the dependent variable. Brand name and laying date are confounding variables. Hold them constant or randomise them.
Using data and the aov() command in R, we could then determine the impact Egg Type has on the price per dozen eggs.
Plot the data first to check spread and outliers. R’s oneway.test() gives Welch’s version, which does not assume equal variances.
Worked Example of a One-Way ANOVA
Imagine a small trial in which 30 adults with low mood are randomly allocated, ten per group, to cognitive behavioural therapy (CBT), medication, or a waiting-list control. The table shows depression scores after eight weeks, where higher means worse. These data are invented for illustration.
| Group | Scores | Mean | SD |
|---|---|---|---|
| CBT | 11, 15, 9, 18, 14, 12, 16, 13, 8, 19 | 13.5 | 3.63 |
| Medication | 15, 20, 13, 22, 17, 14, 21, 18, 12, 23 | 17.5 | 3.92 |
| Waiting list | 20, 27, 17, 30, 24, 19, 28, 22, 16, 26 | 22.9 | 4.84 |
Calculating the F Ratio
The calculation runs in six steps:
- Grand mean: All 30 scores sum to 539, so the grand mean is 17.97.
- Between-groups SS: Each group mean’s squared distance from the grand mean, weighted by n = 10, gives 445.07.
- Within-groups SS: Each score’s squared distance from its own group mean, summed, gives 467.9.
- Degrees of freedom: 2 between groups (3 − 1) and 27 within groups (30 − 3).
- Mean squares: 445.07 / 2 = 222.53 between groups, and 467.9 / 27 = 17.33 within groups.
- F ratio: 222.53 / 17.33 = 12.84, which beats the .05 critical value of 3.35 (p = .00012).
The null hypothesis of equal population means is rejected. In APA style, the result reads F(2, 27) = 12.84, p < .001.
Effect Size and Follow-Up Tests
A significant F does not say how large the difference is. An effect size does. Eta squared (η²) is the share of total variability linked to group membership: 445.07 / 912.97 = .49.
Sample eta squared is biased upward, so omega squared (ω²) corrects for chance differences between means. Here ω² is .44.
Cohen (1988) suggested benchmarks for small, medium and large effects, shown here in terms of Cohen’s f and η².
| Effect | Cohen’s f | η² (approx.) |
|---|---|---|
| Small | .10 | .01 |
| Medium | .25 | .06 |
| Large | .40 | .14 |
Cohen’s f here is 0.98, far above the large benchmark.
A Tukey HSD test then compares every pair of means. Any gap larger than 4.62 points is significant.
The waiting-list group scored higher than the CBT group (9.4 points, p < .001) and the medication group (5.4 points, p = .019). CBT and medication did not differ significantly (4.0 points, p = .099).
A significant F with one non-significant pair is a normal outcome, not an error.
ANOVA vs. t-test?
T-tests and ANOVA tests are both statistical techniques used to compare the means of groups.
The t-test compares exactly two groups. ANOVA is used when you want to compare more than two levels of an independent variable.
Referring back to our egg example, testing Non-Organic vs. Organic would require a t-test while adding in Free Range as a third option demands ANOVA.
Rather than generate a t-statistic, ANOVA results in an f-statistic to determine statistical significance.
With two groups, F equals t squared. The p-values match.
Why Not Run Several t-Tests?
Each test carries its own 5% chance of a false positive (a Type I error), and these chances accumulate. If the tests were independent, the overall risk would climb like this:
| Groups | Pairwise comparisons | Chance of at least one false positive |
|---|---|---|
| 3 | 3 | About 14% |
| 4 | 6 | About 26% |
| 6 | 15 | About 54% |
Because pairwise comparisons share data, the true rate is a little lower, but the inflation is real.
A single ANOVA is one decision about all the means. It holds the false-positive rate at 5% however many groups there are.
FAQS
What does anova stand for?
ANOVA stands for Analysis of Variance. It’s a statistical method to analyze differences among group means in a sample. ANOVA tests the hypothesis that the means of two or more populations are equal, generalizing the t-test to more than two groups.
It’s commonly used in experiments where various factors’ effects are compared. It can also handle complex experiments with factors that have different numbers of levels.
When to use anova?
ANOVA should be used when one independent variable has three or more levels (categories or groups). It’s designed to compare the means of these multiple groups.
What does an anova test tell you?
An ANOVA test tells you if there are significant differences between the means of three or more groups. If the test result is significant, it suggests that at least one group’s mean differs from the others. It does not, however, specify which groups are different from each other.
Why do you use chi-square instead of ANOVA?
You use the chi-square test instead of ANOVA when dealing with categorical data to test associations or independence between two categorical variables. In contrast, ANOVA is used for continuous data to compare the means of three or more groups.
Key Takeaways
- Purpose: ANOVA compares the means of two or more groups on one continuous outcome. With three or more groups it replaces repeated t-tests.
- F Ratio: F divides the variance between group means by the variance within groups. A value near 1 fits chance.
- Omnibus Result: A significant F says at least one mean differs, not which. Post hoc tests or planned contrasts locate it.
- Assumptions: Scores must be independent, roughly normal and similar in variance. Equal group sizes are desirable, not required.
- Robustness: ANOVA tolerates non-normality, but not unequal variances combined with unequal group sizes. Welch’s F is the recommended default (Delacre et al., 2019).
- Effect Size: Report eta squared, omega squared or Cohen’s f with F, because significance alone does not show how large a difference is.
References
Blanca, M. J., Alarcón, R., Arnau, J., Bono, R., & Bendayan, R. (2017). Non-normal data: Is ANOVA still a valid option? Psicothema, 29(4), 552–557. https://doi.org/10.7334/psicothema2016.383
Blanca, M. J., Alarcón, R., Arnau, J., Bono, R., & Bendayan, R. (2018). Effect of variance ratio on ANOVA robustness: Might 1.5 be the limit? Behavior Research Methods, 50(3), 937–962. https://doi.org/10.3758/s13428-017-0918-2
Box, G. E. P. (1954). Some theorems on quadratic forms applied in the study of analysis of variance problems, I. Effect of inequality of variance in the one-way classification. The Annals of Mathematical Statistics, 25(2), 290–302. https://doi.org/10.1214/aoms/1177728786
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
Delacre, M., Leys, C., Mora, Y. L., & Lakens, D. (2019). Taking parametric assumptions seriously: Arguments for the use of Welch’s F-test instead of the classical F-test in one-way ANOVA. International Review of Social Psychology, 32(1), Article 13. https://doi.org/10.5334/irsp.198
Tukey, J. W. (1949). Comparing individual means in the analysis of variance. Biometrics, 5(2), 99–114. https://doi.org/10.2307/3001913
