Discriminant Validity

Discriminant validity, also known as divergent validity, is the extent to which a measure does not correlate strongly with measures of different, unrelated constructs. Here, a construct is a behavior, attitude, or concept, particularly one that is not directly observable.

The primary method for assessing discriminant validity is examining the correlation coefficients between the measure in question and measures of different constructs.

Weak or low correlations, typically closer to 0, suggest good discriminant validity. These correlations are sometimes called discriminant validity coefficients.

For example, a test measuring introversion (target construct) should have low correlations with a test measuring mathematical ability (comparison construct), because these are distinct constructs.

If the correlation is high, it suggests the introversion test lacks discriminant validity and may actually be measuring something else, like general intelligence.

Examples of discriminant validity

• A study might assess the discriminant validity of a new measure of self-esteem (target construct) by correlating it with a measure of social desirability (comparison construct). A low correlation would suggest that the self-esteem measure is not simply reflecting a tendency to present oneself in a positive light, but rather a distinct construct.
• Researchers developing a new measure of job satisfaction (target construct) might want to ensure it has discriminant validity from measures of organizational commitment (comparison construct). This means demonstrating that the job satisfaction measure is not simply reflecting a general positive attitude towards the organization, but rather a specific assessment of satisfaction with one’s job.
• In a study on personality traits, researchers might find that a measure of extraversion (target construct) has a low correlation with a measure of conscientiousness (comparison construct), suggesting that these are indeed separate constructs.

Discriminant vs. Convergent Validity

Discriminant validity and convergent validity are both crucial aspects of construct validity.

Together, they show the extent to which a test or measure accurately assesses the underlying construct it is designed to measure.

They provide complementary information about the extent to which a measure is assessing the intended construct and differentiating it from other constructs:

What Each Type Examines:

• Discriminant validity examines the relationship between a target measure and comparison measures of different, theoretically distinct constructs.
• Convergent validity examines the relationship between a measure and other measures of the same construct, often using different methods.

Expected Correlations:

• Discriminant validity is supported by weak or low correlations between the target measure and measures of dissimilar constructs. This indicates that the measure is not capturing variance from those unrelated constructs.
• Convergent validity is supported by strong, positive correlations between the target measure and different measures of the same construct. This suggests that the measures are capturing overlapping aspects of the target construct.

Why Each Matters:

• Discriminant validity helps demonstrate the uniqueness of the construct being measured, showing it is distinct from other constructs.
• Convergent validity helps establish that the measure is actually capturing the intended construct, as evidenced by its agreement with other measures of the same construct.

A measure needs both kinds of evidence because each rules out a different failure. Correlating with everything is just as damaging to construct validity as correlating with nothing.

Neither extreme is safe. A measure that moves in step with every other scale captures nothing specific to its own construct.

Most psychological constructs have no single, independently verifiable standard to check a new measure against. Validity is instead built up piece by piece.

Discriminant evidence rules out overlap with the wrong constructs; convergent evidence confirms overlap with the right ones.

How is discriminant validity measured?

Discriminant validity is a crucial aspect of construct validity that helps ensure a test is truly measuring its intended construct and not being influenced by other, unrelated constructs.

Discriminant validity is not an all-or-none property. It is a matter of degree, and the strength of the evidence can vary.

Multiple sources of evidence should be considered to develop a strong argument for the discriminant validity of a measure.

1. Define the Target and Comparison Constructs:

• Target Construct: Clearly articulate the specific theoretical attribute or quality your measure is intended to assess. What does this construct encompass, and what does it exclude?
• Comparison Constructs: Identify other constructs that are theoretically distinct from your target construct but potentially related. These comparison constructs will be used to assess discriminant validity.

Here is an example to illustrate this:

Imagine researchers are developing a new measure of job satisfaction. To assess its discriminant validity, they might choose to compare it with a measure of organizational commitment.

• Target Construct: Job satisfaction, defined as the level of contentment and fulfillment an individual experiences in their job.
• Comparison Construct: Organizational commitment, defined as the degree to which an individual identifies with and feels loyal to their organization.

While these constructs are related, they are also distinct.

Someone might be very committed to their organization but dissatisfied with their specific job due to factors like lack of autonomy or challenging work tasks.

• Expected Finding: If the job satisfaction measure has good discriminant validity, it should show a weak or low correlation with the organizational commitment measure.
• Interpretation: A low correlation would suggest the job satisfaction measure is assessing a construct that is separate from general organizational commitment, supporting its discriminant validity.

2. Select Valid Measures:

• Choose reliable and valid measures for both the target and comparison constructs. The quality of the comparison measures directly impacts the assessment of discriminant validity.
• If the comparison measures are unreliable or not valid for their intended constructs, the resulting correlations will be misleading.
• Consider using multiple methods (e.g., self-report, other-report, behavioral observation) to measure each construct to mitigate method variance, which can inflate correlations and obscure discriminant validity.

3. Administer Measures and Collect Data:

• Administer the selected measures to a sample that is relevant to the constructs being studied.
• Ensure appropriate data collection procedures to minimize potential sources of error or bias.

4. Analyze Correlations:

• Calculate the correlations between the target measure and the measures of the comparison constructs.
• Interpreting Correlations:
  • Low correlations between the target measure and measures of dissimilar constructs generally indicate good discriminant validity. This suggests the target measure is not capturing variance from those unrelated constructs. The exact threshold for “low” correlations can vary depending on the field and the specific constructs involved, but correlations below 0.3 are often considered acceptable.
  • High correlations may signal potential issues with discriminant validity, raising concerns that the measure is not sufficiently distinct from the comparison measures. However, it is crucial to consider the theoretical relationships between constructs when interpreting correlations.
    • For example, moderate correlations between measures of anxiety and depression might be acceptable given the known comorbidity between these constructs.
    • It’s also important to consider the context of the research. What constitutes acceptable levels of correlation might differ across settings and populations.
• Statistical Techniques:
  • The most common approach is to examine Pearson correlation coefficients.
  • Multitrait-Multimethod (MTMM) Matrix: This method involves measuring multiple traits with multiple methods, allowing researchers to systematically analyze patterns of correlations and assess both convergent and discriminant validity while controlling for method variance. The MTMM matrix can be evaluated using various statistical techniques, such as confirmatory factor analysis (CFA).
  • Factor Analysis: Both exploratory and confirmatory factor analyses can be used to assess discriminant validity. By examining factor loadings, researchers can see how clearly the target measure loads onto a distinct factor from the comparison measures.

5. Consider Theoretical Implications:

• Theory First: Statistical analyses must always be interpreted in light of theoretical expectations.
  • A high correlation between two measures might not necessarily indicate poor discriminant validity if the constructs are theoretically expected to overlap.
  • Conversely, a low correlation might not be sufficient evidence for discriminant validity if there are strong theoretical reasons to expect a relationship between the constructs.
• Researchers should consider potential confounding variables or alternative explanations for the observed correlations.

6. Examine Response Processes (If Feasible):

• Gathering evidence about how respondents interpret and respond to test items can provide valuable insights into discriminant validity.
• Techniques like think-aloud protocols, cognitive interviews, and analysis of response times can help determine whether individuals are relying on the intended construct when answering questions.
• This type of evidence can be challenging to obtain but can be highly informative.

7. Beyond Correlations: The Evolving Landscape of Discriminant Validity

While correlations remain a key tool, the sources also highlight a shift in thinking about discriminant validity:

• Moving from Correlations to Causation: Researchers are increasingly focusing on understanding the causal mechanisms that link the constructs to measurement outcomes. This involves:
  • Developing theories about how the target construct should influence responses to test items.
  • Gathering evidence to support those theories, going beyond simple correlations.
  • Ruling out alternative explanations for the observed relationships.
• Focus on Item Response Processes: There is a growing emphasis on examining the cognitive processes involved in responding to test items as a way to strengthen claims about discriminant validity.

Embracing these evolving perspectives means combining statistical and theoretical reasoning. Together, the two approaches help researchers assess discriminant validity more effectively and strengthen the quality of their research findings.

limitations of discriminant validity?

Reliance on Correlations

The primary method for assessing discriminant validity involves examining correlations between measures, but several factors can influence those correlations, including:

• Measurement Error: Unreliability in either measure can attenuate correlations, making it harder to draw firm conclusions about discriminant validity.
• Restriction of Range: A sample with limited variability on either construct lowers the correlations, which can make discriminant validity look stronger than it really is.
• Sample Characteristics: What counts as a “low” correlation varies across populations, so the specific research context matters when interpreting it.
• Method Variance: Using the same method (e.g., self-report) for both measures can inflate correlations and obscure true discriminant validity.

Beyond Correlations: The Need for Theoretical Justification

Low correlations alone are not sufficient evidence for discriminant validity. Interpretation must always be grounded in a strong theoretical framework.

• Acceptable Levels of Correlation: What counts as “acceptable” varies with the specific constructs and the theory behind them. Constructs expected to overlap may show moderate correlations without harming discriminant validity.
• Potential for Misinterpretation: A rigid focus on correlations alone can wrongly conclude two constructs are distinct. A true but weak relationship cannot be ruled out from statistics alone.

Addressing the Limitations of Discriminant Validity

A multifaceted approach to validity assessment moves researchers beyond the limitations of relying solely on correlations. This gives a fuller understanding of how distinct and meaningful their constructs really are.

1. Go Beyond Simple Correlations: Embrace a Multifaceted Approach

• Recognize the Influence of Measurement Error: Unreliable measures attenuate correlations, so weak reliability alone can look like poor discriminant validity. Correction-for-attenuation techniques help adjust for this.
• Address Restriction of Range: A sample with limited variability lowers correlations and can make discriminant validity look stronger than it really is. Disattenuation formulas can correct for this.
• Account for Sample Characteristics: What counts as a “low” correlation varies across populations, so findings should not be generalized beyond the sample studied without further evidence.
• Control for Method Variance: Using the same method (e.g., self-report) for both measures can inflate correlations. Multitrait-multimethod (MTMM) designs disentangle trait variance from method variance.

2. Ground Interpretation in Strong Theory: Correlations Are Not Enough

• Set Clear Theoretical Expectations: Before checking correlations, state how the target and comparison constructs should relate: completely distinct, moderately overlapping, or highly related.
• Determine Acceptable Correlation Levels: There is no universal cutoff for discriminant validity. What counts as “low” depends on the constructs and the theory behind them.
• Consider Alternative Explanations: A low correlation alone does not prove two constructs are distinct. Weak measurement, restriction of range, or suppressor variables could explain it too.

3. Investigate Response Processes: Unveiling the “Black Box”

• Go Beyond Statistics: Statistical relationships alone cannot settle discriminant validity. Examining how people interpret test items reveals whether they are really responding to the intended construct.
• Use Qualitative Techniques: Think-aloud protocols, cognitive interviews, and focus groups show how people understand items, revealing construct-irrelevant variance that statistics alone would miss.
• Seek Participant Feedback: Engage in member checking, so participants can confirm the researcher’s understanding of the construct matches how they themselves experience it.

4. Address Construct Underrepresentation and Irrelevant Variance

• Expand the Item Pool: Make sure the item pool samples the full breadth of the target construct. A measure that is too narrow misses important facets of it.
• Control Irrelevant Variance: Check for construct-irrelevant variance, such as response styles or test anxiety. Factor analysis can flag items unduly influenced by it for revision.
• Consider Contextual Factors: Construct validity is not absolute. Cultural factors, social norms, and situational variables can shape it, so generalize across contexts cautiously.

5. Embrace the Evolving Nature of Constructs

• Constructs Change Over Time: Psychological attributes are not static. Their definitions evolve with new research, cultural shifts, and better measurement, so measures may need revising.
• Ongoing Evaluation: Discriminant validity is not a one-time determination. It needs continuous reassessment as knowledge of the construct grows and new evidence emerges.

Additional Considerations:

• Argument-Based Validation: Build a clear interpretive argument linking a test’s intended use, the inferences drawn from it, and the evidence supporting those inferences.
• Consequences of Test Use: Even a valid test can have unintended effects. Weigh its impact on different stakeholder groups and work to minimize harm.

Reading List

Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in
variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1),
115-135. https://doi.org/10.1007/s11747-014-0403-8

Lucas, R. E., Diener, E., & Suh, E. (1996). Discriminant validity of well-being measures. Journal of Personality and Social Psychology, 71(3), 616-628. https://doi.org/10.1037/0022-3514.71.3.616

Mathieu, J. E., & Farr, J. L. (1991). Further evidence for the discriminant validity of measures of organizational commitment, job involvement, and job satisfaction. Journal of Applied Psychology, 76(1), 127-133. https://doi.org/10.1037/0021-9010.76.1.127

Reichardt, C. S., & Coleman, S. C. (1995). The criteria for convergent and discriminant validity in a multitraitmultimethod matrix. Multivariate Behavioral Research, 30(4), 513-538. https://doi.org/10.1207/s15327906mbr3004_3

Rönkkö, M., & Cho, E. (2022). An updated guideline for assessing discriminant validity. Organizational Research Methods, 25(1), 6-14. https://doi.org/10.1177/1094428120968614

Shaffer, J. A., DeGeest, D., & Li, A. (2016). Tackling the problem of construct proliferation: A guide to
assessing the discriminant validity of conceptually related constructs. Organizational Research Methods, 19(1), 80-110. https://doi.org/10.1177/1094428115598239