Discriminant Validity

Discriminant validity, also known as divergent validity, is the extent to which a measure does not correlate strongly with measures of different, unrelated constructs.

Here, a construct is a behavior, attitude, or concept, particularly one that is not directly observable.

The primary method for assessing discriminant validity is examining the correlation coefficients between the measure in question and measures of different constructs.

Weak or low correlations, typically closer to 0, suggest good discriminant validity. These correlations are sometimes called discriminant validity coefficients.

For example, a test measuring introversion (target construct) should have low correlations with a test measuring mathematical ability (comparison construct), because these are distinct constructs.

If the correlation is high, it suggests the introversion test lacks discriminant validity and may actually be measuring something else, like general intelligence.

Examples of discriminant validity

Here are a few examples:

• A study might assess the discriminant validity of a new measure of self-esteem (target construct) by correlating it with a measure of social desirability (comparison construct). A low correlation would suggest that the self-esteem measure is not simply reflecting a tendency to present oneself in a positive light, but rather a distinct construct.
• Researchers developing a new measure of job satisfaction (target construct) might want to ensure it has discriminant validity from measures of organizational commitment (comparison construct). This means demonstrating that the job satisfaction measure is not simply reflecting a general positive attitude towards the organization, but rather a specific assessment of satisfaction with one’s job.
• In a study on personality traits, researchers might find that a measure of extraversion (target construct) has a low correlation with a measure of conscientiousness (comparison construct), suggesting that these are indeed separate constructs.

Discriminant vs. Convergent Validity

Discriminant validity and convergent validity are both crucial aspects of construct validity, which aims to determine the extent to which a test or measure accurately assesses the underlying construct it is designed to measure.

They provide complementary information about the extent to which a measure is assessing the intended construct and differentiating it from other constructs:

Focus:

• Discriminant validity examines the relationship between a target measure and comparison measures of different, theoretically distinct constructs.
• Convergent validity examines the relationship between a measure and other measures of the same construct, often using different methods.

Expected Correlations:

• Discriminant validity is supported by weak or low correlations between the target measure and measures of dissimilar constructs. This indicates that the measure is not capturing variance from those unrelated constructs.
• Convergent validity is supported by strong, positive correlations between the target measure and different measures of the same construct. This suggests that the measures are capturing overlapping aspects of the target construct.

Purpose:

• Discriminant validity helps demonstrate the uniqueness of the construct being measured, showing it is distinct from other constructs.
• Convergent validity helps establish that the measure is actually capturing the intended construct, as evidenced by its agreement with other measures of the same construct.

How is discriminant validity measured?

Discriminant validity is a crucial aspect of construct validity that helps ensure a test is truly measuring its intended construct and not being influenced by other, unrelated constructs.

Discriminant validity is not an all-or-none property. It is a matter of degree, and the strength of the evidence can vary.

Multiple sources of evidence should be considered to develop a strong argument for the discriminant validity of a measure.

1. Define the Target and Comparison Constructs:

• Target Construct: Clearly articulate the specific theoretical attribute or quality your measure is intended to assess. What does this construct encompass, and what does it exclude?
• Comparison Constructs: Identify other constructs that are theoretically distinct from your target construct but potentially related. These comparison constructs will be used to assess discriminant validity.

Here is an example to illustrate this:

Imagine researchers are developing a new measure of job satisfaction. To assess its discriminant validity, they might choose to compare it with a measure of organizational commitment.

• Target Construct: Job satisfaction, defined as the level of contentment and fulfillment an individual experiences in their job.
• Comparison Construct: Organizational commitment, defined as the degree to which an individual identifies with and feels loyal to their organization.

While these constructs are related, they are also distinct. Someone might be very committed to their organization but dissatisfied with their specific job due to factors like lack of autonomy or challenging work tasks.

• Expected Finding: If the job satisfaction measure has good discriminant validity, it should show a weak or low correlation with the organizational commitment measure.
• Interpretation: A low correlation would suggest the job satisfaction measure is assessing a construct that is separate from general organizational commitment, supporting its discriminant validity.

2. Select Valid Measures:

• Choose reliable and valid measures for both the target and comparison constructs. The quality of the comparison measures directly impacts the assessment of discriminant validity.
• If the comparison measures are unreliable or not valid for their intended constructs, the resulting correlations will be misleading.
• Consider using multiple methods (e.g., self-report, other-report, behavioral observation) to measure each construct to mitigate method variance, which can inflate correlations and obscure discriminant validity.

3. Administer Measures and Collect Data:

• Administer the selected measures to a sample that is relevant to the constructs being studied.
• Ensure appropriate data collection procedures to minimize potential sources of error or bias.

4. Analyze Correlations:

• Calculate the correlations between the target measure and the measures of the comparison constructs.
• Interpreting Correlations:
  • Low correlations between the target measure and measures of dissimilar constructs generally indicate good discriminant validity. This suggests the target measure is not capturing variance from those unrelated constructs. The exact threshold for “low” correlations can vary depending on the field and the specific constructs involved, but correlations below 0.3 are often considered acceptable.
  • High correlations may signal potential issues with discriminant validity, raising concerns that the measure is not sufficiently distinct from the comparison measures. However, it is crucial to consider the theoretical relationships between constructs when interpreting correlations.
    • For example, moderate correlations between measures of anxiety and depression might be acceptable given the known comorbidity between these constructs.
    • It’s also important to consider the context of the research. What constitutes acceptable levels of correlation might differ across settings and populations.
• Statistical Techniques:
  • The most common approach is to examine Pearson correlation coefficients.
  • Multitrait-Multimethod (MTMM) Matrix: This method involves measuring multiple traits with multiple methods, allowing researchers to systematically analyze patterns of correlations and assess both convergent and discriminant validity while controlling for method variance. The MTMM matrix can be evaluated using various statistical techniques, such as confirmatory factor analysis (CFA).
  • Factor Analysis: Both exploratory and confirmatory factor analyses can be used to assess discriminant validity. By examining factor loadings, researchers can see how clearly the target measure loads onto a distinct factor from the comparison measures.

5. Consider Theoretical Implications:

• Crucially, statistical analyses must be interpreted in light of theoretical expectations.
  • A high correlation between two measures might not necessarily indicate poor discriminant validity if the constructs are theoretically expected to overlap.
  • Conversely, a low correlation might not be sufficient evidence for discriminant validity if there are strong theoretical reasons to expect a relationship between the constructs.
• Researchers should consider potential confounding variables or alternative explanations for the observed correlations.

6. Examine Response Processes (If Feasible):

• Gathering evidence about how respondents interpret and respond to test items can provide valuable insights into discriminant validity.
• Techniques like think-aloud protocols, cognitive interviews, and analysis of response times can help determine whether individuals are relying on the intended construct when answering questions.
• This type of evidence can be challenging to obtain but can be highly informative.

7. Beyond Correlations: The Evolving Landscape of Discriminant Validity

While correlations remain a key tool, the sources also highlight a shift in thinking about discriminant validity:

• Moving from Correlations to Causation: Researchers are increasingly focusing on understanding the causal mechanisms that link the constructs to measurement outcomes. This involves:
  • Developing theories about how the target construct should influence responses to test items.
  • Gathering evidence to support those theories, going beyond simple correlations.
  • Ruling out alternative explanations for the observed relationships.
• Focus on Item Response Processes: There is a growing emphasis on examining the cognitive processes involved in responding to test items as a way to strengthen claims about discriminant validity.

By embracing these evolving perspectives and employing a combination of statistical and theoretical reasoning, researchers can more effectively assess discriminant validity and enhance the quality and meaningfulness of their research findings.

What are the limitations of discriminant validity?

Reliance on Correlations

The primary method for assessing discriminant validity involves examining correlations between measures. However, correlations can be influenced by various factors, including:

• Measurement Error: Unreliability in either the target measure or the comparison measures can attenuate correlations, making it difficult to draw firm conclusions about discriminant validity.
• Restriction of Range: If the sample used to calculate correlations has limited variability on either the target or comparison constructs, the resulting correlations will be lower than they would be in a more heterogeneous population. This can lead to an overestimation of discriminant validity.
• Sample Characteristics: Correlations can vary across different populations or subgroups. What constitutes a “low” correlation for one group might not be appropriate for another. This highlights the importance of considering the specific context of the research.
• Method Variance: Using the same method (e.g., self-report) to measure both the target and comparison constructs can inflate correlations due to shared method variance. This can obscure true discriminant validity.

Beyond Correlations: The Need for Theoretical Justification

Low correlations alone are not sufficient evidence for discriminant validity. Interpretation must always be grounded in a strong theoretical framework.

• Acceptable Levels of Correlation: What constitutes an “acceptable” level of correlation can vary widely depending on the specific constructs being studied and the theoretical expectations. Constructs that are theoretically expected to overlap may exhibit moderate correlations without necessarily indicating poor discriminant validity.
• Potential for Misinterpretation: A rigid focus on correlations can lead to the erroneous conclusion that two constructs are distinct simply because their measures exhibit low correlations. The possibility of a true, but weak, relationship cannot be ruled out solely based on statistical evidence.

Addressing the Limitations of Discriminant Validity

By adopting a multifaceted approach to validity assessment, researchers can move beyond the limitations of relying solely on correlations and gain a more comprehensive understanding of the distinctiveness and meaningfulness of their constructs.

1. Go Beyond Simple Correlations: Embrace a Multifaceted Approach

• Recognize the Influence of Measurement Error: Always consider the reliability of both the target measure and the comparison measures when interpreting correlations. Unreliability can attenuate correlations, leading to an underestimation of the true relationship between constructs. Statistical techniques, such as correction for attenuation, can help to adjust for the impact of measurement error.
• Address Restriction of Range: Be mindful of potential restriction of range in the sample used to calculate correlations. If the variability on either the target or comparison constructs is limited, the resulting correlations will be lower than they would be in a more diverse population. Disattenuation formulas can be used to estimate the correlation in a less restricted population if information about population variability is available.
• Account for Sample Characteristics: Understand that correlations can vary across different populations or subgroups. A correlation that is considered “low” for one group might not be appropriate for another. Avoid generalizing findings about discriminant validity beyond the specific sample used in the study. The sources suggest using more generalizable sampling frames with probability sampling to improve the ability to generalize findings.
• Control for Method Variance: Be aware that using the same method (e.g., self-report) to measure both the target and comparison constructs can artificially inflate correlations. Employ multitrait-multimethod (MTMM) designs, which involve measuring multiple constructs using multiple methods, to disentangle trait variance from method variance and obtain more robust evidence for discriminant validity.

2. Ground Interpretation in Strong Theory: Correlations Are Not Enough

• Establish Clear Theoretical Expectations: Before examining correlations, carefully articulate the theoretical relationships between the target construct and the comparison constructs. Specify whether the constructs are expected to be completely distinct, moderately overlapping, or highly related.
• Determine Acceptable Levels of Correlation: Recognize that there is no universally applicable “cutoff” for determining discriminant validity based on correlations. What constitutes a “low” correlation depends on the specific constructs being studied and the theoretical expectations.
• Consider Alternative Explanations: Do not automatically conclude that two constructs are distinct simply because their measures exhibit low correlations. Explore other possible explanations, such as weak measurement, restriction of range, or the presence of suppressor variables.

3. Investigate Response Processes: Unveiling the “Black Box”

• Go Beyond Statistical Analyses: Recognize that statistical relationships alone cannot fully address the issue of discriminant validity. Examining how individuals interpret and respond to test items can provide crucial insights into the cognitive processes involved and help to determine whether respondents are relying on the intended construct when answering questions.
• Employ Qualitative Techniques: Utilize methods such as think-aloud protocols, cognitive interviews, and focus groups to gather data on how individuals understand test items and the strategies they use when responding. This can reveal potential sources of construct-irrelevant variance and help to refine measures to enhance discriminant validity.
• Seek Participant Feedback: Engage in member checking, involving participants in reviewing findings to assess whether they believe the interpretations accurately reflect their experiences and perspectives. This can help to ensure that the researcher’s understanding of the construct aligns with how it is understood by the individuals being studied.

4. Address Construct Underrepresentation and Irrelevant Variance

• Expand the Item Pool: When developing or refining measures, ensure that the item pool adequately samples the full breadth and depth of the target construct. Avoid creating measures that are too narrow in focus and fail to capture important facets of the construct.
• Identify and Control Irrelevant Variance: Systematically examine potential sources of construct-irrelevant variance, such as response styles, test anxiety, and specific item content. Utilize statistical techniques, such as factor analysis, to identify items or subscales that are unduly influenced by irrelevant variance and consider revising or removing these items.
• Consider Contextual Factors: Recognize that construct validity is not an absolute property. The meaning and measurement of constructs can be influenced by cultural factors, social norms, and situational variables. Be cautious about generalizing findings across different contexts without carefully considering potential moderating variables.

5. Embrace the Evolving Nature of Constructs

• Acknowledge That Constructs Change: Psychological attributes are not static entities. Their definitions and conceptualizations can evolve over time, influenced by new research findings, cultural shifts, and changes in measurement practices. Be prepared to revise measures and reinterpret findings as our understanding of constructs develops.
• Engage in Ongoing Evaluation: Discriminant validity is not a one-time determination. It requires continuous evaluation and refinement as our knowledge of constructs grows. Regularly reassess the validity of measures and interpretations in light of new evidence.

Additional Considerations:

• Utilize Argument-Based Approaches to Validation: Develop clear interpretive arguments that outline the intended uses of test scores, the inferences that will be drawn, and the evidence supporting those inferences. This structured approach helps to ensure that validity claims are transparent and well-supported.
• Examine the Consequences of Test Use: Recognize that even valid test interpretations can have unintended or adverse consequences. Consider the potential impact of test use on different stakeholder groups and strive to minimize any negative consequences that might arise.

Reading List

Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in
variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1),
115-135. https://doi.org/10.1007/s11747-014-0403-8

Lucas, R. E., Diener, E., & Suh, E. (1996). Discriminant validity of well-being measures. Journal of Personality and Social Psychology, 71(3), 616-628. https://doi.org/10.1037/0022-3514.71.3.616

Mathieu, J. E., & Farr, J. L. (1991). Further evidence for the discriminant validity of measures of organizational commitment, job involvement, and job satisfaction. Journal of Applied Psychology, 76(1), 127-133. https://doi.org/10.1037/0021-9010.76.1.127

Reichardt, C. S., & Coleman, S. C. (1995). The criteria for convergent and discriminant validity in a multitraitmultimethod matrix. Multivariate Behavioral Research, 30(4), 513-538. https://doi.org/10.1207/s15327906mbr3004_3

Rönkkö, M., & Cho, E. (2022). An updated guideline for assessing discriminant validity. Organizational Research Methods, 25(1), 6-14.

Shaffer, J. A., DeGeest, D., & Li, A. (2016). Tackling the problem of construct proliferation: A guide to
assessing the discriminant validity of conceptually related constructs. Organizational Research Methods, 19(1), 80-110. https://doi.org/10.1177/1094428115598239