Questionnaire Method In Research

Questionnaires provide a relatively cheap, quick, and efficient way of obtaining large amounts of information from a large sample of people.

Questionnaires as a Self-Report Method

Questionnaires belong to a family of three self-report methods, alongside rating scales such as the Likert scale and interviews. All three rely on people describing their own experiences, attitudes, feelings, and behaviors.

Self-report is the main way to measure things that cannot be observed directly. Examples include attitudes, opinions, personality traits, moods, symptoms, and intentions.

There is a catch. The participant is both the instrument and the object of measurement, which opens the door to response biases.

Data can be collected quickly without the researcher present.

This is useful for large populations when interviews would be impractical.

The trade-off is accuracy.

However, a problem with questionnaires is that respondents may lie due to social desirability. Most people want to present a positive image of themselves. They may lie or bend the truth to look good, such as pupils exaggerating how long they revised.

Social desirability is not the only distortion: several other response biases can affect questionnaire answers.

• Social Desirability: Wanting to look good, respondents exaggerate approved behaviors and hide disapproved ones. Paulhus (1984) separated deliberate faking (impression management) from an honestly held, overly positive self-view (self-deceptive enhancement).
• Acquiescence: Agreeing with almost any statement regardless of content; reverse-worded items help cancel this out.
• Central-Tendency Bias: Respondents avoid the extreme options and cluster around the neutral midpoint, which compresses variation in the data.
• Extreme Responding: Respondents pick only the endpoints, such as “strongly agree” or “strongly disagree,” whatever the item says. This tendency varies across individuals and cultures.
• Demand Characteristics: Guessing the study’s purpose and adjusting answers to fit it, or deliberately spoiling it.
• Reference-Group Effect: Respondents judge themselves against their own social or cultural group, so identical ratings from different groups can mean different things.

Questionnaires can effectively measure relatively large subjects’ behavior, attitudes, preferences, opinions, and intentions more cheaply and quickly than other methods.

Often, a questionnaire uses both open and closed questions to collect data. This is beneficial as it means both quantitative and qualitative data can be obtained.

One example makes this concrete.

Closed Questions

A closed-ended question requires a specific, limited response, often “yes” or “no” or a choice that fits into pre-decided categories.

Data that can be placed into a category is called nominal data. Two options, such as yes and no, make it dichotomous. More options make it polytomous, a longer list of alternatives for the respondent to choose from.

Closed questions can also provide ordinal data, which can be ranked. This often involves using an ordinal rating scale, such as a Likert scale, to measure the strength of attitudes or emotions.

Response options are pre-set. For example: strongly agree, agree, neutral, disagree, strongly disagree, or unable to answer.

Researchers have used closed questions to study type A personality (Friedman & Rosenman, 1974). Other studies have applied them to stressful life events (Holmes & Rahe, 1967) and attachment (Fraley, Waller, & Brennan, 2000).

Strengths

• Economical: Large samples are affordable, so results can generalize to the wider population.
• Easy to Quantify: Fixed responses convert directly into numbers (e.g., counts of “yes” or “no”), enabling statistical analysis.
• Standardized: Every respondent answers the same questions in the same order, so another researcher can repeat the questionnaire to check reliability.

Limitations

• Less Detail: Because the responses are fixed, there is less scope for respondents to supply answers that reflect their true feelings on a topic.

Open Questions

Open questions allow for expansive, varied answers without preset options or limitations. Respondents answer in their own words.

Open-ended questions work well for detail respondents could not fit into preset categories. For example: “can you tell me how happy you feel right now?”

Open questions suit complex topics that cannot be reduced to a few simple categories, and they work best when you want in-depth answers.

Lawrence Kohlberg presented his participants with moral dilemmas. One of the most famous features a character called Heinz. He must choose between watching his wife die of cancer or stealing the only drug that could help her.

Participants were asked whether Heinz should steal the drug or not and, more importantly, for their reasons why upholding or breaking the law is right.

Strengths

• Rich Qualitative Data: Open questions allow respondents to elaborate on their answers, producing rich qualitative data. This means the research can determine why a person holds a certain attitude.

Limitations

• Time-Consuming to Collect: Open questions take longer to complete, which can shrink the sample size.
• Time-Consuming to Analyze: Coding qualitative answers into categories is slow and often subjective. Smith (1992) devoted an entire book to thematic content analysis, describing 14 different scoring systems for open-ended questions.
• Literacy-Dependent: Open questions demand strong writing and verbal skills, disadvantaging less-educated respondents.

Questionnaire Design

With some questionnaires suffering from a response rate as low as 5%, a questionnaire must be well designed.

There are several important factors in questionnaire design.

• Aims: Make sure that all questions are asked to address the research aims. However, each item should cover only one feature of the construct you are investigating.
• Length: The longer the questionnaire, the less likely people will complete it. Questions should be short, clear, and concise; any unnecessary questions/items should be omitted.

Pilot Study

Run a small-scale practice study to ensure people understand the questions. People can also give detailed, honest feedback on the questionnaire design.

Cognitive interviews are one useful tool. Ask a handful of respondents, typically 5 to 25, to talk through the draft so that ambiguous wording, misread items, and unintended interpretations come to light.

Item analysis is another. On a multi-item scale, an item that correlates below roughly 0.3 with the total score adds noise rather than signal and is a candidate for deletion.

For example, a short attitude scale might be piloted on about 30 students. Cognitive interviews fix the wording, and item analysis drops any weak items before the main study.

Question Order

Questions should progress logically, from the least sensitive to the most sensitive. Order should also move from factual and behavioral content to cognitive content, and from general questions to specific ones.

The researcher should ensure that previous questions do not influence the answer to a question.

Question order effects

• What They Are: An earlier question can change how someone answers a later one, at any stage from interpretation to reporting.
• Three Types: Unconditional effects shift answers regardless of the prior response; conditional effects depend on what was answered; associational effects change the correlation between two questions.
• Where They Occur: Found across social attitudes, health, and vignette research, and moderated by respondent age, education, and prior knowledge.
• Minimizing Them: Avoid dependent questions, separate reactive ones, randomize order, and design with the respondent in mind.

Terminology

• Minimal Jargon: Keep questions simple and match the vocabulary of the group being studied.
• Consistent Interpretation: Word statements so different respondents read them the same way.
• Match the Audience: Adjust language for the respondent’s age, education, and social background.

Avoiding Biased Wording

• Double-Barrelled Questions: Avoid asking two things in one item (e.g., “Do you find this app fast and easy to use?”), which cannot be answered clearly if the two parts diverge.
• Leading Questions: Avoid wording that steers the respondent toward a particular answer (e.g., “How much did you enjoy the excellent lecture?”).
• Double Negatives: Avoid double negatives and jargon, which respondents often misread or misinterpret.

Presentation

Make sure it looks professional and includes clear and concise instructions. If sent through the post, ensure the envelope does not signify ‘junk mail.’

Response Options and Item Polarity

Response options shape the data. An odd-numbered scale (5 or 7 points) offers a neutral midpoint, which suits constructs where “no opinion” is a real position. The midpoint can also become a dumping ground for indifferent or evasive respondents, which produces central-tendency bias.

An even-numbered, forced-choice scale (4 or 6 points) removes the midpoint. That pushes people to lean one way and increases variance. It also risks inventing an attitude in people who genuinely have none.

There is no universal answer. The right choice depends on whether “neutral” is a meaningful state for the construct.

Match the response format to the construct: agreement (as in a Likert scale), frequency, evaluation, or a continuous line. Mix favorable and unfavorable wording as well. Reverse-scored items break acquiescence, because agreeing with everything no longer inflates the total.

Ethical Issues

• Confidentiality: The researcher must ensure that the information provided by the respondent is kept confidential, e.g., name, address, etc.
• Sensitive Topics: Respondents tend to be more honest when they cannot be identified, so questionnaires suit sensitive topics.
• Reduced Harm: Keeping the questionnaire confidential should also reduce the likelihood of psychological harm, such as embarrassment.
• Right to Withdraw: Participants give informed consent before completing the questionnaire and can withdraw their data at any time during the study.

Problems with Postal Questionnaires

At first sight, postal questionnaires seem to avoid interviewer bias by removing the researcher from the process. They are also cheaper than face-to-face interviews and can reach many respondents across a wide area quickly.

However, these advantages come with practical problems of their own. A lack of involvement by the researcher means there is little control over the information-gathering process.

The data might not be valid (i.e., truthful). We can never be sure the intended person actually completed the questionnaire.

That assumes a reply arrives in the first place. Low response rates are one of the most persistent problems with mailed questionnaires, and this diminishes the reliability of the data.

Postal questionnaires may also fail to represent the population being studied, for several reasons.

Non-Response Bias in Postal Questionnaires

• Lost in the Post: Some questionnaires may be lost in the post, reducing the sample size.
• Wrong Respondent: The questionnaire may be completed by someone not a member of the research population.
• Self-Selection: Those with strong views on the questionnaire’s subject are more likely to complete it than those without interest.

Benefits of a Pilot Study

A pilot study is a practice / small-scale study conducted before the main study.

It allows the researcher to try out the study with a few participants first. Adjustments can then be made before the main study, saving time and money.

Why Run a Pilot Study

• Terminology: Check that respondents understand the terminology used in the questionnaire.
• Emotive Questions: Check that emotive questions are not used, as they make people defensive and could invalidate their answers.
• Leading Questions: Check that leading questions have not been used as they could bias the respondent’s answer.
• Time Frame: Ensure the questionnaire can be completed in an appropriate time frame (i.e., it’s not too long).

Evidence Check: Does Self-Report Hold Up?

Verdict: Supported with qualifications.

Self-report is a reliable and useful way to measure internal states. The best available evidence shows two limits. Answers can diverge substantially from objectively measured behavior. The quality of data collected online also depends on the platform that supplies the respondents.

Self-Reports Can Depart From Behavior

Level of evidence: Meta-analysis or systematic review; not yet replicated.

The claim is that self-report answers are the participant’s own account and may not match what people actually do. Honesty does not guarantee accuracy.

Prince et al. (2020) ran a systematic review and meta-analysis comparing self-reported with device-measured sedentary time in adults.

• Studies: The review included 185 unique studies. Of these, 123 studies (173 comparisons, 55,199 participants) tested criterion validity.
• Average gap: Self-report fell 105.19 minutes per day below the device measure (95% CI -127.21 to -83.17), an underestimate of about 1.74 hours a day.
• Format: Single-item questions did worse than multi-item questionnaires, momentary-assessment methods, and diaries. Compared with inclinometers, those three did not differ significantly on average.
• Spread: Even so, the three varied by up to 6 hours a day within individual studies, with roughly half over-reporting and half under-reporting.
• Reliability: Across 54 studies, the average test reliability was good (ICC = 0.66).

Connelly and Ones (2010) add a second line of evidence: three meta-analyses of 263 independent samples and 44,178 target individuals. Ratings of personality made by other people predicted behavior well. For academic achievement and job performance, they gave predictive validities greater than, and incremental to, self-ratings.

The evidence supports the claim in a qualified form. A gap of this size against a device is not a rounding error.

But the error is not a fixed property of self-report. The better formats showed no significant average gap. Within studies, individual respondents erred in either direction.

Connelly and Ones fit the claim without proving it, since other raters may simply add independent information. The claim is therefore real but narrower than a blanket “self-report does not match behavior.” It is tested directly only for sedentary time. For personality and attitudes it holds by inference.

Prince et al. is the most trustworthy piece. It tests the claim directly against an objective criterion with the largest sample.

Its limits are specific. The abstract reports no heterogeneity statistics. An inclinometer also measures posture, not the whole construct. The result therefore tests sedentary-time questionnaires, not self-report in general.

Online Data Quality Varies by Platform

Level of evidence: Correlational or self-report study; not yet replicated.

The claim is that online surveys are not one method. Data quality differs by platform and can be recovered through screening.

Peer et al. (2021) compared Amazon Mechanical Turk, CloudResearch, and Prolific (platforms) with Qualtrics and Dynata (panels). They measured attention, comprehension, honesty, and reliability in two studies of about 4,000 participants in total, with and without approval-rating filters.

• Overall: The sites differed considerably, especially in comprehension, attention, and dishonesty.
• Without filters: Only Prolific gave high quality on all measures.
• With filters: CloudResearch and Prolific did. Amazon Mechanical Turk remained of low quality even with filters.
• Approval rating: It did not predict quality, but frequency and purpose of use did.
• Lowest quality: Amazon Mechanical Turk participants who used the site as their main income while spending few hours on it each week.

Chmielewski and Kucker (2020), described below, point the same way for that platform across time.

The finding supports a cautious reading. The same broad pattern appeared with and without filters, which is hard to explain as noise.

The lowest quality came from a definable subgroup, so who responds and why matter as much as where people are recruited.

The design is correlational, and participants chose their platform. The safest reading is therefore that quality differed between these platforms at that time, not that any platform has a fixed rank. The income motive and low engagement are patterns in the data, not something the study manipulated.

The scale is a strength. The weaknesses matter more. The abstract does not say whether the studies were preregistered, and it gives no effect sizes or confidence intervals. The samples are online volunteers, and platforms change quickly. It is a single team’s work with no independent replication.

Overall, the first limit rests on a meta-analysis and is secure for sedentary behavior. The second rests on one unreplicated comparison. Hold it loosely.

Critical Evaluation of Questionnaires

Questionnaires are one of the cheapest ways to collect large amounts of data, but that convenience comes with real trade-offs in accuracy and depth. Weighing these strengths and limitations matters when choosing whether a questionnaire is the right tool for a study.

Strengths and Limitations

Strengths

• Access to Internal States: Attitudes and feelings that cannot be observed directly are only available by asking.
• Cheap and Scalable: Large samples can be surveyed at low cost, especially with closed and Likert-type items that are quick to score.
• Flexible Depth: Open questions capture rich, detailed answers, while closed items produce data that is quick to compare statistically.
• Mature Measurement Science: Established reliability and validity procedures give researchers principled ways to build and test good questionnaires.

Limitations

• May Not Match Behavior: Self-report answers are the respondent’s account, not a direct observation, so they do not always match what people actually do.
• Limits of Self-Insight: Respondents may not have accurate access to the causes of their own behavior, so honesty does not guarantee accuracy.
• Response Biases: Social desirability, acquiescence, and the other biases described above can all distort scores.
• Literacy and Culture: Closed-format items are harder for respondents with lower abstract-thinking ability, such as young children, and cultural differences in response style can make cross-cultural comparisons unreliable.
• Reductionism: Turning a complex attitude into a single number or rating can lose important nuance.

Reliability and validity, however, are not just background assurances.

Cronbach’s alpha (Cronbach, 1951) quantifies how well a scale’s items hang together. Values above 0.8 are generally considered very good, and 0.7 to 0.8 acceptable.

Alpha has a catch. Because its value scales with the number of items, a long questionnaire can post a respectable alpha even when its items barely correlate (Cortina, 1993). Compute alpha separately for each subscale.

Test-retest reliability checks a different thing: whether the same respondents give stable scores when they complete the questionnaire again later.

Validity asks whether a questionnaire measures what it claims to measure. Researchers look for four kinds of evidence.

• Content Validity: Experts judge whether the items cover the full construct.
• Construct Validity: Evidence, often from factor analysis, that the scale measures the intended construct and relates as expected to other measures.
• Criterion Validity: The new measure correlates with an established benchmark taken at the same time, known as concurrent validity.
• Face Validity: The questionnaire appears, on the surface, to measure what it claims. This is the weakest evidence, but it affects how engaged respondents are.

Self-Report Compared With Other Methods

Self-report is not the only way to measure attitudes, traits, or emotions. Each alternative reaches a different layer of the same phenomenon.

Baumeister, Vohs, and Funder (2007) reviewed the methods of personality and social psychology. They documented a decades-long shift away from direct observation toward self-report, hypothetical vignettes, and questionnaire ratings.

Their point was not that self-report should be abandoned. Private thoughts and intentions leave no behavioral trace to observe. But a measure of what someone says they would do is not interchangeable with a measure of what they do.

Mauss and Robinson (2009) reached a similar verdict on emotion. They concluded that there is no single gold standard. Physiological indices correlate only modestly with self-reported feelings and with each other, because experience, arousal, and expression only partly overlap.

None of these methods simply replaces self-report. The strongest designs triangulate, pairing self-report with at least one alternative.

Contemporary Research

Online surveys are now the default way to run a questionnaire study. They give cheap access to large, geographically diverse samples, but they also raise new concerns about representativeness, bots, and inattentive responding.

Data Quality Across Online Platforms

“Online survey” is not one method with fixed properties. Chmielewski and Kucker (2020) tracked data quality on Amazon Mechanical Turk (MTurk) across four waves of data collection. The waves spanned a documented mid-2018 disruption to the platform.

Response-validity pass rates fell markedly, as did the reliability and validity of a widely used personality measure. The disruption also produced failed replications of well-established effects. Screening on response-validity indicators substantially reduced the damage.

Platforms differ too. Peer et al. (2021) found substantial differences between sites in attention, comprehension, and honesty, with MTurk the weakest even after filtering.

Together, the studies show that online data quality varies by platform, drifts over time, and can be recovered through active screening. It cannot simply be assumed from the format.

Detecting Careless Responding

Careless responding is a particular risk when nobody is watching. Modern studies screen for it directly.

Researchers now embed attention-check items and measure response times to catch participants who click through without reading. They also compute indices such as long-string scores (identical answers given in a row) and Mahalanobis distance, a statistical flag for unusual response patterns.

Person-fit statistics check whether a respondent’s pattern of answers fits what the scale would predict. One study compared them directly with other detection methods.

The study’s strength is that it tested the methods against genuinely induced carelessness, which exposed the gap between real and simulated sensitivity. Its manipulation targeted only speeded responding, though, so the results may not extend to long-string acquiescence or deliberate mischief.

Curran (2016) reviewed the wider problem. No single index reliably separates careless responding from genuine but atypical responding. Most methods therefore trade false positives against false negatives, and the threshold is a judgment call.

Item Response Theory and Adaptive Testing

Scale construction is also changing. Classical test theory scores a questionnaire by simply summing item responses.

Item Response Theory (IRT) instead models each response using the respondent’s trait level and each item’s difficulty and discrimination. Discrimination is how sharply an item separates people.

This lets researchers run adaptive tests that present only the most informative items.

Two studies show what IRT makes possible.

• Colledani et al. (2018): An IRT model applied to 590 people on the 100-item Eysenck Personality Questionnaire-Revised yielded a 48-item short form with no misfit.
• Second sample: In 300 further respondents, the short form matched or exceeded the original reliability and tracked full-length scores more closely than the existing short form.
• Loe et al. (2017): Fitting 17 of the 20 CES-D depression items to an IRT model, they simulated a computerized adaptive test.
• Adaptive result: Adaptive scores correlated highly with full-scale scores while needing under 12 items on average.
• Where it helped most: The largest precision gains came at the highest, clinically most important, end of the depression range.

The payoff is the same in both studies. A calibrated item bank lets a shorter, adaptively chosen item set do the work of a longer fixed scale.

The cost is the extra calibration work that IRT modeling requires.