Questionnaire Method In Research

Questionnaires provide a relatively cheap, quick, and efficient way of obtaining large amounts of information from a large sample of people.

Data can be collected relatively quickly because the researcher would not need to be present when completing the questionnaires. This is useful for large populations when interviews would be impractical.

However, a problem with questionnaires is that respondents may lie due to social desirability. Most people want to present a positive image of themselves, and may lie or bend the truth to look good, e.g., pupils exaggerate revision duration.

Social desirability is not the only distortion. Several other response biases can distort questionnaire answers, including demand characteristics and acquiescence.

• Acquiescence: Agreeing with almost any statement regardless of content; reverse-worded items help cancel this out.
• Central-Tendency Bias: Clustering around the neutral midpoint instead of choosing an extreme answer, which compresses variation in the data.
• Extreme Responding: The opposite pattern: consistently picking only the endpoints, such as “strongly agree” or “strongly disagree.”
• Demand Characteristics: Guessing the study’s purpose and adjusting answers to fit it, or deliberately spoiling it.

Questionnaires can effectively measure relatively large subjects’ behavior, attitudes, preferences, opinions, and intentions more cheaply and quickly than other methods.

Often, a questionnaire uses both open and closed questions to collect data. This is beneficial as it means both quantitative and qualitative data can be obtained.

Aim: Gosling et al. (2010) investigated the connection between personality and online behavior.

Method: Participants completed a questionnaire measuring their position on the Five-Factor Model of personality. They then self-reported their Facebook behaviors using a set of Likert-scale questions.

Results: FFM scores correlated with Facebook activity. Extraverts had more Facebook friends and commented more on other pages, while openness was linked to adding and replacing photographs frequently.

Conclusion: Offline personalities are consistent with online behavior.

Closed Questions

A closed-ended question requires a specific, limited response, often “yes” or “no” or a choice that fits into pre-decided categories.

Data that can be placed into a category is called nominal data. The category can be restricted to as few as two options, i.e., dichotomous (e.g., “yes” or “no,” “male” or “female”). Or it can include more complex lists of alternatives, i.e., polytomous, from which the respondent chooses.

Closed questions can also provide ordinal data (which can be ranked). This often involves using an ordinal rating scale, such as a Likert scale, to measure the strength of attitudes or emotions.

For example, strongly agree / agree / neutral / disagree / strongly disagree / unable to answer.

Researchers have used closed questions to study type A personality (Friedman & Rosenman, 1974). Other studies have applied them to stressful life events (Holmes & Rahe, 1967) and attachment (Fraley, Waller, & Brennan, 2000).

Strengths

• Economical: Large samples are affordable, so results can generalize to the wider population.
• Easy to Quantify: Fixed responses convert directly into numbers (e.g., counts of “yes” or “no”), enabling statistical analysis.
• Standardized: Every respondent answers the same questions in the same order, so another researcher can repeat the questionnaire to check reliability.

Limitations

• They lack detail. Because the responses are fixed, there is less scope for respondents to supply answers that reflect their true feelings on a topic.

Open Questions

Open questions allow for expansive, varied answers without preset options or limitations.

Open questions allow people to express what they think in their own words. Open-ended questions enable the respondent to answer in as much detail as they like in their own words. For example: “can you tell me how happy you feel right now?”

Open questions will work better if you want to gather more in-depth answers from your respondents. These give no pre-set answer options and instead, allow the respondents to put down exactly what they like in their own words.

Open questions are often used for complex questions that cannot be answered in a few simple categories but require more detail and discussion.

Lawrence Kohlberg presented his participants with moral dilemmas. One of the most famous features a character called Heinz. He must choose between watching his wife die of cancer or stealing the only drug that could help her.

Participants were asked whether Heinz should steal the drug or not and, more importantly, for their reasons why upholding or breaking the law is right.

Strengths

• Rich qualitative data is obtained as open questions allow respondents to elaborate on their answers. This means the research can determine why a person holds a certain attitude.

Limitations

• Time-Consuming to Collect: Open questions take longer to complete, which can shrink the sample size.
• Time-Consuming to Analyze: Coding qualitative answers into categories is slow and often subjective. Smith (1992) devoted an entire book to thematic content analysis, describing 14 different scoring systems for open-ended questions.
• Literacy-Dependent: Open questions demand strong writing and verbal skills, disadvantaging less-educated respondents.

Questionnaire Design

With some questionnaires suffering from a response rate as low as 5%, a questionnaire must be well designed.

There are several important factors in questionnaire design.

Ethical Issues

• The researcher must ensure that the information provided by the respondent is kept confidential, e.g., name, address, etc.
• This means questionnaires are good for researching sensitive topics as respondents will be more honest when they cannot be identified.
• Keeping the questionnaire confidential should also reduce the likelihood of psychological harm, such as embarrassment.
• Right to Withdraw: Participants give informed consent before completing the questionnaire and can withdraw their data at any time during the study.

Problems with Postal Questionnaires

At first sight, postal questionnaires seem to avoid interviewer bias by removing the researcher from the process. They are also cheaper than face-to-face interviews and can reach many respondents across a wide area quickly.

However, these advantages must be weighed against the practical problems of conducting research by post. A lack of involvement by the researcher means there is little control over the information-gathering process.

The data might not be valid (i.e., truthful). We can never be sure the intended person actually completed the questionnaire.

That assumes a reply arrives in the first place. Low response rates are one of the most persistent problems with mailed questionnaires, and this diminishes the reliability of the data.

Postal questionnaires may also fail to represent the population being studied, for several reasons.

Non-Response Bias in Postal Questionnaires

• Some questionnaires may be lost in the post, reducing the sample size.
• The questionnaire may be completed by someone not a member of the research population.
• Those with strong views on the questionnaire’s subject are more likely to complete it than those without interest.

Benefits of a Pilot Study

A pilot study is a practice / small-scale study conducted before the main study.

It allows the researcher to try out the study with a few participants first. Adjustments can then be made before the main study, saving time and money.

Why Run a Pilot Study

• Check that respondents understand the terminology used in the questionnaire.
• Check that emotive questions are not used, as they make people defensive and could invalidate their answers.
• Check that leading questions have not been used as they could bias the respondent’s answer.
• Ensure the questionnaire can be completed in an appropriate time frame (i.e., it’s not too long).

Critical Evaluation of Questionnaires

Questionnaires are one of the cheapest ways to collect large amounts of data, but that convenience comes with real trade-offs in accuracy and depth. Weighing these strengths and limitations matters when choosing whether a questionnaire is the right tool for a study.

Strengths and Limitations

Strengths

• Access to Internal States: Attitudes and feelings that cannot be observed directly are only available by asking.
• Cheap and Scalable: Large samples can be surveyed at low cost, especially with closed and Likert-type items that are quick to score.
• Flexible Depth: Open questions capture rich, detailed answers, while closed items produce data that is quick to compare statistically.
• Mature Measurement Science: Established reliability and validity procedures give researchers principled ways to build and test good questionnaires.

Limitations

• May Not Match Behavior: Self-report answers are the respondent’s account, not a direct observation, so they do not always match what people actually do.
• Limits of Self-Insight: Respondents may not have accurate access to the causes of their own behavior, so honesty does not guarantee accuracy.
• Response Biases: Social desirability, acquiescence, and the other biases described above can all distort scores.
• Literacy and Culture: Closed-format items are harder for respondents with lower abstract-thinking ability, such as young children, and cultural differences in response style can make cross-cultural comparisons unreliable.
• Reductionism: Turning a complex attitude into a single number or rating can lose important nuance.

Contemporary Research

Online surveys are now the default way to run a questionnaire study. They give cheap access to large, geographically diverse samples, but they also raise new concerns about representativeness, bots, and inattentive responding.

Careless responding is a particular risk when nobody is watching. Modern studies screen for it directly.

Researchers now embed attention-check items and measure response times to catch participants who click through without reading. They also compute indices such as long-string scores (identical answers given in a row) and Mahalanobis distance, a statistical flag for unusual response patterns.

Scale construction is also changing. Classical test theory scores a questionnaire by simply summing item responses.

Item Response Theory instead models each response using the respondent’s trait level and the item’s difficulty. This lets researchers run adaptive tests that present only the most informative items.