A pilot study, also known as a feasibility study, is a small-scale preliminary study conducted before the main research to check the feasibility or improve the research design.
Pilot studies are important before conducting a full-scale research project, helping researchers design the methods and protocol.
Key Takeaways
- Purpose: A pilot study is a small-scale trial run of a study’s procedures, used to catch design flaws before the main research begins.
- Floor & Ceiling: A pilot can reveal when a task is too hard (a floor effect) or too easy (a ceiling effect) for participants.
- No Hypothesis Testing: Because sample size and power are not calculated, a pilot study should test feasibility, not the study’s hypotheses.
- Avoiding Bias: Pilot participants should closely resemble the target population but should not be reused in the final sample.
- Main Benefits: Piloting improves the reliability and validity of the main study and helps researchers estimate time, cost, and sample size accurately.
How a Pilot Study Works
Pilot studies are a fundamental stage of the research process. They can help identify design issues and evaluate a study’s feasibility, practicality, resources, time, and cost before the main research is conducted.
Identifying Flaws and Ambiguities
A pilot study involves selecting a few people and trying out the study on them. The idea is simple. This saves time and, in some cases, money by identifying flaws in the researcher’s procedures before the main study begins.
It can also help the researcher spot ambiguities, confusing instructions given to participants, or problems with the task itself. Two problems come up often.
Floor and Ceiling Effects
Sometimes the task is too hard, and the researcher may get a floor effect. None of the participants can score at all or complete the task, so all performances are low.
The opposite is a ceiling effect. Here the task is so easy that all achieve virtually full marks, “hitting the ceiling.”
Benefits for the Main Study
Piloting lets researchers predict an appropriate sample size, budget accordingly, and improve the study design before the full-scale project runs. It also provides preliminary data, giving early insight into the likely results of the proposed experiment.
Piloting is not for hypothesis testing. The appropriate power and sample size are not calculated at this stage, so pilot studies should not be used to test hypotheses. Their real job is to assess feasibility.
By piloting first, researchers face the main study’s challenges better prepared and grow more confident with the instruments they will use to collect data.
Good Practice: Avoiding Bias and Using Feedback
To avoid bias, pilot studies are usually carried out on people who closely resemble the target population, but who will not join the final sample. Multiple pilot studies may be needed, and qualitative and/or quantitative methods may be used.
Feedback closes the loop. Feedback from pilot participants can improve the experience for participants in the main study. For example, it might reduce their burden, improve the instructions, or flag ethical issues.
Pilot Study for Experiments
In a pilot study with an experimental design, check that your measures of the independent variable (IV) and dependent variable (DV) are reliable and valid. You should also confirm that you can manipulate the IV effectively and control for potential confounding variables.
Confounding variables are the usual culprit. A confounding variable is an extraneous factor that varies systematically with the IV, offering an alternative explanation for the results. Piloting catches this early, since researchers can fix the design before a confound contaminates the real data.
A pilot study also allows the research team to gain experience and training, which can be particularly beneficial if new experimental techniques or procedures are used.
Pilot Study for Questionnaires
Conduct a questionnaire pilot study for these reasons:
- Terminology Check: Confirms that respondents understand the terminology used in the questionnaire.
- Avoid Emotive Wording: Checks that emotive questions are not used, since they make people defensive and can invalidate their answers.
- Avoid Leading Questions: Checks that leading questions have not been used, since they could bias the respondent’s answer.
- Time to Complete: The questionnaire should be completed in a reasonable time. A version that is too long risks lower response rates and weaker data quality.
Advantages of Pilot Studies
- Reduces Errors: By identifying and addressing issues in the pilot study, researchers can reduce errors and risks in the main study, improving the reliability and validity of its results.
- Feasibility Check: Assesses the practicality and feasibility of the main study.
- Instrument Testing: Tests the efficacy of the research instruments.
- Weakness Detection: Identifies and addresses any weaknesses or logistical problems.
- Preliminary Data: Collects preliminary data ahead of the main study.
- Time and Costs: Estimates the time and costs the project will require.
- Resource Planning: Determines what resources the study needs.
- Procedure Fixes: Flags procedures that need to be modified because they do not elicit useful data.
- Credibility: Adds credibility and dependability to the study.
- Interview Pretesting: Pretests the interview format.
- Consistent Practice: Helps researchers develop consistent practices and become familiar with the protocol.
- Safety Checks: Addresses safety issues and management problems.
Limitations of Pilot Studies
- Extra Costs: Piloting requires extra costs, time, and resources on top of the main study.
- No Guarantee: A successful pilot does not guarantee the main study will succeed.
- Contamination: Using pilot data or pilot participants in the main study results counts as contamination.
- Funding Risk: Funding bodies may be reluctant to fund a further study once the pilot results are published.
- Low Power: A small pilot sample has limited power to detect real effects. Cohen (1962) found that typical psychology studies had under a 50% chance of detecting a medium-sized effect.
Examples
- Viscocanalostomy: A Pilot Study (Carassa, Bettin, Fiori, & Brancato, 1998)
- WHO International Pilot Study of Schizophrenia (Sartorius, Shapiro, Kimura, & Barrett, 1972)
- LaBerge (1985): Stephen LaBerge of Stanford University tested time perception during lucid dreaming. Participants signaled the start and end of a ten-second count with pre-set eye movements recorded on an EOG.
- Negative Word-of-Mouth by Dissatisfied Consumers: A Pilot Study (Richins, 1983)
- A pilot study and randomized controlled trial of the mindful self‐compassion program (Neff & Germer, 2013)
- Pilot study of secondary prevention of posttraumatic stress disorder with propranolol (Pitman et al., 2002)
- Unstructured Observation: With no coding system, researchers record all relevant behavior, but may capture too much and miss what matters. A pilot study helps identify which behaviors are actually worth recording.
- Perspectives of the use of smartphones in travel behavior studies: Findings from a literature review and a pilot study (Gadziński, 2018)
Further Information
- Lancaster, G. A., Dodd, S., & Williamson, P. R. (2004). Design and analysis of pilot studies: recommendations for good practice. Journal of Evaluation in Clinical Practice, 10(2), 307-312.
- Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., … & Goldsmith, C. H. (2010). A tutorial on pilot studies: the what, why and how. BMC Medical Research Methodology, 10(1), 1-10.
- Moore, C. G., Carter, R. E., Nietert, P. J., & Stewart, P. W. (2011). Recommendations for planning pilot studies in clinical and translational research. Clinical and Translational Science, 4(5), 332-337.
References
Carassa, R. G., Bettin, P., Fiori, M., & Brancato, R. (1998). Viscocanalostomy: a pilot study. European Journal of Ophthalmology, 8(2), 57-61.
Gadziński, J. (2018). Perspectives of the use of smartphones in travel behaviour studies: Findings from a literature review and a pilot study. Transportation Research Part C: Emerging Technologies, 88, 74-86.
In, J. (2017). Introduction of a pilot study. Korean Journal of Anesthesiology, 70(6), 601–605. https://doi.org/10.4097/kjae.2017.70.6.601
LaBerge, S., LaMarca, K., & Baird, B. (2018). Pre-sleep treatment with galantamine stimulates lucid dreaming: A double-blind, placebo-controlled, crossover study. PLoS One, 13 (8), e0201246.
Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45(5), 626–629. https://doi.org/10.1016/j.jpsychires.2010.10.008
Malmqvist, J., Hellberg, K., Möllås, G., Rose, R., & Shevlin, M. (2019). Conducting the Pilot Study: A Neglected Part of the Research Process? Methodological Findings Supporting the Importance of Piloting in Qualitative Research Studies. International Journal of Qualitative Methods. https://doi.org/10.1177/1609406919878341
Neff, K. D., & Germer, C. K. (2013). A pilot study and randomized controlled trial of the mindful self‐compassion program. Journal of Clinical Psychology, 69(1), 28-44.
Pitman, R. K., Sanders, K. M., Zusman, R. M., Healy, A. R., Cheema, F., Lasko, N. B., … & Orr, S. P. (2002). Pilot study of secondary prevention of posttraumatic stress disorder with propranolol. Biological Psychiatry, 51(2), 189-192.
Richins, M. L. (1983). Negative word-of-mouth by dissatisfied consumers: A pilot study. Journal of Marketing, 47(1), 68-78.
Sartorius, N., Shapiro, R., Kimura, M., & Barrett, K. (1972). WHO International Pilot Study of Schizophrenia. Psychological Medicine, 2(4), 422-425.
van Teijlingen, E., & Hundley, V. (2001). The importance of pilot studies. Social Research Update, (35).
