Questionnaire Construction & Validation

In short:   Psychology research is built on survey instruments like the PHQ-9, the Y-BOCS, the PANAS, and hundreds of others. Each was developed by someone who had studied the existing surveys, designed a new one, and then validated their new survey with a sample of participants. This type of “psychometric” IP is a specialized form of quantitative research. It’s rarely done at the Masters level, as it requires knowledge of validity and reliability as well as tools like factor analysis. But it can be immensely rewarding to create and validate your own scale – and someday see it used by others!

Key Components:

• Intro & review of conceptual space
• Review of existing instruments & measures
• Initial item pool development
• Expert and non-expert feedback
• Pilot testing
• Psychometric testing
• Analysis of reliability and validity
• Discussion

The whole story:  Questionnaire construction & validation involves building a novel survey measurement tool and evaluating its reliability, validity, and usefulness in capturing a psychological construct -- hopefully advancing research or practice for a defined population or purpose in the field of clinical psychology. 

The process begins with articulating the construct you wish to measure, and the constellation of constructs related to your topic. This means a comprehensive review of existing empirical literature that has shaped how your topic has been described and measured. For example, resilience to trauma may have six different scales to measure it, as well as related constructs like “grit” with important conceptual overlap. Your lit review should map the conceptual space around your construct, and then describe the gaps and limitations in existing tools and constructs that can justify why your new questionnaire instrument is needed. 

When reviewing existing tools, be sure to include the most often used tools as well as any popular revised and newer versions, and the limitations that those versions were developed to address. 

You must next construct your questionnaire, an iterative process of several steps.

It starts with creating an initial pool of possible items that capture every aspect of the construct you wish to measure. You can rely on existing surveys, relevant theory, or the opinions of people involved with the topic to generate your pool of items. Aiming for a 15-item finished survey? Be sure to start with a pool of at least 40 to 50 questions. You’ll also need to choose a response format; often this is a classic 5- or 7-item Likert Scale with anchors from strongly agree to strongly disagree, but other formats are possible. 

The first test of your item pool is often an expert panel, knowledgeable people in the field you recruit to evaluate test items to ensure they are relevant and representative of the entire domain being measured – also known as content validity. Without a panel, however, sponsor feedback can serve as a primary source of content validity. It is also useful to test your item pool with 10 to 15 non-expert respondents who can help identify confusing items and give feedback on the format. 

The pilot testing of your new instrument follows, providing quantitative evidence of your survey’s performance on issues such as floor or ceiling effects (responses bunching at one extreme or another). Items with highly skewed or very low variance may be discarded, along with redundant items. Included in a pilot test will often be one or two established questionnaires, so you can gauge convergence or divergence with related concepts. For example, how do my questions on sadness relate to existing questionnaires about something related like grief… or about something opposite like happiness?

Next up, full psychometric testing of your survey usually requires sample sizes in excess of 200 respondents, so recruitment is a major issue. Should you pull this off, however, the wealth of resulting data allows for the exciting subsequent steps of exploratory and confirmatory factor analysis, which provide further basis for winnowing items and understanding underlying subscales inside your item pool. Note that subscales within the larger questionnaire should have an adequate number of items, typically 4 to 5, such that the reliability of each subscale can be statistically evaluated. 

The results of your psychometric testing also allow various forms of validity to be assessed, including convergent validity with other related scales and divergent validity with measures of unrelated constructs. The reliability of your survey, such as whether the same person responds similarly at later time points, must also be assessed with standard procedures.

IRB approval is required for questionnaire development. You must submit an IRB protocol that describes their validation study, their approach to collecting and handling data, and otherwise demonstrates appropriate management of any potential risks of study participation to ensure ethical standards are met. 

The final integrative project should follow a clear structure (e.g., abstract, introduction, methods, results, discussion, references) and provide well-organized tables or figures as applicable. All references should be properly listed and all writing should be in APA style. All questionnaire construction & validation studies must include an appendix with all items, scoring procedures, and psychometric results for the final version of the questionnaire, such that it can be utilized by others in the future.

To learn more: Our brief outline merely sketches the process for students. More detailed guidance can be found in sources such as these:

• Boateng et al’s (2018) Best Practices for Developing and Validating Scales, a straightforward guide with easy-to-understand graphics.
• A step-by-step guide from Raphael Valdece Sousa Bastos from a freely-available online book, Advances in Psychometric Theory and Measurement for Psychological Sciences.
• Kalkbrenner’s MEASURE approach to scale development

Relevant Courses at TC:

• Psychological Measurement (HUDM)