Survey Design - Wikimedia€¦ · “Hourglass” notion of research e.g., design a survey then sample data. 13 Survey types 14 Types of surveys Self - administered Interview - administered

Lecture 2Survey Research & Design in Psychology

James Neill, 2015

Survey Design

Lecture 1 Summary Survey research

1. Research types (3)1. Experimental2. Quasi-experimental3. Non-experimental

2. Purposes (4)1. Information gathering (2)

1. Exploratory2. Descriptive

2. Theory testing (2)1. Explanatory2. Predictive

Lecture 1 Summary Survey research

1. What is a survey?1. A standardised stimulus used as a social

science measurement tool

2. Survey research1. Pros

1. Ecological validity2. Cost-efficient3. Can obtain lots of data

2. Cons1. Low compliance2. Reliance on self-report

Overview

1. Learning outcomes2. Research process3. Survey types – Interview vs. self-administered

4. Survey construction5. Levels of measurement6. Biases7. Sampling

Learning outcomes

Understanding of the learning outcomes for this lecture and associated readings should be demonstrated via the lab report assessment exercise.

Learning outcome 1: Research process

Understand recommended research process steps involved in survey research studies including planning, developing and implementing research questionnaires.

Learning outcome 2: Survey administration methods

Consider the pros and cons of common survey administration methods:

1. Interview-based survey2. Self-report survey

Learning outcome 3: Questionnaire design

Examine the nuts & bolts of questionnaire design including:

1. Question style (open/closed, objective/subjective)

2. Response formats3. Levels of measurement4. Layout 5. Pre- and pilot-testing

Learning outcome 4:Survey implementation issues

Consider survey research implementation issues, including:

1. Sampling methods2. Sample size and return rates3. Representativeness4. Biases

Research process

Examples of iterative research process models and where survey design and sampling fits in.

A typical scientific survey research process

Reality / observation / theory →

Problem definition / hypotheses →

Research method design (incl. survey) →

Collect data →

Analyse →

Discuss (generalise / apply) →

Disseminate (get reviewed / publish) →

New study?

“Hourglass” notion of research

e.g., design a survey

then sample data

Survey types

Types of surveys

Self -administered

Interview -administered

Postal Delivered andcollected

Telephone Face to facestructuredinterview

Online

Email Web Mobile

Advantages and disadvantages of

self- and interview-administered surveys

Put “low” or “high” in each cell and indicate for each row, which is more desirable.

● Self-administered surveys:• Pros:

- cost- demand characteristics- access to representative sample- anonymity

• Cons:- non-response- adjustment to cultural differences, special needs

• Interview-administered surveys have the opposite pros and cons

Summary: Survey types

Survey construction

Survey construction1. Survey design is science and art2. Questionnaire development

1. Parts of a survey2. Order, flow and structure3. Demographics and personal information4. Ending the survey5. Layout6. Pre- and pilot-testing

3. Writing questions1. Types of questions2. Response formats

“Surveys are a mixture of science and art, and a good

researcher will save their cost many times over by knowing

how to ask the correct questions.”

- Creative Research Systems (2008)

Surveys are science and art

1. Formulategeneric

questionnaire

2. Expand the

questionnaireTurn into separatesections based onstudy objectives.

Draft qs & responseformats

4. Finalise questionnaire& implement

Question order &

funnel qs

Questionnaire development

3. Pre-test,pilot test,& redraft

Parts of a survey

• Participant information sheet• Informed consent form• Survey with sections containing

measurement items for each objective• End page(s)

– Indication of survey end– Instructions for returning survey

Parts of a survey

• Participant information sheet• Informed consent form• Survey with sections containing

measurement items for each objective• End page(s)

– Indication of survey end– Instructions for returning survey

• Make it look clean and easy– Large size (14 pt) and clear font type– Minimise number of pages– Logical flow/order– High contrast e.g., avoid text in

coloured boxes, etc.– Number the questions

Layout

Participant information sheetOutline details of research project e.g.,:• Who are you? Are you bona fide?• Purpose of survey?• What's involved?• Explain any risks/costs/rewards• How will results be used?• Human ethics approval #• More info: Complaints, how to obtain results,

contact details etc.

Informed consent formA separate page or screen following the participant information sheet which allows participants to indicate whether they consent or do not consent to participation in the study:• How is consent given / not given? (Can be active

consent or passive consent)• Statement should include that participants are free

to not participate in any part of the study and to withdraw at any time

Ethical considerations: How to treat respondents

• Informed consent• Minimise risk / harm to respondents• Confidentiality / anonymity• No coercion• Minimal deceit• Fully debrief• Honour promises to provide respondents

with research reports• Be aware of potential sources of bias /

conflicts of interest

• Provides consistency - helps to ensure standard conditions across different administrations

• Few will read it without good prompting and easy-to-read instructions

• Explain how to do the survey in a user-friendly manner, possibly with examples

Instructions

• Start gently; ease respondent in• Group similar questions together• Consider order effects:

– Habituation e.g. → polarisation of responses, yea-saying, nay-saying

– Fatigue– Minimise switching between response

formats

• Consider counter-balanced orders

Order, flow and structure

• Single section, usually at beginning or end of questionnaire

• Only include personal questions that are justified by the research question(s)

Demographics andpersonal information

• Space for comments?• Indicate the end• Say thanks!• Provide instructions about how to

return the survey or submit responses

• Details about how to contact researchers, obtain results, make complaint etc.

• Debriefing or referral information

Ending the survey

Pre-testing• Pre-test items on convenient others -

ask for feedback• Revise items e.g.,

– Which don’t apply to everybody– Are redundant– Are misunderstood– Are non-completed

• Reconsider ordering & layout

Pilot-testing• Pilot test on a small sample from the

target population• Analyse data• Revise survey

How to write good survey questions

How to write good survey questions:

Overview

1. Survey question tips2. Objective vs. subjective questions3. Open- vs. closed-ended questions4. Closed-ended response formats5. Improving survey questions

(Exercise)

• Be direct: Focus directly on topic/issue

• Be clear: Brief and readable– Avoid big words– Use simple and correct grammar

● Define target constructs: be as concrete and unambiguous as possible; the meaning must be clear to all respondents

Survey question tips

Survey question tips● Related tools: Check similar surveys● Relate to objectives: Only ask

questions that relate to research objectives

● Ask questions: Phrase as questions

• Brevity: Keep questions as short as possible

Survey question tips

• Applicability: Questions must be applicable to all respondents (or use skip rules).

• Exhaustive: Response options must be exhaustive (i.e., provide options for suitable for each respondent) and mutually exclusive (i.e., not overlapping)

• Demand: Recall of detail must not be unnecessary or excessive

Watch out for questions which are...

Double-barrelled: Questions which contain more than one concept or purpose should be simplified or split into separate questionse.g., “What do you think the speed limit should be for cars and trucks?” vs.“What do you think the speed limit should be for cars?”“What do you think the speed limit should be for trucks?”

Watch out for questions which are...

Double negative: Negatively worded questions are often confusing because responding "no" creates a double negative. e.g., “Do you disapprove of gay marriage?” vs “Do you approve of gay marriage?”

Watch out for questions which are...

Leading: A question that suggests the answer the researcher is looking for e.g., “Do you agree that psychologists should earn more than they are currently paid?” vs.

“Do you think that psychologists' wages are lower than they should be, higher than they should be, or about right?”

“What dangers do you see with the new policy?” vs.

“What do you think about the new policy?”

Watch out for questions which are...

Loaded: A questiopn that suggest socially desirable answers or is emotionally charged. e.g.,

“Have you stopped beating your wife?” vs“Have you ever physically struck your partner?”

“Do you advocate a lower speed limit in order to save human lives?” vs “What speed limit is required for traffic safety?”

Objective questions

● A verifiably true answer exists (i.e., factual info).

● An observer (in theory) could provide an accurate answer.

e.g.,How many times during 2014 did

you visit a general medical practitioner? ______

Subjective questions• Asks about fuzzy personal perceptions• There is no “true”, factual answer• Many possible answers• Can't be accurately answered by an

observer. e.g.,Think about the visits you made to a GP

during 2014. How well did you understand the medical advice you were given?perfectly very well reasonably poorly not at all

Open-ended questions: Examples

What are the main issues you are currently facing in your life?

How many hours did you spend studying last week? _________

Closed-ended questions

• Important information may be lost forever

• Useful for hypothesis testing• Easy and objective to analyse• Time efficient

Closed-ended response formats1.Dichotomous2.Multichotomous3.The list (multiple response)4.Ranking5.Verbal frequency scale6.Likert scale7.Graphical rating scale8.Semantic differential9.Non-verbal (idiographic)

DichotomousTwo response options e.g.,

Excluding this trip, have you visited Canberra in the previous five years? (tick one)

__ Yes __ No

Provides the simplest type of quantification (categorical LOM).

MultichotomousChoose one of more than two possible answers e.g.,What type of attractions in your current trip to Canberra most appeal to you? (tick the most appealing one)

__ historic buildings

__ museum/art galleries

__ parks and gardens

The list (multiple response)Provides a list of answers for respondents to choose from e.g.,Tick any words or phrases that describe your perception of Canberra as a travel destination:

__ Exciting __ Important

__ Boring __ Enjoyable

__ Interesting __ Historical

RankingHelps to measure the relative importance of several itemsRank the importance of these reasons for your current visit to Canberra (from 1 (most) to 4 (least)):

__ to visit friends and relatives

__ for business

__ for educational purposes

__ for holiday/ sightseeing

Verbal frequency scaleOver the past month, how often have you argued with your intimate partner? (circle one)1. All the time2. Fairly often3. Occasionally4. Never5. Doesn’t apply to me at the moment

Likert scale

1 2 3 4 5strongly disagree neutral agree stronglydisagree agree

Measures strength of feeling or perception.Indicate your degree of agreement with this statement:

“I am an adventurous person.”(circle the best response for you)

AGREEMENT ABOUT SOMETHING2-Categories

DISAGREE AGREE

3-Categories DISAGREE NEUTRAL AGREE

4-Categories STRONGLY MILDLY MILDLY STRONGLY

DISAGREE DISAGREE AGREE AGREE

5-Categories STRONGLY MILDLY MILDLY STRONGLY DISAGREE DISAGREE NEUTRAL AGREE AGREE

Number of response options?Likert scale example

How many response options?• Minimum = 2• Average = 3 to 9• Maximum = 10?

Basic guide: 7 +/- 2

Number of response options?

Watch out for too many or too few response options

“Capital punishment should be reintroduced for serious crimes”

1 = Agree 2 = Disagree1 = Very, Very Strongly Agree 7 = Slightly Disagree2 = Very Strongly Agree 8 = Disagree3 = Strongly Agree 9 = Strongly Disagree4 = Agree 10 = V. Strongly Disagree5 = Slightly Agree 11 = V, V Strongly Disagree6 = Neutral

Sensitivity & reliability

• Scale should be sensitive yet reliable.

• Watch out for too few or too many options.

Graphical rating scale

Rate your enjoyment of the movie you just saw. Mark your response with a cross (X) on the line below.

not very enjoyable enjoyable

What is your view of tobacco smoking? Place one tick on each row to show your

opinion.

Bad ___:___:___:___:___:___:___ GoodStrong ___:___:___:___:___:___:___ WeakMasculine ___:___:___:___:___:___:___ FeminineUnattractive ___:___:___:___:___:___:___ AttractivePassive ___:___:___:___:___:___:___ Active

Semantic differential

Summary: Survey questions

1. Objective vs. subjective questions1. Objective – there is verifiably true

answer2. Subjective – based on perspective

of respondent2. Open vs. closed

1. Open – empty space for answer2. Closed – pre-set response format

options

Summary: Response formats1. Dichotomous and Multichotomous2. Multiple response3. Verbal frequency scale (Never... Often)4. Ranking (in order → Ordinal)5. Likert scale (equal distances →

Interval, typically with 3 to 9 options)6. Graphical rating scale (e.g., line)7. Semantic differential (opposing words)8. Non-verbal (idiographic)

How could these survey questions

be improved?

Example: How could this question be improved?

How old are you?

___ 18-20

___ 20-22

___ 22-30

___ 30 and over

Are you satisfied with your marriage and your job?__________________________

Example: How could this question be improved?

You didn’t think the food was very good, did you? (tick your answer)_____ Yes _____ No

Example: How could this question be improved?

Environmental issues have become increasingly important in choosing hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation? (tick an answer) ____ Yes ____ No

Example: How could this question be improved?

How did you hear about this restaurant?(please tick appropriate spaces) ____ yellow pages ____ Internet ____ word of mouth

Example: How could this question be improved?

Levels of measurement

● Nominal / Categorical

● Ordinal

● Interval

● Ratio

Each level has the properties of the precedinglevels, plus something more!

Ratio data are continuous.

Nominal, ordinal and interval dataare discrete.

Categorical / nominal

• Conveys a category label• (Arbitrary) assignment of #s to

categories e.g. Gender

• No useful information, except as labels

Ordinal / ranked scale

• Conveys order, but not distance e.g. in a race, 1st, 2nd, 3rd, etc. or

ranking of favourites or preferences

Interval scale

• Conveys order & distance• 0 is arbitrary• e.g., interval scale 1 2 3 4 5STRONGLY MILDLY MILDLY STRONGLY DISAGREE DISAGREE NEUTRAL AGREE AGREE

• For data analysis assumption testing, usually treat as continuous if > 5 intervals are used.

Ratio scale• Conveys order & distance• Meaningful 0 point e.g. height, age, weight, time, number of times

an event has occurred

• Continuous (i.e., there can be fractional amounts / decimal places)

• Ratio statements can be madee.g. X is twice as old (or high or heavy) as Y

Why do levels of measurement matter?

Different analytical proceduresare used for different

levels of data.

More powerful statistics can be applied to higher levels

Summary: Level of measurement1. Categorical/Nominal

1. Arbitrary numerical labels2. Could be in any order

2. Ordinal1. Ordered numerical labels2. Intervals may not be equal

3. Interval1. Ordered numerical labels2. Equal intervals

4. Ratio1. Meaningful 02. Data are continuous

Quiz question 1: What level of measurement are

the following questions?

Quiz question 2: What level of measurement is used for this survey question?

How well do you think you have understood this lecture about survey design so far?

perfectly very well reasonably poorly not at all

Quiz question 3: What level of measurement is used for this survey question?

Rate your view about this statement:Australia should provide residency to more

asylum seekers.

1 2 3 4 5strongly disagree neutral agree stronglydisagree agree

Quiz question 4: What level of measurement is used for this survey question?

What is your favourite primary colour? (choose one of the following options)

• Red• Yellow• Blue

Sampling

Sampling:Overview

1. Sampling terms2. What is sampling?3. Why sample?4. Sampling methods5. Example: Shere Hite’s survey

Sampling terms• Target population

– To whom do you wish to generalise?

• Sampling frame– Who has a chance of being selected?

• Sample– Who was selected and responded?

• Representativeness– To what extent is the sample a good

indicator of the target population?

What is sampling?

“Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.”- Trochim (2006)

Why sample?

• Reduces cost, time, sample size etc.

• If the sample is representative, the sample data allows inferences to be drawn about the target population.

Sampling process

● Identify target population and sampling frame

● Select sampling method● Calculate sample size for desired

power.● Maximise return rate

Representativeness of a sample depends on:

• Adequacy of sampling frame• Sampling method• Adequacy of sample size• Response rate – both the % &

representativeness of people in sample who actually complete survey

It is better to have a small, representative sample than a large, unrepresentative sample.

Sampling methodsTypes of probability sampling:• Simple random• Systematic random• Stratified randomTypes of non-probability sampling:• Convenience• Purposive• Snowball

Probability sampling

• Each unit has an equal chance of selection

• Selection occurs entirely by random chance

Simple random sampling• Everyone in the target population

has an equal chance of selection• Useful if clear study area or

population is identified• Similar to a lottery:

–List of names are assigned #s and then randomly #s are used to select respondents

–Random selection can be manual using a table of random #s or by computer

Systematic random sampling

• Respondents (units) are selected from a list e.g., list of students

• Useful when target population closes matches a list

• Select the sample at regular intervals e.g., every 5th person on a list (starting at a random number between 1 and 5)

Stratified random sampling

• Sub-divide population into strata (e.g., gender, age, or location)

• Then randomly select from within each stratum

• Improves representativeness• e.g., Telephone interviews

conducted use using post-code strata

Non-probability sampling• Useful for exploratory research

and case study research• Able to get large sample size

quickly• Limitations include potential bias

and non-representativeness

Convenience sampling

• Sampling is by convenience (i.e., whoever is available) rather than randomly e.g. surveying visitors to a tourist attraction over one weekend

• Less cost/time involved than random sampling

• Subject to sampling bias

Purposive sampling

• Respondents are selected for a particular reason e.g., because they are “typical” respondents

• e.g., for a tourism study, select a sample of tourists aged 40-60 for interviews as this is the typical age group of visitors to Canberra

• e.g., Contacting Frequent Flyer members to participate in a survey about service quality in an airline setting

Snowball sampling

• Respondents are asked to recommend other respondents

• Useful for difficult to access populations e.g., illegal immigrants, illegal drug users

• e.g., in studying ecstasy users, a research may gain trust of a few potential respondents and ask then these respondents to recommend the researcher to other potential respondents

Summary: Sampling1. Key terms

1. (Target) population2. Sampling frame3. Sample

2. Sampling–Probability (random)

1. Simple2. Systematic3. Stratified

2. Non-probability1. Convenience2. Purposive3. Snowball

100

Biases

101

BiasesBiases which can influence survey research data:• Non-sampling biases

– Instrumentation reliability and validity– Response biases

• Sampling biases– Sample does not represent target

population

102

Response biases• Acquiescence

– yea- and nay-saying• Order and fatigue effects• Demand characteristics• Hawthorne effect• Self-serving bias• Social desirability

103

Demand characteristicsParticipants form an interpretation of the researcher's purpose and unconsciously change their behaviour to fit that interpretation.

Interview• Higher demand characteristics

Questionnaire• Lower demand characteristics

104

Maximising response rate

• Layout and design is key• Respondent’s level of interest• Rewards• Accompanying letter / introduction• Colour of paper• Mail surveys - self-addressed

stamped return envelope• Reminders or follow up calls

105

References• Alreck, P. & Settle, R. (1995). The survey

research handbook (2nd ed.). New York: Irwin.

• Reeve, J. (2009). Understanding motivation and emotion (5th ed.). Hoboken, NJ: Wiley.

• Stevens, S.S. (1946). On the theory of scales of measurement. Science, 103, 677-680.

• Trochim, W. M. K. (2006). . In Research Methods Knowledge BaseSampling.

• Wikipedia (2009). Shere Hite - Methodology.

Survey Design - Wikimedia€¦ · “Hourglass” notion of research e.g., design a survey then sample data. 13 Survey types 14 Types of surveys Self - administered Interview - administered - [PDF Document] (2025)

References