Quality Assurance Processes in Computer Assisted Telephone Interview (CATI) Surveys

Why Quality?

• Asking someone a question is easy
  
• It is more difficult to work out if the person is 'typical' and to determine the quality of the answer they give

Possible Response Patterns

• Chronic non-responders ~10%
• Whim responders ~ 5%
• Cautious responders ~20%
• Compliant responders ~65%

The percentage of cautious responders may be increasing

Possible Response Patterns

• Of those who do respond, what percentage are genuinely interested?
• Probably few are genuinely interested
• Therefore:
  All aspects of survey must be designed to maximise both response and data quality

Improving Quality

• Persuading people to interview
• Pilot Testing
• After respondent agrees to interview

Improving Quality: Persuading People to Interview

• The questionnaire introduction
• Interviewer attributes / personality
• Interviewers' training
• Supervisor's skills

The Introduction

• Most refusals occur very early
• First sentences critical in limiting non-response
• Small wording changes can influence response rate greatly
• Also important in establishing early rapport between interviewer and respondent

Interviewer Training

• Important in improving interviewer persuasiveness
• Also alerts interviewers to impact of other issues on response rates
• QH runs a full day of training before each survey and provides an interviewer training manual to staff

Training Manual Topics

• Background to survey
• General rules for telephone interviewing
• Reasons for piloting and feedback we would like
• How to handle refusals
• Answers to questions respondents frequently ask eg how did you get my silent number?
• Correct use of disposition codes
• Other technical matters

Interviewer Attributes

• Voice / style
  • Some voices encourage compliance
• Accent
  • Lower response for interviewers with "foreign" accents
• Age / sex
  • Lower response for young male interviewers

Supervisor

• Must be adept at dealing with respondents who are suspicious or want information
• Good supervisor needs :
  • Awareness of health and survey issues
  • People management skills
  • Computing aptitude

Improving Quality: Pilot Testing

• Trains interviewers
• Tests the questionnaire
• Interviewers provide very useful feedback on wording / placement of questions
• Analysis of data can provide insights into respondent mis-interpretation of questions
• Calling-back respondents enables question reliability to be checked

Improving Quality: After Person Agrees to Interview

• Monitoring interviewer productivity
• Personnel management
• Speech pace and rhythm
• Questionnaire and survey length
• Questionnaire design

Monitoring Interviewer Productivity

Three ways of monitoring productivity:

• Reports generated by management software
  • Capabilities should be considered when choosing software
• Monitoring while interviewing
  • Additional software and hardware required
• Random audits of respondents
  • Gives feedback on interviewer and questionnaire

Personnel Management

• Interviewers are a valuable resource
• Can provide quick feedback on problems with wording and mis-interpretation
• If staff feel valued productivity improves
• Don't under-estimate the importance of staff morale to your survey

Speech Pace and Rhythm

• At the start of a long survey:
  • median interview length - long
  • variance - large
• Towards the end of the survey
  • median interview length - much shorter
  • variance - much tighter

Is this a quality improvement?

Speech Pace and Rhythm

• Over time interviewers develop a rhythm in question delivery
• This tends to force the speed of the interview along
• The longer the survey period, the greater the effect
• Particularly problematic where there is a long series of similar types of question

Questionnaire and Survey Length

Long questionnaires = respondent fatigue = low quality data

Long survey periods = interviewer fatigue = low quality data

Where possible interviewers should be rested or rotated but also work a minimum per week

Questionnaire Design

• Many aspects of questionnaire design impact on data quality, including:
  • question wording
  • question placement
  • question type (closed vs open etc)
• Important to consider which subjects are suited to CATI and which are not

Conclusion

• Impossible to accurately quantify data quality
• A balance is always struck between quality, productivity and cost
• Topics covered in this presentation represent strategies which QH takes to maximise data quality

Authorised by: Executive Officer, NPHP
Contact: Darryl Kosch, nphp@dhs.vic.gov.au
Date of publication: 1999
Page last updated: 27 November, 2003

Copyright | Disclaimer | Privacy statement