A Consultant’s Perspective on Conducting a Computer Assisted Telephone Interview (CATI) Population Health Survey - Computer Assisted Telephone Interview (CATI) Population Health Surveys Forum : Public Health Division, Department of Human Services, Victoria, Australia

A Presentation to the Computer Assisted Telephone Interview (CATI) Population Health Surveys Forum

A Consultant's Perspective on Conducting a Computer Assisted Telephone Interview (CATI) Population Health Survey

Presented by Tony Quint
Managing Director, TQA Research Pty. Ltd.

I'd like to cover ...

1. Value of population health surveys

2. Value of co-operation and syndication

3. An underestimated danger with CATI

4. Working with Government

5. Victorian Pilot Study - RDD vs. EWP

6. The balance between cost and perfection

Value of Population Health Surveys

Of key strategic benefit
You can't manage public health if you don't know what the people think and how they behave
Need:

The right information
Accurate information
Timely information
Affordable information

What do people think on the broad issues?

Huge danger ...

Never just rely on one or two 'simple' questions for complex issues
Many mistakes made in health policy areas because of poor or inadequate market research (usually over-simplification)
- Use simple questions to get the 'broad fix' and for tracking purposes
- Complex issues and policies generally require detailed research

A fortune gets spent on Surveys which make people better informed, but not much wiser.

'Wise' surveys tell you:

what's going on
why it's going on
what you could do to change what's going on
behaviours of key segments of population
what's changing over time (accurately)

An insight to strategy and progress

Value of Co-operation and Syndication in Health Surveys

'Health Care & Insurance - Australia'
Syndicated Survey conducted by TQA Research
Biennial since 1987 (conducted odd numbered years)
5,400 interviews across Australia
80 issues; 260 page report
Costs approximately $320,000 to conduct
Participation costs subscribers $10,000 to $44,000 (depending on States/Territories covered)
Syndication saves subscribers about 80%

Co-operation and syndication make a great deal of sense

Economies
Benchmarking
States can still have their own 'questions'

CATI ... powerful, but with dangers

Don't get overawed by the power and advantages of CATI

Automatic question skips
Custom-tailored question wording depending on respondent category
Automatic rotation of questions
Results available 'instantly'
Powerful contact status analysis

It's really the interviewers who are the vital cog in a high quality survey

Working with Government Agencies

The Working Party or Technical Reference Group
'Token' membership can be a problem
What members' job is and isn't

... not to make life as difficult as possible for the Consultant

... to ensure that the project is as good as it can be

Select members very carefully
Far better to have fewer members, but who are really interested in the project.

Victorian Pilot Population Health Survey 1998 (CATI)

Evaluation of Random Digit Dialling (RDD) versus Electronic White Pages (EWP)

Fieldwork:	9 November - 9 December
Top-Line Results:	18 December

Aims

To compare outcomes of RDD sampling versus EWP

Variation between samples (non-response bias)
Cost-effectiveness
Whether characteristics of people with a silent telephone number are different

Literature Search Outcomes

More than 200 articles reviewed. Many leave the reader 'up in the air'.
General consensus that RDD allows contact with households having unlisted numbers ('silent' or not yet listed)
Estimated that 10-15% of Australian households have silent primary number
Jury still out on whether RDD generates a different sample to EWP.

Methodology - Sample Size

Victoria Wide Survey - Metropolitan and non-Metropolitan

Methodology - Key Points

6 call-backs to make contact
Exhaust all call attempts before generating a replacement
Aiming for 80% response rate (from households where contact is made)
Sample frames drawn from EWP and RDD
7 languages including English
Pre-pilot (20 interviews)
Introductory letters sent out
1800 Hotline
Person selected for interview
- 18+ years
- First birthday on or after a specific date
Validate 10% of interviews (with call-back)
- Check if appropriate person interviewed (birthday check)
- Check another question

Generating RDD Numbers

Aim:

Keep the whole random dialling process purely random ...
but eliminate numbers that will not be legitimate residential telephone numbers

Telephone Number

Prefix

4 or 5 digits

Suffix

3 or 4 digits

Random prefix + random suffix = random number

Step 1 : Draw random prefix:

Convert all prefixes to 5 digit to 'equalise probabilities'
Draw random number and if match Eligible Prefix File, keep; otherwise discard and draw again

Step 2 : Draw random suffix

Draw 3 digit random number
Convert to 4 digit suffix by inserting last digit of prefix at 'front'
If matches Eligible Prefix/Suffix File, keep

Important:

Need to ensure have all eligible prefixes
Recently issued prefixes will generally be in new areas (e.g. new housing estates)

Scope for bias if leave out

A Very Recent Version of EWP is Preferable

About 5% of domestic numbers change each year
Ideally, want to 'wash' RDD sample against up-do-date EWP

to mail out letter

to calculate real proportion of households whose primary private line is 'silent'

Key Issue:

Which version of Electronic White Pages?

'Official' Telstra version
'Pirate' versions (DTMS and Australia on Disk)

Significant implications for project

So we need ...

Comprehensive list of prefixes
Cross-link between eligible area (STD) codes and eligible prefixes
List of suffixes or ranges of suffixes which are: (a) reserved always for business
(b) not used at all (i.e. 'empty')

How Do We Make RDD as Efficient as Possible?

Eliminating zero probability 'hits'
Eliminating low probability 'hits'

Cost vs. perfection

'Randomising' selections from EWP, e.g. draw a number and replace last two digits at random

RDD - How Many Numbers Required to Complete a Population Survey?

Huge variation in interview rate per RDD number:
- Government Statistician's Office Queensland - 1997 - (1:2.47)
- USA 1986 - highly respected Centre for Survey Research, University of Nevada (1:7)
- New South Wales Health Department - 1997/98 - (1:4.5)
Depends on:
- nature of survey (including interview length)
- where conducted
- steps taken to make the RDD process more 'efficient'

Techniques to Improve the RDD 'Hit Rate' - How Far Do We Go?

Techniques employed by others include:

Suffix 'occupancy density' method
- Use EWP to map the proportion of numbers in a given suffix range (combined with a prefix) which are listed in EWP
- Call only those suffixes where proportion of numbers listed ('density') exceed, say, 20%
Limited 'snowball sampling'
- If achieve a 'hit' with a particular RDD, add a digit and achieve another interview. Add another digit . . . . etc.
- Some overseas researchers believe can achieve up to 10 interviews in each 'snowball', without introducing undue bias.

Matching of RDDs With EWP to Get Addresses

Silent numbers won't be matched
Recent connections won't be matched.
The 'address problem' for people living in units and flats:
- unit number rarely listed in White Pages
- possible solution
Major issues:
(a) How many RDDs do we need?
(b) What proportion of listed vs unlisted numbers should we achieve?

Several subtleties which cannot be overlooked