Conduct of Sample Surveys by Queensland
What is a survey?
Surveys provide a means of measuring a population’s characteristics, self-reported and observed behaviour, awareness of programs, attitudes or opinions, and needs. Repeating surveys at regular intervals can assist in the measurement of changes over time. These types of information are invaluable in planning and evaluating Government policies and programs.
Unlike a census, where all members of a population are studied, sample surveys gather information from only a portion of a population of interest. The size of the sample depends on the purpose of the study.
In a statistically valid survey, the sample is objectively chosen so that each member of the population will have a known non-zero chance of selection. Only then can the results be reliably projected from the sample to the population. The sample should not be selected haphazardly or only from those who volunteer to participate.
When to use a survey
When determining the need for a survey, departments/agencies should first check that the required information is not already available (for example, conduct library searches, refer to the Office of Economic and Statistical Research.
The option of collecting the required information using existing administrative records should also be explored. Using existing data or records provides considerable advantages in terms of cost, time and the absence of respondent burden. The major disadvantage is the lack of control over the data collected. If existing data is not available or suitable, a number of factors must then be considered when determining which type of survey, if any, is appropriate. For example:
Practicality
-
Can the information be collected cost effectively and accurately via a survey?
-
Consider the complexity and sensitivity of the topic.
-
Do respondents have access to the required information?
-
Will they be willing to supply the information?
-
Will their responses to the questions be valid?
Resources
-
Are the necessary financial, staff, computer or other resources available?
Timing
-
When is the information required?
-
Is enough time available to ensure that data of sufficient quality can be collected and analysed?
-
When is the best time to conduct the survey? (For example, need to allow for seasonality, impact of school holiday periods etc).
Survey requirements
-
Do you want to use this information to target program improvements? If so, you may need to identify the key sub-groups you wish to report on (for example, geographic areas, age groups, sex, industry and size of business) and obtain sufficient responses for each group to ensure results are accurate enough for your needs.
Accuracy
-
What level of error can be tolerated? This depends on how and for what purposes you intend to use the survey results.
Frequency
-
Is the survey to be repeated? How often?
Legislative powers
-
Does the department/agency have authority to collect the information through either a compulsory or voluntary survey? For example, the Statistical Returns Act 1896 empowers the (Queensland) Government Statistician to collect statistical information on a wide range of matters on a compulsory basis.
Ethical consideration
Ethical considerations must be observed during the survey exercise. This includes that data, where appropriate, is treated confidentially, and that where information is sought on the understanding that the respondent cannot be identified, that such anonymity is preserved. Other ethical considerations include:
-
Do you need identifiable information (for example, names, addresses, telephone numbers) relating to respondents for follow-up research or matching with other data? If so, you need to clearly explain why you need such details and obtain their consent.
-
Will respondents be adversely affected or harmed as a direct result of participating in the survey?
-
Are procedures in place for respondents to check the identity and bona fides of the researchers?
-
If the survey is being conducted on a voluntary basis respondents must not be misled to believe it is compulsory when being asked for their co-operation.
-
If it is necessary to interview children under 14 years the consent of their parents/guardians/responsible adults must be obtained.
These factors must all be taken into consideration when developing an appropriate sample design (that is, sample size, selection method, etc.) and survey method. Depending on the expertise within the department/agency, this may be done with advice or assistance from the Office of the Government Statistician or a consultant.
Survey process
The following is an outline of the general process to be followed once the need for a survey has been determined. Some steps will not be necessary in all cases and some processes can be carried out at the same time (for example, data collection and preparation for data entry and processing).
A sample survey is cheaper and more timely than a census but still requires significant resources, effort and time. The survey process is complex and the stages are not necessarily sequential. Pilot testing of, at least, key elements such as the questionnaire and survey operations is an essential part of the development stage. It may be necessary to go through more than one cycle of development, testing, evaluation and modification before a satisfactory solution is reached.
The entire process should be planned ahead, including all critical dates. It is always beneficial to approach the Office of the Government Statistician (OGS) or prospective consultants as early as possible during this planning stage.
The time required from initial planning to the completion of a report or publication may vary from several weeks to several months according to the size and type of survey.
Key steps in the survey process include:
Planning and designing
1. Define the purpose, objectives and the output required. Experience has shown that well-defined output requirements at the outset minimise the risk of the survey producing invalid results.
2. Design collection methodology and sample selection method.
3. Develop survey procedures. Design and print test questionnaires and any other documentation (for example, instructions for interviewers and introductory letters).
Testing and modifying
4. Pilot test all aspects of the survey if possible. As a minimum, a small-scale pre-test of questionnaires can reveal problems with question wording, layout, understanding or respondent reaction.
5. Analyse test results (completed questionnaires, response rate etc). Obtain feedback from respondents and/or interviewers.
6. Modify procedures, questionnaires and documentation according to test evaluation.
7. Repeat steps 1-6 if necessary.
Conducting the survey
8. Finalise procedures, questionnaires and documentation.
9. Select sample.
10. Train interviewers (if interviewer-based)
11. Conduct the survey (that is, mail out questionnaires or commence interviewing) including follow-up of refusals and non-contacts, supervision and checks of interviewers’ work.
Processing and analysing
12. Prepare data entry, estimation and tabulation systems.
13. Code, enter and edit data.
14. Process data—calculate population estimates and standard errors, prepare tables.
15. Prepare report of survey results.
16. Prepare technical report. Evaluate and document all aspects of the survey for use when designing future surveys.
Collection method
Commonly used collection methods include telephone interviews, self-completion (for example, mail, email), face-to-face interviews or combinations of these.
Each has advantages and disadvantages in terms of the cost, time, response rate and the type of information that can be collected.
Self-completion Surveys are generally the least expensive, particularly for a widespread sample.
They allow respondents time to consider their answers, refer to records or consult with others. (These can be advantages or disadvantages, depending on the survey’s objectives).
They also eliminate interviewer errors and reduce the incidence of selected people (or units) being unable to be contacted.
A major disadvantage of self-completion surveys is the potentially high non-response. In such cases, substantial bias can result if people who do not complete the survey have different characteristics from those who do. However, response can be improved using techniques such as follow-up, well-written introductory letters and incentives for timely return of questionnaires.
In self-completion surveys there is no opportunity to clarify answers or supplement the survey with observational data. In mail surveys the questionnaire usually has to be simple and reasonably short, particularly when surveying the general community.
Email and Internet Surveys are becoming increasingly popular particularly for surveying internal clients or staff within organisations and allow more complex questionnaires to be used than mail surveys.
Interviewer-based Surveys (that is, face-to-face or telephone surveys) generally allow more data to be gathered than self-completion surveys, including the use of more complex questionnaires.
Interviewers can reduce non-response by answering respondents’ queries or concerns. They can often pick up and resolve respondent errors.
Face-to-face Surveys are, however, usually more expensive than other methodologies. Poor interviewers can introduce additional errors and, in some cases, the face-to-face approach is unsuitable for sensitive topics.
Telephone Surveys are generally cheaper and quicker than face-to-face surveys, and are being increasingly used. They are well suited for situations where timely results are needed and the length of the interview is not too long.
However, non-response may be higher than for face-to-face surveys as it is harder for interviewers to prove their identity, assure confidentiality and establish rapport.
Telephone surveys are not suited for situations where the respondents need to refer to records extensively. Also, the questionnaires must be simpler and shorter than for face-to-face surveys and prompt cards cannot be used.
COMPUTER ASSISTED TELEPHONE INTERVIEWING (CATI) helps to address some of these limitations. With CATI, interviewers use a computer terminal. The questions appear on the computer screen and the interviewers key in the responses directly into the computer. The interviewer’s screen is programmed to show questions in the planned order. Interviewers cannot inadvertently omit questions or ask them out of sequence. Online edits warn interviewers if they key in invalid values or unusual values. Most CATI systems also allow many aspects of survey operations to be automated, e.g. rescheduling of call-backs, engaged numbers and “No Answers”, and allow automatic dialling and remote supervision of interviewer/ respondent interaction. |
It is estimated that about 97% of Queensland households have a connected telephone. However about 15% of households have a silent number. If the Electronic White Pages are used to select a sample of households then the sample will not include households with silent numbers. It will also not include households with recent new connections or recent changes to existing numbers.
If the likely bias due to such under-coverage is considered to be significant, a method called Random Digit Dialling may be used where the sample is selected from ranges of numbers which include all connected telephone numbers.
Research by the Office of the Government Statistician has shown that about 25% of households picked up using the Random Digit Dialling method are not listed in the latest Electronic White Pages. The main disadvantage of using Random Digit Dialling samples is the extra cost involved in screening out disconnected and business numbers.
Combinations of collection methods, such as interviewers dropping off a questionnaire to be mailed back or returning to pick it up, a mail survey with telephone follow-up, or an initial telephone call to obtain cooperation or name of a suitable respondent followed by a mail survey are sometimes used to obtain higher response rates to a survey.
If in-depth or purely qualitative information is required, consider alternative research methods. Focus groups, observation and in-depth interviewing are all useful when developing a survey or initially exploring areas of interest. They can also be a valuable supplement to survey data. However, results from such studies should not be considered representative of the entire population of interest. |
|
Sources of error
Whether a survey is being conducted by departmental/agency staff or by consultants, it is important to be aware of potential sources of error and strategies to minimise them.
Errors arising in the collection of survey data can be divided into two types—sampling error and non-sampling error.
Sampling Error occurs when data is collected from a sample rather than the entire population. The sampling error associated with survey results for a particular sub-group of interest depends mainly on the number of achieved responses for that sub-group rather than on the percentage of units sampled. Estimates of sampling error, such as standard errors, can be calculated mathematically. They are affected by factors such as:
-
Sample size—increasing the sample size will decrease the sampling error;
-
Population variability—a larger sampling error will be present if the items of interest vary greatly within the population;
-
Sample design—standard errors cannot be calculated if the probability of selection is not known (for example, quota sampling).
All other errors associated with collecting survey data are called NON-SAMPLING ERRORS. Although they cannot be measured in the same way as sampling errors, they are just as important.
The following table lists common sources of non-sampling error and some strategies to minimise them.
| Source of Error | Examples | Strategies to minimise error |
| Planning and interpretation | Inadequate definitions of concepts, terms or populations. | Ensure all concepts, terms and populations are defined precisely through consultation between data users and survey designers. |
| Sample selection | Inadequate list from which sample is selected; biased sample selection. | Check list for accuracy, duplicates and missing units; use appropriate selection procedures (see “Bias and Accuracy” below). |
| Survey methods | Inappropriate method (e.g., mail survey for a very complicated topic). | Choose an appropriate method and test thoroughly. |
| Questionnaire | Loaded, misleading or ambiguous questions, poor layout or sequencing. | Use plain English, clear questions and logical layout; test thoroughly. |
| Interviewers | Leading respondents, making assumptions, misunderstanding or misrecording answers. | Provide clear interviewer instructions and appropriate training, including exercises and field supervision. |
| Respondents | Refusals, memory problems, rounding answers, protecting personal interests or integrity. | Promote survey through public media; ensure confidentiality; if interviewer-based, use well-trained, impartial interviewers and probing techniques; if mail-based, use a well-written introductory letter. |
| Processing | Errors in data entry, coding or editing. | Adequately train and supervise processing staff; check a sample of each person’s work. |
| Estimation | Incorrect weighting, errors in calculation of estimates. | Ensure that skilled statisticians undertake estimation. |
If a consultant conducts the survey, departmental/agency staff should have input into the questionnaire design, participate in testing and attend interviewer training and debriefing.
Details of techniques used to minimise non-sampling error should be requested.
Bias and accuracy
Non-response occurs in virtually all surveys through factors such as refusals, non-contact and language difficulties.
It is of particular importance if the characteristics of non-respondents differ from respondents. For example, if high-income earners are more likely to refuse to participate in an income survey, the results will obviously be biased towards lower incomes.
For this reason, all surveys should aim for the maximum possible response rate, within cost and time constraints, by using techniques such as call-backs to non-contacts and follow-up of refusals. The level of non-response should always be measured.
Hint |
Bias can also arise from inadequate sampling frames, the lists from which respondents are selected. Household and business telephone listings and electoral rolls are often used as sampling frames, but they all have limitations. Telephone listings exclude respondents who do not have telephones and can exclude those with “silent” or unlisted numbers. Electoral rolls exclude respondents aged less than 18 years of age, migrants not yet naturalised and others ineligible to vote.
Once again, if these people are of interest and have different characteristics to those included on the frame, bias will be introduced.
One selection method often used by researchers is quota sampling. Interviewers are instructed to obtain a certain number of interviews, often with respondents in particular categories. For example, 30 interviews with females aged 18 to 25 years and 20 interviews with males aged 18 to 25 years etc. Interviewers may interview anyone fitting these criteria. Unfortunately, people who are most easily contacted or most approachable may have different opinions or behaviour to those not interviewed, introducing potential bias. As each person’s chance of selection is not known, standard errors cannot, strictly speaking, be calculated. Consequently, the accuracy of the survey results cannot be determined.
Hint |
If constraints are such that a probability sample is impractical, other research methods—such as focus groups or purposive sampling—should be considered. It must, however, be remembered that results from these procedures cannot be assumed to be representative of a broader population.
Given that a probability sample has been undertaken, standard errors should be calculated to check the accuracy of all results.
Hint |
|
The role of the Office of the Government Statistician (OGS)
The role of the Office of the Government Statistician is to coordinate statistical activities of the Queensland Government and to provide a policy framework for the collection and management of statistics. This includes compliance with Government directives on the management of statistics, the reduction of waste through unnecessary duplication and monitoring the impact of data collection on those who respond to surveys, particularly businesses.
The OGS is part of the Office of Economic and Statistical Research (OESR) which provides a commercial statistical service including assistance to agencies conducting surveys by:
-
Conducting all or part of the survey
In consultation with departmental/agency staff, OESR can design the survey and questionnaire, pilot test, train interviewers, enumerate, process and analyse the data;
-
Assisting in selection or evaluation of a consultant
Agencies may prefer to engage specialised consultants or the OESR may not be able to conduct the survey due to other priorities. In these cases the OESR can assist with all aspects of engaging consultants, including assistance with preparing briefs, selecting consultants and evaluating their work; or
-
Assisting agencies to conduct their own surveys
Budgetary or other constraints may require agencies to conduct their own surveys. The OESR can still provide advice on all aspects of conducting a sample survey.
Engaging consultants / contractors
The following information on preparing briefs may be of particular assistance in the engagement of consultants or contractors to conduct all or part of a survey. Please refer to the “Engaging and Managing Consultants” Guide for information on the general process and requirements for engaging consultants or contractors.
To obtain the highest quality proposals, it is important to provide contenders with the maximum amount of relevant information. Concentrate on clearly stating aims, objectives and requirements.
The following (additional) information should be included in a brief for the conduct of a survey:
-
Background
Give relevant background details such as previous research and where the survey fits into the department’s/agency’s program.
-
Objectives
Outline the specific purpose and objectives of the survey.
-
Population
Indicate the population and/or sub-populations of interest. For example, all Queensland women aged 18 years or over.
-
The Consultancy
Provide details of the survey content and preferred method (if appropriate). Include a list of data items and output specifications (for example, tables and analyses, including accuracy requirements). Attach a draft questionnaire if prepared. Specify reporting requirements including the calculation of standard errors and details of response rates and techniques used to minimise non-sampling error. Clearly indicate the parts of the project which will be the responsibility of the consultant.
-
Timing
Include dates, such as:
-
receipt of proposals;
-
engagement of consultant;
-
commencement and completion of pilot testing;
-
commencement and completion of survey; and
-
presentation of results or report.
-
Proposals
Indicate proposal requirements, such as:
-
details of proposed method;
-
full breakdown of costs (list categories of interest);
-
details of interviewers, if appropriate (for example, number, location, training, experience);
-
details of previous relevant work;
-
names and backgrounds of staff who would be responsible for the project;
-
details of any part of the project which would be subcontracted to another organisation; and
-
proposed timetable.
-
Selection Criteria
-
Selection Criteria should also cover:
-
quality, clarity and relevance of proposed survey design;
-
expertise in technical and operational aspects of sample surveys; and
- demonstrated ability in undertaking similar work.
Where can I get more help?
- Requests for advice or further information about conducting surveys should be directed to:
statsconsulting@treasury.qld.gov.au
Telephone: (07) 3224 5326
Facsimile: (07) 3227 7437
See also OESR'sSurveypage on this site.
Last reviewed: Feb 14, 2007, Last modified: Feb 14, 2007
Other languages