Quantitative survey

What is the quantitative survey?

Surveys are a popular method of collecting primary data. The broad area of survey research encompasses any measurement procedures that involve asking questions of respondents. They are a flexible tool, which can produce both qualitative and quantitative information depending on how they are structured and analysed. In this section we focus on the quantitative use of surveys, and in later sections we explore the more qualitative use of survey methods.

When should it be used?

What do I need to consider?

In undertaking a survey it is important to understand who you want to survey, how you are going to select them, how you are going to survey them, what you want to ask them and how you are going to organise the task. The following section outlines some key considerations that need to be made before embarking on a large-scale survey.

Population – A number of questions about the proposed popula­tion for a survey need to be considered. Such as:

Can the population be counted? Some populations will be easy to count, in a given geographical area there will be secondary data sources that will give you a population count (Census), in a membership organisation there may be a list of all members, however in a newly arrived ethnic community such as the recent arrivals of Polish and Eastern European communities there is less chance that you can obtain a reliable count of the population. A bias in your survey results can occur if the survey sample does not accurately represent the population. Having a count of the population is also important in order to establish the significance of your results to allow a generalisation to the population as a whole.

Are there language issues? Respondents may have vary­ing capacities for being able to complete written surveys or questionnaires. While telephone and street surveys do not require the respondent to be able to read or write in English, postal surveys involve respondents completing the survey or questionnaire themselves. You should consider the offer of help in self-administered surveys for respondents to complete a form either in person or over the telephone, this will help address potential language or basic skills issues. If surveying an ethnic minority population you may wish to translate questionnaires into community languages, or have people who speak the com­munities’ language to assist where necessary.

What are the geographic restrictions? The geographic spread of the population to be surveyed will determine the method used for collecting your data. If you are surveying people from a particular location or organisation it may be possible to conduct a survey using an interviewer, however if you have a population sample that is geographically dispersed then you would look to use a different method, such as a telephone or postal survey.

Sampling

The sample is the section of the wider population that will be engaged in the survey and sampling is the process of identifying who you will aim to contact from that population. The word ‘population’ is used to describe the target group, and while this may be the national popula­tion as a whole, it may also be a smaller group such as lone parents, or business members of a Chambers of Commerce in a particular location. Detailed consideration of sampling needs to be made to ensure the validity of your results, and the following issues need consideration:

Who is the respondent? The first thing you need to under­stand is who your respondent is going to be. This is the person that will provide the data you are asking for. If the survey is distributed amongst households, who in particular will be filling in the survey? Do you want to specify who the survey is to be completed by? And do you understand why you are specifying this person? The same is true when surveying organisations or groups. A survey will have much greater success if it is directed to the right respondent. Identifying the person best suited to completing a survey will help to increase the response rate and generate more accurate data.

What is your sampling frame? A sampling frame is a list of members of a population from which members of a sample are then selected. A sampling frame needs to be accurate, complete, up-to-date and relevant to the purposes of the survey for which it is to be used. Once you have an established sampling frame, depending on its size you may need to adopt a sampling technique to extract your final sample. For example random sampling, simple random sampling or stratified sampling (see further reading for more details on sampling techniques).

Are response rates likely to be a problem? With any survey, you need to look at the profile of the people who did responded and satisfy yourself that they are about the same as the people who didn’t respond – and also, that they’re about the same as the overall population that you’re sampling. If you send out a survey to a population, which is 50% male, and 50% female, but your responses are 80% from females then your findings will not represent your target population. Response rates can be low for surveys, under 20% for a postal survey is not uncom­mon. However, all the considerations in this section can help to improve your response rate.

Statistical significance: Understanding your population, sample size, and response rates are important for calculating

interval and confidence levels, which are vital in determining how many people you need to interview in order to get results that reflect the target population as precisely as needed. You can use online calculators to establish this type of information, but it is important to understand the terms and the reasons for doing this (see section on statistical analysis for more detail).

Format

It is important to understand what format of survey you are looking to undertake. There are broadly two survey formats that you may use and it is important to understand which you are using:

Cross-sectional surveys are used to gather information on a population at a single point in time. An example of a cross­sectional survey would be a questionnaire that collects data on peoples’ experiences of a particular initiative or event. A cross­sectional survey questionnaire might try to determine the relationship between two factors, like the impact of a pro­gramme of activity on the level of benefits claims for example.

Longitudinal surveys gather data over a period of time. This would allow analysis of changes in the population over time and attempt to describe and/or explain them. The three main types of longitudinal surveys are trend studies, cohort studies, and panel studies (for more details see further reading). A longitu­dinal study will also seek to determine the relationship between factors, but the difference is that the examination will be of a change in factors over time, so for example the relationship between health and employment.

Questions

There are a whole range of questions to be asked in survey design, such as: What types of questions can be asked? How complex will/can the questions be? Will screening questions be needed? Can question sequence be controlled? Will lengthy questions be asked? Will long response scales be used? Here we outline the main types of questions used in quantitative surveys:

Closed questions – these have a number of possible answers in a list for respondents to choose from (e.g. a closed question about the sources of funding for a community project would ask respondents to choose from a list of categories, such as New Deal for Communities, Neighbourhood Renewal Funding and so on). Usually, closed questions include an ‘other’ option to enable respondents to add any categories that have been omitted;

Ranking scales – these are most commonly used when trying to ascertain the level of importance of a number of items. A list of choices are provided and respondents are asked to put them in order (e.g. when undertaking a feasibility study for a new town centre, a question using a ranking scale may show a list of items that are commonly found in town centres and ask respondents to rank which ones are most important to them);

Sliding scales – these are used to discover respondents’ strength of feeling towards an issue. Respondents are given a series of statements and asked how much they agree or disagree with the statement by using a sliding scale where numbers represent different strengths of feelings. For example, 1 = strongly agree and 5 = strongly disagree.

Write questions that are clear, precise, and relatively short

Because every question is measuring something, it is important for each to be clear and precise. Your goal is for each respondent to inter­pret the meaning of each survey question in exactly the same way. If your respondents are not clear on what is being asked in a question, their responses may result in data that cannot or should not be applied in your survey findings.

Do not use “loaded” or “leading” questions

A loaded or leading question biases the response given by the respond­ent. A loaded question is one that contains loaded words. Loaded or leading questions may hint to the respondent how you expect the question answered, for example ‘Do you think your neighbourhood is still run down?’, by including the word ‘still’ a bias is introduced as it presupposes that the respondent thought the area was previously run down.

Ambiguous or compound questions can be confusing, leaving respond­ents unsure as to how to answer. Compound questions are ones that ask several things which might require different answers, for example ‘Would you like to see more community support officers on the streets, allowing a reduction in investment in CCTV?’. The respondent may wish to provide multiple answers to this question, answering yes to having more community support officers, but disagreeing with the reduction in investment for CCTV. See the section on further reading for more information on question types and constructing survey ques­tions.

Administration

The costs, required facilities, time, and personnel needed to conduct an effective survey are often underestimated. The most common resource underestimated is time. You need to factor in time to pilot or test your survey, time to deliver your survey, time to give respondents to complete surveys and then have them returned (this may be via mail and therefore take time to return), and you also need to factor in the time required to analyse surveys. When conducting a large scale survey, inputting data to generate your analysis can be very time consuming. The best approach is to often work up your timeline backwards from when you need your results, calculating the time required for each step, this way you can establish when things need to start by.

What is the output?

Survey data is the question answers, such as ‘yes’ or ‘no’ or perhaps a number, where a person has ranked a question on a scale. The survey data output will depend on the way in which the survey was construct­ed, it will be shaped by the survey questions asked, the format of the survey itself and the method in which data was collected. For example, if the survey was completed by the respondent, in a written form, then you will have a collection of written documents which require analysis of the question answers. If the survey has been completed by a re­searcher, then a more sophisticated method of data collection may have occurred e.g. tallies and counts of responses. If using an internet or email survey, a computer programme may have collected the data in a format which can easily be analysed. Consideration of the output needs to be made at the outset of the process, and time considerations need to be given as to how this data will be collected and analysed.

SurveyMonkey
SurveyMonkey is an online survey tool that enables people of all experience levels to create their own surveys quickly and easily.
It has an online survey designer, which contains many questions and formats. It collects responses and analyses them in real time, producing charts and graphs with available information. All responses can be downloaded in a variety of formats to allow further statistical analysis in computer packages such as SPSS.
www.surveymonkey.com

How should it be analysed?

Before you can input your data in a computer program or application you will need to undertake a process of coding. This involves assign­ing a code (often numeric) to each possible answer in your survey. So if question 1 in your survey asked the gender of the respondent, you may seek to code the answer ‘male’ with the number 0, while you may seek to code the answer ‘female’ with the number 1. Establishing these ‘codes’ on the distributed questionnaire can help at data entry time, but obviously has the downside of putting numbers on the questionnaire that are of no relevance to the respondent and therefore could make the questionnaire look more confusing than it needs to.

Web based programmes

Internet based survey tools can distribute your survey via email and also collect your results, often allowing you to view your results as they are collected in real-time. You can download live graphs and charts of the responses, and often filter the responses and dig down to get individual responses. While this offers significant benefits there needs to be careful consideration of the pros and cons of email or internet surveys and whether this method of collection suits the population you are targeting.

Excel

Microsoft Excel is useful for data summary, presentation, and for other basic statistical analysis. The program provides a set of data analysis tools called the Analysis ToolPak which you can use to save steps when you develop complex statistical analyses. You provide the data and pa­rameters for each analysis and the tool uses the appropriate statistical macro functions and then displays the results in an output table. Some tools generate charts in addition to output tables. The Analysis ToolPak is not loaded by default, instructions for installing it, along with guides on how to use it can be found on the Microsoft website.

SPSS (Statistical Package for Social Scientists)

SPSS is among the most widely used program for statistical analysis in social science. This is a data analysis package for quantitative research. It is particularly useful for the analysis of survey data as it covers a broad range of statistical procedures. There are other packages available such as SAS, Stata or Minitab however all are expensive to purchase, especially if only to be used for a one off survey. It may be possible to work with an academic institution to utilise their statistical packages, and organisations such as the Cathie Marsh Centre for Census and Sur­vey Research (CCSR) provide training on the use of these packages.

For more detail on data analysis see section on Statistical Analysis.

Further reading

A practical guide to sampling, National Audit Office 

Question Bank is an information resource, in the field of social re­search, with a particular emphasis on quantitative survey methods. http://qb.soc.surrey.ac.uk/

A general introduction to the design of questionnaires for sur­vey research, University of Leeds top2.pdf

The Centre for Applied Social Surveys, University of Southampton – runs a programme of short courses in survey methods across the UK. programme.php

Click to download PDF version.