April 15, 2020 9:02AM

The Science of Survey Chunking

Authored by Dr. Kelvin Goh, PhD in Physics

Internet users today have shorter attention span than a goldfish. To get them to complete an online survey, we need to keep the questionnaire short. However, that can often be challenging, especially when transitioning long offline surveys to online.

Offline surveys are often very long (more than 20 questions). This is because the cost to acquire samples offline is very high, so clients often wants to maximize their return by asking as many questions as possible. However, long questionnaires have high rates of dropout, where respondents stop responding to the survey before the end of the survey.

The dropout rate is even higher for Internet users because of their short attention span. Furthermore, the few who would complete long questionnaires often speed through the questionnaire as they start to lose patience after progressing halfway. This would make the data collected from the final half of the questionnaire virtually useless, because most respondents would have simply answered them. Ultimately, it will be a waste of everyone’s time and client’s money.

It is often recommended that questionnaires should be kept short and succinct. However, this may not be possible for every survey.

Introducing survey chunking

There is another alternative solution to this problem. It is called survey chunking, which is to split a questionnaire into smaller chunks and get different respondents of identical demographics to answer each chunk. This would keep the questionnaire short for each respondent, reduce dropout rate and questionnaire speeding. This will ensure higher quality data and more accurate insights.

Is a chunked survey equivalent to a non-chunked survey?

On first glance, it might seem that having multiple different respondents completing a questionnaire might be scientifically wrong, even if they have identical demographics. However, one should remember the definition of a sample:

Sample is a pool of people who are randomly drawn from the target population to represent the greater population.

A respondent does NOT represent a population, a sample does. Let me give a simple example:

Assume a survey with 4 samples (A, B, C, D) and a 40-questions Questionnaire Q. Each sample has a sample size of 1000 and has identical sample definition Z.

Now let’s assume the following 3 scenarios:

Scenario 1:

All 4 samples answer the same question, e.g.: the 1st question of Questionnaire Q.

Scenario 2 (chunked):

All 4 samples answer a different chunk of Questionnaire Q, where Questionnaire Q is split into chunks of 10 questions.

Scenario 3 (non-chunked):

Sample A answers the entire Questionnaire Q.

In the case of Scenario 1, it is obvious that the 4 sets of identical samples that are asked the same question should give similar results that are within the sampling error margin.

Therefore, in the case of Scenario 2 (chunked), 4 sets of identical samples that are asked different chunks of the questionnaire should give similar results as Scenario 3 (non-chunked method). The only difference is that Scenario 2 will have higher quality data than Scenario 3 due to lower questionnaire fatigue.

If you are unable to replicate the above scenario, it is likely because your survey sampling method doesn’t have high degree of random sampling due to its small survey sampling frame. 

How to chunk a survey?

The only trade-off for having higher quality data with survey chunking is that you can’t directly cross tabulate questions from different chunks. However, this problem can be solved with careful planning of your survey using the following procedure:

1.    Use Opening Chunk

Opening Chunk contains a set of compulsory questions where every respondent will need to answer before progressing to their respective chunks. The typical questions that are placed in the opening chunk are demographic, psychographic and other profiling questions that you want to cross tabulate with questions in all other chunks. 

2.    Know your research objective and deliverables

By having a clear picture on the insights that you want to gain from your questionnaire, you may then structure your questionnaire with a goal of splitting the questionnaire into as small chunk as possible without compromising your research objective.

This can be done by placing all the questions that you want to cross tabulate with in the same chunk, while the questions that you want to cross tabulate across multiple chunks is placed in the Opening Chunk.

3.    Use indirect cross tabulation

If there is an oversight during the research planning stage and you would like to cross tabulate two questions’ data from different chunks after the fieldwork is completed, you can use Indirect Cross Tabulation to correlate both questions via Opening Chunk questions. 

Vodus Solution

Vodus survey platform have dedicated features to implement survey chunking to help clients gain high-quality survey data. For more info on how to utilize Vodus platform to generate fast and representative insights using survey chunking, please reach out at contact@vodus.com.