World Giving Index – Methodology

Image via Wikipedia

Part two of (probably) three discussing the World Giving Index, published by the Charities Aid Foundation (CAF). This post covers the report’s methodology.

Methodology

This post is based on the methodology section of the World Giving Index report (pages 2 and 3).

Secondary Analysis

The World Giving Index report uses data from Gallup Worldview about charitable behaviour and well-being, so in my opinion qualifies as secondary analysis. This isn’t necessarily a criticism, but it does mean it’s unclear how closely the research goals are reflected in the questions asked by Gallup, so just be mindful.

Sample Size

Responses were collected in 153 countries. As I’ve said before, there are between 192 and 203 states depending on who you ask. Since these are the most populous countries, they represent the vast majority – over 95% – of the world’s population.

In terms of sample size, between 500 and 2,000 responses were completed for each country, depending on the size of the population of the country. For example, samples of 2,000 were collected for larger countries, such as China and Russia, with smaller samples taken for smaller countries. This is a huge-scale survey (approximately 153,000 responses!).

The problem with the sample, though, isn’t the size, but how it was collected. Here there are some quite problematic limitations. As any statistician will tell you, it’s not the number of responses, but how people are selected to take part that makes the crucial difference. A sample size of 1,000 will obviously give you greater confidence in your results than a sample size of only 100, but the difference is marginal, and certainly not ten times more. Have a look at the wikipedia page on sample size for more details, but to sumarise, a sample size of 400 would mean we can be confident the population mean lies within 5% of our calculated sample mean, and a sample size of 1,000 gives us a 3% confidence interval. For us to be sure the population mean lies within 1% of of our calculated, sample mean, we’d need a sample size of 10,000. As you can see it’s not the sample size that limits the generalisability of the research.

Sampling Technique

The sample size is fine, but the way respondents were recruited has some quite significant limitations. For the assumptions about sample sizes above to work, the responses must be randomly selected from the population. Trouble is, they aren’t.

First, respondents were selected from urban centres only. This means that the results are not generalisable to the whole country in any way, only to urban dwellers in each country. This neglects whole swathes of the world’s population.

Second, I’m concerned that, on page 2 of the report, ‘at least 2,000 [responses] are collected.’ This might just be a poorly-phrased sentence, but I fear that this means in order to get the number of responses up to 2,000, much larger numbers of questionnaires were sent out. This is worrysome because a response rate of 50% for a questionnaire is high, suggesting that over 4,000 people might have been asked to reply. The trouble is, the type of people who respond to surveys are very different to people who do not. This is a significant problem in statistical surveys, and the reason why efforts should always be made to get the response rate as high as practically possible, rather than send more questionnaires out to get the number of responses desired.

Questions

The questions the survey asks have their pros and cons. The questions, principally, are:

1. Have you donated money to an organisation?
2. Have you volunteered time to an organisation?
3. Have you helped a stranger, or someone you didn’t know, who needed help?

These questions are excellent in that they recognise that giving is not just about donating money, although it is obviously important. Volunteering money and doing good deeds are important ways in which people can give to others.

My problem with the questions as they are asked is that, quite simply, they’re exactly the sort of thing people will exaggerate. Any responses are highly likely to be over-estimations of the true figure for each country. However, assuming people from each country exaggerate by the same amount, it is still possible to make comparisons between regions, bearing in mind true figures are probably lower than those reported.

Finally

Despite the issues I’ve raised here, I think the results this survey generated are still valuable. They should be insightful and indicative, but not really generalisable across the complete population of each country.