This section takes you through what to watch out for when you analyse your results and how to use your data to improve what you do.
Julie gets the survey results from all her groups and starts to analyse the data.
Julie analyses the results from the question she designed about optimism. The women show more improvement than men when it comes to feeling optimistic. So she realises the centre could do some research about why optimism doesn’t increase as much for men, and find new ways to address that.
Julie looks at the different responses to the open questions. A lot of the women who report big improvements in optimism are also part of a women’s peer-to-peer support group at the centre.
She decides to ask the men who took part in the survey if they would be happy to discuss the results, and how the centre could help them feel more optimistic. For example, could they start a similar support group for men?
The exercise classes are targeted at a specific age group. So Julie wants to know how people from the classes scored compared to other older people locally and nationally, in relation to the ONS life satisfaction question.
First, she compares the wellbeing scores of other older people living in the local authority area using the benchmark data. The exercise class scores are lower than the average for the local authority area. This is surprising because, from the conversations she had with people in the class, she expected their wellbeing scores to be higher than average.
So Julie then compares the local authority average with the national average, and it’s much higher. The local authority area is comparatively wealthier than the UK overall, so this isn’t so surprising. But Padley Heath isn’t as wealthy as the rest of the local authority area – it’s similar to the UK overall. So Julie decides it’s probably more appropriate to compare her exercise class scores with the national average. And they are higher, so she won’t be able to use this data to help with any funding bids.
To benchmark and report on her evaluation, Julie decides to present her data with both the local authority average and the national average, and explain the differences.
As new people join the class, Julie asks them to answer a few quick questions online including demographics, the ONS wellbeing questions and questions on social connection. She makes sure they know how the data will be used. All 50 of the new attendees are happy to fill in the questionnaire.
She compares her results to the benchmark in the UK. She can see from the baseline data that those joining the computer course have lower than average wellbeing and lower than average scores on social connection.
Julie asks people the same questions again after six months into the programme, plus a new open question about what positive changes people have experienced. She reminds the group their responses are private and she won’t use their names. A few of the adults have dropped out of the programme now – but only four, so she’s happy she can use the results.
The results show that there was an increase in life satisfaction and an increase in purpose. She checked using Excel, which shows her this is a significant change.
She wants to be clear why the changes in wellbeing have taken place. This was her thinking behind including the questions and social connection. She wants to understand the intermediate changes that have been taking place in their lives and in turn have led to higher wellbeing.
Julie finds that the adults:
From this, Julia realises that:
Julia checks this idea with the answers to her open questions. In their answers, many people – especially people who felt isolated before – mention how positive their new friendships are in their lives, and/or having the space to ‘check in’ with other people. So Julie knows she can start to look at new ways of targeting more isolated people to get them involved in the class.
Julia also notices an increase in anxiety for the female members of the group. It isn’t a big increase, but she decides to run a short focus group to find out which aspects of the programme could be linked to it.
Some women in the focus group say they worry about keeping up with the pace of the class or finishing the assignment. Julie records this so the centre can make sure women in the next programme can go at their own pace and get enough support.
Other people in the group also mention on their survey that they like the sessions in the room with the window more than the ones in the basement. So Julie notes down that one more thing she can do to improve things is to rethink her booking of the basement room.
Now that Julie has finished her survey she knows how successful the services – especially the computer class – have been when it comes to improving people’s wellbeing. She also knows what changes she can make to try and improve wellbeing even more.
She is going to:
For the purposes of this guide, we’ve assumed you’ve done other evaluations before and that you know how to:
Here are a few things to think about:
Who is going to use the results? Make sure what you produce will be useful for them – how you represent data makes a difference.
If you need more guidance: Go to NCVO Know-How Non-profit for some great advice and guidance on:
Even if you just want to prove your impact, your survey data can also help you improve what you do to make an even bigger difference to wellbeing. For example, it might show you that you could:
What you can improve and how
Analysing data from different types of question will help you find out what you could change and how:
When you deliver services and activities, your work exists within a wider system. So it’s important to understand what contribution other people or groups make to any changes in wellbeing as well. When you evaluate your activities, ask:
These questions are a great starting point for thinking about attribution – in other words how much credit you can take for changes that a person or community goes through.
Your answers will help you work out:
This is important both for improving your services, as well as proving your impact.
How to identify attribution
Here are some suggestions for how to get an idea of your contribution at the start of your project:
Approach | Method | Opportunities and limitations |
---|---|---|
Informed guess or expert judgement | Ask your staff or other experts to judge how much credit you can take. It could be quantitative – for example, “50% because we provide support 3 days a week”. | This can help, when you pitch or design a new project, to acknowledge the role your work plays in a person’s life. But you need to be clear in your report that it’s based on assumptions and experience not data. |
Ask people about the support they get and whether it’s made a difference to their lives | Work with participants to map out all the people and organisations that support them in their lives. | This could help you prepare a quantitative or qualitative representation of the contribution you make. It allows people to tell you how your project interacts with their lives. It’s an asset-based approach, which acknowledges and builds on what you already deliver, but recognises gaps that need to be filled. You need to be very clear in your report that this is the participants’ subjective account of their experiences. |
Benchmarking from national data or previous projects | Look at national data to see if other organisations have already collected data on a similar project (for example using a control group) and use this data to make an informed judgement. | Make sure you pick a project or service similar to yours. And be very clear in your report about any assumptions you make. |
Set up or use an existing control group (for exact comparison) | Use a control group for your project. You need to measure the same wellbeing data for people involved in your activities and those in the control group. | This approach can help you isolate and measure the changes you contribute to. But it often takes significant skills and resources. |
A regression using survey data (for general information) | You can use this to understand what differences in wellbeing can be linked to, for example, a certain activity rather than the different characteristics of the group. | This is only likely to be relevant if: 1. your survey covers lots of people 2. you’re using existing survey data to find out how important something is for wellbeing (in general) if everything else is equal. 3. You can do some valuable analysis without this. Only do this if you’re confident or have enough resources to bring an expert in. See ‘Advanced analysis’ for more details. |
This section is only for people or organisations who:
Ordinary least squares regression
Differences in wellbeing between different groups can sometimes be down to the different characteristics of people within the groups – like age, gender or income. But we often want to test whether wellbeing is different if every other characteristic is equal.
For example, your data may show that people living in areas with lower air pollution have higher wellbeing than people in areas of high air pollution – and it may be a big difference. However, if they live in areas with better air quality, they may also have things like higher income, better facilities and higher-quality jobs. So the difference in wellbeing is showing the combined impact of air quality plus all these other factors.
You can do this with ordinary least squares regression.
What to control for
Wherever possible, try to control for:
You need to interpret the results of a regression carefully. For example, if you find that ethnicity no longer predicts wellbeing when controlling for income, ethnicity has no direct effect on wellbeing. But ethnicity still affects income, which in turn affects wellbeing.
To find out more about this approach go to:
(S)WEMWBS: What does a change mean?
This information is from the Warwick-Edinburgh Mental Wellbeing Scale (WEMWBS) User Guide Version 2 May 2015:
This has not yet been published for the short version of WEMWBS (SWEMWBS).
Remember you are looking at the total scores across all the questions.
At a group level, in keeping with other studies, changes of half a standard deviation or more are likely to be important. The standard deviation depends on the number of participants so the more participants the smaller the change which can be statistically significant. At an individual level a change of three points or more in WEMWBS is regarded as the ‘minimumally important difference’. Below that level improvement is quite likely just to be normal fluctuation. The minimally important difference for SWMEWBS is 2.
Benchmarking is about understanding how your group compares to others in the UK. Here’s how you can compare your scores for the ONS wellbeing questions and WEMWBS.
Benchmarking your results: the ONS wellbeing questions
The ONS wellbeing questions are included on national surveys. So you can compare the wellbeing of people using your services to the average scores for your area or the category of people you’re dealing with.
This can be useful when it comes to:
These spreadsheets show the scores for the four ONS wellbeing questions. They have been split to show the differences for protected characteristics including:
There is a separate Excel spreadsheet for each of the four wellbeing questions so you can compare each one.
There are a few other ways the data has been split to allow for benchmarking:
These spreadsheets show mean as well as threshold data.
If you would like to research wellbeing from the Annual Population Survey, you can see the End User license version of the dataset at the UK Data Service.
Benchmarking your results: SWEMWBS
The population norms, split by groups, are here.
The SWEMWBS isn’t split up as much as the four ONS questions because the sample size is smaller.
Many organisations measure wellbeing to understand the difference they’re making to wellbeing, and expect their evaluation to show they’re improving wellbeing.
But what if it doesn’t?
Try and learn from your data and think about what it tells you. Discuss it with staff and people who use your services. Think about why things haven’t improved, and what this might say about your organisation, your services and the people you support.
Qualitative feedback or focus groups can help you understand what’s happening. You may find out that parts of someone’s life are getting better, even though their overall wellbeing scores are lower.
After carrying out your analysis and delving deeper by using focus groups, you may find you haven’t had an impact on wellbeing. In this case you might need to think about changing your your services to better suit the people you support.
We hope this guide is useful for you, and that you can use your results to prove and improve the effect you have on people’s wellbeing.
We also hope you’ll help us build a national evidence base of what really works to improve wellbeing. If everyone who uses this guide shares their results with us, we can find out how we can all make more of a difference to wellbeing across the UK.
So if you’re measuring wellbeing already or you plan to measure wellbeing using this guide, we’d love to hear from you. You can send your results summaries, case studies and evaluation reports to:
Julie gets the survey results from all her groups and starts to analyse the data.
Julie analyses the results from the question she designed about optimism. The women show more improvement than men when it comes to feeling optimistic. So she realises the centre could do some research about why optimism doesn’t increase as much for men, and find new ways to address that.
Julie looks at the different responses to the open questions. A lot of the women who report big improvements in optimism are also part of a women’s peer-to-peer support group at the centre.
She decides to ask the men who took part in the survey if they would be happy to discuss the results, and how the centre could help them feel more optimistic. For example, could they start a similar support group for men?
The exercise classes are targeted at a specific age group. So Julie wants to know how people from the classes scored compared to other older people locally and nationally, in relation to the ONS life satisfaction question.
First, she compares the wellbeing scores of other older people living in the local authority area using the benchmark data. The exercise class scores are lower than the average for the local authority area. This is surprising because, from the conversations she had with people in the class, she expected their wellbeing scores to be higher than average.
So Julie then compares the local authority average with the national average, and it’s much higher. The local authority area is comparatively wealthier than the UK overall, so this isn’t so surprising. But Padley Heath isn’t as wealthy as the rest of the local authority area – it’s similar to the UK overall. So Julie decides it’s probably more appropriate to compare her exercise class scores with the national average. And they are higher, so she won’t be able to use this data to help with any funding bids.
To benchmark and report on her evaluation, Julie decides to present her data with both the local authority average and the national average, and explain the differences.
As new people join the class, Julie asks them to answer a few quick questions online including demographics, the ONS wellbeing questions and questions on social connection. She makes sure they know how the data will be used. All 50 of the new attendees are happy to fill in the questionnaire.
She compares her results to the benchmark in the UK. She can see from the baseline data that those joining the computer course have lower than average wellbeing and lower than average scores on social connection.
Julie asks people the same questions again after six months into the programme, plus a new open question about what positive changes people have experienced. She reminds the group their responses are private and she won’t use their names. A few of the adults have dropped out of the programme now – but only four, so she’s happy she can use the results.
The results show that there was an increase in life satisfaction and an increase in purpose. She checked using Excel, which shows her this is a significant change.
She wants to be clear why the changes in wellbeing have taken place. This was her thinking behind including the questions and social connection. She wants to understand the intermediate changes that have been taking place in their lives and in turn have led to higher wellbeing.
Julie finds that the adults:
From this, Julia realises that:
Julia checks this idea with the answers to her open questions. In their answers, many people – especially people who felt isolated before – mention how positive their new friendships are in their lives, and/or having the space to ‘check in’ with other people. So Julie knows she can start to look at new ways of targeting more isolated people to get them involved in the class.
Julia also notices an increase in anxiety for the female members of the group. It isn’t a big increase, but she decides to run a short focus group to find out which aspects of the programme could be linked to it.
Some women in the focus group say they worry about keeping up with the pace of the class or finishing the assignment. Julie records this so the centre can make sure women in the next programme can go at their own pace and get enough support.
Other people in the group also mention on their survey that they like the sessions in the room with the window more than the ones in the basement. So Julie notes down that one more thing she can do to improve things is to rethink her booking of the basement room.
Now that Julie has finished her survey she knows how successful the services – especially the computer class – have been when it comes to improving people’s wellbeing. She also knows what changes she can make to try and improve wellbeing even more.
She is going to: