Section 9: Analyse and use your results

This section takes you through what to watch out for when you analyse your results and how to use your data to improve what you do.

More from Julie

Julie gets the survey results from all her groups and starts to analyse the data.

 

The results from quantitative questions

Julie analyses the results from the question she designed about optimism. The women show more improvement than men when it comes to feeling optimistic. So she realises the centre could do some research about why optimism doesn’t increase as much for men, and find new ways to address that.

The results from qualitative questions

Julie looks at the different responses to the open questions. A lot of the women who report big improvements in optimism are also part of a women’s peer-to-peer support group at the centre.

She decides to ask the men who took part in the survey if they would be happy to discuss the results, and how the centre could help them feel more optimistic. For example, could they start a similar support group for men?


Comparing results for the over-60s

The exercise classes are targeted at a specific age group. So Julie wants to know how people from the classes scored compared to other older people locally and nationally, in relation to the ONS life satisfaction question.

First, she compares the wellbeing scores of other older people living in the local authority area using the benchmark data. The exercise class scores are lower than the average for the local authority area. This is surprising because, from the conversations she had with people in the class, she expected their wellbeing scores to be higher than average.

So Julie then compares the local authority average with the national average, and it’s much higher. The local authority area is comparatively wealthier than the UK overall, so this isn’t so surprising. But Padley Heath isn’t as wealthy as the rest of the local authority area – it’s similar to the UK overall. So Julie decides it’s probably more appropriate to compare her exercise class scores with the national average. And they are higher, so she won’t be able to use this data to help with any funding bids.

To benchmark and report on her evaluation, Julie decides to present her data with both the local authority average and the national average, and explain the differences.                                                                     


Showing the impact of the computer class

As new people join the class, Julie asks them to answer a few quick questions online including demographics, the ONS wellbeing questions and questions on social connection. She makes sure they know how the data will be used. All 50 of the new attendees are happy to fill in the questionnaire.

She compares her results to the benchmark in the UK. She can see from the baseline data that those joining the computer course have lower than average wellbeing and lower than average scores on social connection.

Julie asks people the same questions again after six months into the programme, plus a new open question about what positive changes people have experienced. She reminds the group their responses are private and she won’t use their names. A few of the adults have dropped out of the programme now – but only four, so she’s happy she can use the results.

The results show that there was an increase in life satisfaction and an increase in purpose. She checked using Excel, which shows her this is a significant change.

She wants to be clear why the changes in wellbeing have taken place. This was her thinking behind including the questions and social connection. She wants to understand the intermediate changes that have been taking place in their lives and in turn have led to higher wellbeing.

Julie finds that the adults:

  • who show an increase in wellbeing also show an increase in social connections
  • who started with the lowest scores in social connection show the biggest increase in wellbeing after six months.

From this, Julia realises that:

  • bringing people together through the service could be an important factor in their improved wellbeing
  • she might be able to increase the service’s impact by making sure the centre is targeting people with lower social connection, who may need the service the most.

Julia checks this idea with the answers to her open questions. In their answers, many people – especially people who felt isolated before – mention how positive their new friendships are in their lives, and/or having the space to ‘check in’ with other people. So Julie knows she can start to look at new ways of targeting more isolated people to get them involved in the class.

Improving wellbeing by improving the computer class

Julia also notices an increase in anxiety for the female members of the group. It isn’t a big increase, but she decides to run a short focus group to find out which aspects of the programme could be linked to it.

Some women in the focus group say they worry about keeping up with the pace of the class or finishing the assignment. Julie records this so the centre can make sure women in the next programme can go at their own pace and get enough support.

Other people in the group also mention on their survey that they like the sessions in the room with the window more than the ones in the basement. So Julie notes down that one more thing she can do to improve things is to rethink her booking of the basement room.

What Julie has learnt

Now that Julie has finished her survey she knows how successful the services – especially the computer class – have been when it comes to improving people’s wellbeing. She also knows what changes she can make to try and improve wellbeing even more.

She is going to:

  • look into how the centre can help men feel more optimistic
  • make sure the centre targets more isolated people whose wellbeing would really benefit from the computer classes
  • look at the day-to-day aspects of the services – like what rooms they’re in and the support people get form the centre.

    SEE ALL OF JULIE’S STORY ON ONE PAGE

1. Before you start analysing

For the purposes of this guide, we’ve assumed you’ve done other evaluations before and that you know how to:

  • use Excel
  • do basic data analysis
  • communicate the difference you’ve made.

Here are a few things to think about:

  • From your work in Section 4, you already know what you want to find out, what you want to measure, and why. Recap on these things before you start so you don’t analyse anything unnecessarily.
  • You need reliable findings. Think about whether the data you have will allow you to come to specific conclusions. For example, the findings are not reliable if you work out average wellbeing scores based on the responses of just six people.

Who is going to use the results? Make sure what you produce will be useful for them – how you represent data makes a difference.


If you need more guidance: 
Go to NCVO Know-How Non-profit for some great advice and guidance on:

 

2. Using your data to improve your services

Even if you just want to prove your impact, your survey data can also help you improve what you do to make an even bigger difference to wellbeing. For example, it might show you that you could:

  • change your target audience to focus on people who need the most help or who show the greatest improvement from your activities
  • change the focus between activities, if your analysis shows some services are more effective than others
  • change how you run your services, if your analysis suggests that, for example, certain groups don’t benefit in the same way as others or people prefer certain aspects of a service
  • share findings with other organisations in your sector to have a bigger impact together.

What you can improve and how 

Analysing data from different types of question will help you find out what you could change and how:

  • Quantitative questions are things like the Office of National Statistics (ONS) questions or the closed questions you designed. They can help you identify areas where you could improve people’s wellbeing. They can also help you see if you’re targeting the people who need your help the      most. For example, if your work focuses on helping people in crisis, you might want to target people who get low scores for domains related to resilience.
  • Qualitative questions are the open questions (Section 4). You can use the answers to see what you can do to improve your services and, in turn, people‘s wellbeing.
3. Who takes credit for the results?

When you deliver services and activities, your work exists within a wider system. So it’s important to understand what contribution other people or groups make to any changes in wellbeing as well. When you evaluate your activities, ask:

  • who and what else is involved in supporting this person or community?
  • how can we understand their contribution?

These questions are a great starting point for thinking about attribution – in other words how much credit you can take for changes that a person or community goes through.

Your answers will help you work out:

  • if you can make sense of the contribution you make without over-claiming
  • if there’s a link between short-term changes and long-term impact, and any existing evidence to justify this link
  • where you can draw the line of accountability for a particular project – for example you can help someone prepare for an interview but once they’re in the interview room, you can’t do any more for them and it’s up to them
  • how to use short-term information (for example if the number of people taking part in an activity changes over a few months) in longer-term decision making

This is important both for improving your services, as well as proving your impact.

How to identify attribution

Here are some suggestions for how to get an idea of your contribution at the start of your project:

ApproachMethodOpportunities and limitations
Informed guess or expert judgementAsk your staff or other experts to judge how much credit you can take. It could be quantitative – for example, “50% because we provide support 3 days a week”.This can help, when you pitch or design a new project, to acknowledge the role your work plays in a person’s life. But you need to be clear in your report that it’s based on assumptions and experience not data.
Ask people about the support they get and whether it’s made a difference to their livesWork with participants to map out all the people and organisations that support them in their lives.This could help you prepare a quantitative or qualitative representation of the contribution you make.

It allows people to tell you how your project interacts with their lives.

It’s an asset-based approach, which acknowledges and builds on what you already deliver, but recognises gaps that need to be filled. You need to be very clear in your report that this is the participants’ subjective account of their experiences.
Benchmarking from national data or previous projectsLook at national data to see if other organisations have already collected data on a similar project (for example using a control group) and use this data to make an informed judgement.Make sure you pick a project or service similar to yours. And be very clear in your report about any assumptions you make.
Set up or use an existing control group (for exact comparison)Use a control group for your project. You need to measure the same wellbeing data for people involved in your activities and those in the control group.This approach can help you isolate and measure the changes you contribute to. But it often takes significant skills and resources.
A regression using survey data (for general information)You can use this to understand what differences in wellbeing can be linked to, for example, a certain activity rather than the different characteristics of the group.This is only likely to be relevant if:
1. your survey covers lots of people

2. you’re using existing survey data to find out how important something is for wellbeing (in general) if everything else is equal.

3. You can do some valuable analysis without this. Only do this if you’re confident or have enough resources to bring an expert in. See ‘Advanced analysis’ for more details.
4. Advanced analysis

This section is only for people or organisations who:

  • are confident and experienced in data analysis
  • or have an academic partner who is experienced in more advanced analysis
  • or have resources to bring in an expert in analysis.

Ordinary least squares regression

Differences in wellbeing between different groups can sometimes be down to the different characteristics of people within the groups – like age, gender or income. But we often want to test whether wellbeing is different if every other characteristic is equal.

For example, your data may show that people living in areas with lower air pollution have higher wellbeing than people in areas of high air pollution – and it may be a big difference. However, if they live in areas with better air quality, they may also have things like higher income, better facilities and higher-quality jobs. So the difference in wellbeing is showing the combined impact of air quality plus all these other factors.

You can do this with ordinary least squares regression.

What to control for

Wherever possible, try to control for:

  • Health and disability (using objective measures where you can)
  • Relationship status
  • Economic activity or employment
  • Income
  • Age and age squared
  • Gender
  • Ethnicity
  • Religion
  • Highest qualification level
  • Dependent children
  • Occupational group or status
  • Housing tenure
  • Interview type
  • Region or location

You need to interpret the results of a regression carefully. For example, if you find that ethnicity no longer predicts wellbeing when controlling for income, ethnicity has no direct effect on wellbeing. But ethnicity still affects income, which in turn affects wellbeing.

To find out more about this approach go to:

(S)WEMWBS: What does a change mean? 

This information is from the Warwick-Edinburgh Mental Wellbeing Scale (WEMWBS) User Guide Version 2 May 2015:

  • Remember you are looking at the total scores across all the questions.
  • At a group level, in keeping with other studies, changes of half a standard deviation or more are likely to be important.
  • At an individual level a change of three points is regarded as the ‘minimum clinically important difference’.

This has not yet been published for the short version of WEMWBS (SWEMWBS).

Remember you are looking at the total scores across all the questions.

At a group level, in keeping with other studies, changes of half a standard deviation or more are likely to be important. The standard deviation depends on the number of participants so the more participants the smaller the change which can be statistically significantAt an individual level a  change of three points or more in WEMWBS is regarded as the ‘minimumally  important difference’. Below that level improvement is quite likely just to be normal fluctuation. The minimally important difference for SWMEWBS is 2.

5. Comparing your results to national data

Benchmarking is about understanding how your group compares to others in the UK. Here’s how you can compare your scores for the ONS wellbeing questions and WEMWBS.

Benchmarking your results: the ONS wellbeing questions

The ONS wellbeing questions are included on national surveys. So you can compare the wellbeing of people using your services to the average scores for your area or the category of people you’re dealing with.

This can be useful when it comes to:

  • understanding the differences in wellbeing in your community, and tracking them over time
  • making the case for more funding for your organisation to improve wellbeing levels (if scores are lower than the benchmark scores)
  • using it alongside expert judgement for understanding attribution (see ‘How to identify attribution’).

These spreadsheets show the scores for the four ONS wellbeing questions. They have been split to show the differences for protected characteristics including:

  • age
  • religion
  • marital status
  • ethnicity
  • self-reported health
  • economic activity
  • reason for economic activity
  • reason for part-time work
  • local authority and other geographies.

There is a separate Excel spreadsheet for each of the four wellbeing questions so you can compare each one.

There are a few other ways the data has been split to allow for benchmarking:

These spreadsheets show mean as well as threshold data.

If you would like to research wellbeing from the Annual Population Survey, you can see the End User license version of the dataset at the UK Data Service.

Benchmarking your results: SWEMWBS

The population norms, split by groups, are here.

The SWEMWBS isn’t split up as much as the four ONS questions because the sample size is smaller.

 

6. What if your evaluation shows that wellbeing drops after your activities?

Many organisations measure wellbeing to understand the difference they’re making to wellbeing, and expect their evaluation to show they’re improving wellbeing.

But what if it doesn’t?

Think about why this might be

Try and learn from your data and think about what it tells you. Discuss it with staff and people who use your services. Think about why things haven’t improved, and what this might say about your organisation, your services and the people you support.

  • It’s not realistic to expect improvements or high scores for everyone.
  • Many people have complex lives and you probably can’t influence the negative impact this can have on wellbeing. But you may be able to help people maintain their wellbeing level, rather than improve it. If people have high anxiety, this could mean you’re reaching the people who need support the most.
  • Wellbeing isn’t always stable. Even when someone’s wellbeing is improving overall, there will still be some dips – which may be why a score is lower than you expect.
  • Scores can drop for various reasons, unrelated to your intervention. For example, someone could feel more comfortable being honest about how they feel as they develop closer relationships with people.

Qualitative feedback or focus groups can help you understand what’s happening. You may find out that parts of someone’s life are getting better, even though their overall wellbeing scores are lower.

If you’ve made no impact

After carrying out your analysis and delving deeper by using focus groups, you may find you haven’t had an impact on wellbeing. In this case you might need to think about changing your your services to better suit the people you support.

Help us find out what works

We hope this guide is useful for you, and that you can use your results to prove and improve the effect you have on people’s wellbeing.

We also hope you’ll help us build a national evidence base of what really works to improve wellbeing. If everyone who uses this guide shares their results with us, we can find out how we can all make more of a difference to wellbeing across the UK.

So if you’re measuring wellbeing already or you plan to measure wellbeing using this guide, we’d love to hear from you. You can send your results summaries, case studies and evaluation reports to:

Email evaluate@whatworkswellbeing.org

More from Julie

Julie gets the survey results from all her groups and starts to analyse the data.

The results from quantitative questions

Julie analyses the results from the question she designed about optimism. The women show more improvement than men when it comes to feeling optimistic. So she realises the centre could do some research about why optimism doesn’t increase as much for men, and find new ways to address that.

The results from qualitative questions

Julie looks at the different responses to the open questions. A lot of the women who report big improvements in optimism are also part of a women’s peer-to-peer support group at the centre.

She decides to ask the men who took part in the survey if they would be happy to discuss the results, and how the centre could help them feel more optimistic. For example, could they start a similar support group for men?


Comparing results for the over-60s

The exercise classes are targeted at a specific age group. So Julie wants to know how people from the classes scored compared to other older people locally and nationally, in relation to the ONS life satisfaction question.

First, she compares the wellbeing scores of other older people living in the local authority area using the benchmark data. The exercise class scores are lower than the average for the local authority area. This is surprising because, from the conversations she had with people in the class, she expected their wellbeing scores to be higher than average.

So Julie then compares the local authority average with the national average, and it’s much higher. The local authority area is comparatively wealthier than the UK overall, so this isn’t so surprising. But Padley Heath isn’t as wealthy as the rest of the local authority area – it’s similar to the UK overall. So Julie decides it’s probably more appropriate to compare her exercise class scores with the national average. And they are higher, so she won’t be able to use this data to help with any funding bids.

To benchmark and report on her evaluation, Julie decides to present her data with both the local authority average and the national average, and explain the differences.                                                                     


Showing the impact of the computer class

As new people join the class, Julie asks them to answer a few quick questions online including demographics, the ONS wellbeing questions and questions on social connection. She makes sure they know how the data will be used. All 50 of the new attendees are happy to fill in the questionnaire.

She compares her results to the benchmark in the UK. She can see from the baseline data that those joining the computer course have lower than average wellbeing and lower than average scores on social connection.

Julie asks people the same questions again after six months into the programme, plus a new open question about what positive changes people have experienced. She reminds the group their responses are private and she won’t use their names. A few of the adults have dropped out of the programme now – but only four, so she’s happy she can use the results.

The results show that there was an increase in life satisfaction and an increase in purpose. She checked using Excel, which shows her this is a significant change.

She wants to be clear why the changes in wellbeing have taken place. This was her thinking behind including the questions and social connection. She wants to understand the intermediate changes that have been taking place in their lives and in turn have led to higher wellbeing.

Julie finds that the adults:

  • who show an increase in wellbeing also show an increase in social connections
  • who started with the lowest scores in social connection show the biggest increase in wellbeing after six months.

From this, Julia realises that:

  • bringing people together through the service could be an important factor in their improved wellbeing
  • she might be able to increase the service’s impact by making sure the centre is targeting people with lower social connection, who may need the service the most.

Julia checks this idea with the answers to her open questions. In their answers, many people – especially people who felt isolated before – mention how positive their new friendships are in their lives, and/or having the space to ‘check in’ with other people. So Julie knows she can start to look at new ways of targeting more isolated people to get them involved in the class.

Improving wellbeing by improving the computer class

Julia also notices an increase in anxiety for the female members of the group. It isn’t a big increase, but she decides to run a short focus group to find out which aspects of the programme could be linked to it.

Some women in the focus group say they worry about keeping up with the pace of the class or finishing the assignment. Julie records this so the centre can make sure women in the next programme can go at their own pace and get enough support.

Other people in the group also mention on their survey that they like the sessions in the room with the window more than the ones in the basement. So Julie notes down that one more thing she can do to improve things is to rethink her booking of the basement room.

What Julie has learnt

Now that Julie has finished her survey she knows how successful the services – especially the computer class – have been when it comes to improving people’s wellbeing. She also knows what changes she can make to try and improve wellbeing even more.

She is going to:

  • look into how the centre can help men feel more optimistic
  • make sure the centre targets more isolated people whose wellbeing would really benefit from the computer classes
  • look at the day-to-day aspects of the services – like what rooms they’re in and the support people get form the centre.