Cumbria CVS: Evaluating a participant-led employment programme

We helped Cumbria Community and Voluntary Service (CVS) to understand and evidence the impact of a multi-million-pound participant-led employability programme. 

Using mixed methods research, validating our decision to form a partnership, and a collaborative approach to delivery with co-design and co-analysis at its heart, we told the story of a complex programme through comprehensive analysis, hard-hitting observations and a range of reports and presentations for various audiences. 

The impact of social research takes time and can be hard to demonstrate. But our research is already being used to shape the way similar projects are delivered in the area and strengthen funding bids. It has led to Cumbria CVS partnering with the NHS to address barriers to employment locally. All of which should positively impact the local community long-term. 

Our support

●      Summative evaluation

●      Data analysis and survey design

●      Qualitative research and analysis

●      Evaluation and learning report

  
We decided to ‘go it alone together’, setting up PS Research as one research collaboration between two independents, bringing together the best of what each of us do: quant and qual. This combination lent itself perfectly to a project like the evaluation of Cumbria’s ‘My Future’ employment programme. The nature of such a large European-funded project meant they were sat on a lot of data, but it was also a complex, challenging topic where understanding lived experience and outcomes through qualitative methods was key. Blending the two resulted in a rounded picture of the programme and some clear recommendations and next steps.   

We delivered mixed methods research to evaluate the impact of an innovative, multi-million-pound participant-led employment programme in Barrow-in-Furness, one of the most deprived areas in the country. At a time when our nation is going through a cost-of-living crisis with rising poverty levels, this kind of research has never been more important. As researchers, whilst we don’t directly work with people in poverty, we believe we can play an integral role in supporting and shaping the organisations that do.  

What was our role?

Our role was to understand and evidence what a successful outcome is for the project. It wasn’t as simple as measuring how many participants moved into employment. It was about understanding the context and the lives of the people entering the programme and evidencing the many barriers they faced to getting a job. Our analysis showed that people entering the programme had been ‘economically inactive’ for an average of 18 months and one in four did not have any secondary school qualifications. Two in five participants lived in one of the most deprived areas in the country and seven in ten lived in a jobless household. Using qualitative methods with project staff and then beneficiaries we were able to show the human stories behind these statistics and showcase the importance of identifying alternative measures of success, such as how the support from the programme increased self-confidence, with one participant feeling confident enough to start making eye contact with others again. 

Between October 2022 and March 2023, we worked with Cumbria CVS and three local delivery partners supporting these long-term unemployed residents. We ensured our research had impact by going beyond what was asked of us and doing what was right for the project, using our expertise to shape the evaluation. Our research approach covered: 

A facilitated ‘discovery’ session 

This was an early opportunity to collaborate, listen and learn. We facilitated a session with the entire partnership team, enabling partners to reflect on their experience of the project from the start. We used a range of projective techniques to understand the journey of the programme (see image 3), consider and discuss its successes, areas of challenge past and present and to share learning and frustrations. This set the tone and became the foundation for the subsequent research approach and methods. 

Data analysis 

Whilst not included in our original proposal, it was clear early on that the project and team would benefit from a better understanding of their local area through the lens of data. We undertook population health analysis using sources including the Index of Multiple Deprivation, Census 2021, Nomis labour market statistics and Public Health local area profiles. 

As a European Social Fund (ESF) project, the team collected an awful lot of data about their participants and programme activities. Previously used only for grant monitoring and mandatory ESF returns, we used the data to build a clear picture of their programme and the participants they supported. Bringing it into SPSS analysis software, running postcode lookups against other datasets such as deprivation levels and urban-rural classifications and cross-tabulating the data to compare different groups of participants, e.g. by length of time in the programme. 

Stakeholder survey 

An online questionnaire was developed to give a wide set of stakeholders the opportunity to feed-back on the programme. The focus was on capturing outcomes beyond employment status, such as confidence, optimism for the future and engagement in their community. It also acted as a useful recruitment tool for qualitative research with beneficiaries. Engagement and response rates were challenging, a reflection of both the limited budget and general disengagement from some participants who exited the programme. 

Qualitative research 

The qualitative research was key to providing insight into the impact of the project on delivery partners and participants. 

We used a range of qualitative methods including online workshops with the project team and delivery partner leads, depth interviews with key workers and participants and observations at programme activities. Including periods of observation at activities proved invaluable, allowing us to build a rapport with often vulnerable participants prior to conducting individual interviews.  

Sharing the lived experiences of participants with wider stakeholders not only brought the project to life but showcased how this project differed from other employability programmes (which tend to be short-term and focus solely on ‘getting people into a job’) and the importance of investing in long-term innovative projects to support people furthest away from employment, for example evidencing that moving someone closer to employment often starts with improving their self-confidence.

Impact and learning 

This research was about learning, sharing and making improvements for the long-term. As the evaluation developed, it became clear that there were also opportunities for us to leave our mark on the third sector and research practice in the area.  

Our work is already strengthening organisational and partnership approaches in the area and project delivery and development: 

Through our presentations at Learning Events and accessible reports for a range of audiences, we were able to share the learning more widely with community leaders and decision-makers to make a positive difference to the local VCSE sector. For example, the insights and data highlights on the local area are being used by other local organisations, including the Poverty Truth Commission, to support their own work. 

Our collaborative and open approach when working with Cumbria CVS has helped them to understand how to improve data collection and research practices internally:

The quality of the research has also enabled Cumbria CVS to lobby funders and has been shared widely with VCSE and academic partners. The learning and data contributes to knowledge about unemployment in the locality, the needs of people furthest from the labour market and offers up learning to improve practice in relation to project participants. It helps local organisations to better understand the issues and barriers faced and some of the potential solutions. The NHS are currently using the research to better understand how to attract and retain under-represented people into jobs in south Cumbria and the sharing of the research has enabled Cumbria CVS to foster a stronger relationship with the NHS. They have subsequently been invited to work collaboratively to address barriers to employment. 

Social value 

We wanted to give something back as we try to deliver meaningful social value for the projects we work on. This wasn’t a planned outcome at the point we were commissioned; it was something that emerged and developed as we embedded ourselves in the project. At the learning events, we identified a gap in research and evaluation knowledge across the sector. Community workers approached us and asked for our advice on collecting useful data, what research methods to adopt during their projects and more. So we worked with Cumbria CVS to deliver some pro-bono sessions for these small organisations to drop-in and speak to us about their research and evaluation challenges. Following these sessions, we developed some broad advice and guidance for organisations that couldn’t attend. We are now building this approach into other projects with third sector networks and infrastructure organisations.