A blog post from Marianna Hay, Co-Director at Take Note and Founding Director of Orchestras for All
What is the difference we’re trying to make with this project? How do we know our project is having a positive impact on people’s lives? What evaluation techniques can we use to measure this change?
Questions like this are likely to be familiar to anyone responsible for social impact projects. Across the sector, it is widely accepted that it is only through understanding the impact of our work that we can develop, adapt and continually improve it in order to give participants the best possible experience.
That said, evaluating – or measuring – the impact of this kind of work is often easier said than done; and when it comes to creative activity it is notoriously challenging.
Creative work is not the kind of activity that is easily measurable in numbers (which is not to say that numbers don’t matter).
Soft outcomes, evaluation, and setting up Orchestras for All
It is often the case that outcomes for people taking part in creative projects are ‘soft’ and can manifest over a longer period of time than the duration of the activity itself. Typical outcomes might include changes to participants’ confidence, self-esteem, sense of belonging, isolation, motivation and aspiration, to name but a few.
I have to confess that in the early days of setting up Orchestras for All (OFA), the whole concept of evaluation sent me into something of a panic. I founded OFA 12 years ago, becoming first Artistic Director and Chief Executive and latterly its Founding Director. OFA is a national youth music charity that aims to break down barriers to give all young people the life-changing experience of making music together.
At OFA, we know that music, and in particular making music together with others, can have a transformative, life-changing impact on young people, and yet all too often the experience of orchestral music-making is only available to a privileged few.
We set out to change that, and our programmes – National Orchestra for All (NOFA), our schools-based Modulo Programme and our Music Leadership Training Programme (MLT) – are all designed to be fully inclusive and accessible to any young person, regardless of the barriers they face.
As the charity found its feet, it became clear that if we wanted to achieve our short- and long-term aims, we would have to engage in evaluating how we were moving towards them. It also became clear that funders required a proper understanding of our impact if they were to continue supporting us, so we would have to engage with evaluation whether we liked it or not.
I should also confess that in those early days, measuring OFA’s impact was hard to do in a way that didn’t feel tacked on as an afterthought to the creative activity itself.
At times, our overstretched team could feel quite negative about it: tick box forms and interview techniques, when not used sensitively, can undermine the participant’s role in the project.
Moreover, some methods of evaluation do not naturally engender a deep level of imaginative thought and responses can be too easily swayed (unintentionally) by those asking the questions.
Over the years, however, and with some outstanding professional help, we came to think about the role of evaluation quite differently.
Reframing evaluation: How do people feel?
Clearly, evaluation should never simply be a tick box exercise for funders, but our change in attitude has gone much further than that.
Perhaps counterintuitively, it started with much greater emphasis on a key ‘soft’ question: “How do we want people to feel when they take part in our activities?” and with much greater focus on and close attention to their answers.
This led in turn to an improved evaluative practice which not only told us what we were doing well/less well, but also inspired new ways of approaching our work.
To give one example: encouraging composition has always been part of NOFA’s work but in the early days, it was confined to a small number of young enthusiasts. As an experiment we then introduced an element of whole group composition to the entire orchestra.
The evaluation of this activity showed us that co-creating music as a full ensemble had far greater impact on our young players than any of us had expected, particularly in terms of feelings of agency and ownership of the orchestra. So we moved this part of our activity to centre stage, introducing new elements to collaborative composition while placing feedback and evaluation at the heart of the process.
Music and poetry
Recently, for example, as part of NOFA’s 2021-2022 ‘The Way We See It’ artistic season, our young musicians learnt and performed an arrangement of Benjamin Britten’s Young Person’s Guide to the Orchestra.
In traditional performances of this piece – a work originally written to introduce children to the different sections of an orchestra – a narrator performs alongside the musicians explaining each variation of the piece and the different instrument groups (wind, brass, strings, percussion) as they appear along the way.
For our performance of this work, however, the young people of NOFA collaborated with Nottingham-based poet, Ben Macpherson, to co-create a new piece of spoken word poetry to accompany the music. Through the poem, we wanted to capture – in their own words – what being in an orchestra meant to them and the impact it had.
Their words and ideas were incorporated into the new poem, which was then performed by young people across the orchestra to accompany their performance of Britten’s music. The effect was profound and moving as well as a creative manifestation of NOFA’s impact.
NOFA cellist and Orchestras for All youth board member, Chloe, told attendees: “Playing in an orchestra is playing with pieces of puzzles. These pieces are like a well-lit forest: enormous and sometimes scary. It’s super important to keep counting. One musical family, playing equally important roles, working in unison to create and develop an atmosphere: diverse and talented, very colourful and fast.”
NOFA violinist, Alasdair, added: “Synchronised swimming in sound form. A triangle of unity, mistakes and waves. All of it feels good. The power of an orchestra when everyone works as a team. Work harder, challenge myself more, sitting next to a great musician, forgetting to count to four.”
Trust and confidence
Encouraging 11-18 year olds to open up in this way, to share their feelings about their experiences of NOFA, and to give them the confidence to stand up and perform in front of an audience clearly required a high level of trust from our orchestra members in us, and in the process overall.
To gain this trust, we have to think deeply about the culture we are creating at any NOFA event and understand how the young people feel when they are taking part.
Do they feel valued as individuals? Do they feel listened to? Do they feel safe? Identifying and evaluating what these ‘experiences’ or ‘change mechanisms’ are for them becomes a fundamentally important part of a feedback loop and the creative process itself.
There is no doubt that evaluating impact in this way requires commitment, trust and resource, particularly the often-scarce resource of time. However, it is worth also noting that funders – especially those funding arts for social impact projects – are increasingly as interested in how you are measuring your impact and the techniques you are using, and will support the resource required.
In OFA’s case though, the real reward of putting evaluation at the heart of its activity is that it not only allows a deeper understanding of our impact, but has made us listen more closely to our participants with responses like this one from NOFA keyboardist, Troy:
Music is me, it makes me who I am.
The people I see, it’s all part of the plan.
The positivity is huge, and the community is a wonder.
The orchestra is an experience like no other.
That’s why I love music a lot. It really gave me a shot.”
For more information about Orchestras for All and our approach to research and evaluation, take a look at the Impact page on the OFA website.
