SECTION: PLANNING

20309128 How to Demonstrate your Organisation's Outcomes

Published: 03.09.2012 |
Last Updated: 18.12.2015
carmichael
carmichael
Sandra Velthuis

I help not-for-profits to build a better world by channelling their passion, measuring their impact and putting it all into words. See http://www.whitebarn.info for my biography, list of services, clients, testimonials and blog.

Recent research (Velthuis, 2011) has shown that social impact measurement in the Republic of Ireland is significantly underdeveloped.  This is not to say that it is not happening at all, because there are some excellent examples.  Nor is it to suggest that there is no appetite for it, because there clearly is, as shown by numerous conferences, courses and publications on the topic.  It is simply that times are very challenging, most people are extremely busy and many of them do not know how to start measuring and demonstrating their outcomes, even if they might want to.

However, it is time to stop … and take stock.  Not only because funders will increasingly demand that community and voluntary organisations get better at demonstrating their outcomes and impacts.  And not only because infrastructure bodies, consultants, overseas organisations and the media are saying that assessing and reporting outcomes and impacts is essential.  Even if you ignore all of these external pressures, your main internal goal remains the same, which is doing the best that you can to improve the lives of your beneficiaries (or meeting some sort of other defined social/environmental need).  But how do you know if you are doing the best you can if you do not have the evidence for this?

Take a step back to the time when your organisation was founded.  You may or may not have been there yourself.  At some stage, an individual or a group of people became so vexed or excited by a particular issue that they decided that setting up an organisation to deal with the issue was the best way forward.  They wanted something to change.  What was that change? Is that the change that is still needed today?  Has the issue altered or does it remain the same?  It is vital that everyone in your organisation is agreed on this and that you can all clearly and succinctly articulate what success ‘looks like’.  This does not mean saying what activities your organisation undertakes or what services it provides.  Your activities are merely the vehicle for you to reach your vision and these may change over time, as you find out what is, and what is not, working in terms of achieving your ultimate goal.

Vision, outcomes, impacts, activities, goals … this whole area of work is, unfortunately, littered with jargon and there is no absolute agreement about what all the terms mean and how they are interrelated.  The following simplified example of an organisation that runs training courses for unemployed people might shed some light on it all:

  • The INPUTS are the funding, premises, equipment, volunteers, etc, that the organisation uses to do its work.
  • These enable the organisation to undertake ACTIVITIES, such as planning, publicising, coordinating and delivering the training courses.
  • The OUTPUTS generated by these activities are the number of people who attended the courses, the number of certificates they achieved, and so on.
  • These will hopefully lead to the desired OUTCOME, which is people who are more ready for work than they were before undertaking the training course.
  • In time, and almost certainly backed up by a range of additional interventions, the IMPACT will be reduced unemployment.

There should be a logical link between what you do and what change you wish to see happen and you need to be very clear about the assumptions that underpin your reasoning.  It is rarely as simple and as linear as the above example suggests.  Simply running some training courses is not necessarily going to get people into jobs, particularly if they are long-term unemployed and are experiencing other forms of social exclusion.  There are likely to be a number of stepping-stones, or intermediate outcomes, along the way, such as boosting their self-confidence or attending further training.  Also, whether or not unemployed people gain employment is not entirely the result of upskilling them, as there are other factors at play too, such as the actual availability of employment opportunities and the chances of availing of affordable childcare.  It is only when the training programme is viewed in the wider context of other programmes run by the organisation (and/or run by other actors also working in the field) that there is a chance that it will contribute to the overall impact of reduced unemployment.  Using a ‘theory of change’ model can be a powerful way to plan, manage and track outcomes (see later).

None of this should be viewed as extra work.  Most organisations are already aware of the need to plan their work and the need to monitor and evaluate it, and indeed, most do so (some more effectively than others).  I think it is vital that the processes of planning, implementing and evaluating are aligned fully and that when you undertake them, outcomes are at the forefront of your mind.  In this way:

Again, this is an oversimplification (for clearly evaluation should inform planning in an ongoing loop), but it stresses that these processes are all interrelated and gives some indication as to how this is so.

Your organisation already makes decisions each and every day about how time of staff and volunteers and other resources are spent.  By becoming much more systematic about whether your tasks, services and products are actually making a positive difference, you will be able to make better decisions and continue to improve, eventually reaching your end goal.  This involves a range of prerequisites, including: being able to embrace change; being prepared to think about outcomes and impact in all that you do; understanding that positive outcomes rely on you being an organisation that values quality; and crucially, involving your stakeholders.  A stakeholder is anyone who has a legitimate interest in what you do and this includes beneficiaries themselves (members, clients, their families and friends), those who fund the organisation, those who work for the organisation both in a paid and unpaid capacity, sister organisations, policy-makers, etc.

Demonstrating your outcomes is not just about measurement and you should not spend an undue proportion of your time on measuring: delivering impact is most important!  However, some measurement is of course necessary, for how else do you know if you are creating positive change?  Do not just measure inputs (for example, number of staff hours devoted to a short-term project) and outputs (for example, number of calls made to a helpline), but also try and find ways of assessing outcomes.  Some ‘hard’ outcomes may be relatively easy to measure (for example, an increase in literacy rates), but invariably there will be ‘soft’ outcomes that are more challenging to gauge (for example, changes in an individual’s thought and behaviour patterns).  The latter are often at the very heart of what community and voluntary organisations are aiming to achieve, so just because it is difficult, this does not mean that you should not try.  It is important that you develop appropriate indicators and proxies for this purpose; that is, signals or hints that can fairly be assumed to show whether change has indeed taken place.  For example, a health indicator might be the number of doctor or hospital visits or an environmental quality indicator might be the number of trees that were planted that have successfully established.  Try to use at least two, and ideally three, measurements or indicators per variable.  If at all possible try and measure change pre- and post-intervention and between groups who received the intervention and those who did not.  Collect data rigorously and analyse it properly.

There are many different pre-existing approaches, frameworks, methods, tools and systems for measuring change.  These can be very helpful.  If you decide to use any of these, ensure they are the right fit for your organisation, which, despite sharing commonalities with many organisations, is unique.  Do not be lured into a trap of believing that there is a technological solution to measuring impact.  Certainly, there are good systems that make it easier to track and analyse data, especially if your data sets are large.  However, by themselves, these will not help you to understand the change process and make the required judgments.

Which tools you choose will depend on what resources you have and what you need most:

• Is it quality?  If so, consider a system such as PQASSO (Practical Quality Assurance System for Small Organisations developed by Charities Evaluation Services in the UK and promoted by Carmichael Centre for Voluntary Groups), but remember that outcomes are only one of many quality areas covered.
• Is it strategic management capability?  If so, consider a system such as SPEAK (Strategic Planning Evaluation And Knowledge developed in Ireland by Nexus Research) which is used by Family Resource Centres, Sports Partnerships and others.
• Is it outcomes for individuals?  If so, consider using a system such as the Outcomes Star (developed by Triangle Consulting in the UK), which is now available for fourteen different social subsectors.
• Is it social value?  In other words, here you are interested not only in what changed, but how valuable this change was in the eyes of different stakeholders.  If so, consider using the Social Return On Investment approach, which is one in which I have a particular interest.  There are other social value approaches in existence as well.

There are also tools for measuring change in community regeneration, international development programmes, local economies, voluntary sector infrastructure, volunteering, and much more.  It is likely that you will want to use a range of tools.  Start small and build on that.  Find out what works and what does not work and make changes on the basis of evidence.

Then you have a duty to demonstrate your impact, both internally and externally.  Communicate to your board or committee at each of its meetings about how you are progressing, possibly using some sort of ‘dashboard’ approach.  However, also communicate your impact to all of your staff (if you have any), your volunteers and your clients.  Doing so will boost morale, aid learning and enable continuous improvements to take place.  Doing so externally is in any case probably part of your funding requirements, but it will also increase your credibility, allow you to attract further support and help you to share your lessons learnt with others. Communicate truthfully and be transparent both about things that went well and areas where further work is needed.  Use numbers, stories and images/videos of change to illustrate the difference made.  Tailor your format and your message to the intended audience.

In addition to the further reading suggestions highlighted above and the references listed below, there are many different resources available on the internet.  Use them: there is no point wasting your limited resources by reinventing the wheel.  Ones that I particularly recommend are:

• National Council for Voluntary Organisations https://www.ncvo.org.uk/practical-support/monitoring-and-evaluation

• New Economics Foundation http://www.proveandimprove.org

• Foundation Center http://trasi.foundationcenter.org

• NPC http://www.thinknpc.org/

• Center for Theory of Change: http://www.theoryofchange.org

For most organisations, there is no reason why they cannot do all of this alone.  However, if you have little capacity or capability in-house, or if you are contemplating using approaches that are particularly large-scale or complex, it may be worthwhile involving people with external expertise.  If you are going to do this, I would urge you to do so at the start of any process, rather than at the end.  Even limited third party input at the beginning of your strategic planning process or when you are developing a funding proposal for a new programme, can reap dividends.  This is often money much better spent than bringing in a consultant to undertake an expensive evaluation of a programme once it has ended.

I wish you every success in demonstrating your organisation’s outcomes.

 

My Top Tips
Top Tips
1
Let every decision within your organisation be made in the context of ‘will this deliver better outcomes for our beneficiaries?’
2
Be open to the fact that the change your organisation creates can be both unintended and negative, as well as intended and positive.
3
Do not ignore the evidence: if something is shown not to be having a significant and positive impact, stop doing it.
4
Use the help that is out there: you do not need to do this alone.
Suggested reading
1
It Ain’t What You Give: It’s The Way That You Give It. Giving Evidence. London.
Fiennes, C. (2012)
2
Leap of Reason: Managing to Outcomes in an Era of Scarcity. Venture Philanthropy Partners. Washington.
Morino, M. (ed.) (2011)
3
Knowing and Showing your Outcomes and Impacts. The Wheel. Dublin.
Velthuis, S., (2012)
References
References
1
• Velthuis, S., (2011) Demonstrating Impact: Current Practice Amongst Social Purpose Organisations in the Republic of Ireland. The Wheel. Dublin.

https://www.wheel.ie/sites/default/files/Demonstrating_Impact_Report_Final-1.pdf

SECTION 1: PLANNING

20309128 How to Demonstrate your Organisation's Outcomes

Published: 03.09.2012 |
Last Updated: 18.12.2015
carmichael
carmichael
Sandra Velthuis

I help not-for-profits to build a better world by channelling their passion, measuring their impact and putting it all into words. See http://www.whitebarn.info for my biography, list of services, clients, testimonials and blog.

My Top Tips
My Top Tips
My Top Tips
1
Let every decision within your organisation be made in the context of ‘will this deliver better outcomes for our beneficiaries?’
2
Be open to the fact that the change your organisation creates can be both unintended and negative, as well as intended and positive.
3
Do not ignore the evidence: if something is shown not to be having a significant and positive impact, stop doing it.
4
Use the help that is out there: you do not need to do this alone.
Suggested reading
Suggested Reading
Suggested Reading
1
It Ain’t What You Give: It’s The Way That You Give It. Giving Evidence. London.
Fiennes, C. (2012)
2
Leap of Reason: Managing to Outcomes in an Era of Scarcity. Venture Philanthropy Partners. Washington.
Morino, M. (ed.) (2011)
3
Knowing and Showing your Outcomes and Impacts. The Wheel. Dublin.
Velthuis, S., (2012)

Recent research (Velthuis, 2011) has shown that social impact measurement in the Republic of Ireland is significantly underdeveloped.  This is not to say that it is not happening at all, because there are some excellent examples.  Nor is it to suggest that there is no appetite for it, because there clearly is, as shown by numerous conferences, courses and publications on the topic.  It is simply that times are very challenging, most people are extremely busy and many of them do not know how to start measuring and demonstrating their outcomes, even if they might want to.

However, it is time to stop … and take stock.  Not only because funders will increasingly demand that community and voluntary organisations get better at demonstrating their outcomes and impacts.  And not only because infrastructure bodies, consultants, overseas organisations and the media are saying that assessing and reporting outcomes and impacts is essential.  Even if you ignore all of these external pressures, your main internal goal remains the same, which is doing the best that you can to improve the lives of your beneficiaries (or meeting some sort of other defined social/environmental need).  But how do you know if you are doing the best you can if you do not have the evidence for this?

Take a step back to the time when your organisation was founded.  You may or may not have been there yourself.  At some stage, an individual or a group of people became so vexed or excited by a particular issue that they decided that setting up an organisation to deal with the issue was the best way forward.  They wanted something to change.  What was that change? Is that the change that is still needed today?  Has the issue altered or does it remain the same?  It is vital that everyone in your organisation is agreed on this and that you can all clearly and succinctly articulate what success ‘looks like’.  This does not mean saying what activities your organisation undertakes or what services it provides.  Your activities are merely the vehicle for you to reach your vision and these may change over time, as you find out what is, and what is not, working in terms of achieving your ultimate goal.

Vision, outcomes, impacts, activities, goals … this whole area of work is, unfortunately, littered with jargon and there is no absolute agreement about what all the terms mean and how they are interrelated.  The following simplified example of an organisation that runs training courses for unemployed people might shed some light on it all:

  • The INPUTS are the funding, premises, equipment, volunteers, etc, that the organisation uses to do its work.
  • These enable the organisation to undertake ACTIVITIES, such as planning, publicising, coordinating and delivering the training courses.
  • The OUTPUTS generated by these activities are the number of people who attended the courses, the number of certificates they achieved, and so on.
  • These will hopefully lead to the desired OUTCOME, which is people who are more ready for work than they were before undertaking the training course.
  • In time, and almost certainly backed up by a range of additional interventions, the IMPACT will be reduced unemployment.

There should be a logical link between what you do and what change you wish to see happen and you need to be very clear about the assumptions that underpin your reasoning.  It is rarely as simple and as linear as the above example suggests.  Simply running some training courses is not necessarily going to get people into jobs, particularly if they are long-term unemployed and are experiencing other forms of social exclusion.  There are likely to be a number of stepping-stones, or intermediate outcomes, along the way, such as boosting their self-confidence or attending further training.  Also, whether or not unemployed people gain employment is not entirely the result of upskilling them, as there are other factors at play too, such as the actual availability of employment opportunities and the chances of availing of affordable childcare.  It is only when the training programme is viewed in the wider context of other programmes run by the organisation (and/or run by other actors also working in the field) that there is a chance that it will contribute to the overall impact of reduced unemployment.  Using a ‘theory of change’ model can be a powerful way to plan, manage and track outcomes (see later).

None of this should be viewed as extra work.  Most organisations are already aware of the need to plan their work and the need to monitor and evaluate it, and indeed, most do so (some more effectively than others).  I think it is vital that the processes of planning, implementing and evaluating are aligned fully and that when you undertake them, outcomes are at the forefront of your mind.  In this way:

Again, this is an oversimplification (for clearly evaluation should inform planning in an ongoing loop), but it stresses that these processes are all interrelated and gives some indication as to how this is so.

Your organisation already makes decisions each and every day about how time of staff and volunteers and other resources are spent.  By becoming much more systematic about whether your tasks, services and products are actually making a positive difference, you will be able to make better decisions and continue to improve, eventually reaching your end goal.  This involves a range of prerequisites, including: being able to embrace change; being prepared to think about outcomes and impact in all that you do; understanding that positive outcomes rely on you being an organisation that values quality; and crucially, involving your stakeholders.  A stakeholder is anyone who has a legitimate interest in what you do and this includes beneficiaries themselves (members, clients, their families and friends), those who fund the organisation, those who work for the organisation both in a paid and unpaid capacity, sister organisations, policy-makers, etc.

Demonstrating your outcomes is not just about measurement and you should not spend an undue proportion of your time on measuring: delivering impact is most important!  However, some measurement is of course necessary, for how else do you know if you are creating positive change?  Do not just measure inputs (for example, number of staff hours devoted to a short-term project) and outputs (for example, number of calls made to a helpline), but also try and find ways of assessing outcomes.  Some ‘hard’ outcomes may be relatively easy to measure (for example, an increase in literacy rates), but invariably there will be ‘soft’ outcomes that are more challenging to gauge (for example, changes in an individual’s thought and behaviour patterns).  The latter are often at the very heart of what community and voluntary organisations are aiming to achieve, so just because it is difficult, this does not mean that you should not try.  It is important that you develop appropriate indicators and proxies for this purpose; that is, signals or hints that can fairly be assumed to show whether change has indeed taken place.  For example, a health indicator might be the number of doctor or hospital visits or an environmental quality indicator might be the number of trees that were planted that have successfully established.  Try to use at least two, and ideally three, measurements or indicators per variable.  If at all possible try and measure change pre- and post-intervention and between groups who received the intervention and those who did not.  Collect data rigorously and analyse it properly.

There are many different pre-existing approaches, frameworks, methods, tools and systems for measuring change.  These can be very helpful.  If you decide to use any of these, ensure they are the right fit for your organisation, which, despite sharing commonalities with many organisations, is unique.  Do not be lured into a trap of believing that there is a technological solution to measuring impact.  Certainly, there are good systems that make it easier to track and analyse data, especially if your data sets are large.  However, by themselves, these will not help you to understand the change process and make the required judgments.

Which tools you choose will depend on what resources you have and what you need most:

• Is it quality?  If so, consider a system such as PQASSO (Practical Quality Assurance System for Small Organisations developed by Charities Evaluation Services in the UK and promoted by Carmichael Centre for Voluntary Groups), but remember that outcomes are only one of many quality areas covered.
• Is it strategic management capability?  If so, consider a system such as SPEAK (Strategic Planning Evaluation And Knowledge developed in Ireland by Nexus Research) which is used by Family Resource Centres, Sports Partnerships and others.
• Is it outcomes for individuals?  If so, consider using a system such as the Outcomes Star (developed by Triangle Consulting in the UK), which is now available for fourteen different social subsectors.
• Is it social value?  In other words, here you are interested not only in what changed, but how valuable this change was in the eyes of different stakeholders.  If so, consider using the Social Return On Investment approach, which is one in which I have a particular interest.  There are other social value approaches in existence as well.

There are also tools for measuring change in community regeneration, international development programmes, local economies, voluntary sector infrastructure, volunteering, and much more.  It is likely that you will want to use a range of tools.  Start small and build on that.  Find out what works and what does not work and make changes on the basis of evidence.

Then you have a duty to demonstrate your impact, both internally and externally.  Communicate to your board or committee at each of its meetings about how you are progressing, possibly using some sort of ‘dashboard’ approach.  However, also communicate your impact to all of your staff (if you have any), your volunteers and your clients.  Doing so will boost morale, aid learning and enable continuous improvements to take place.  Doing so externally is in any case probably part of your funding requirements, but it will also increase your credibility, allow you to attract further support and help you to share your lessons learnt with others. Communicate truthfully and be transparent both about things that went well and areas where further work is needed.  Use numbers, stories and images/videos of change to illustrate the difference made.  Tailor your format and your message to the intended audience.

In addition to the further reading suggestions highlighted above and the references listed below, there are many different resources available on the internet.  Use them: there is no point wasting your limited resources by reinventing the wheel.  Ones that I particularly recommend are:

• National Council for Voluntary Organisations https://www.ncvo.org.uk/practical-support/monitoring-and-evaluation

• New Economics Foundation http://www.proveandimprove.org

• Foundation Center http://trasi.foundationcenter.org

• NPC http://www.thinknpc.org/

• Center for Theory of Change: http://www.theoryofchange.org

For most organisations, there is no reason why they cannot do all of this alone.  However, if you have little capacity or capability in-house, or if you are contemplating using approaches that are particularly large-scale or complex, it may be worthwhile involving people with external expertise.  If you are going to do this, I would urge you to do so at the start of any process, rather than at the end.  Even limited third party input at the beginning of your strategic planning process or when you are developing a funding proposal for a new programme, can reap dividends.  This is often money much better spent than bringing in a consultant to undertake an expensive evaluation of a programme once it has ended.

I wish you every success in demonstrating your organisation’s outcomes.

 

References
References
References
1
• Velthuis, S., (2011) Demonstrating Impact: Current Practice Amongst Social Purpose Organisations in the Republic of Ireland. The Wheel. Dublin.

https://www.wheel.ie/sites/default/files/Demonstrating_Impact_Report_Final-1.pdf