Are we there yet? Evaluating NFP outputs, outtakes, outcomes & impact

 

Evaluation – Part 2

Evaluation is one of the central elements in the EDM (Evaluate, Direct, Monitor) Governance Model, but its role in governance (and management) is often obscured by the use of other terms, like ‘problem-solving’ or ‘decision-making’. The importance of evaluation in non-profit governance is highlighted in the extracts from AICD’s NFP Governance Principles illustrated below.

Part 1 (https://bit.ly/41GhRCr) in this 2-part series on Evaluation mainly focussed on directors using evaluation measures to address their performance and conformance roles. The diverse nature of evaluative activities carried out across the organisation would acknowledge the wide scope of work implied by the following definition of ‘evaluation’. That broader scope is the theme explored in Part 2 of the series.

As well as seeking recognition of this wider scope of evaluation activities, the shift of emphasis in recent years towards the evaluation of outcomes and impacts also requires that we understand these terms, so that shared understandings inform board and management deliberations.

Integrated evaluation framework

There are many types of evaluation activity and numerous methods to choose from. Evaluation is used in virtually all aspects of organisational life, yet there are few organisations with (monitoring and ) evaluation frameworks or policies to guide directors and staff in this key aspect of their work.

There are numerous ways one could approach the development of an integrated framework, and each has its merits and drawbacks. Looking through multiple lenses may help to overcome some of these drawbacks (think blind people each touching a different part of an elephant). Here are just some of the lenses that could be employed in thinking about evaluation in a non-profit entity.

Working with directors and managers in many organisations, the existence of ‘evaluation silos’ has been evident. It is often the case that people involved in internal audits see no connection between their work and that of their colleagues involved in program evaluation, risk management, performance management, tendering, or project management.

Much of the evaluation literature focuses on development projects or education, and there is relatively little which is overtly identified as relevant to evaluation across the non-profit organisation. Some boards have adopted a monitoring and evaluation framework to bring structure and consistency to evaluations conducted by or for the board, however, these tend to focus on indicators attached to their strategy, along with selected dashboard elements (like solvency and membership trends).

Thinking of evaluation only in terms of directors using data from monitoring activities to determine whether and how well strategic and operational outcomes were achieved, and to guide future strategy, is a limited view of the role played by a spectrum of evaluation activities, some of which are described with different names.

Boards wishing to ensure that a coherent approach to evaluation is taken throughout their organisation may wish to consider the development of an integrated evaluation framework, which will help to ensure that the results of evaluative activities are presented to directors in a more coherent form. For such a framework to apply across the spectrum of evaluation activities undertaken, it would doubtless need to focus on a set of evaluation principles rather than any single approach. Here are two sample sets of such principles which may offer starting points for your organisation’s Monitoring and Evaluation Framework.

Critical thinking

In Bloom’s (original) Taxonomy of Educational Objectives, evaluation was at the top of the hierarchy of thinking skills – the pinnacle of critical thinking. Perhaps partly in recognition that evaluation did not necessarily solve problems or result in innovation, Bloom’s Revised Taxonomy (2001) added ‘creating’ to the top of the hierarchy.

The EDM (Evaluate, Direct, Monitor) Governance model already recognised that evaluation was not the last or highest step in governance thinking. Once the ‘What?’ and ‘So What?’ questions have been answered via monitoring and evaluation, the ‘Now What?’ question remains to be answered by the board setting directions for future action (creating). (See header image above.)

Evaluation for quality

A parallel can be seen in the operational uses of evaluation, where conclusions drawn about the value, standard, or status of a matter within a given ‘silo’ are only some aspects of quality assurance and precursors to quality improvement.

Given its significant role in shaping the insights which inform future plans and activities, the recent shift in evaluation practice to an outcomes focus is a welcome development.

Thinking of evaluation only in terms of directors using data from monitoring activities to determine whether and how well strategic and operational outcomes were achieved, and to guide future strategy, is a limited view of the role played by a spectrum of evaluation activities, some of which are described with different names.

Evaluation logic

Attempting to devise an organisational evaluation framework or model that accommodates this wider collection of evaluative activities could run the risk of oversimplifying various parts of the evaluative ecosystem. Failure to seek a coherent framework, however, could miss the opportunity to see relationships and patterns offering significant insights into organisational development opportunities and enhanced quality management. The logic framework suggested below packs a lot of detail into a single chart, but hopefully offers insights that will be helpful to your board and senior managers as they seek to improve the efficiency and effectiveness of your non-profit or for-purpose entity.

When we recognise our organisation as a system comprising interdependent sub-systems and relationships, we break down silos and challenge narrow views about how people, systems, processes, and technology interact to achieve our purpose/s or execute strategy.

Measuring success, progress, and impact

The evaluation methods, metrics, and milestones identified in your evaluation plan will benefit from looking beyond mere outputs, to identify lessons learned along the way (outtakes), outcomes achieved for stakeholders, and the longer-term impact of your strategy, service model, and campaign activities (where applicable). Identifying your evaluation criteria after the initiative or program has concluded runs the risk of hindsight bias clouding the picture. Using a logic framework like the one suggested above should help to avoid that risk.

https://www.aes.asn.au/talking-about-evaluation
https://www.councilofnonprofits.org/tools-resources/evaluation-and-measurement-of-outcomes

https://balancedscorecard.org/bsc-basics-overview/

https://www.criticalthinking.org/pages/index-of-articles/1021/

https://tinyurl.com/yresy54e

Risk Assessment Matrix

 

Summary

A risk matrix analyzes project risks based on likelihood and severity. Once you map your risks, you can calculate overall impact and prioritize risks accordingly. In this piece, you’ll learn how to create a risk matrix template and how to use the information from this analysis tool to develop a comprehensive risk management plan.

Risks are a part of any project, and there’s no surefire way to know which ones will occur and when. Sometimes, you'll get through an entire project without experiencing a single hiccup. Other times, you’ll feel like all the odds are against you. Without the help of a crystal ball, the only way to prevent project risks is to proactively prepare for them. 

A risk matrix helps you analyze risk by assigning each event as high, medium, or low impact on a scale of one through 25. Once you assess the severity and likelihood of each risk, you’ll prioritize your risks and prepare for them accordingly. In this article, we’ll explain how to create a risk matrix template and offer helpful tools for turning your results into action.

What is a risk matrix in project management?

A risk matrix is a risk analysis tool to assess risk likelihood and severity during the project planning process. Once you assess the likelihood and severity of each risk, you can chart them along the matrix to calculate risk impact ratings. These ratings will help your team prioritize project risks and effectively manage them. 

Types of risks

As part of the process, you’ll need to brainstorm a list of risks to chart in your risk matrix. The risks you may face will likely fall into these categories:

• Strategic risk: Strategic risks involve performance or decision errors, such as choosing the wrong vendor or software for a project.

• Operational risk: Operational risks are process errors or procedural mistakes, like poor planning or a lack of communication among teams.

• Financial risk: Financial risk can involve various events that cause a loss of company profit, including market changes, lawsuits, or competitors.

• Technical risk: Technical risk may include anything related to company technology, such as a security breach, power outage, loss of internet, or damage to property.

• External risk: External risks are out of your control, like floods, fires, natural disasters, or pandemics. 

There are other risk categories to consider depending on your work industry. For example, if you have government clients, then you also want to brainstorm legal risks. If your company sells a physical product, you may have to think about manufacturing risks.

How to create a risk matrix template

When creating your risk matrix template, you’ll first identify your scale of severity, which you’ll place in the columns of your matrix. ​​The scale of severity measures how severe the consequences will be for each risk. In a five-by-five matrix, there are five levels in your scale of severity. 

• Negligible (1): The risk will have little consequences if it occurs.

• Minor (2): The consequences of the risk will be easy to manage.

• Moderate (3): The consequences of the risk will take time to mitigate.

• Major (4): The consequences of this risk will be significant and may cause long-term damage.

• Catastrophic (5): The consequences of this risk will be detrimental and may be hard to recover from.

You’ll then identify your scale of likelihood, which you’ll place in the rows of your risk matrix template. The scale of likelihood identifies the probability of each risk occurring.  

• Very likely (5): You can be pretty sure this risk will occur at some point in time.

• Probable (4): There’s a good chance this risk will occur.

• Possible (3): This risk could happen, but it might not. This risk has split odds.

• Not likely (2): There’s a good chance this risk won’t occur.

• Very unlikely (1): It’s a long shot that this risk will occur.

When you place a risk in your matrix based on its likelihood and severity, you’ll find the level of risk impact. The risk impact is both color-coded from green to red and rated on a one through 25 scale. 

• Low (1-6): Low-risk events likely won’t happen, and if they do, they won’t cause significant consequences for your project or company. You can label these as low priority in your risk management plan.

• Medium (7-12): Medium-risk events are a nuisance and can cause project hiccups, but if you take action during project planning to prevent and mitigate these risks, you’ll set yourself up for project success. You shouldn’t ignore these risks, but they also don’t need to be a top priority.

• High (13-25): High-risk events can derail your project if you don’t keep them top of mind during project planning. Because these risks are likely to happen and have serious consequences, these are most important in your risk management plan.

 You don’t have to stick to the labels above for your risk matrix template if they don’t feel right for your company or project. You can customize the size and terminology of your matrix to your needs.

How to use a risk matrix

Once you’ve created a risk matrix, you can use it as a comprehensive analysis tool. The best part about a risk matrix template is that you don’t need to change it for every project. Once you have one, you can reuse it and share it with others. 

1. Identify project risks

You’ll need a list of potential risks to make use of your risk matrix. In this step, you’ll determine what risks may affect the specific project you’re working on. 

To come up with relevant risks for your project, you’ll need to understand your project scope and objectives. This includes the project’s:

• Timeline

• Budget

• Resources

• Constraints

Using your project scope as a guide, think of risky situations that might affect your project. If you’re not sure where to start, try brainstorming techniques like mind mapping or starbursting to list as many risks as you can under each risk type. 

2. Determine severity of risks

When you created your risk matrix, you defined the criteria for your risk severity and likelihood. Now that you have a list of project risks, categorize them using the matrix criteria. Start with the scale of severity and go through each risk you’ve listed. Consider the following questions:

• What is the most negative outcome that could come from this risk?

• What are the worst damages that could occur from this risk?

• How hard will it be to recover from this risk?

• Which of the five severity levels most closely matches this risk?

You may not always have the perspective you need to know how severe the consequences of a risk are. In that case, work with other project stakeholders to determine the potential risk impact.

3. Identify likelihood of risks

Once you’ve defined the severity of each risk, you’ve completed half of the risk analysis equation. Next, identify the likelihood of each risk. To do this, consider the following questions:

• Has this risk occurred before and, if so, how often?

• Are there risks similar to this one that have occurred?

• Can this risk occur, and if so, how likely is it to occur?

Team collaboration is also crucial in this step because you may not have a good idea of similar risks that have occurred in past projects. Make sure to reference past projects and analyze the probability of each risk with your team in order to create a more accurate mitigation plan.

4. Calculate risk impact

The last part of your risk analysis equation is to calculate risk impact. The equation you’ll use is:

Likelihood x severity = risk impact 

Place each risk in your matrix based on its likelihood and severity, then multiply the numbers in the row and column where it lands to find the level of risk impact. For example, if you think the risk of a data breach is of major severity (4) and probable likelihood (4), you’d multiply four by four to get a risk impact of 16. This is considered a high-risk impact. 

5. Prioritize risks and take action

You should now have a risk impact level on a scale of 1–25 for each risk you’ve identified. With these number values, it’s easier to determine which risks are of top priority. When you have risks with the same risk impact score, it will be up to you and your team to determine which risk to prioritize. Risks with equal risk impact may require equal attention as you create your action plan. 

Your risk response plan should include steps to prevent risk and ways to mitigate risk if unfortunate events occur. Because so much goes into project planning, the best strategy when tackling risks may be to divide and conquer.

Risk assessment matrix template

The size of your risk matrix template determines how closely you can analyze your project risks. A larger risk matrix template offers more room on the risk impact spectrum, while a smaller risk matrix template keeps your risk impact rating simpler and less subjective. 

Each square in your matrix represents a risk level of likelihood and severity, so you shouldn’t make your risk matrix smaller than three squares in length and width.

A five-by-five risk matrix is ideal so you can further analyze each risk. Once you chart your risks along your finished risk matrix template, this matrix creates a larger color spectrum to see the impact of each risk as high, medium, or low. 

The example below shows a five by five risk matrix template.

Pair your risk matrix template with a work management tool

You can use the same risk matrix template when measuring risk across multiple projects. However, it’s important to remember that the risks you face will evolve. The environment changes, technology becomes smarter, and the workplace grows. Every project faces unique risks, and you must reevaluate these risks year after year.

https://tinyurl.com/2s3vy782

How To Use a Risk Assessment Matrix (With Example)

Risk management tools, such as a risk assessment matrix, can help identify the risks associated with a project and how to address them.In this article, we explain what a risk assessment matrix is, explain the benefits of using one and show you how to use one to evaluate potential risks to your project.

What is a risk assessment matrix?

Many companies use a risk management tool, such as a risk assessment matrix, in the risk evaluation process to determine the right steps in business decisions.A risk assessment matrix can come in the form of a chart, where you plot the severity of possible risk on one axis and the probability of this event occurring on another. You could also format your matrix as a table by listing your potential risks in rows and entering the probability and severity information as columns.By providing a visual representation of complex data, you can use a risk assessment matrix to facilitate and simplify the risk evaluation process and help you make more informed decisions related to your business.

The benefits of using a risk matrix

There are several benefits to creating and using a risk matrix to evaluate projects, including that they help:

• Identify areas to reduce risk quickly and easily
• Explain specific risks in a clear way
• Prioritize and group project event outcomes
• Outline a foundational resource for subsequent detailed analysis

How to use a risk assessment matrix

To use a risk assessment matrix during the risk evaluation process effectively, take the following steps:

1. Identify all potential risks

The first step in the risk assessment process is to identify potential risks. To maintain a structure that is easy to manage, the risk assessment process offers a way to prioritize risks by evaluating potential risks. After you identify all risks, the next step is to order risks from most impactful to least impactful.

2. Sort risks according to probability and impact

Now you are ready to sort risks according to their probability and impact.

Probability

This describes the likelihood of a risk occurring. You can use different approaches to sort risk probability. Some companies, for instance, assign potential risks a probability percentage that ranges from 0%—that is, no possibility of the risk occurring—to 100%, in which case the risk is certain. Or, you can sort risks according to categories, such as:

• Unlikely: Put potential risks in this category if they are highly unlikely to materialize.
• Seldom: This category is for uncommon risks that have a small chance of materializing.
• Occasional: Sort risks in this category that have a roughly 50-50 chance of taking place.
• Likely: If a risk is probably to occur, you should place it in this category.
• Definite: This is for risks that are going to occur. When coupled with high impact, you should regard this kind of risk as a priority, and address it right away.

Impact

This aspect of risk points to how severe the impact will be if a potential risk actually manifests. The impact of a specific risk materializing could influence various aspects of the project, and potentially, the company as a whole. In project management, companies often evaluate risk impact according to the negative effect it may have on three important aspects:

• Schedule: Will it negatively affect time frames for delivery?
• Cost: Will you have to adjust the budget?
• Technical performance: If the risk occurs, how will it affect performance?

As is the case with evaluating the probability of a risk, you could sort the severity of risk impact in the following ways:

• Insignificant: Place risks that will have little to no negative impact on a project in this category.
• Minor: Place risks that may have a slight negative impact on a project but will not likely cause any major disruptions in this category.
• Moderate: This category is for risks that pose a moderate threat to operations.
• Critical: Place risks that pose a significant threat to the successful execution of the project in this category.
• Catastrophic: This category is for risks that will in all likelihood jeopardize the whole project and significantly impact daily operations should they occur. These risks are high-priority.

3. Decide on risk ranking

Next, plot the risks according to their probability and impact on the risk assessment matrix. After you plot the information, you will have a clear visual representation of what priorities the potential risks should have.For instance, risks that are very likely to occur and will have an extremely negative impact on operations will appear as the highest-priority risks on the matrix. On the other hand, those that are both unlikely to occur and pose no significant threats should they occur will fall under low-priority risks.

4. Decide on preventative measures

Draw up contingency plans to deal with worst-case scenarios. This last step in the risk assessment process helps you determine how you should deal with middle- and high-ranked risks.

Example of a risk assessment matrix

Here is an example of risk impact/probability chart that consists of varying degrees of risk probability and risk impact:

The four corners of a risk impact/probability matrix show extremes that typically have the most actionable insight and include:

• Low probability/ low impact: Risks in this corner of the chart are both low probability and low impact. You do not need to pay attention to these risks.
  
• High probability/ low impact: This kind of risk poses a moderate threat to operations. Although you should try to minimize the possibility of such events occurring, you can manage these risks if and when they take place.
  
• Low probability/ high impact: This type of event will have a high impact on operations, but the probability of them materializing is unlikely. In order to avoid such risks occurring, you should take all possible preventative steps. You should also put contingency plans in place to minimize the severity of the impact should the risk manifest.
  
• High probability/ high impact: The risks in this category are the highest-priority risks because they have a high probability of occurring and would also have a severely negative effect on operations. This means that you should give these risks the most attention and should take them into consideration in the daily decision-making process.

Medium-priority risks could seriously impact the profitability and overall successful implementation of a project, the occurrence of high-priority risks may not only potentially signal the end of a project, but could also have a serious impact on the organization as a whole.

https://tinyurl.com/5cafcst9

Evaluation of Risks in Complex Problems

• Gregory S. Parnell

https://tinyurl.com/5n93x83k