The Scales of Governance: Weighing options, arguments, evidence & consequences

 

Evaluation – Part 1

We use the term ‘on balance’ as a shorthand way of saying that we have come to a decision or choice after considering the power, influence, or ‘weight’ of both sides of a question or issue. This invokes metaphoric reference to a set of balance scales – as in the ‘scales of justice’ (see header image).

Evaluation skills, sometimes described as good judgment, are fundamental to good governance. The EDM (Evaluate, Direct, Monitor) Governance Model acknowledges this. This article is the first in a short series looking at some aspects of evaluation in the work of non-profit directors and managers.

Weighing options

Regrettably, most non-profit governance and management decisions involve more than two options or alternatives. Simple choices between good and bad options are rare. If the issues were that simple, then they could probably be resolved by reference to a checklist or filter system, without having to include them on a board agenda. Often enough in governance deliberations, we can also be faced with a choice between ‘least worst’ options rather than ‘best case’ scenarios.

Debates over complex public policy issues inevitably involve more than two perspectives, unless they are seen through the lens of a polarising media story or a ‘school debating club’ approach. These perspectives contrive to restrict debate to black and white positions, with ‘government’ and ‘opposition’ sides taking a stance for or against a given claim or contention.

Even the choice between action and inaction usually involves additional sub-options, such as whether to act one way or another. For inaction, you could choose to leave the matter off the agenda, or include it, but recommend that the situation simply be monitored for significant developments.

We usually employ arguments for and against each of the options to develop a collective view on which of the options is the most robust, and therefore likely to offer the most satisfactory response to the situation. The criteria we employ to make judgments regarding our option preferences, are discussed further below under ‘weighing evidence’.

Weighing arguments

Deliberative processes in non-profit and for-purpose settings are much less about winning an argument than they are about negotiating best possible outcomes for key stakeholders.

Simply counting the number of arguments for or against the proposal would not pay sufficient regard to the relative value of some criteria compared with others. That’s where recognition of the evidence called upon to support each argument comes into play.

The importance of employing evidence-based decision-making has been formally recognised by the International Standards Organisation with the adoption of ISO9000.

Weighing evidence

Identification of arguments for or against a particular proposal or position, and mapping these so they can be fully examined is a good start in weighing the arguments, but when we acknowledge that not all arguments are supported by the same standard of evidence, we recognise that we need to attach some form of importance ranking or weight to the criteria we apply to our decision making. This would tilt the scales in recognition of our values, strategic priorities, and commitment to evidence-informed decision-making.

When we establish a tender process or initiate CEO recruitment and selection, we are happy to identify criteria for use by a tender committee or selection panel. This helps to ensure that an objective decision can be reached on the preferred candidate. Skill in crafting such criteria exists in most boards and senior management teams, however, the establishment of evaluation criteria for other kinds of decisions is not always addressed. Taking the time to agree on the evaluation criteria, and their relative importance, will be rewarded when the time comes to make a decision.

For complex and high-value decisions, I have found argument maps to be a helpful aid to the deliberative process. There are numerous desktop and online mapping systems available, but I have preferred Rationale* and Bcisive* for many years. The capacity to unpack the debate, capture the supportive and opposing arguments, identify the evidence underpinning those arguments, and the sources of that evidence, is particularly helpful to a board seeking to weight or rank the arguments according to the standard of evidence they rest upon.

*No referral fees or commission arrangements apply.

When assessing whether a board sought access to relevant data and analyses to support their decisions, courts will seek to confirm that directors informed themselves to a level expected by a reasonable person before making their decision. Mere access to the relevant data would not be sufficient of course. The extent to which probing questions were asked and answered also enters into consideration.

Weighing consequences

Certainly, when we assess the likelihood and severity of adverse consequences from action (or inaction) on a given issue, we are weighing the consequences. This only considers the question of what could go wrong of course, and a balanced approach to deliberation would also look at the value proposition, and ‘benefit dividend’ for our client, member, or community. The sweet spot which (at minimum) balances benefits, costs, and risks, should be identified in board decision making, with a preference for proposals in which benefits outweigh costs and risks.

Higher-order governance

The evaluative skills involved in weighing options, arguments, evidence, and consequences are examples of higher-order critical thinking skills. This aspect of evaluation will be explored further in Part 2 of this series.

https://tinyurl.com/bp6ewc4s

What is a Decision Matrix? Criteria Rating Form, Weighted Ranking.

 DECISION MATRIX. Also called: Pugh matrix, decision grid, selection matrix or grid, problem matrix, problem selection matrix, opportunity analysis, solution matrix, criteria rating form, criteria-based matrix

A decision matrix evaluates and prioritizes a list of options and is a decision-making tool. The team first establishes a list of weighted criteria and then evaluates each option against those criteria. This is a variation of the L-shaped matrix.

WHEN TO USE A DECISION MATRIX

• When a list of options must be narrowed to one choice
• When the decision must be made on the basis of several criteria
• After a list of options has been reduced to a manageable number by list reduction

Typical situations are:

• When one improvement opportunity or problem must be selected to work on
• When only one solution or problem-solving approach can be implemented
• When only one new product can be developed

DECISION MATRIX PROCEDURE

1. Brainstorm the evaluation criteria appropriate to the situation. If possible, involve customers in this process.
2. Discuss and refine the list of criteria. Identify any criteria that must be included and any that must not be included. Reduce the list of criteria to those that the team believes are most important. Tools such as list reduction and multivoting may be useful.
3. Assign a relative weight to each criterion, based on how important that criterion is to the situation. This can be done in two ways:
   1. By distributing 10 points among the criteria, based on team discussion and consensus.
   2. By each member assigning weights, then the numbers for each criterion for a composite team weighting. 
4. Draw an L-shaped matrix. Write the criteria and their weights as labels along one edge and the list of options along the other edge. Typically, the group with fewer items occupies the vertical edge.
5. Evaluate each choice against the criteria. There are three ways to do this:
   

   Method 1: Establish a rating scale for each criterion. Some options are:

   1. 1, 2, 3 (1 = slight extent, 2 = some extent, 3 = great extent)
   2. 1, 2, 3 (1 = low, 2 = medium, 3 = high)
   3. 1, 2, 3, 4, 5 (1 = little to 5 = great)
   4. 1, 4, 9 (1 = low, 4 = moderate, 9 = high)
   
   It is important that your rating scales are consistent. Word your criteria and set the scales so that the high end of the scale (5 or 3) is always the rating that would tend to make you select that option: greatest impact on customers, greatest importance, least difficulty, greatest likelihood of success.

   Method 2: For each criterion, rank-order all options according to how well each meets the criterion. Number them with 1 being the option that is least desirable according to that criterion.

   Method 3 (Pugh matrix): Establish a baseline, which may be one of the alternatives or the current product or service. For each criterion, rate each other alternative in comparison to the baseline, using scores of worse (-1), same (0), or better (+1). Finer rating scales can be used, such as 2, 1, 0, -1, -2 for a five-point scale or 3, 2, 1, 0, -1, -2, -3 for a seven-point scale. Again, be sure that positive numbers reflect desirable ratings.

6. Multiply each option’s rating by the weight. Add the points for each option. The option with the highest score will not necessarily be the one to choose, but the relative scores can generate meaningful discussion and lead the team toward consensus

DECISION MATRIX EXAMPLE

Figure 1 shows a decision matrix used by the customer service team at the Parisian Experience restaurant to decide which aspect of the overall problem of "long wait time" to tackle first. The problems they identified are customers waiting for the host, the waiter, the food, and the check.

The criteria they identified are "Customer pain" (how much does this negatively affect the customer?), "Ease to solve," "Effect on other systems," and "Speed to solve." Originally, the criteria "Ease to solve" was written as "Difficulty to solve," but that wording reversed the rating scale. With the current wording, a high rating on each criterion defines a state that would encourage selecting the problem: high customer pain, very easy to solve, high effect on other systems, and quick solution.

Figure 1: Decision Matrix Example

"Customer pain" has been weighted with 5 points, showing that the team considers it by far the most important criterion, compared to 1 or 2 points for the others.

The team chose a rating scale of high = 3, medium = 2, and low = 1 and used it for the problem. "Customers wait for food." In this example, the customer pain is medium (2), because the restaurant ambiance is nice. This problem would not be easy to solve (low ease = 1), as it involves both waiters and kitchen staff. The effect on other systems is medium (2), because waiters have to make several trips to the kitchen. The problem will take a while to solve (low speed = 1), as the kitchen is cramped and inflexible.

Each rating is multiplied by the weight for that criterion. For example, "Customer pain" (weight of 5) for "Customers wait for host" rates high (3) for a score of 15. The scores are added across the rows to obtain a total for each problem. "Customers wait for host" has the highest score at 28. Since the next highest score is 18, the host problem probably should be addressed first.

DECISION MATRIX CONSIDERATIONS

• A very long list of options can first be shortened with a tool such as list reduction or multivoting.
• Criteria that are often used fall under the general categories of effectiveness, feasibility, capability, cost, time required, and support or enthusiasm (of team and of others). Other commonly used criteria include:
  
  For selecting a problem or an improvement opportunity:
  
  • Within control of the team
  • Financial payback
  • Resources required (e.g., money, people)
  • Customer pain caused by the problem
  • Urgency of problem
  • Team interest or buy-in
  • Effect on other systems
  • Management interest or support
  • Difficulty of solving
  • Time required to solve
  
  For selecting a solution:
  • Root causes addressed by this solution
  • Extent of resolution of problem
  • Cost to implement (e.g., money, time)
  • Return on investment; availability of resources (e.g., people, time)
  • Ease of implementation
  • Time until solution is fully implemented
  • Cost to maintain (e.g., money, time)
  • Ease of maintenance
  • Support or opposition to the solution
  • Enthusiasm by team members
  • Team control of the solution
  • Safety, health, or environmental factors
  • Training factors
  • Potential effects on other systems
  • Potential effects on customers or suppliers
  • Value to customer
  • Potential problems during implementation
  • Potential negative consequences

Additional considerations

• While a decision matrix can be used to compare opinions, it is better used to summarize data that have been collected about the various criteria when possible.
• Sub-teams can be formed to collect data on the various criteria.
• Several criteria for selecting a problem or improvement opportunity require guesses about the ultimate solution. For example: evaluating resources required, payback, difficulty to solve, and time required to solve. Therefore, your rating of the options will be only as good as your assumptions about the solutions.
• It’s critical that the high end of the criteria scale (5 or 3) always is the end you would want to choose. Criteria such as cost, resource use and difficulty can cause confusion (for example, low cost is highly desirable). Avoid this by rewording your criteria: Say "low cost" instead of "cost"; "ease" instead of "difficulty." Or, in the matrix column headings, write what generates low and high ratings. For example:

   

  Importance	Cost	Difficulty
  low = 1 high = 5	high = 1 low = 5	high = 1 low = 5

• If individuals on the team assign different ratings to the same criterion, discuss until the team arrives at a consensus. Do not average the ratings or vote for the most popular one.
• In some versions of this tool, the sum of the unweighted scores is also calculated and both totals are studied for guidance toward a decision.
• When this tool is used to choose a plan, solution, or new product, results can be used to improve options. An option that ranks highly overall but has low scores on criteria A and B can be modified with ideas from options that score well on A and B. This combining and improving can be done for every option, and then the decision matrix used again to evaluate the new options.

https://asq.org/quality-resources/decision-matrix

Criteria Rating Form, Weighted Ranking

Use the criteria rating form when:

• You have to select among several alternatives
• You want to make a decision objectively
• You want your group to agree on a decision

1 Start the session and list the alternatives available

 

2 Brainstorm decision criteria

You will be judging your alternatives against what you feel are the most important qualities each one should have. These qualities are called decision criteria. Brainstorming may be a useful way for a group to agree appropriate criteria.

 

3 Determine the relative importance of each criterion.

Rank the criteria and assign a relative importance (weight) to each. The total of the assigned weights should equal 100.

 

4 Establish a rating scale; rate the alternatives.

A suitable rating scale might be, for instance: 1= low, 10=high. each alternative should be weighed against each criterion, using the same scale for each.

 

5 Calculate the final score.

Multiply the weight for each alternative by the score and write this in brackets. Add up the numbers in brackets for each alternative and write the sums in the appropriate total boxes. Add any summary comments in the appropriate summary box.

 

6 Select the best alternative.

Select the alternative with the highest score. this alternative may not be the one ultimately chosen - if the group disagrees with the choice, they should review the weighting of the criteria and make the necessary changes. if necessary, repeat the process.

 

References

• Chang, RY., and Niedzwiecki, ME., "Continuous Improvement Tools", Volume 1. 1993, 1995. Kogan Page Ltd. London

https://www.ifm.eng.cam.ac.uk/

Criteria rating form

Criteria rating forms help individuals and groups decide the best option or options among a group of options. In problem solving groups, they are often used in problem and solutions selection.

 

The criteria rating form can be used anytime there are criteria that will be used to inform the decision making process.  It is often used by interview teams when selecting a candidate for any position or when selecting new materials/textbooks. 

1. The criteria are selected by the group and a rating scale is defined.  In most cases a scale of 1 to 5 is used with 5 being the most desirable.

2. Weights are assigned to each criteria depending on its importance relative to the other criteria.

3. Each potential solution is also given a rating for each criterion, and the rating is multiplied by the weight of the criterion. 

4. The weighted ratings are totaled.  In this case, a 5 on the scale was most desirable so the solution with the highest total is judged the best option.  (I typically use the .5 to 2.0 rating scale.)

Remember that this is only a tool to collect data so be sure to discuss the final outcome before making the final decision, especially if the ratings are close.

 

Criteria Rating Solutions

There are a number of general criteria that can be considered when judging solutions.  What other criteria might you consider.  Criteria are personalized to each process.

•        Control         Is the group in a position to implement the solution? 

•                                                Effectiveness To what extend does the solution solve the problem?  (How likely is it to achieve the desired state?)

•        Customer Satisfaction Will the solution result in increased satisfaction of parents, community members, staff, students, Central Office or others?

•        Time            How long will it take to implement the solution?   (Some solutions          may take less time than others.)

•        Cost of Quality To what extent will the solution reduce the cost of non-conformance?

•        Cost            Are the financial resources available to support the initiative?            

•        Acceptability Will those responsible for implementing the solution accept                             the solution? 

        

Criteria & Scale	Weighting	Potential Solutions
Brainstorm Criteria	.5 -`- 2.0
1  2  3  4  5
1  2  3  4  5
1  2  3  4  5
1  2  3  4  5
1  2  3  4  5
1  2  3  4  5
1  2  3  4  5
TOTAL

 Evaluating Alternatives:  Criteria Rating (Grid Analysis)

The Criteria Rating Matrix, or Grid Analysis, is a tool for objectively reviewing each of your solution options as related to the various criteria you need to consider in order to come to a decision about which solution to implement

For this analysis, list your options as rows on a table, and the criteria you need to consider as columns.  Each option is then rated by how well it satisfies each criterion

Criteria:	A	B	C	D	E	Total
Option 1
Option 2
Option 3

 Steps:

1. List the alternatives you are considering.
2. Brainstorm decision criteria (remember, identify all possibilities before critiquing them).
3. Discuss the suggested criteria.

ü Is this criterion clear & unmistakable in its meaning?

ü Will this criterion be observable?

4. Narrow the list of criteria to 3-6 criteria. 
5. Establish a rating scale (e.g., 0-5 or 1-10).
6. Rate each of the alternatives on each of the criteria.
7. Calculate the final score.

Select the best alternative

Commonly Used Criteria
·       Ease of implementation ·       Cost ·       Ability to meet customer requirements ·       Equipment/resources required ·       Resource Availability ·       Lowest Risk ·       Fastest to implement ·       Long-term workability ·       Effective resource use	·       Impact on employee morale ·       Level of complexity ·       Human resources time required ·       Time required for implementation ·       Degree of control by the team ·       Political support ·       Disruption caused by change ·       Impact on the problem (high, medium, low) ·       Resistance of Stakeholders

Evaluating Alternatives:  Criteria Rating (Grid Analysis)

Example:

 A kayaking enthusiast, Patrick, is getting ready to buy a new car.  He needs one that will carry his kayak but will also be good for business travel.  He has always loved and wanted a convertible sports car.  So far, no one car he has looked at seems to fit all three criteria.

The vehicles Patrick is considering are an SUV/4x4, a comfortable 'family car', a station wagon, and a convertible sports car.  His decision criteria are cost, ability to carry a kayak safely, ability to store his equipment securely, comfort over long distances, attractive look, and fun.

Patrick draws up a table with the vehicle options and the decision criteria as shown below.  He then scores each option, 0-5, by how well it satisfies each criterion.  At this point he does not consider the relative weights of the factors

Criteria/Factors:	Cost	Kayak	Storage	Comfort	Fun	Look	Total
Sports Car	1	0	0	2	5	5	13
SUV/4x4	2	5	4	4	4	4	23
Family Car	3	2	1	4	0	0	10
Station Wagon	4	5	5	5	1	1	21

Based on this calculation, Patrick should buy the    ___________________________

Evaluating Alternatives:  Criteria Rating (Grid Analysis)

Weighted Criteria Rating

In many situations we also want to ask ourselves, “Are some of the criteria more important than others?”  Essentially, what you are exploring is the relative importance of each criterion.  For example, could it be that carrying the kayak and cost are more important to Patrick than comfort and look?

If that is the case, then you will want to weight the criteria, i.e. a weighted criteria rating matrix

Factors/ Criteria:	A	B	C	D	E	Total
WEIGHTS:	#	#	#	#	#
Option 1	Rating X weight	Rating X weight	Rating X weight	Rating X weight	Rating X weight
Option 2	Rating X weight	Rating X weight	Rating X weight	Rating X weight	Rating X weight
Option 3	Rating X weight	Rating X weight	Rating X weight	Rating X weight	Rating X weight

    Steps:

1.     List the alternatives you are considering.

2.     Brainstorm decision criteria (remember, identify all possibilities before critiquing them).

3.     Discuss the suggested criteria.

4.     Narrow the list of criteria to 3-6 criteria that will be most appropriate for identifying the best solution. 

5.     Determine the relative importance of the criteria & assign weights. 

6.     Establish a rating scale.

7.     Rate each of the alternatives on each of the criteria.

8.     Calculate the final score (remember to multiply the rating by the criterion’s weight).

9.     Select the best alternative.

       Back to our Example:

Patrick reflects on this decision a bit more and realizes that, in fact, the criteria are not equally important to him. So he decides to determine the relative weights for each of the criteria and recalculates using the ratings he previously identified

Factors:	Cost	Kayak	Storage	Comfort	Fun	Look	Total
Weights:	5	5	3	4	2	3
Sports Car	5	0	0	8	10	15	38
SUV/4x4	10	25	12	16	8	12	83
Family Car	15	10	3	16	0	0	44
Station Wagon	20	25	15	20	2	3	85

      

       Based on this calculation, Patrick should buy the    ___________________________.

The following is a sample of a 5 Point Numerical / Narrative Rating Scale including sample narrative ratings and definitions for each point value on the scale.

 

 

Scale	Rating	Definitions (Choose and/or Modify as Appropriate)
5 points (Pass)	Excellent. Exceptional Mastery. Much more than acceptable.	Should ensure extremely effective performance. Significantly above criteria for successful job performance. Surpassed expectations. Reserved for the exemplary set of skills that yield a particularly sophisticated approach to handling the situation. Meets all major / essential / core criteria or acceptable equivalents and met three or more additional criteria.
4 points (Pass)	Very Good. Full Performance Behaviours. Above average.	More than adequate for effective performance Generally exceeds criteria relative to quality and quantity of behaviour required for successful job performance. Meets all of the major / essential / core criteria or acceptable equivalents and several of the minor / addiitional criteria. No major deficiencies exist in the areas assessed. Consistently demonstrated better than average level of performance. Describes / demonstrates the full range of skills appropriate for handling the situation and the desired result, or outcome is obtained.
3 points (Pass)	Good. Acceptable. Satisfactory Average	Should be adequate for effective performance. Meets criteria relative to quality and quantity of behaviour required for successful job performance. Meets several of the major / essential / core criteria one or two of the minor / additional criteria or acceptable equivalents. Describes / demonstrates a sufficient range of skills for handling the situation and the desired outcome is obtained. Some of the major and minor criteria were met; some deficiencies exist in the areas assessed but none of major concern.
2 points (Fail)	Weak. Less than Acceptable	Insufficient for performance requirements. Generally does not meet criteria relative to quality and quantity of behaviour required for successful job performance e.g. meets half or less of criteria. Does not describe / demonstrate a sufficient range of skills appropriate for handling of the situation, or describes plausible but inappropriate behaviours for handling the situation or the desired result or outcome is not obtained.
0 – 1 point (Fail)	Unacceptable. Poor. Much less than acceptable	Significantly below criteria required for successful job performance. Few or no criteria met. Many deficiencies. A major problem exists. No answer or inappropriate answer. Describes/demonstrates counter-productive behaviours that have negative outcomes or consequences (make the situation worse).