In our post on May 29 we introduced the problem of finding the best solution to the problem of a high number of road accidents on a certain stretch of road in the Adelaide Hills. The Government had decided to address the problem but what was most important to them – reducing the number of accidents, maintaining speed of traffic without congestion, aesthetics (or the ‘general look’ of the solution), minimising social disruption, or containing the cost?
Multiple Criteria Analysis
In this case a multiple criteria solution was adopted. Each option was analysed by specialists in the particular criteria who estimated the expected outcomes for their criteria, e.g. likely number of accidents reduced, or likely impact on aesthetics, so that each option ended up with five scores, one for each of the criteria examined. No option clearly dominated all others on the five criteria.
To weight or not to weight
At this stage there was a staff debate. Some thought that each criteria should be allocated a weight which would enable them to reduce each solution option to just one score and thus they would be able to rank the options, as is often recommended in the textbooks on the subject.
Others thought that they, the staff, were not the right group to assign weights but rather that this was the right and responsibility of the decision makers themselves (i.e. the government decision group). They pointed out that in previous circumstances where staff had assigned their own weights and ranked the options accordingly, the decision makers frequently ignored their ranking recommendations. Moreover they argued, since the decision makers were unable to see the analysis that had been done on each criteria, it was not taken into consideration and so was wasted. This group argued that the decision makers should be presented with the results and allowed to decide what weights to assign.
Good question, as this issue crops up quite frequently in local govt when we are considering various options. In essence it is not a matter of who has the right as they are all stakeholders.
The real question is the balance of risk v benefits. I find that developing a set of scenarios that encompass each stakeholders issues and then looking at its probability of occurrence, its consequences (good and bad), possible mitigation strategies and the probability to detect (or become aware of the issue) need to be calculated.
This calculation is my version of generating a possible score (for risk) for rating the options. What it does is forces each stakeholder to place all their cards on the table for the decision to be made – ie exposes all the upside (which is usually over-rated in most analyses) and the downsides (which is usually under-rated). It is a push more towards evidence-based decision making ie no surprises (now or in the future).
You also consider these scenarios over the expected lifetime of the proposals – not just a convenient budget timescale…..and the scenarios must be as realistic as possible (include black swan events as these are the ones that usually catch us out).
Admittedly this methodology does assume some linearity in the variables considered in the scenario and in real life this is often not the case. There are numerous techniques available from data science to expand this model to cater for non-linearity and multiple interactions between the variables but as a first approximation it should suffice. A good indicator of non-linearity is to carry out a sensitivity analysis and if the solution is unstable (as evidenced by widely varying results) then these data science techniques can be used (happy to discuss these techniques at a later time).
An important note – please consider the customer/user as a stakeholder in these scenarios – it is not just the staff and decision makers.
End note- this methodology is structured around overcoming the inherent biases we all suffer from. Wikipedia has a very good section on the biases that have been cataloged.
The example where recommended weightings have been ignored (or not understood) may be more related to the familiarity of the process than with the specific analysis itself. I am aware of a similar decision using a complex multi criteria analysis for selection of a software system that was difficult to gain approval – because the analysis method was not well understood and had not been used before (or after, as it happened).
Tender processes often consider weighted attributes, and a common methodology with the weighting is to set them based on accepted criteria (before issuing the tender of course). The deciding group is familiar with the weighted attributes method and so may challenge the occasional ranking but generally accept the recommendations that arise.
So in response to the question of which group has the right of it: I would undertake a separate process of seeking the decision group’s input into forming the weightings. Only after that is resolved would a specific project / decision / service get presented using the methodology. Ideally many such decisions would be made in a year using the same methodology.