Quantifying the Unquantifiable

Quantify the unquantifiable and you change the way that you are able to state goals, track performance, benchmark, evaluate,  communicate, and measure outputs not just inputs.

Costs are easy to measure. they are quantitative.  The aspects of performance that we are interested in, however, are nearly always qualitative.  We want to know what effect our actions have had. Has this action of ours made things better or worse?  Such qualitative measures are harder.  This is why we are much more likely to default to measuring efficiency (a supply side cost issue) than effectiveness (a demand or user side performance issue).  Consider some of the major measures that we use to track how we are going, as a country, or as an organisation – e.g. GDP, life cycle costing.  Even benefit-cost analysis will traditionally focus on the measurable dollar amounts and fudge the unquantifiable externalities (both benefits and costs).

We can tie ourselves up in knots trying to express everything in dollar terms and to the extent that we succeed, we may be unaware of the elements inadvently omitted.

So today I want to introduce a technique that was introduced to me 20 years ago by Norman Eason,  the Founder and First President of the UK Institute of Asset Management.  He gave me this work to publish and I present it to you.  If anybody would like to take up the challenge to develop it and take it further, now that technology has made tasks like this so much easier, maybe turn it into an App, I would be happy to put you in touch with the author.

Norman called his technique CAT – Comparative Assessment Technique.  It enables you to determine  whether qualitative outcomes are improving,  whether goals have been reached. – and, if not, how far we are away from the desired end-state (and how much it will cost to get there!)

CAT is based on the simple, but operationally powerful, idea that any qualitative output can be expressed in terms of its worst state, its best, and all possible states in between.

Being able to measure qualitative outcomes/outputs makes it possible to

• Track developments over time
• Compare outcomes with other organisations
• Relate activities to outcomes
• Relate the cost of activities to the improvement in outcomes

So how is it done?

Suppose we use as an example of intangible factors the difficulty in measuring the Cleanliness of a Reception Area.  We could list all the possible states, from worst state to best state, as follows: –

Ladder – Cleanliness of a Reception Area

1. The filthy nature of the area is seriously affecting our relationships with staff/clients.
2. We are always receiving complaints about how filthy the area is.
3. We often receive complaints about how filthy the area is.
4. We have had complaints about how filthy the area is.
5. Although we have not yet received complaints from staff/clients about the dirty condition of the area, we know that this is likely to happen.
6. We feel that as a company we are embarrassed by the dirty condition of the area.
7. We feel that staff/clients believe that the area is very often dirty.
8. We feel that staff/clients believe that the area is usually clean, but is often embarrassingly dirty.
9. We believe that staff/clients think that the area is almost always clean.
10. We believe that staff/clients think that the area is always clean.
11. We never have any problems with the cleanliness of the area.
12. We believe that staff/clients think that the cleanliness of the area is always excellent.
13. We are always receiving favourable remarks from staff/clients about how clean the area is.

There is no limitation on the number of steps to be used

Note that we have not attempted to limit the number of steps to ten; this would have been the natural tendency, but it is pointless and an artificial irrelevance to the definition of the states. The CAT term for this list is a ladder and each listed state is a step on that ladder.

Setting Target Step

We can now designate the Target Step that the user and supplier agree should be met. Suppose this is Step 11.  We never have any problems with the cleanliness of the area.

Setting Actual Step

The user and supplier can now select a step that matches the actual position at any one time. This is the Actual Step. Suppose this is selected to be Step 5. Although we have not yet received complaints from staff/clients about the dirty condition of the area, we know this is likely to happen.

Comparing the Two Steps

We can now obtain a measurement of the actual condition in relation to the target condition, thus Step 5/Step 11 = 0.45 so we are 45% of the way there.

Client-Contractor Relationships

Ladders such as this, especially if developed with joint input, have the potential for far better communication and understanding between client and contractor.   It is possible for both to see clearly when a goal has not been achieved (with potential for service improvement) and when it has been over- achieved (with potential for cost savings)

Two interesting things become obvious from our earlier calculation. 

• We now have a numerical value of something that previously could only be considered in subjective terms. (Target 45% achieved)
• We now have a decimal value of the relative position of the actual state against the target state, even though we did not start with a scale of ten.

Note:

• Any ladder can be of any length. In fact, the user and the supplier of the service determine the length by incorporating as many steps as are required to fully describe the activity.

• It doesn’t matter that the Target Step is not the top step of the ladder. Target Steps can be moved as new goals are agreed. Note also that there need not  be  equal  increments  between  steps,  although equal steps would be ideal.

• The technique is not restricted to Building Services. Ladders can be used to measure any activity in any one of a number of areas. These areas are not limited to commerce or industry; Local and Central Government are also relevant.

CAT can be applied to the measurement of anything that is intangible

What do you wish to measure?

In assigning CAT to an application area, it is helpful to consider what precisely is being measured. This may seem to be an obvious statement, but there is often confusion in this respect whether tangible or intangible measurements are required.

For example, do you wish to measure the performance of a group, or the results of their work? The two are not necessarily the same. CAT accommodates these more precise definitions of activities by providing for three types of ladder.

CAT can measure

• the performance of a task,
• the effectiveness (or result) of the task, and
• the  consequence  of  the  overall  activity,  e.g., what do the users think of the results?

These ladders can be used separately or combined into groups – any groups that the user organisation feels are important to measure or compare. Also, the activities associated with  ladders  can  have  parent/child  hierarchies  so that, for example, measurements can be rolled up from individuals to groups, to departments, to sites, to divisions and ultimately to the overall company or organisation.  In both cases, users also need to agree the weights to be given to each ladder and activity.

Ladders can also be used for comparison in benchmarking exercises. For benchmarking, it is necessary for user organisations to agree on the standardisation of ladders.

 If you would like to see the complete report, which contains examples of CAT ladders for consequences, performance, and effectiveness, email me at penny@TalkingInfrastructure.com

Do you know of similar work or perhaps other means of quantifying the unquantifiable?