3.2.3 Interpreting Customer Feedback Surveys
It wasn’t until 1997 that an IBM supercomputer, named appropriately Deep Blue, finally out smarted a chess Grand Master, Gary Kasparov. The fundamental reason why it took so long for a computer to achieve this is explained by an experiment carried out by Herbert Simon, the 1978 Nobel Laureate in Economics.
Simon was interested in decision-making using artificial intelligence and reasoned that before one could develop computers with this attribute, one needed to understand how the human brain makes decisions.
Simon asked chess Grand Masters to look briefly (for ten seconds) at chess games that were already in progress with around twenty pieces remaining on the board. He found that despite this very brief look, Grand Masters could recall the locations of virtually all the pieces. When Class A players, one below Grand Masters, were asked to do the same thing, they fared much less well.
However, if twenty or so chess pieces were placed at random on the board, neither the Grand Masters nor the A class players performed well. What emerged from this research was that the Grand Masters were much superior to the A class players in recognising patterns of play. Their innate ability and their experience enabled them to store in their long-term memory around 50,000 patterns compared to the 2,000 patterns of A class players.
Armed with this prodigious memory bank, these chess players could consistently beat the most powerful computers of the day because of course, the computers could only operate by examining all possible moves and counter-moves. So to beat Gary Kasparov, what the IBM programmers had to do was to equip Deep Blue with a similar library of patterns to drastically reduce the number of possible moves that the computer had to evaluate. Even so, it needed a computer of exceptional power to defeat him.
The fundamental notion behind this experiment is that people who have a large vocabulary of patterns stored away on a particular topic or discipline can extract many more insights from the same data as a person whose pattern vocabulary is more limited. So stab wounds to a forensic pathologist yield a mine of information about the crime and profit & loss accounts reveal more to an accountant or a stock analyst than they do to the average person.
I have recently completed a customer feedback survey for a client. It’s the third one in a series and by the completion of the second, it was possible to divine from the feedback what steps the client had to take to raise satisfaction levels. The table that provided these answers looked like this.
Attribute |
Average |
Product availability |
9.3 |
Accurate paperwork |
9.1 |
Problem handling |
9.0 |
Pro-activeness |
9.0 |
Flexibility |
8.9 |
Order lead-time |
8.8 |
Ease of order placement |
8.7 |
Price |
8.5 |
Product support |
7.9 |
Product range |
7.3 |
Representation |
6.1 |
Average |
8.4 |
Each one of the above attributes comes with a definition that is read out to the customer to ensure uniformity of understanding. The respondent is asked to rate the importance to them of each supplier attribute on a scale of 1 - 10 with 10 being high.
Respondents are then asked to rate my client’s performance on the same scale against the same attributes. Again, a sentence of definition is read out to them.
To interpret this table, one needs to look at the absolute number and the relative position of each attribute. Then one has to tap into the vocabulary of patterns to determine what’s the same and what’s difference compared to many other surveys.
One point that usually surprises clients is the relative position of price - eighth out of eleven in this case. It is actually fairly rare to see the position of price any higher because it has to be borne in mind that we are talking to customers who, by definition, are currently using your product or service. If we were to ask prospective customers how important price was in their purchasing decision, there would be more variation - less in the absolute importance attached to it but more to its position in the decision-making hierarchy.
There are many factors that influence the importance of price but two stand above the others. One is the degree to which the product or service can be differentiated in the eyes of the prospective customer and the other is the percentage of total costs that the purchase of the product represents.
In the example in question the value of price at 8.5 is a very high one given the lowest that we have experienced is 7.0 for a supplier of consulting engineering services. This indicates that either we are measuring price in relation to a “commodity product” or one that represents a high percentage of total costs. In this case, a rating of 8.5 confirms that customers see my client's product as one of low differentiation. The highest rating for price that I have seen is one of 9.3 for a supplier of feed to the pig industry.
In this case, feed accounts for up to 80% of production costs and many respondents regarded feed as a commodity product - and if one supplier’s product is viewed as much the same as another’s why would anyone want to pay more? One conclusion that can be drawn from a very high rating for price is that the band of market prices between the lowest and the highest will be a narrow one. Conversely, where the absolute rating is low - as for the consulting engineer - the band of market prices is much broader.
It might be tempting to conclude from the example table that Representation at 6.1 is not that important in comparison with such factors as Product availability and Accurate paperwork. Nothing could be further from the truth. When interpreted in conjunction with other feedback, Representation emerges as the key point of differentiation. The rationale is that many of those factors rated higher than Representation are the price of entry into this market but, as is so often the case these days, approximate parity exists between suppliers on these factors.
Hence the quality of representation becomes a potential source of competitive advantage. Given this conclusion - and barring any significant deficiencies in attributes ranked above Representation - my client should seek to improve the quality of representation to gain a competitive advantage. Another section of the study investigated what customers viewed as the most useful roles a representative could play.
Most people would not rate accuracy of paperwork as a key factor in achieving and maintaining high levels of customer satisfaction yet for businesses where many invoices are involved and where the diversity of products purchased is high, accurate paperwork is more important now than it has ever been. Why? Because whilst the likelihood of mistakes has declined due to computerisation, when they do occur, they are a lot harder to sort out.
One might think that Product range, valued at 7.3, also has the potential to be a differentiator, given the fact that it is rated just above Representation in the hierarchy. On the contrary, past experience or the vocabulary of patterns indicates that this is not the case. The attitude of most customers is that “provided you have the products that I want, I don’t care how extensive your range is”. There are exceptions to this thinking and from experience, I know what these are.
It’s one thing collecting customer feedback and quite another interpreting it. Experience counts for a lot in forensic marketing.