Big data is a route to Abundance, but let us take care, data makes a great servant but a dangerous master.

If we respond to data outputs uncritically we make data our master.  Yes, algorithms are becoming ever more complex and difficult to understand – and therein lies the problem.  Unless we are able to understand HOW a decision was made, algorithms are just dead dangerous.   There are really serious problems of people being denied access to medical or social assistance or being jailed because an algorithm deemed it so.  Less serious – but funny! – is the story of two booksellers on Amazon using pricing algorithms that resulted in a 1992 out of print book on the evolution of flies rising to over $28m. A post graduate student interested to buy the book first noticed that where second hand copies were listed at $35, new copies were then over $2,000.  When he went back the next day, the price had risen even more, and so it continued. Studying the prices, the student realised that one seller had set his pricing algorithm to be fractionally less than his main competitor in order to attract sales. the other bookseller had set his algorithm to price this book at slightly more than his main competitor.   Why?  We don’t know but the student suggests (in his blog where he writes it all up) it might have been because the seller didn’t actually have a copy of the book and would have to source it should anyone order. Since the premium of the one seller exceeded the discount of the other, the price kept rising. It took this research student to notice what was happening – not the booksellers, who were, of course, on autopilot.  What damage could occur in your work if you are on autopilot?

The ability of big data to deal with increasing complexity is nevertheless a great advantage. Consider how we have moved since the first industrial revolution.

Our data culture is shaped by the first 3 revolutions.  This has given rise to two attitudes or mindsets that are no longer acting to our advantage.

1. A perception of scarcity and the idea that adversarial competition is good.  But we are about to embark on a new world of abundance – as well described in Jeremy Rifkin’s “Zero Marginal Cost Society”.  This is requiring businesses to change their business models.  We know it as ‘disruption’ but it is more than that, it is opportunity.
2. With increasing specialisation we have developed a culture of fragmentation and the idea that knowledge means control.   Now that we are able to share knowledge so much more freely, we are able to develop better models, a more fruitful culture.
3. This means that we can start to explore replacing these old ideas with a better idea, that humility, gratitude and kindness create abundance and by helping each other we can meet the challenges of an uncertain future.  These are not really new ideas. During and just after the second world war, Londoners who had survived the blitz, now faced the problem of recovery. Instinctively they knew that, just as they had survived the war through helping each other, this was also the way that they would meet the uncertainty of the future.  But as people became richer, they became more selfish and the initial rate of gain slowed.  The way of the future is through collaboration. Harvard studies, as reported by Shawn Achor in  “Big Potential”, have clearly demonstrated that teams of collaborating individuals have it all over geniuses.  In fact a team of average intelligent people who ‘get on with each other’ well exceed the performance of a team of geniuses.
4. This is the real message of ‘big data’ – together we can do more than we can do alone.