Jill Lepore has an excellent take-down in The New Yorker of Clay Christensen’s The Innovator’s Dilemma. Yet I am unconvinced.

I thought I was convinced when I read it. It’s a brilliantly done piece, examining Christensen’s evidence, questioning his methods, and drawing appropriate lessons, including wondering why we accepted the Innovator’s Dilemma for decades without critically examining it. (Christensen became so famous for it that his last name isn’t even flagged as a spelling error on my Mac.)

I got de-convinced by a discussion on a mailing list I’m on that points to some weaknesses in Lepore’s own argument, including her use of “cherry-picked” examples — a criticism she levels at Christensen — and her assumption that the continuity of companies, as opposed to their return on assets, is the right measure. As a person on the mailing list points out, John Hagel, John Seely Brown and Lang Davison take return on assets as a key metric in their book The Big Shift. And then someone else maintained that ROA is a poor measure of networked phenomena. That morphed into a discussion about the pragmatic value of truth: Does disruption provide a helpful framing for the New York Times as it considers its future?

The problem is that brains are truthy. They are designed to pay attention to things that seem to matter to us, bending our world around our concerns and interests. And brains are associative, so they make sense of the world — maybe even at the level of perception — by finding the relationships that seem to matter to us. In Heidegger’s terms, we are not indifferent knowing machines, but are creatures that care about what happens to us and to others. The brain is an unreliable narrator.

We now have access to an unfathomable sea of information that can contradict anything we settle on. That sea has been assembled by caring creatures and their minions, but it is so vast and global that it contains information beyond the caring and linking of any one of us. Every understanding can be subverted with a wink and a hand wave because all understanding simplifies a world that is resolutely and even necessarily complex. The universe outruns us.

Now we have machines that can look at masses of data and escape from our temptation to turn everything into a narrative. But those machines are limited by our decision about which data is worth gathering and connecting. There is hope in this direction, but it’s not clear whether we are capable of accepting the findings of machines that correlate without stories.

TL;DR: Our brains are truthy and the world is too big to make sense of. Not that that will stop us from trying.

[June 20:] Clay Christensen has cried foul in an interview.

Categories: big data, business, too big to know dw