Creeping Complexity

The optimized analog world of forty or so years ago was based on networks in which the nodes were humans. Sometimes they were imaginative, clever. I read about a Cambridge don who sent himself reminders to non-existent addresses around the world, depending on the postal system to return them to him at the appropriate time. At a company where I worked in the 1980s, their distribution system depended on high school kids on roller skates putting paper slips into pigeonholes. Try as we could, we could find no digital solution within even one order of magnitude on cost.

Analog systems like this had one huge advantage: they could deal with ambiguity. That ability allowed them to be much simpler: ambiguity allows simplicity. The network nodes, aka human beings, could look at information, realize something was wrong and correct it. Software nodes can only do what they have been programmed to do. Thus, seemingly simple tasks became complicated. A rule of thumb was that more than 90% of every software application deals with error handling rather than the actual task.

I was forced to deal with this full on thirty years ago at Sun Microsystems. In the spring of 1989 we had a disastrous quarter, and the initial answer the company gave was that new manufacturing software failed. This was correct but misleading. We replaced an existing enterprise resource planning system (ERP) with a new one. The existing system was much less sophisticated and relied upon humans making common sense decisions. The data had to be roughly right. The new system did much more, but the data had to be precisely right. The new system failed because it could not handle ambiguity and failed because of error rates approaching 100%.

Digitization of the world has thus driven increased complexity relentlessly. Compare the sophistication of the systems at Amazon or Shopify with that of a traditional retailer. The state of the art retailers have real time views by store and SKU of everything they sell. Walmart started keeping every register tape by customer three decades ago and was a total leader in this space. This type of analysis led to the realization, for example. that the item most commonly sold with disposable diapers was beer. Something only data could suggest.

Brains differ. Some brains deal with complexity better than others, and there is a relative scarcity of people with such brains. The result not surprisingly is that compensation for such people keeps increasing but decreasing for most everyone else. Most tasks…