On Complexity

This is about programming or at least solutions materialized using computers and software. Apply it to other areas at your own risk.

I know how to program. Its not what I do for a living anymore.

For many problems within my field of expertise I can give a rough estimate, and with rough I mean that I get it right within a factor of 2. Ask me the price for building something and I can tell you 8h, 16h, 40h, 80h, 160h, 400h, 800h, 1600h… you get it. The estimate also includes generally what tools to use (C, ASP.NET, Linux+bash, LAMP).

My observation: many people – professionals – cant get this right.

I believe it has to do with (a lack of) intuition for complexity:

  • Software-related problems are much more non-linear that most real-world-problems.
  • For software, the hardest part of a problem tends to determine the hardness of the whole problem
  • Solving two problems together, or merging the solutions of two problems, tends to generate a much more than twice-the-size challenge

The result: people who fail to thoroughly understand the above can not reasonably (within a factor of 2) estimate the cost of a software solution.

There are simple solutions, and complex solutions – often to the same problem. Once I watched news about the unmanned subway trains in Copenhagen. They interviewed the responsible manager and asked him: is it not dangerous with no one to brake the train? He answered: no it is safe, we have a very complex system monitoring the trains and the safety. Horror – I thought! I wish he said they had a very SIMPLE system. Because to me

Simple means many people are able to understand and verify the correctness of the system.
Complex means few people (and maybe no single person) are able to understand and verify the correctness of the system.

Sometimes I draw this matrix:

Simple Solution Complex Solution
Simple Problem Yes No
Hard Problem Yes No

In Einsteins words: Everything should be made as simple as possible, but not simpler.

Where is this going? Well, my question today is:

Why do people choose to attempt unnecessarily complex solutions to problems?

In other words: Why do so many computer professionals suck so bad at their job?

First lets just despise those who intentionally make money by selling more hours than necessary. But those with good intent, who are just bad?

I think they simply fail to understand complexity, as I mentioned before, so they simply can not tell a simple solution from a complex solution. Obvious.

Whats worse, I believe that they prefer complex solutions, based on the real-world fact that more-is-more.

Whats even worse, and this is just my conspiracy theory, is that they prefer complex solutions because; not understanding a complex thing is normal, but not understanding a simple thing is obviously stupid. Thus, stupid people are drawn to complicated stuff to not reveal their stupidity. It is just a survival strategy in the job market, to level the field a bit.

  1. On complexity:

    For many years, I have considered IT people as some of the most incompetent people due to their seemingly missing values in business. Your explanation really fills in a lot of holes in my thinking and I found it very entertaining and informative.

    Thanks for your observations.

    Charlie Morrison

  2. Glad you liked it!

  3. Brilliant and entertaining!

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Time limit is exhausted. Please reload CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.