The Paradox of Competency

It has been argued that the most difficult thing to do in sports is to hit a ball off of a major league pitcher. I will respectfully disagree. Once every four years, I stare mesmerized at the image of young women competing at the Summer Olympics in the vault competition. The girls sprint down a narrow strip, jump onto a launching pad, get sprung ten feet in the air, land on a narrow apparatus, then push themselves back into the air from a handstand, perform multiple spins and twists, and are expected to nail the landing. For the life of me, I cannot think of anything that could be more difficult.

And yet… Should one of these ladies hop or slightly waive an arm, you can audibly hear the commentator suck in their breath and say, “Oooooohhh, it was a good jump but there was that little hop at the end. The judges are definitely going to deduct for that.”

There are perhaps one in a million young women with the requisite talent and training to be able to accomplish this mind blowing feet. These girls spend literally years of their lives training and preparing for this moment. All those years, all those workouts, all those sessions in the gym come down to those few seconds in front of the judges. Their goal is to do the near impossible and to make the impossible look easy.

No commentator has ever said, “Wow! That was an amazing vault. She did three somersaults with a twist! Who cares if she hopped a little on the landing? It was still incredible!”

I think the judges are so adamant that the girls make it look easy because we all implicitly know that what they are doing is both difficult and dangerous. Making it look easy is part of the competition. On the opposite end of the spectrum, in software engineering, software engineers are often judged by people who know little about software engineering. In order to be a good software engineer, in their minds, software engineers need to make their jobs look hard.

If an engineer takes a look at a difficult problem, thinks about it for a while, quietly sits and cranks out some code, writes a few tests, and then silently checks it in - the equivalent of a young woman nailing the perfect vault, most managers will think that it is easy and be sadly unimpressed with the accomplishment.

On the other hand, if the same engineer had called countless design sessions, stayed some late nights, checked in code and saw it fail, reworked it several times, and then after a few weeks got it to work; the manager judging the engineer would probably give him high marks. This would be the equivalent of the gymnast flailing around in the air and landing on her butt, yet receiving a perfect ten.

I like to call this the “Paradox of Competency”. In most pursuits, accomplishing a task with grace and style is seen as a good thing. Sadly, in this godforsaken industry, most managers feel developers are interchangeable parts. To a bad manager, a software engineer is like the example of a monkey randomly hitting keys on a keyboard and hammering out a work of Shakespeare. Given enough time, randomly, that monkey will do it by accident. In fact, I have worked on plenty of systems that looked like they were developed by monkeys banging on keyboards.

THE NASH EQUILIBRIUM

Mathematician John Nash, the subject of the movie “A Beautiful Mind”, won the Nobel Prize in Economics for his work on the Nash Equilibrium. It is a framework for evaluating the decision making process of parties involved in a non-cooperative game. In layman’s terms, it is often used to describe the “prisoner’s dilemma”.

Both prisoners would be best off if they could confess without the other one confessing. Both prisoners, unfortunately, know the stakes and if they both confess they face ten years of hard time. Rationally, both of them keep quiet and accept the one year of jail. In this framework, each prisoner has to take in account the decision making process of the other participant and they settle on the most likely, least bad outcome. According to the Economist Magazine, this framework helps to describe why “the decisions that are good for the individual can be terrible for the group.” (Source)

While I am a student of applied game theory and a lover of all things rational, I once joked that I had to leave the consulting industry because I had ethical problems. Mainly, I have ethics. Sadly, I have started to realize that game theory and questionable ethics mix quite often in software companies, especially in the presence of a non-technical manager. Mainly, doing a competent job and creating simple, maintainable solutions is not rewarded. Effort and complexity are rewarded. It is, therefore, in the best interest of the individual to make their solution as complicated as possible. As the Nash Equilibrium predicts, what is good for the individual is bad for the organization.

I worked in a group dominated by the Comic Book Guy. Well over forty, never had a family or wife. As far as I know, never had a girlfriend. The only thing he did, to the best of my knowledge, outside of work was collect comic books and talk about collecting comic books. He was an absolute delight to work with. He, somehow, had managed to convince just about the entire company that he was the ONLY person who could possibly understand all the data going into the data warehouse. Therefore, however he wanted to design the data warehouse was up to him.

With two members of the crack team of architects dedicated to him, two consulting firms, countless dollars, endless resources, and two full years to create the data warehouse; it is not complete. With less than 50 gigabytes (not a big amount in terms of data warehousing), it is not uncommon for queries to run for over an hour. By any reasonable measure, the data warehouse in question is an absolute disaster. And yet… The Comic Book Guy is still calling the shots. Our shared manager went out of his way to call out how good a job he and his small crew were doing giving they were new to Redshift. They had been working with Redshift for two years now.

On the other hand, my small crew and I pulled together a platform that would hopefully lead to new revenue. We did it in two months using technology we had never used before. The CEO announced it, PR Newswire carried a press release for it, and not a word was said to the two guys who actually wrote it. We made a mistake. We made it look easy.

Like the gymnasts hurling themselves at the vault, writing software is not easy. Making it look easy, sticking to schedules, and actually delivering should be rewarded like the gymnast who sticks the landing. Hopefully, the non-technical managers of the world can do a better job evaluating teams and eliminate the Paradox of Competency.