Tuesday, September 5, 2017

Moving Parts

"For each ticket you do, please ask yourself when you get done: 'Did I make the entire system I am working on better?' If you cannot answer, 'Yes,' I implore you to address that in the craftsmanship of your work.

You see, ultimately you, the engineer, are the arbiter of _done_. Thus, you are the ultimate source of all technology success. It’s a heavy burden. Carry it well." - A really good person that will remain anonymous unless he/she wants to claim credit for this quote

Elon Musk founded the company Tesla on an absurd idea. Taking a bunch of Lithium Ion laptop batteries, bundling them together, and creating a pure electric car was absolutely nuts. And yet… Tesla keeps making cars that are critical (Source) and commercial successes and their automobiles exceed all expectations. As of this writing, the Tesla Model S, with the performance package and dual motors, is now capable of accelerating from 0 to 60 in  “Ludicrous Mode” in 2.4 seconds (Source). This time beats the pants off of established players in the performance game like Porsche, Lamborghini, Ferrari, and any other car company - and they are doing this in a luxury sedan, not a sports car.

Not only do they provide blazing fast acceleration and near 300 miles of range on a charge, Tesla has established themselves as a “tech” centric car company. Their vehicles have gigantic 17” monitors in the console. Heck, some cars even drive themselves (Source). As a Tesla owner and a tech enthusiast, I find all of these extras amazing or annoying; depending on the time of day.

Recently, there was a software upgrade that was received by my car. After the upgrade, the handles on the driver’s side of the car stopped being usable. Ordinarily, the handles lie flush with the car. When the car is stopped and the driver approaches the car, the handles automatically pop out. Once seated with the doors closed, the handles retract. It’s truly magical, right up to the point when it stops working.

The Model S first arrived on the scene in 2012. In the years leading up to the launch, I am sure that there was an engineer who came up with the idea of the retractable door handles and was very proud of this accomplishment. After all, when the door handles lie flush with the body, it does cut down on the coefficient of drag thereby extending the overall range of the vehicle and the number one concern of making an all electric car was winning the battle with the consumer over the dreaded “range anxiety” phobia (Source). Anything that extended the driving range of the vehicle had to be good. Right?

After my incident with the aforementioned software update and the inability to open the door on the driver’s side, I spent a decent amount of time on various Tesla forums. It became readily apparent that for all of the great, innovative things that Tesla does, the least liked feature is the door handles (Source). If the door handles are broken, the owner is looking at a roughly $1,000 bill (Source). Sure, when you first go to the Tesla showroom and you see the door handles magically pop out, you think, “Cool!” But now that the same feature is costing you a grand, you think it sucks.

In reality, the novelty wears off pretty quickly. After a while, as a driver you cease to even notice - that is until it stops working. Does it lower the coefficient of drag? Well, yes. But… I cannot find any relevant sources that will quantify by how much. Is this adding 1 mile of range or 10? My guess is it’s probably close to less than one and has more to do with the coolness factor than any practical application.

As for my experience with the dreaded driver’s side door handles - I called Tesla as I am still under warranty. I was advised to, and I’m not making this up, reboot my car. As luck would have it, rebooting the car did the trick and the door handles now work as expected. No repairs were needed.

However, I thought a lot about this experience. An engineer came up with a design that would ever so slightly improve something that actually mattered. However, in doing so, the engineer introduced, in this case literally, moving parts. The benefit of the moving parts were minor, but the complexity of the feature was not insignificant. This feature has cost Tesla some customer loyalty and satisfaction and tarnished their reputation somewhat.

In software, this kind of thinking happens all the time. There is almost never a “right” solution. There are multiple solutions to every problem. There are tradeoffs to every proposed solution like how long will it take to implement and how well it will perform, but what is often overlooked is the question, “Who is going to maintain this?”

In the godforsaken software engineering industry, according to the folks at Fast Company, it is advisable to change jobs every three years (Source). They go on to say that workers who stay in their jobs make up to 50% less. While I am not claiming the information in said article is accurate and I cannot provide an actual metric to how often software engineers change jobs, I will say, anecdotally, it happens all the time.

So, if a really smart engineer looks at problem and designs a solution that does a couple of unimportant things really well, but is super complicated - it might not be the right solution. As I sit here and reflect, midway through my own career I had a startling thought. I am over twenty years into a career in a field where intellect is fetishized and shiny objects du jour rule decision making. And yet, I have a below average intelligence, barely able to walk and chew gum at the same time. The thought crossed my mind that my lack of brilliance is actually an asset and not a weakness as I tend to write extremely simple solutions without a lot of moving parts.

Instead of over optimizing for theoreticals that don’t really matter, I spend most of my time making sure I nail all the must haves. I add in a little documentation and then I move on to the next thing. I have this overall philosophy that as I sit and write this, there is a kid that is in their senior year of UT having a drink on Sixth Street. While he or she may be care free right now, enjoying their last year of college, in a few months this individual will be wanting to start their career. I would be honored if a reasonably smart recent grad with a good attitude was handed my work to extend and maintain.

Being replaced by a fresh out of college graduate is the furthest thing from an insult. This is my mission at work. In all likelihood, a fresh out will destroy me in any discussion of algorithms or Big O notation or any of the other academic stuff I never studied. However, they have likely not worked together as a team, done pull requests, tried to get business requirements out of business users, or understood how difficult it is to get stuff done in the work environment. My code repositories do what they say they and even have a little documentation and some comments to help a newbie get started. I always expect to have someone else maintain it later, so I think a new grad should be able to clone my work and add a feature. There is a ton of value to this and I would take it as the ultimate compliment if they could be successful. It means that I understood the original problem, wrote maintainable code, and got things to a certain point that it could be turned over to someone to continue working. The heavy lifting is done and I can move on to the next problem.

Recently, I had an epiphany on how we could do data extracts at work. I was handed a list of challenges like managing the connections to the MySQL database and being able to handle a certain volume of customers. I gave it some thought and came up with a design that seemed very clean, powerful, and yet extremely simple. It used various components of AWS, scaled out beautifully, and would cost pocket change to execute. I did a quick whiteboarding session with some coworkers and once I pitched it, most people seemed pretty enthusiastic. I took the whiteboard sketch and put it into a more formal architecture diagram along with a high level overview of the problems the solution was trying to solve and published it internally. I received a lot of compliments and was shocked at the end of the day I was told that we should be using a different tool.

I don’t want to go into the specifics, but the proposed tool is intended to be used for situations where there are well over 1,000 transactions per second. The typical use case would be for an Internet of Things application or aggregating web logs. Our use case was nowhere near this threshold. In frustration, I tried to explain that the counter to my solution was like having a problem with how should you cut a cake.

Me: Let’s use a knife.

Architecture: Let’s use a chainsaw.

Me: OK, I can’t deny that a chainsaw can be used to cut a cake, but doesn’t that seem like overkill? I don’t even know how to use a chainsaw and while I can learn, it just seems completely unnecessary.

Architecture: But chainsaws are much better at cutting things than a knife.

Me: I’m not trying to cut through a log. I want to cut a cake.

Architecture: What are you going to do when you’re cake gets to be 10 feet tall?

Me: No one is ever going to make a 10 foot tall cake. It’s not going to happen. You clearly decided you wanted to use a chainsaw and now you’re making up hypothetical situations to validate your decision.

Architecture: Not ah.

Me: I’m taking my ball and going home.

Software is dominated by smart people doing dumb things. I would like to think that what sets me apart is that I am a dumb person doing smart things. There are infinite ways to take a relatively simple problem and provide a complicated solution. To be truly successful, a good engineer must break the problem down and provide a simple and maintainable solution no matter how complicated the problem. There are no bonus points for using a shiny object and even if it looks cool on paper, remember the Tesla door handle. It provides very little practical benefit besides looking cool. Eventually, everyone forgets about it until it stops working.

Tuesday, March 21, 2017

Tesla's Dirty Little Secret - They Suck at Supply Chain

“Hi this is Evan Zlotnick (pause) Zee as in Zebra, L, O, T as in Tom, N, I, C, K… Yes, Zlotnick. Yeah, I brought my car in four weeks ago to get an estimate. Well, that’s the thing, you never called me back. What do you mean, you don’t know when the parts will be here? Did you put in the order? You didn’t get a confirmation and expected delivery date? Look, I’m not a supply chain expert… Oh wait, I am! This doesn’t make any sense…”

That is an actual conversation I had with one of the body shops in Austin. Backing up a little bit, a year ago, I drove from Austin to Dallas and picked up my midlife compromise car. I was driving a 4 wheel drive Cayenne which seemed to make a lot of sense when I lived in Seattle, but in Texas, I no longer needed four wheel drive. When the air conditioning on it died and I was looking at spending another $1500 - $2000 on fixing it, making payments for a year, and then trying to sell it with over 100,000 miles on it for the amount I had sunk into it over the course of the year in the form of car payments - I did what every sensible person would do. I bought a new, used car.

I did my research. I had a list of requirements. The new car must be capable of getting all four members of my family in it, so the back seats had to be actually usable. It had to be fast. I needed a great stereo. Since I traded rain and snow for good weather, I naturally wanted a convertible. Lastly, I didn’t want to spend too much money.

While Austin is the 29th most populous city in the United States with limited inventory, within three hours are the larger cities of San Antonio, Houston, and Dallas. I decided to buy a three year old Mercedes E550. This underrated beast sports a twin turbo 407 horsepower engine, surprisingly spacious seating, reasonable gas mileage (22 mpg with me constantly flooring it), a kickass Harmon Kardon stereo, and my payments would be lower than I was making on the Cayenne.

It didn’t take long for me to fall in love with the Mercedes. With all the power and performance of a Porsche 911 and none of the stigma at a fraction of the price I couldn’t believe I was able to find exactly what I was looking for. Unfortunately, our love was not meant to be as six months later, my wife was rear ended at 45 mph. While my wife was unharmed, thanks mostly to the precision German engineered safety equipment, the car was not. It was eventually declared a total loss.

My first thought was to replace it with another Mercedes E550, but I was unable to locate a comparable vehicle in the Great State of Texas. I then looked at getting a used Tesla Model S. The used vehicle I eventually purchased was $15k more than the Mercedes and I had to do some mental gymnastics to rationalize the price difference. I figured I was spending roughly $200 - $300 a month on gas and the extra money spent on the new car payment would be offset by the savings I was about to receive from no longer needing gas.

All was well for about a week when the new, used Tesla got rear ended in a parking lot. While still drivable, there was a giant dent in the rear of my new vehicle. The insurance company provided an estimate and I went to one of only a handful of body shops in the area certified to do repairs on Teslas. Six weeks went by and after multiple phone calls, I never heard back from the shop. So, I brought it to a different shop. The new shop provides plenty of status messages to me and will return my calls, but the status is still the same. The shop is waiting for parts from Tesla.

The shop has gone so far as to provide proof that the part was ordered on 2/6. There is a confirmation number on the order, except in subsequent follow ups with Tesla, the response was, “We lost your order.”

It turns out that my experience of waiting on parts is pretty much par for the course. A writer for the financial news site, the Motley Fool, penned the seminal article explaining the situation:

The author, who had only a slightly more damaged rear fender than I did, waited a full eight months for his repair. In fact, these kind of waits for repairs seem common amongst Tesla owners. While it is inexplicable that someone could be kept waiting this long for a repair depending on simple parts, what is more perplexing is that these supply chain issues seem to be caused by Tesla themselves…

A few years ago, articles started popping up about how Tesla wrote their own ERP (Enterprise Resource Planning) software (Source). Rather than use the industry standard SAP or Oracle, Tesla decided to do it themselves. Heck, they got it done in just four months because, well, they’re Tesla and clearly they know what they are doing. But not really. Maybe I am a bit jaded but in all of my years working in supply chain and using SAP, I have never, not once, heard of an order “lost in the system”.

I had high hopes for owning a Tesla. I love the fact that it gets nearly 300 miles worth of range with zero gas. However, if the company can’t handle something as simple as taking an order and actually delivering on said order, everything else they do is moot. At this point, all I can do is wait, wish I still had the Mercedes, and wonder why anyone would be so arrogant as to write an entire ERP system from scratch.



Monday, February 13, 2017

Make America Great Again: Immigration

My grandfather left Russia to escape religious persecution. My ancestors have, literally, been kicked out of every decent nation in Europe. He landed on American shores and was refused entry. Eventually he was taken in by Canada, but legally emigrated to the United States several years later. He started a business, saved as much as he could, and spent his savings bringing over the extended family he left behind.

Decades after his journey, I met several of the people that his efforts and sacrifice directly affected. He was much beloved by all, so much so that I cannot tell the truth from the myth when it comes to this man that I have never actually met. I do know that the escape from Russia in the dead of winter had a prolonged effect on the man’s health, and unfortunately, he died at the relatively young age of fifty-six.

My uncle was raised in a house where English was not spoken and showed up to kindergarten, in a public school, confused and lost. Eventually, the family my grandfather formed on American shores assimilated as my father, nine years later, was raised speaking English and is not at all fluent in Yiddish. My grandfather worked at the only craft he knew (stereotype/irony alert), making bagels. He socialized amongst other exiled European Jews, talking in Yiddish, and visited the schvitzes (traditional Russian bath houses). My father left the family business and integrated more into the society around him. Eventually moving to Arizona far away from the large Jewish population he knew in New York. We rarely went to temple and I barely knew any other Jews.

While my father had a foot in each culture, I consider myself 100% American. I married an American woman and am raising American kids. I know that my last name was once Zlotnickov, which sounds far more Russian than the bastardized name of Zlotnick I now carry. When my future wife asked if I would change my name back to Zlotnickov my immediate answer was no followed by hell no. I was raised an American. While my family came from Russia, we were Jews, not Russians. Russia did not want us, so fuck them.

I have been to Ellis Island where my grandfather was denied entry. I am beyond proud to be an American and realize that the American story is shaped by several waves of immigration. Anyone who is not of Native American descent, is in fact, a descendent of immigrants as Europeans did not show up in North America until the tail end of the fifteenth century.

Somehow, our country and our messed up electoral college system selected a reality TV show host with delusions of being an actual “business person” to the highest office in the land. The current president received a “small loan” of $1M (you know, like most of us do) from his father to start his real estate business. He additionally received multiple guarantees and loans against his future inheritance (Source). Even after being born on third base and believing he hit a triple, the president has a long history of not paying contractors, being sued, and then settling for pennies on the dollar. He has been personally involved in 3,500 lawsuits (Source). His one sole publicly traded company ended in disaster; but not until it was ruined from the inside by payments of over $64M to Trump in salary and bonus, incredible conflicts of interests with his other businesses, and costing stock and bond holders $1.5 BILLION in the process (Source).

This charlatan and colossal narcissist ran and won on the slogan of “Make America Great Again”. Implying that it once was, but no longer is, great. A key component of what makes America no longer great, according to Trump and his ilk, is the murderers/rapists invading our Southern borders from Mexico and the influx of terrorists from the Middle East ready to attack Americans on our own soil.

Wasting no time, in his first week, Trump issued multiple executive orders. Amongst these orders were a ban on immigrants from seven predominantly Muslim countries and the process to begin building “The Wall” physically fencing off Mexico from the United States. Thus, Trump could say that he made good on his campaign promises and that he was “tough on immigration”.

This grandson of a Russian immigrant, only the second generation to be born on American soil, and a staunch believer of the American dream was thoroughly sickened and appalled. Green card holders and permanent residents were denied entry into this country with the president’s signature on a piece of paper. As citizens protested at airports en mass, the president tried to conjure up images of 9/11. Except that happened almost sixteen years ago and was completely irrelevant to the present day executive order.

As much as I sincerely believe the true purpose of the executive order was to stoke the flames of racial tension and create a perception that we are an imminent danger of a terrorist attack, there is a second component to the president’s tough on immigration act. The president has promised to reform the H1B visa program. As much as I despise the man, as much as I support immigration, and as much as our nation’s history of immigration has personally benefited me and my extended family; I hate to admit this, but he has a point.

WHAT IS THE H1B?

The H1B visa was is not about immigration, the permanent move by someone of a different nationality to a new country, but it does temporarily allow an alien to legally reside in the United States and work for a sponsoring company for a period of three to six years. The program was established with the Immigration Act of 1990. It provides a certain number of Visas to foreign nationals that is currently capped at 65,000 per year with a special provision for an additional 20,000 visas to applicants that have advanced degrees. Since the number of visas that are granted are capped, they are handed out via a lottery system.

Here are some key things to keep in mind…

  • The H1B visa holder may only work for the company that sponsored them
    • If they decide they no longer like their job, they cannot switch jobs unless another company initiates a visa transfer
    • While a visa transfer is, technically possible, it is rare in practice
    • Therefore, H1Bs typically may only work for the company that sponsored them
  • The vast majority of H1Bs are employed in technology
    • In 2014, the mean salary of an H1B employee in software development was 25% less than the average income for software developers (Source)
    • While the H1B program is supposedly in place to make up for a “skills shortage” amongst the US labor pool, 56% of the visas go to the lowest skill level (Level I) individuals
    • The top 10 employers of H1B visa holders are offshore “body shops” such as Cognizant, Infosys, and Tata Consulting (Source)
    • These offshoring companies flood the visa applicant pool. Their entire business model depends on grabbing up as many visas as possible and then selling bodies to US companies at cut rate prices.
    • The program is rife with abuse including, but not limited to, resumes being radically embellished and/or swapping people from the person who interviewed offshore to the person who actually shows up at a client site (Source)

Most H1B’s are employed in the technology sector which accounts for 6.7 million jobs (Source). At any time, there can be well over 1 million H1B’s in the United States. For argument’s sake, let’s say 1 million are employed in tech. That means that roughly 15% of jobs in the tech sector are going to a group of people that has no leverage because they cannot quit their job and they are forced to accept 25% less than someone without the same restrictions.

While tech titans from Bill Gates to Mark Zuckerberg insistently bleat about the need for more H1B’s, we need to acknowledge the program for what it is - a form of indentured servitude that lowers the cost of labor by bypassing the market driven wages created by supply and demand. If this seems to dwell into tin-foil hat conspiracy theory territory, keep in mind that five top tech companies in Silicon Valley recently were ordered to pay $415M to their employees as they conspired to prevent their employees from leaving their respective firms to go to competitors at higher salaries. The ringleader of this scheme was none other than Apple co-founder Steve Jobs (Source).

Tech companies are not desperate to raise the quotas because they cannot find employees qualified for the jobs they have open. They want the quotas raised because they cannot find employees qualified for the jobs they have open at the rates they are willing to pay. An almost brazen example of this was shown in 2015 when Disney laid off over two hundred tech employees, you know - the employees tech companies absolutely cannot find? Well, before laying off their employees, they had them train their replacements. Their replacements were contractors provided by Cognizant, one of the top ten abusers of the H1B visa in the United States. Of the over two hundred laid off employees, many with excellent job reviews and years of great service, only two were hired back by Disney. In one fell swoop, Disney managed to radically cut costs and lock in their labor pool as they no longer run the risk of having good employees quit (because they can’t), completely breaking the equilibrium that should exist between employees and employers, and lowering their operating costs (Source).

THE LIFE OF AN H1B

You just slip out the back, Jack
Make a new plan, Stan
You don't need to be coy, Roy
Just get yourself free
Hop on the bus, Gus
You don't need to discuss much
Just drop off the key, Lee
And get yourself free.”
  • Paul Simon

While this is anecdotal evidence, I have worked in tech for over twenty years. I have worked alongside plenty of H1Bs. On one particular project at Microsoft, I worked with many H1Bs that was located in a far flung building in Microsoft’s sprawling campus in suburban Redmond off of Willows Road. The Indian people on the project in question lived in some apartments near the building. They walked or road the bus to get around. Four guys were each crammed into a two bedroom apartment. They billed at $30 an hour which was way, way lower than my bill rate.

Seemingly out of the blue, Microsoft moved our space to a newly leased building called the Bravern near Bellevue Square. Most of us thought it was cool that we would now be in the Bravern as it put us in the heart of a hipper area in Bellevue. The Bravern was, well, way nicer. There were bars to go out to after work. There were nice restaurants nearby. While the swanky new digs in Bellevue seemed like a world away from the suburban, strip-mall feel of Redmond; in reality it was a few miles. My commute would be longer, but it would only be a matter of minutes each way.

I couldn’t help but see how dejected the Infosys folks were. I found out later that the office move of a few miles impacted their commute by an hour each way as it had radical implications on the bus routes they depended on. None of them owned a car, nor could they really afford one. While everyone else was happy because we had a nicer work space in a cooler part of town, there was a whole group unhappy with the situation and powerless to do anything about it.

Strangely, I had been in there exact same situation. I was contracting to a company and they offered a position as a salaried employee. I took a hit in compensation, but moved from hourly to a guaranteed (well, as much as a job in today’s world is guaranteed) salary. When I made the switch, I was working and living in Los Angeles. Nothing really changed. A few months later, that project came to an end and I wound up travelling to Sacramento for a new client.

I had travelled for work before and that usually meant a decent hotel and a rental car. This company was so phenomenally cheap and trying to hang on to every nickel that they rented a property known not affectionately as “the happy house”. I was stuck living and sharing a bathroom with my co-workers.  Awkward. Everyone else in the “happy house” was a Malaysian H1B.

Since the company hired from Malaysia, there were some cultural differences. Most Malaysians are Muslim - which is fine. Most Muslims do not consume alcohol and follow Kosher rules - again, this is fine. However, I am not a Muslim and I occasionally drink alcohol and do not follow Kosher rules at all. Except the “happy house” had rules prohibiting the consumption of alcohol or pork products.

So, here I was - a legal adult, forced to cohabitate with strangers, and unable to order a pepperoni pizza and have a beer to watch Monday Night Football after work hours. I did not like the work conditions at all. I expressed my disappointment in my living conditions. When I did not get the privacy that I expected on work travel, I exercised an option that my indentured co-workers did not have, I turned in my resignation.

The morale in the “happy house” was absolutely abysmal. No one wants to work all day and then go live with their co-workers. No one wants to have a fight with a co-worker about using all the hot water for the shower or argue about whose turn it is to take the trash out. These types of fights are reserved for our spouses. What made me feel truly awful for my former co-workers is that they didn’t have the choice that I did. They could not ask for better conditions or more money. They were replaceable. If they caused a problem, they could be on the receiving end of a one-way ticket back to Malaysia. As bad as things might have been here, it was preferable to returning to a third world country.

A lot of people complain about at-will employment. It can seem really cold hearted, but it works both ways. As an employee, if you do not feel like you are getting the compensation or work conditions you want, you have every right to seek employment elsewhere. This forces employers to treat employees a certain way. Granted, there are some employees the employers want to get rid of for performance, but finding decent people is hard. Most employers who are growing and healthy want to hold on to any employee who is productive. When a decent employee quits, it causes all kinds of internal problems. Therefore, it is in an employer’s own best interest to have policies in place that look after their employees. This is the reason most jobs give annual performance increases, bonuses, and offer benefits. It’s cheaper than losing someone and trying to replace them. But would a company give a performance increase to someone who couldn’t leave? I would say it would be very unlikely that they would. Without the power to quit, employers have all the power and that’s not how capitalism is supposed to function. There is an equilibrium point between both parties that ideally keeps both sides happy. If that equilibrium doesn’t exist, then abuses can and will happen.

EVERY SINGLE DAY
"Attracting entrepreneurs and human capital is critical to the tech community."
- Todd Schulte, president of fwd.us an organization started by tech titans including Mark Zuckerberg, Bill Gates, and Melissa Meyers in favor of radically increasing H1B visas

It is my sincere belief that the H1B visa has nothing to do with a so-called skill shortage and everything to do with being a cost control measure. As I am now approaching an age that is considered ancient in the tech industry, there is a near constant flood of low-cost (and of dubious quality) workers willing to do my job. And yet… My life is touched in a profound way by the companies founded by immigrants every single day.

I commute to work in a Tesla. The company that makes the car was founded by Elon Musk who was born in South Africa. At work, I use Google to help find information I need to do my job. Google was founded by Larry Page and Sergey Brin. Mr. Brin was born in Moscow and emigrated to the United States at the age of 6. These companies, founded by immigrants, are revolutionary and making significant impacts in creating jobs and changing lives of Americans. Sergey Brin did not steal jobs, he helped found a company that employees tens of thousands of workers directly. Indirectly, he has created hundreds of thousands of jobs as the income produced by each one of the Google employees was spread throughout Silicon Valley.

Elon Musk is, quite literally, trying to change the world. Tesla has proven beyond the shadow of a doubt that an electric car can be fun, exciting, and safe. Independent ratings firm Consumer Reports has called it “the best car they have ever reviewed” (Source). The kind of innovation and disruption on display from Tesla would have been unlikely from traditional American car companies. I dare to say that no one has ever been passionate or excited about the Chevy Volt. As imitation is the most sincere form of flattery, now that the standard has been set, other manufacturers are trying their best to compete with Tesla. By smashing through preconceived notions of what a car is and how to make it; Tesla is leading the way in making automobiles more environmentally friendly and bringing about an end to the United States addiction to oil.

Clearly, immigrants can and do make a huge impact on present day American lives. Clearly, immigration has long been a part of the American story. Hopefully, I have not come across as anti-immigrant. I would like to be thought of anti-immigrant abuse. There has to be a better system that allows qualified people to come to the United States, be productive, participate in the American dream, and raise American families.

The current system, plain and simple, is nothing more than a distortion to the laws of supply and demand in the labor force. So how do we get the best of both? How do we encourage innovators, dreamers, and skilled workers to become Americans, contribute to our society, and help shape the next century of American living?

WHAT IF?

What if… tomorrow we made every H1B holder a green card holder? What could possibly be the harm? Every single immigrant on US soil has already been vetted. They are here legally. They pay taxes, including Social Security - for which they will never receive benefits unless they become citizens. If every H1B suddenly had a green card, the only change would be that they would no longer be indentured servants to their sponsoring company. They would be considered permanent residents and no longer fear deportation for making their corporate overlords unhappy. They could change jobs, demand higher pay, and live better lives. The distorted supply and demand curve could be fixed that easily.

What if… Going forward, we allowed companies to hire foreigners? I am not against competition, I am against competition with indentured servants. I am against lotteries determining who the competition will be. I am against companies that have a business model of hiring lawyers and lobbyists to abuse the system. Why not let a company hire whoever they want? If the candidate that best fits the role is foreign born, we need a way to do a background check and allow them to immigrate, but with a fast path to citizenship and the freedom of a green card holder. I mean, after all, these are the supposed “best and brightest” that we are trying to bring over here, right? Let’s treat them accordingly. If a company had the ability to hire a foreign born candidate, pay their way over here, ensure a decent background check, and pay a fee (how about $50k?) to hire all the employees required for the background checking; let them. But the moment that employee shows up on US soil, they have a green card. If the job is not working out, let them negotiate with other employers. Let Adam Smith’s invisible hand of capitalism do its job.

I believe if these two changes were enacted there would be a significant improvement for people both American born and immigrants. It would bring an end to the body shop model. It would allow wages in the tech industry to stop stagnating (Source). It would give employers the flexibility to hire whoever they needed for their jobs without undercutting existing employees. Most importantly, for a country built on waves of immigration, it would be the right thing to do.

Monday, January 9, 2017

Cargo Cults

"Those who cannot remember the past are condemned to repeat it." - George Santaya

The very mention of the phrase "cargo cult" will elicit either a knowing smirk or confused ignorance from anyone who hears it.  There is no other known reaction to this phrase.  For those in the know, there is a certain condescending superiority in mocking the ignorance of an apparently primitive people. The sense of superiority is not earned as every person exhibits cargo cult behavior at one time or another.  Being able to identify this behavior is truly a rare skill.

For those who have never heard of cargo cults, prepare to learn an interesting quirk of history and, more importantly, how it applies in everyday life.  The cargo cult phenomena can be traced to the later part of the nineteenth century.  As Europeans and Americans began to interact with the islanders of the South Pacific, they misconstrued the mechanism by which the white people gained their apparent wealth.

To the islanders, Whitey performed certain rituals and the gods rendered unto them the gifts of material wealth or cargo.  Of course, the islanders desired cargo from the gods and began to copy the rituals that delivered the cargo.  The Islanders had been known to create mock airstrips, radios out of coconuts, conduct army field maneuvers, and design air traffic towers.  The fetishization of these rituals in the attempt to appease the gods is what makes it so comical and tragic at the same time.  It's laughable to think that talking into a coconut radio in a crudely constructed air traffic control tower beside a fake runway that has never seen an airplane will produce the desired result.

In addition to mimicking the rituals that produce cargo, the largest active cargo cult centers around the myth of John Frum.  John Frum is the god in their religion - a black man who has mastered the art of making cargo appear.  He had promised the islanders that he would return to them and bring bountiful cargo.  On the island of Vanuata, February 15th is known as John Frum day.  Every year, the islanders gather and conduct mock army drills carrying fake rifles with the letters USA written on their bodies anticipating the prophesied return of John Frum. 

Interestingly enough, the name Frum does not appear in any census in the English speaking world prior to the start of the cargo cult.  The theory is that the cult was founded by a GI who introduced himself to the natives as "John from America".  It's as if everything he said after that was completely misinterpreted by the natives.  John Frum day has been celebrated for over seventy years and John has not returned to the natives nor brought any cargo.  Still, year after year the celebrations continue and the belief persists that John's return is imminent.

Lest any Westerner feel too smug, it has been said that a participant in John Frum day was asked how can he go on participating in the drills waiting for John to come back when no one has heard from him in over seventy years.  The cargo cultist simply replied, "We have only been waiting seventy years for John to return and you have been waiting for over two thousand for your Jesus."  You have to admit, he has a point.  There are plenty of everyday examples of dogmatically following an idea without understanding why it is supposed to work.

In the United States, the diet industry is a $40 billion a year juggernaut.  In its simplest form, weight loss can be achieved by eating fewer calories than a person expends.  It really is that simple.  Yet how many people attend a Weight Watcher's meeting and then eat a Krispe Kreme doughnut?  Going to Weight Watchers is not a bad idea to get some support, learn some diet success tips, and learning about nutrition; but it will not result in weight loss in and of itself any more than speaking into coconut radios will bring cargo.  The meeting does not cause the desired result, it is the behavior modification that matters.

The mistake of thinking that attending the meeting results in weight loss can be found all over corporate America.  For example, the application of Six Sigma processes to software development.  Six Sigma originated at Motorola in the 1980's as a way to improve quality.  The desired goal was to have less than 3.4 defects per million units - a standard of quality so high that testing units would actually result in more defects than not testing.  For defined manufacturing processes where defects followed a normal distribution curve, Six Sigma provided great mathematical tools and concepts to improve efficiency.  The success of Motorola's methodology prompted other companies to start applying what was a manufacturing process to just about everything.  At General Electric, under the leadership of Six Sigma loving Jack Welch, Six Sigma methodology was applied to every aspect of the company.  Recruiters in HR were tasked with creating projects to lower the amount of days it took to hire a candidate for a job posting, which begs the question is it more important to hire someone quickly or to hire someone who is a perfect fit for the role?  Would it be better if the job was open for 30 days and filled with a mediocre candidate or if it took 50 days to find the right candidate?  With mis-applications such as Six Sigma projects for HR, the entire movement quickly became a corporate Cargo Cult.

Six Sigma has five distinct phases:  Define, Measure, Analyze, Improve, and Control.  New buzzwords were created to find out what the real problem to be solved for like root causes, voice of the customer, voice of the business, etc.  Templates were created to capture the information and an emphasis was placed on metrics.  In the HR example, the problem was open positions were not being filled quickly.  A goal was set to reduce this time and an elaborate way of measuring the opening's position was created.  Where the new process gets Cargo Cultish is the moment incentives are put in place is the same moment that people will start gaming the system for their own purposes.  What is in the best interest of the project manager, the HR recruiter, and all involved with the project may or may not be in the best interest of the company.  The project manager will claim that due to their hard work, the average number of days it took to fill a position fell by some percentage saving some amount of money (the savings will be based on a series of assumptions which are dubious at best, but the good thing is no one will ever ask for the numbers let alone challenge them), and he or she will probably get promoted.  The cost and time spent on the project will rarely be factored into the program's true cost and everyone will get a promotion.  Unfortunately, the new system will reward hasty hiring and not good hiring.  Mediocre candidates will receive job offers in order to not exceed the arbitrary goal of leaving a position open.  An analysis of the quality of candidates resulting from the new process will likely not be performed and all attention will be placed on the now critical (but once arbitrary) number of days a position was open metric.

At least when it comes to hiring, the consequences of this example are somewhat harmless.  It has been shown time and time again that the way a candidate performs in an interview situation has no correlation to how they perform on the job.  Where it gets really scary (and even more Cargo Cultish) is when processes like this are applied to cost cutting and used to define Key Performance Indicators (KPIs), and worse circulated in executive scorecards.

The Dreaded KPI and the Birth of Offshoring

The offshoring movement has been going on for decades.  Around the year 2000, it picked up momentum starting with call center jobs.  In theory, it was cheaper to hire people in India to handle customer support calls from America.  Since the consumer in America was calling, not interacting face to face, why not send the call to a country where people speak English (sort of), did not have unions, did not make demands for health care, and were willing to work for a fraction of what an American would ask for?  Call centers sort of work to drive costs down and so long as consumers stick to the script, it kind of works.  Sure, investments must be made in building out the infrastructure, rigorous training, linguistic classes to minimize accents, and management.  However, the workers in the call centers eventually believe they are valuable and have no loyalty, so turnover is high.  To counteract high turnover, wages go up, bringing a price equilibrium to about the cost of keeping the job onshore, but like I said, it kind of works for call centers.  It's low end work and at the end of the day, regardless of where the work is done, no one is ever really happy with the resolution that is provided.

Since offshoring call centers sort of worked, there has been a movement to offshore all kinds of other work on the premise that it will be cheaper.  Take, for example, programming of large-scale enterprise systems applications.  In the fury of systems integration work leading up to the Year 2000 and the dreaded Y2K bug, it was said that implementing Enterprise Resource Planning systems was "the corporate equivalent of a root canal".  The projects were complex and very expensive.  In an effort to lower costs, companies started to offshore what was thought to be low level tactical work.

My theory is that someone took a look at the hourly rate that a programmer was making in 1999.  Let us say, for example, billing rates were between $150 and $200 per hour.  To a manager making less than a six figure salary annually, this looks like an exorbitant number.  Most consultants or programmers who were billing this hourly rate were receiving only a fraction of it, but the manager paying the bill could not comprehend how this person they hired was making nearly $400k per year.  An Indian offshoring company walks in and says we will provide you with programmers and we will charge you $20 per hour.  The manager has no problem paying someone forty grand a year and it seems like a bargain.  Instantly, their bill was lowered by 90%.

Unfortunately, the adage you get what you pay for has never been more true.  In systems integration work, the real problem is rarely writing the code.  The problem lies in understanding what exactly the end user really wants as, often times, it is not what they say they want.  Users, Subject Matter Experts, Customers, Clients, Stakeholders, Business Managers or whatever you call them do not think in terms of code, tables, or design.  They think in terms of high level business scenarios and think a fully defined specification is, "JUST convert the purchase orders from our legacy system." (A real spec I have actually handled in a previous life)

Please note, the above "specification" makes no mention as to what validation rules must be performed on all purchase orders, what tables the legacy purchase orders are stored in, how to map any differences in data between systems, etc.  The very high level and non-actionable requirement is then sent offshore to be acted upon by a software engineer who barely speaks English and has zero practical business experience.  The engineer will predictably do one of two things, he or she will either freeze like a deer in headlights or make wild assumptions and code away merrily.

Sadly, the deer in headlights scenario is the lesser of the two offered evils.  Even though no progress is being made on the activity, the engineer will always report that their status is green and there are no problems.  Culturally, it is career suicide to raise one's hand and say, "I don't get it."  Eventually, the deadline will approach and the onshore program manager will suddenly find out that no progress has been made in weeks.  Panic will ensue, but thankfully, real requirements will be created, a new deadline will be established, and eventually the requirement will be satisfied.  Since no code will be written prior to the panic, there will be no fundamental re-architecting or regression testing required.  In this case, doing nothing with bad requirements is far more preferable to making speculative guesses and reworking the solution once requirements are known...

Which leads us to the alternative to doing nothing.  Doing something, paradoxically, is far, far worse, in this scenario, than doing nothing.  The engineer that does something has no relevant business background or understanding of the rules governing the requirement they have received.  In order to feel productive, they simply make up business rules.  When asked for a status, the engineer says that everything is going fine.  When the deadline comes, sure enough, the legacy purchase orders are converted but the actual results are no where near the expected.  Panic ensues and all the business rules governing the conversion are then captured.  The engineer then has to redo all their work, but rarely starts with a clean slate and instead tries to shoehorn in new requirements on top of their made up requirements resulting in buggy and overall poor code.

So where are the savings?  The hourly rate now moves from $200 per hour to $20 per hour.  However, since engineers are now available at bargain basement rates, usually five offshore resources are hired to do the work of what would have been done by one onshore resource.  With this in mind, the cost now is $100 an hour.  But wait, there's more - it usually takes the offshore team twice as long to get the job done when you account for the do nothing and then get requirements, or do something and redo it all over again methodology. Now, we're right back to we we started.  There is no actual savings taking place, the work takes twice as long, and it requires a lot of sacrifice on behalf of the business users.  Their lives will be disrupted by conference calls in the middle of the night, multiple efforts to give requirements to people who do not understand your requirements, and hours pouring through documents that appear to be in English but in fact are not.  When factoring the cost of the extended team's increased commitment to the project the costs begin to skyrocket.

It all comes back to the KPIs and the Scorecards...  A General Manager can report to their Vice President during their review that they have reduced costs.  The GM can say (and how they can do this with a straight face is beyond me) that they have reduced the cost of development from $200 per hour to $20 per hour.  VPs being VPs never think to ask the next question and ask what the overall program costs are with onshore versus offshore or if the number of resources and hours spent in development were kept the same.  The VP shakes their head and congratulates the GM.  No money was saved, the extended team is burnt out, the code sucks, but the GM gets a big raise and that's what really matters.

Vendor Consolidation and Basic Economics?

As if offshoring is not bad enough, large companies have found a different way to screw themselves and their onshore partners in the process.  Procurement people who are used to making consolidated purchases of materials decided that consulting resources could be purchased in the same way.  Analysis was done on just how much money was being spent on consultants and an across the board number was thrown out, probably so it could be tracked on a scorecard.  A bold proclamation was made, such as cut vendor spend by 15%, or else.  Procurement then decides that since the company is big, they can consolidate to a handful of agencies and give them all their work.  In exchange, the agencies will reduce their rates.  At this point, anyone who has had any experience in dealing with consulting companies, staffing agencies, marketing companies, etc. should burst out laughing.  The problem is, people are not commodities.  The right consultant is worth their weight in gold while the wrong consultant is nothing more than a corporate parasite.

The process starts by procurement, who has never hired consultants, taking a look at the companies that provide consulting services.  From there, they make a list of arbitrary requirements as to who should be included or excluded from the final list of "approved vendors".  If any vendor company finds that they are not on the list, they begin an all-out campaign to be included.  Once the list has been established, it is nearly impossible to be included on future versions.  For those who have managed to make the list, a game of politicking begins to see how many competing firms they can convince procurement to kick out in the future.

Before examining the theoretical cost savings that centralized procurement may or may not provide, it is worth a brief examination of how consulting companies work. They hire a person and pay them $X/ hr (even if it is an annual salary, this still applies - just divide the annual amount by 2000 hours and voila - $X /hr). They then charge a client $Y /hr for client services. Their profits are then $Y - $X x billable hours. No matter how complicated some firms try to make it. Just as every fast food restaurant claims to have a secret sauce and uses mayonnaise and ketchup, it’s the same with consulting firms.

With that in mind, what should people expect if the variable Y now takes a 15% hit, competition is all but removed, and most consulting firms have a KPI for margin? Obviously, X goes down with Y which results in attracting lower performing people. Just as with offshoring, the lower rate now requires more people and more hours resulting in no cost savings, increased risk, and a host of other problems. However, if you focus on the pure dollar per hour cost, it has been reduced.

I have seen this approach used at multiple Seattle based companies. Consulting firms that had been in existence for less than two years magically got on the Approved Vendor List (mainly by kissing the right rings in procurement) and started flooding the market with twenty-four year olds freshly laid off from the now defunct Washington Mutual. However, work still had to get done by someone and with no real competition, they were able to get fat and happy all in the name of cost cutting.

Back to Cargo Cults

“Everyone wants out of the box thinking, right up until you give it to them.” -Me

These are just a handful of observations I have made in my two decade long career as an IT professional working as a Business Analyst, Program Manager, Project Manager, Software Engineer, and Solutions Architect as well as armchair economist (not a professional - just a top seeded amateur). Every observation should be apparently obvious, but somehow isn’t.

Given that making software is difficult, expensive, and risky; maybe it’s time to stop doing things simply because that’s the way it’s always been done. Methodology has moved from the waterfall approach to agile, but neither guarantees results.

“That was a great use of my time!” said no software engineer ever following the daily standup.

Just as performing incantations into a pair of coconuts fashioned into a radio will not bring cargo, neither will holding a daily standup produce great software. Process can be a good thing and can be used to keep teams on track. However, every time something is measured and someone is held accountable to a metric, behavior will change to achieve the desired metric. This behavior may be a good or a bad thing, but management by metrics is both lazy and dangerous. Unintended consequences have to be sought out and accounted for, clear vision and leadership provided, and cost cutting can make things more expensive.