A Guide to Hiring Programmers: The High Cost of Low Quality

I was invited to a wonderful dinner party (I swear it wasn't too spicy Sarah!) with some St. Louis Perl peoples this week while I'm here on business.  At one point we were talking about hiring programmers, specifically Perl programmers.

We agreed on the following:

  • Finding good programmers is hard in any language.  And that a good programmer can be as effective as 5-10 average programmers.
  • Average pay rates between equivalent programmers are out of sync and are based more on the language used than the skill of the programmer.
  • You don't need to hire an expert in language X, you can and should look for expert programmers that are willing to learn language X. An expert can easily cross over from being a novice in any language in a matter of a few weeks.
  • You should seriously consider allowing your expert developers to telecommute full-time. Restricting your search to programmers who live in your area or are willing to move limits the talent you can acquire. Arguments regarding "face time", productivity, etc. can easily be nullified when you look at how some of the largest and most successful Open Source projects such as Linux, Apache, and Firefox are developed by individuals rarely living in the same time zone or even country.
  • We love Perl and think it's a great language that you graduate to after you have been forced to use less agile languages such as Java, C/C++/C#, etc. Not necessarily a first language you get your feet wet with and then move onto a *cough* "real" language.

Many people in the Perl community have been writing on this topic lately and wanted to share my opinions on the subject, as it is one I have put many hours of thought into. Doing my best to keep this language agnostic as I believe these tips can be applied to any programming language. I will however, use Perl in some examples as it is my preferred language.

Why is it so hard to find good programmers?

The simplest reason is when a company finds a good developer they do more to make sure that person is happy which leads to longer tenures. Better salary, more flexible working conditions, good tools, interesting projects, and better perks can often keep a programmer working for you longer.

Another obvious reason is that experts in any field are small in number, so your possible talent pool is limited. This leads managers and HR departments to settle for average or even below average developers.  I believe this is the single biggest mistake a technology oriented company can make, regarding developers, just short of not using a good version control system.

We're not talking about customer service representatives or sales people here. Just having a body to fill the seat is not, I repeat not, always a win for the company. Sub-standard programmers drag down the efficiency of your other developers with beginner questions, poor comments/documentation, and bad code that someone else will later be forced to spend time fixing.

Companies need to stop thinking about their developers as cogs in the machine. They are more akin to artists, authors, designers, architects, scientists, or CEOs.  Would your HR department rush to find the first person who would be willing to take on the role of Chief Scientist, Art Director, or CEO in your company? Of course not, they would spend the time to do a thorough talent search for just the right candidate, court them, and then compensate them appropriately. They realize that having the wrong person in that seat is much worse than having the seat empty. It is absolutely the same with programming.

Anyone who has been a developer or managed developers can tell you that an expert can accomplish as much as 10 average developers.  However, companies typically pay only a 10-20% premium for an expert over the average programmer. Whether or not their title is Lead, Architect, Development Manager, Guru or whatever nomenclature the company uses. I am not saying that if your average developer is paid $50k/year that you should pony up $500k/year for an expert. The employer/employee relationship never works like that, but what employers don't seem to realize is that in the end paying more saves them more.

Let's look at an example.  One common argument from HR departments is that they "can't find any Perl programmers, but they can't swing a cat without hitting a Java developer".  While this is fairly accurate, they are approaching the problem from the wrong direction.  If you fill your shop with 15 average Java developers, paying an average of $60k per developer you have an approximate labor cost of $900k/year for your development staff.  Not considering any non-salary benefits.

Suppose you instead took the time to find 5 experts, or at least above average, Perl developers at $120k each per year. Here is a partial list of the pros and cons of such a scenario:

Cons:

  • You must spend extra time finding, evaluating, and courting these more sought after developers.
  • Your company and what the developer may be asked to build may simply not be attractive to this class of developer.  Very few people want to work for a spammer or a small web design firm that caters solely to freelance accountants for example. Smart people find boring things even more boring than the masses.
  • When one of them leaves the company, there is the feeling that your company's business objectives are more at risk due to having only 4/5ths of your normal resources. Or that a larger chunk of your corporate knowledge just walked out the door. This is more of a perceived problem than an actual one as good developers are better at writing readable/maintainable code, commenting their work, and writing effective documentation.

Pros:

  • Each developer will be more content with their job, due in part to the higher than average salary, but also because his or her co-workers are of a much higher quality which improves anyone's job satisfaction.
  • Development would require less overall communication as there are less people to communicate with.  This obviously improves efficiency as anyone who has been on a 20+ person conference call can attest to. Or read the Mythical Man Month if you want a more in-depth analysis of this phenomenon.
  • Experts travel in the same social circles.  Having one expert on staff makes it much easier to find other experts in the same field, no matter what field that may be.
  • You would save 2/3rds on infrastructure costs.  Things like cubicles, computers, cell phones, free lunches, training costs, travel, office space, air conditioning, electricity, etc, etc. The list is essentially endless.
  • Your HR department would have 1/3rd the number of developers that it would need to take care of. Less paper work, less questions, less everything, and less turn over because of the lower number of employees.
  • Oh and you'd save $300k/year on your labor costs.  Not to mention non-salary benefits such as stock options, retirement matches, health insurance premiums, perks, etc. You could spend as much as $100k/year on your talent searches and still be $200k/year ahead.  Hell, you could dedicate an entire HR person just to this task.

What is an expert programmer?

Experience is key, but not necessarily in ways you might imagine.  Time in the saddle, with a particular language is not as important as diversity of experience.  Someone who has worked in several disparate industries, a generalist, is often a much better developer than one who has spent years in the same industry.  There are exceptions to this, but in general I have found this to be the case.  Bonus points if your developer was a systems administrator in a former life.

Some of the best developers I know were originally trained as journalists, mathmaticians, linguists, and other professions not normally associated with software development.

Experts use better tools and care deeply about their craft.  They aren't assembling bits on an assembly line, they are crafting a unique product to solve a unique problem.  Experts are lazy, they work smarter rather than harder.  Experts prefer the easiest solution that gets the job done. Experts aren't interested in creating complex solutions simply to have the complexity, that misguided egoism is the territory of more junior developers. They often get it right the first try and almost always on the second one.

Simply put, experts write readable code.  They comment and document it appropriately based on the complexity and criticality of that particular piece of code.

All of this pays huge dividends when the next developer has to pick up where they left off. Especially if the next person isn't an expert.

More reasons you want an expert programmer

Is your business technology oriented?  Perhaps the software you create is even your main product. If nothing else I'm sure we can agree that if the software your developers create is to some degree critical to your business.

I've worked in many different environments, with people of every skill level, and it's very easy to tell whether or not a company has expert developers. Do you often find that the software is down? That it has as many bugs or even just idiosyncrasies that make no sense to the user as it does features?  Do the users find it difficult to use?  Is the problem at hand relatively simple compared to the training or documentation necessary to begin using the software?

If you answered yes to any of those questions you more than likely have average or below average developers.

When you work in an environment with experts things simply work.  They are easier to use and require less initial training. The software is easier to modify.  Requested changes happen more frequently and easily.  Things just flow.  It is the difference between Apple and Microsoft.  It's the difference between the iPod and a 400 disc CD changer with 50+ buttons.

As with many things in life, sometimes you get what you pay for. I'd love to hear your comments and opinions on the subject.

UPDATE: I've written a response to some of the questions and comments I've received on this article in a follow up post A follow up to "A Guide for Hiring Programmers"

Tags: business, Featured Posts, perl, programming

Comments

just putting a sales guy in a seat is not a win for the company either. who do you think generates the revenue to allow the company enough funds to hire an 'expert' developer? this was a good write-up, but don't invalidate the effort of the sales team; we make sure the bills get paid.

but we can all hate on hr :-)

by sales guy on Aug 5, 2007 at 6:29 AM

Money alone cannot be motivating factor when it comes to retaining the best talents. Software industry has been plagued by high attrition rates. No company can win the business game with one department even if it means that department is most productive.

by kalpesh on Aug 6, 2007 at 1:52 AM

"The High Cost of Low Quality" really applies to ALL fields. However high quality does demand a higher cost, just not as high as low quality and most companies are reluctant to spend more than the least possible.
As a chemist I would catch mistakes that my peers were routinely making and cause problem because of "backtracking" to fix the problems.

by Maxdogg on Aug 6, 2007 at 2:41 AM

Good Post. The point for job seekers is look for a Company not a Job.

http://jobhacks.wordpress.com/2007/08...

by rem on Aug 6, 2007 at 3:38 AM

Good points - except that Perl is passable only for the most *cough* simple of applications. For scale, managing complexity, readability, nothing beats Java.

by Peet on Aug 6, 2007 at 4:25 AM

You couldn't be more right. A bad programmer is almost worse than no programmer.

by OneYearGoal.com - $100,000 in one year on Aug 6, 2007 at 4:39 AM

I need 10 Outstanding Perl Developers to work for an investment Bank in NYC. We're probably looking at 100K ++. Hopefully some Experience in the Mortgage Business (Application Development for Mortgage Backed Securities- Analytics and High-Speed Trading Systems)

This is going to be the opportuinity of a lifetime for someone.. (It'll certainly make your career) Exected Comp over the Next 5 Years is 1 Million+. No Joke. If you rock at PERL, and have Financial Industry KNowledge/Experience (Or just know alot about Mortgages) email me: agamache@optionsgroup.com
{
Compile your future.
AG
}

by Andrew Gamache on Aug 6, 2007 at 4:40 AM

We had a perl "programmer", he was very fond of the game perl "golf" and tried to play it with our valuable assignments. We got rid of his stupid ass before he had much chance to cause trouble, and from now on we only hire programmers skilled in the dot net toolset, and its served us very well.

by lol @ perl on Aug 6, 2007 at 5:01 AM

So, how do you find expert Perl programmers?? We're always looking!

by David F. Skoll on Aug 6, 2007 at 5:11 AM

Thank you for this post.

I was brought on my current project (non-Perl) when it was six months overdue. The former programmers were novices at best (more likely incompetent), which meant my job entailed progressive rewrites of their code. And, no, I wasn't being snotty about the code. They had used two (sometimes more) functions instead of arguments. BTW, one of those developers gave his notice as soon as I asked for his source code.

The VP didn't want to move me to the project because I was, "too expensive."

In two years, these guys had produced 400 "units" (purposefully vague) for delivery, or 16/month. In less than two months following the restructuring, I and a budding-expert developer produced more than 100, or 50/month. That's better than 3x previous production.

Yet, somehow, I doubt those other guys cost 32% what I did.

[Note: I wouldn't consider myself "expert," but I have a lot of experience working for a variety of clients, and I do try to do things right the first time.]

by JB on Aug 6, 2007 at 5:11 AM

>You don't need to hire an expert in language X, you can and should look for expert programmers
>that are willing to learn language X. An expert can easily cross over from being a novice in
>any language in a matter of a few weeks.

A common notion, but wrong.

Sure, any competent programmer can learn the syntax of a new language in short order, and most of the uber-concepts of programming do not change from language to language. But the *details* of the language take time to learn, and a program is made of details. How do I implement a menu control? How do I listen for mouse events? Is there a maximum string size? Are chars signed by default? Many of these details might be tiny and easily-learned, but there are a LOT of them, and a person who is just learning a new language has to look up most of them. That takes time (and BTW, if you're just learning a language, how do you know whether that example you looked up is well-written or not?). Further, some concepts and details just don't translate into a different language.

Really, the proper learning time to consider is not how long it takes to learn the syntax, but how long it takes to learn a useful subset of whatever standard library the language uses to perform useful tasks. I spent years getting familiar with all the details and gotchas of MFC before I felt confident calling myself an expert. Now, I've been dumped into the Java world, and suddenly, there's an entirely different big, bloated, buggy standard library to learn. AND the new concept of cross-platform issues to deal with.

In MFC, I know how to do lots of useful stuff just off the top of my head. Further, I know where many of the gotchas and pitfalls are, and how to avoid them. I know how to work the IDE. I know none of that for Java, and it will take more than a couple weeks to pick it up, no matter how diligent I am.

It is my considered opinion that no small, agile company on a tight schedule should attempt to write code in any language unless they have at least one bona fide expert in that language on staff, somebody who has spent at a minimum 6-12 months writing almost exclusively in that language (of course, I mean *in addition to* having several years of solid general programming experience).

Anybody who tells you they can learn a new (full-featured) language in a few weeks is either lying, deluded, or unclear on the concept of "learn."

by Dave Palmer on Aug 6, 2007 at 5:12 AM

>You don't need to hire an expert in language X, you can and should look for expert programmers
>that are willing to learn language X. An expert can easily cross over from being a novice in
>any language in a matter of a few weeks.

A common notion, but wrong.

Sure, any competent programmer can learn the syntax of a new language in short order, and most of the uber-concepts of programming do not change from language to language. But the *details* of the language take time to learn, and a program is made of details. How do I implement a menu control? How do I listen for mouse events? Is there a maximum string size? Are chars signed by default? Many of these details might be tiny and easily-learned, but there are a LOT of them, and a person who is just learning a new language has to look up most of them. That takes time (and BTW, if you're just learning a language, how do you know whether that example you looked up is well-written or not?). Further, some concepts and details just don't translate into a different language.

Really, the proper learning time to consider is not how long it takes to learn the syntax, but how long it takes to learn a useful subset of whatever standard library the language uses to perform useful tasks. I spent years getting familiar with all the details and gotchas of MFC before I felt confident calling myself an expert. Now, I've been dumped into the Java world, and suddenly, there's an entirely different big, bloated, buggy standard library to learn. AND the new concept of cross-platform issues to deal with.

In MFC, I know how to do lots of useful stuff just off the top of my head. Further, I know where many of the gotchas and pitfalls are, and how to avoid them. I know how to work the IDE. I know none of that for Java, and it will take more than a couple weeks to pick it up, no matter how diligent I am.

It is my considered opinion that no small, agile company on a tight schedule should attempt to write code in any language unless they have at least one bona fide expert in that language on staff, somebody who has spent at a minimum 6-12 months writing almost exclusively in that language (of course, I mean *in addition to* having several years of solid general programming experience).

Anybody who tells you they can learn a new (full-featured) language in a few weeks is either lying, deluded, or unclear on the concept of "learn."

by Dave Palmer on Aug 6, 2007 at 5:12 AM

You have convinced me about the need to hire experts but how do I figure out if someone is an expert or not?

I am going to be hiring a few programmers in the near future but am not a professional programmer myself. I have some knowledge in sys admin and web development but know little to nothing about more complex languages like C, C#, C++, Java, etc.

How should I go about determining if someone is an expert. What kinds of questions should I ask and what kind of responses do I want.

Thanks for your post!

by Ethan on Aug 6, 2007 at 5:20 AM

Hmm, I see a void forming...

How do we get expert developers? It seems to me that all expert developers were once average developers. If not all, then I would say most. I'll admit that there are exceptions, geniuses. There are also those who got to be experts by working hard as average developers to improve their skills. This is where I see a void forming in your article.

I agree that an expert work force is ideal. I am saying that that ideal isn't really feasible, because it leads to elite-ism. I would say that the ideal is to have a few experts and several very eager average developers. People who will willingly submit to constructive criticism and learn to better discipline themselves, so that they can become expert developers. That road is messier, but it also leads to more expert developers.

If we are to ever meet your ideal of an expert workforce I think it must be by following this road. I say this because I know that there is only so much time that a person will devote to full time education before seeking real world experience. I think we are lucky to have a few CS Masters graduates that are expert developers.

Perhaps I am a little bias on this last point. I am a Systems Administration Bachelor graduate myself, who has been tasked with a few programming projects. I consider myself a fairly eager amateur, and had I tuned by education to software engineering I might go as far as to call myself average. If I am now an average developer it is only because I am so eager, and have been trying to teach myself the software engineering skills that you have in part described in this article.

To summarize, I think that society can only effectively handle putting people in to so much full time education. There has to be a point where real world experience becomes that teacher, along with the tutelage of the experts. However, this sort of path of advancement is circumvented by what I see as an elitist mentality of only hiring the experts.

by Mikey on Aug 6, 2007 at 5:21 AM

I think, this applies just the same to many other professions. Sysadmins for sure. Probably -- other engineers too...

by Міша on Aug 6, 2007 at 5:24 AM

AMEN - I quit progamming after 20 years not because my skills went stale but because my experience and expertise were not being valued. It seemed that additional experience was not rewarded. Companies would hire 5 junior programmers instead of 2 senior programmers. I predicted that this would come to a crisis in code someday. This article tells me that it is about to.

by Susan M on Aug 6, 2007 at 5:30 AM

This is a great post. Thanks for putting my thoughts down on paper :)

by OJ on Aug 6, 2007 at 5:38 AM

As the size of the project grows, the less likely it is that the tradeoff of a few highly skilled people vs. many lesser skilled people holds. If for no other reason than larger projects have a lot of uninteresting code (e.g., a lot of the UI) that isn't amenable to an expert finding an easy, compact, elegant solution. This is one of the factors that makes estimating software projects so tricky. On the plus side, hiring a few highly skilled programmers for a project that eventually requires a lot of uninteresting code is a correctable mistake. The experts come up with an elegant solution and it's possible to bring some additional people on to flesh out the rest of the program. Hiring a bunch of mediocre programmers for a project that needs a few experts usually results in a need for more people and a longer schedule with the resulting project being a bug infested piece of unmaintainable bloatware (think Microsoft products).

Cheers,
Dave

by David G. Miller on Aug 6, 2007 at 5:48 AM

"Anyone who has been a developer or managed developers can tell you that an expert can accomplish as much as 10 average developers."

One of the least fact based generalizations I have ever read. I believe an "expert" writer could have gotten your point across with 10 times less verbiage than you.

by Bad data on Aug 6, 2007 at 5:48 AM

Thanks for spelling out a lot of what I have intuited but not formalized in a decade of working with and managing programmers.

by Robert on Aug 6, 2007 at 5:50 AM

Part of me thinks this article is more sour grapes about low perl developer salaries vs. java developer salaries, otherwise why mention java vs. perl at all and instead, say, "average x programmer vs. expert x programmer" .. then I think your point would be more clearly heard. The problem we have in the java world is overpaid average programmers.

Honestly, salary is more about demand, and less about skill. Technical merits of either language aside, Java is a high demand language with a vast array of available cool projects. Perl not so much. Most perl jobs I've come across have been maintaining someone's glue code. I only know one Perl programmer with a 6 figure income, and his project is being converted to java. I know several Java developers with 6 figure incomes.

But here's another thing, I'll take 4 good programmers who can work well together, vs. the 1 hotshot, because the hotshot is the one that's going to have a huge ego, be a pain in the ass to work with, and end up hurting the company more than he helps. Ideally, I want an expert who can work well with people, and raise the skill level of his team members.

by Alex on Aug 6, 2007 at 5:50 AM

I agree with most of the article. Unfortunately, most of the people doing the hiring don't get it. It is difficult being an expert in a company when there are no others, even worse if your superiors don't recognize the expert. I love the bit about experts being lazy. I tell my boss that all the time, when it comes to my design decisions and implementations...I don't want to revisit the code, and if I do, it is well documented, flexible and extensible. To the expert, it's just a language, architecture, operating system, etc. The expert has full understanding of languages, architectures, operating systems, programming, etc.

I have to totally agree with the simplicity. Much too often I see "engineers" trying to reinvent the wheel. Many times, what you are trying to do, someone else has had to deal with too and the solution is out there (many times it has many hours of use, debugging, etc behind it). Simple is always better. If I want to have fun with the egoist guys, I show my full understanding of the grammar of the language and write totally obfuscated code that is both syntactically correct and grammatically correct.

I know this article was about hiring, but you have hit a sore nerve with me about the state of software development, how it is handled and how it is approached by companies.

I'll stop rambling now.

by Foo, the Cal Poly Computer Engineering grad on Aug 6, 2007 at 6 AM

I agree with most of the article. Unfortunately, most of the people doing the hiring don't get it. It is difficult being an expert in a company when there are no others, even worse if your superiors don't recognize the expert. I love the bit about experts being lazy. I tell my boss that all the time, when it comes to my design decisions and implementations...I don't want to revisit the code, and if I do, it is well documented, flexible and extensible. To the expert, it's just a language, architecture, operating system, etc. The expert has full understanding of languages, architectures, operating systems, programming, etc.

I have to totally agree with the simplicity. Much too often I see "engineers" trying to reinvent the wheel. Many times, what you are trying to do, someone else has had to deal with too and the solution is out there (many times it has many hours of use, debugging, etc behind it). Simple is always better. If I want to have fun with the egoist guys, I show my full understanding of the grammar of the language and write totally obfuscated code that is both syntactically correct and grammatically correct.

I know this article was about hiring, but you have hit a sore nerve with me about the state of software development, how it is handled and how it is approached by companies.

I'll stop rambling now.

by Foo, the Cal Poly Computer Engineering grad on Aug 6, 2007 at 6:01 AM

I agree with most of the article. Unfortunately, most of the people doing the hiring don't get it. It is difficult being an expert in a company when there are no others, even worse if your superiors don't recognize the expert. I love the bit about experts being lazy. I tell my boss that all the time, when it comes to my design decisions and implementations...I don't want to revisit the code, and if I do, it is well documented, flexible and extensible. To the expert, it's just a language, architecture, operating system, etc. The expert has full understanding of languages, architectures, operating systems, programming, etc.

I have to totally agree with the simplicity. Much too often I see "engineers" trying to reinvent the wheel. Many times, what you are trying to do, someone else has had to deal with too and the solution is out there (many times it has many hours of use, debugging, etc behind it). Simple is always better. If I want to have fun with the egoist guys, I show my full understanding of the grammar of the language and write totally obfuscated code that is both syntactically correct and grammatically correct.

I know this article was about hiring, but you have hit a sore nerve with me about the state of software development, how it is handled and how it is approached by companies.

I'll stop rambling now.

by Foo, the Cal Poly Computer Engineering grad on Aug 6, 2007 at 6:01 AM

"Some of the best developers I know were originally trained as journalists, mathmaticians, linguists, and other professions not normally associated with software development."

Sorry, don't agree with this statement, nor the one about Perl being some great language. Every language has it's place, and you need to use the right one for the job, which a whole course was dedicated to in my BSCS.

One way to identify these develpers is to ask them what tools THEY have created to save time and effort.

by Mycroft on Aug 6, 2007 at 6:03 AM

"They are more akin to artists, authors, designers, architects, scientists, or CEOs."

I appreciated this line. Poor draftmanship of architecture drawings causes a lot of grief and time spent chasing the error in all it instances. What's worse is if the error isn't caught and affects construction. It can result in possible failures and lawsuits.

In regards to programming, we had one individual modify AutoCad in significant ways with no documentation. Suddenly he is no longer with the company, and there is no one that knows what he did. Those of us in the company who use AutoCad are now struggling to change things to suit our needs, but none of us are programmers by trade. We do our best and have found some loopholes that we can use to bypass his modifications.

Documentation is very important! Not all of us have time on our hands to decipher what happened.

by Jesika on Aug 6, 2007 at 6:06 AM

One thing I find very wrong with this article is it continues to perpetuate the myth of the hero. First of all a hero is a double edge sword. The down side of the hero is that they usually have a very narrow domain of expertise. This might be fine in a mega-corp, but most shops need smart flexible jack-of-all-trades type to survive in a constantly shifting competitive world. In other words experts/heroes can be anchors tying you down to one approach or one technology.

“Experts are lazy, they work smarter rather than harder.”

There’s a mythical statement if I have ever seen one. I thought the whole “work smarter not harder” mantra went out of style when the buzzword “fruition” died 8 years ago. This author is so confused they don’t know which way is up! Most of your “lazy experts” are just specialization whores. They have a very narrow band they work in. They are “all problems are solved by using X” types. They are NOT necessarily bright or talented. They live and thrive in a small pond.

"Anyone who has been a developer or managed developers can tell you that an expert can accomplish as much as 10 average developers."

Not the places I have worked. I would say an expert can at times generate up to 3 times a much output, but 10 times is ludicrous.

I’ve worked in top 5 software companies and I can tell this author hasn’t.

There is a lot more to software economics than mythology. I hope some day industry professionals wake up and get serious.

by BeenThereDoneThat on Aug 6, 2007 at 6:12 AM

The place I started at a year ago (OpenAir) is a text-book implementation of points made in this essay. Best job I've ever had, for all the reasons mentioned here.

And we're hiring. http://jobs.perl.org/job/5848 or click the link below.

by Pete Kruckenberg on Aug 6, 2007 at 6:16 AM

Great points on the business efficiency of an "expert" versus the "average" (debugged features / dollar / hour), I totally agree. However, I bet either your Perl friends are just average Java developers, or you have too many cats. The same "our product is superior" attitude seems prevalent in many expert groups. That misguided egoism is the territory of more expert developers. I know a lot of crappy Perl guys and if they just used Java the whole world would be happy. ;)

by Jason on Aug 6, 2007 at 6:17 AM

You stated "Is your business technology oriented? Perhaps the software you create is even your main product. If nothing else I'm sure we can agree that if the software your developers create is to some degree critical to your business.

I've worked in many different environments, with people of every skill level, and it's very easy to tell whether or not a company has expert developers. Do you often find that the software is down? That it has as many bugs or even just idiosyncrasies that make no sense to the user as it does features? Do the users find it difficult to use? Is the problem at hand relatively simple compared to the training or documentation necessary to begin using the software?

If you answered yes to any of those questions you more than likely have average or below average developers. "

From my point of view as a Perl programmer, these sound like management flaws...not programmer flaws.

by Mike on Aug 6, 2007 at 6:17 AM

I often hear about these enormous factors of efficiency differences between average and expert programmers, and to some extent I believe them and have witnessed them personally. But at the same time, it always reminds me of Amdahl's Law. I can easily believe that programmer A is a full order of magnitude more effective than programmer B, but *only while working at full speed*. And there are a lot of things that a programmer must do besides working, and many more that a programmer might do: meetings, scheduling, bug triage, slashdot, lwn and other technical sites, day care disasters, etc.

I'm not even saying that all of those things are a waste of time -- I doubt someone can become an expert programmer without spending some amount of time keeping up with the state of the art on various web sites. But the fact remains that a superstar programmer reading slashdot is getting no more accomplished than a halfwit programmer reading slashdot. And I'm willing to bet that non-programming tasks are well above 30% of any programmer's actual work hours. Sure, for a week or sometimes even a month that might be the case, but not over the course of a year. And... let's see... at 30% time, that 10x boost is now down to a 2.7x boost.

You have to consider what percentage of the overall job of a programmer is actually subject to the 10x (or whatever) speed difference.

That said, I fully agree with your basic premise. :-) I suspect that if orders of magnitude really *were* possible, then this wouldn't even be a debatable point. But the real-life differences in performance are still easily large enough that a 100% salary premium should be considered a cheap way to boost productivity.

by Steve Fink on Aug 6, 2007 at 6:23 AM

I often hear about these enormous factors of efficiency differences between average and expert programmers, and to some extent I believe them and have witnessed them personally. But at the same time, it always reminds me of Amdahl's Law. I can easily believe that programmer A is a full order of magnitude more effective than programmer B, but *only while working at full speed*. And there are a lot of things that a programmer must do besides working, and many more that a programmer might do: meetings, scheduling, bug triage, slashdot, lwn and other technical sites, day care disasters, etc.

I'm not even saying that all of those things are a waste of time -- I doubt someone can become an expert programmer without spending some amount of time keeping up with the state of the art on various web sites. But the fact remains that a superstar programmer reading slashdot is getting no more accomplished than a halfwit programmer reading slashdot. And I'm willing to bet that non-programming tasks are well above 30% of any programmer's actual work hours. Sure, for a week or sometimes even a month that might be the case, but not over the course of a year. And... let's see... at 30% time, that 10x boost is now down to a 2.7x boost.

You have to consider what percentage of the overall job of a programmer is actually subject to the 10x (or whatever) speed difference.

That said, I fully agree with your basic premise. :-) I suspect that if orders of magnitude really *were* possible, then this wouldn't even be a debatable point. But the real-life differences in performance are still easily large enough that a 100% salary premium should be considered a cheap way to boost productivity.

by Steve Fink on Aug 6, 2007 at 6:23 AM

Every wants to hire an expert, but no one wants to let more junior people get the experience that allows them to become experts. I have seen this for over 5 years now, so when I hear people complaining about the lack of talented coders, I feel no pity for them.

by James on Aug 6, 2007 at 6:25 AM

Five to ten times? You must be joking. I know someone who worked on a project (a joint effort between Ericsson and Siemens, called Ellemtel) with 500 other developers. At the end of the project, fully half the code delivered was his. Since then he wrote an operating system; every time you make a cell phone call in Europe, you're billed by a machine running his code. He's humble about it, though, because he knows somebody who codes ten times faster than he does, and wears out two keyboards per year. Somebody five times better than average is barely a blip.

That said, it doesn't matter so much how good the Perl coder you hire may be, because when he's done all you've got is Perl code. No IHOP can hire a great chef, and nobody would notice if they did.

by Nathan Myers on Aug 6, 2007 at 6:28 AM

IMHO, good developers are motivated by: (1) interesting work to do, (2) good colleagues to do it with, and (3) compensation commensurate with contribution, in that order. On point 2, we use the test (C/A > 1.0), where C is competence and A is arrogance, as a great short-hand screen. You can't just hire the smart ones...a collection of egotistical prima donnas doesn't get much work done, either. If you hire raw talent, expertise will follow quick enough.

by Dan Murphy on Aug 6, 2007 at 6:32 AM

There are some good points here. However, in addition to expert programmers, something that I find is often overlooked in software development is managers who are competent to make software development decisions. You can have excellent programmers (granted you probably won't if your managers are terrible) but if the person making the ultimate decisions doesn't have a clue what those decisions mean, then you're not going to produce very good software.
I have been surprised more than once to find that the person making the important decisions about software has little or no knowledge of good software development practices, or even of more general computer technologies (i.e. the difference between the web browser and the web server).
It appears that this is less true in other fields, such as bio-tech or engineering (the physical kind).

by Ben Smith on Aug 6, 2007 at 6:34 AM

When selling high-priced proprietary software, sales people are definitely not interchangeable. As a developer-turned-sales-guy I can appreciate both the art of expert developers and the skill required to help a client see how a piece of technology can improve on "what ain't broke". Putting an egocentric sales person in the seat can be, in my experience, even more devastating to the bottom line than hiring a distracting junior programmer. But there's no question, I would rather have 3 Picards over and army of Wesley Crushers.

by Scotty C on Aug 6, 2007 at 6:36 AM

When selling high-priced proprietary software, sales people are definitely not interchangeable. As a developer-turned-sales-guy I can appreciate both the art of expert developers and the skill required to help a client see how a piece of technology can improve on "what ain't broke". Putting an egocentric sales person in the seat can be, in my experience, even more devastating to the bottom line than hiring a distracting junior programmer. But there's no question, I would rather have 3 Picards over and army of Wesley Crushers.

by Scotty C on Aug 6, 2007 at 6:36 AM

When selling high-priced proprietary software, sales people are definitely not interchangeable. As a developer-turned-sales-guy I can appreciate both the art of expert developers and the skill required to help a client see how a piece of technology can improve on "what ain't broke". Putting an egocentric sales person in the seat can be, in my experience, even more devastating to the bottom line than hiring a distracting junior programmer. But there's no question, I would rather have 3 Picards over and army of Wesley Crushers.

by Scotty C on Aug 6, 2007 at 6:36 AM

In regards to the Apple and Microsoft, I'm not so sure the issue with Microsoft is the developers as it is the manangement and the at times poisoness structure of that company (reading MS blogs you can pick up on such things).

Plus there are ( a few) examples of great products at Microsoft that solved difficult problems well and far better than anyone else had done up to that point, and there are some examples of lackluster Apple prodcuts.

It's quite enjoyable to poke fun at Microsoft but I really held myself back until I was willing to look at what they do correctly and learn from it, this otherwise excellent article helps contribute to that all to familiar religious war I tire of seeing.

by ryaninfinite on Aug 6, 2007 at 6:38 AM

Rag on the sales all you want...

:)

That's not to say that sales are necessary, just merely that they deserve some ragging on. Simply because they are the tail end of the whole process "sales" people tend to get most of the perks and the rewards. Sales people would not have a job if developers weren't making a product. Developers could still have a need without a sales team if the product itself is necessary. Obviously though, one is only reaping a minimal harvest without a sales team. But sales people receive a disproportionate reaping of perks.

My mom works for a subsidiary of D&B. Her company had a good year. Her department which makes the product got lunch. The sales team got sent on a cruise. Likewise, I am a developer and work for a company with a large amount of sales people. But us developers don't receive the 1/2 day off if we get a lot of lines of code done in a given week. We don't see the bonuses or perks that sales people recurringly receive.

So feel free to give them a little ragging on. We're not dismissing their necessity - just the workplace's tendency to only reward sales people and not the rest of the people who make it all happen.

;-)

by The Saj on Aug 6, 2007 at 6:38 AM

I think you have oversimplified 'expert' and 'average' programmers. I agree that for the most part the best programmers I have had the pleasure of working with have experience in multiple languages and understand the fundamentals of programming more than any given language ( not to say they don't know the language they are using at the time ).

I think companies should look more to their developers for guidance in how the software should be presented and utilized rather than having sales or even marketing hands digging into the layout. Giving a programmer the ability to mold his sculpture is excellent. Stay away from letting others come in and tell them how the sculpture should turn out.

by Ryan on Aug 6, 2007 at 6:38 AM

I'll only disagree with 2 things:

1. To hire someone who has been doing something else and train them, especially in programming is truly cheap.
2. The right tool for the right job. Obviously Perl can do some things better than other languages, but to consider
Perl good for everything is just plain egotistic.

Most everything else I'll agree with, mind you the article as a whole completely dismisses the free market.
Corporations will always pay less, plain and simple. And most of the time pay more to "kiss ass" drones.
I'm coming from the free market side, it's much harder work, higher paying, and just plain enjoyable :)

Do a job search for cryin out loud.

Perl 5000+ jobs
Java 16000+ jobs
Guess what I write bank apps, engineering apps, gps, data analyzing, military, etc... and yes web apps in?

Later

by Joe Blow on Aug 6, 2007 at 6:44 AM

I disagree with the premise of this article. This article assumes that it's easy to determine who is and who isn't a good programmer and then goes on to discuss the benefits of choosing a good programmer over an average programmer... I find this discussion meritless because it's first assumption is, in my view, plain wrong. It's difficult to hire good to great programmers because generally, it is virtually impossible to determine how good a programmer really is based on an interview, a resume, the person's current salary at his/her existing job, or references. It is this fact that it's so hard to tell who is good and who is not that makes hiring difficult. Hence, you get the statistical mix of good and bad programmers after you've hired enough people. If it was easy to determine the quality of each programmer, don't you think everyone would prefer to hire the better programmer? Hence, I think the rest of your article is rather moot. Maybe a more interesting article would be an exploration of how we can reduce the costs of trying to figure out who is good and who is average to assist those who are in the market to hire programmers.

by Foobar on Aug 6, 2007 at 6:47 AM

One funny thing you miss - hiring expert java developers is no easier. In other words, just because you can swing a cat and hit a java developer doesn't mean that they are any good. In fact, the problem is that you can spend a lot more time trying to figure out who to hire when trying to hire java guys, as there are just so many people to interview. The real trick is coming up with a good enough interview process such that you only end up with the best.

by Matt on Aug 6, 2007 at 6:47 AM

Now we just have to find a way to get the Alpha type expert programmers to play well together.

by Rob Phillips on Aug 6, 2007 at 6:49 AM

I'm a former systems administrator who went back to school to get a degree in mathematics. I'd like to use it to do computer programming, especially for math-based applications. The trouble is, I have only classroom experience in programming, and little at that since I was busy learning deep mathematics. Thus far, I feel like the opportunity to become an expert isn't there just because the first thing companies balk at is my paucity of direct programming experience. Maybe right out the door an expert programmer would be 10 or 100 times as productive, but people with entry-level skills still need a place to start becoming experts. How does the high cost of low quality compare to the potential for making a high return by taking a bet on a promising beginner?

by Christopher on Aug 6, 2007 at 6:53 AM

I'm a former systems administrator who went back to school to get a degree in mathematics. I'd like to use it to do computer programming, especially for math-based applications. The trouble is, I have only classroom experience in programming, and little at that since I was busy learning deep mathematics. Thus far, I feel like the opportunity to become an expert isn't there just because the first thing companies balk at is my paucity of direct programming experience. Maybe right out the door an expert programmer would be 10 or 100 times as productive, but people with entry-level skills still need a place to start becoming experts. How does the high cost of low quality compare to the potential for making a high return by taking a bet on a promising beginner?

by Christopher on Aug 6, 2007 at 6:54 AM

This article is right on. With one exception to the rule though. I consider myself to be one of those good programmers, and having done many interviews trying to find additional good programmers for the company, I know how hard that can be.

Having said that, I find that taking on smart kids straight out of school that have a good handle on programming and mentoring them closely, you can actually "create" new good programmers. So long as they want to learn and are interested in programming and developing their careers, I find you can often mold them into the programmer you need, that overtime actually creates good code. From the get go, teaching and enforcing good habits from the start of their career can make a huge difference.

by John Q Programmer on Aug 6, 2007 at 6:59 AM

It's not just money or how careful you are in hiring. It is also how important quality work is for the business. Good programmers/engineers are created not born. It takes a lot of experience, study, and work to become excellent. The question I have is, what is most important to you, quality or schedule? Throughout virtually all of my career, the organizations I have worked for have made it clear that they value schedule over everything else and they will not work to train or enhance employees skills. Even discussion of how to improve is frowned upon. I can not count the number of times that managers have come to me to ask how long a project is going to take. I say 6 months (or whatever seems reasonable) and the predictable reply is 'What would we have to do to get it done in half that time?' 'Well, I can throw some crap together, forget documentation, forget most of the testing and hope it works; is that what you want?' 'What ever it takes, we just have to have it ready in 3 months or we will lose a tremendous amount of revenue.' So I do as asked, throw some junk together to get it out ASAP all the while hating myself for doing what I know is just shameful work. And often as not, 3 months after I turn it over, I ask how that project is working out and get the reply, 'Oh! we're still waiting on approval to put it in production. Management is still negotiating which division is going to run it.'

Companies don't get high quality programmers/code because they don't cultivate it or understand what it is. Companies get what they set as their goals. If excellence is what they strive for, they will get excellent programmers.

by Don on Aug 6, 2007 at 6:59 AM

Programmers being artists - That is a true statement. I have seen some great code coming from artist-programmers. Code that was efficient, and for the most part bug free. My company was too stupid to try to keep the really great programmers. They preferred to hire 2-3 average joes to replace the really good person they drove off.

by Frank on Aug 6, 2007 at 7 AM

There is just a hidden problem with your approach.
The subtle difference between "Expert" and "Superstar".
WARNING: The Superstar looks like a great "Expert" even to a good HR department. And he will be hired.

The Superstar has his tools.
The Superstar has his code-indentation.
The Superstar creates his own template class to represent an ASCII char.
The Superstar uses UML to study his Char class.
The Superstar writes 10 lines of comments for the "Char Char::toLowercase()" function.
The Superstar blames anything/anyone not being like he is.
The Superstar refuses to code in Perl since Java is more OO.
The Superstar rewrites perfectly working code because it does not use the latest funky tech.
The Superstar is an artist and he writes complex code no one else can maintain.
The Superstar is INFECTIVE and will spread his religion amongst other developers.
The Superstar you want to avoid like pest.

P.S. Programmers are no artists. Art is not rational. Feel free to flame me ;)

by DDT on Aug 6, 2007 at 7:01 AM

Companies and hiring manager and every one else loves to talk about the product and whine about poor programmers out there etc.
Problem with programming is that they are never trained after they graduate. I am a Java programmer so all I know about is Java in this regard. How many people are sent out for training on Spring, EJB or AJAX JSF?
On the other hand managers go for training to
learn PMP crap? Desktop Admins are being trained at WIndows Vista............
And programmers are never given a good orientation they must have in a company to have a good start. People see a website or application all of a sudden they want ur programmer to imitate that or integrate with that ..........

by Programmer on Aug 6, 2007 at 7:08 AM

Ultimately, you will get that guy who says: "but I don't want the Cadillac". And in fact, there is a market for substandard software as there is a market for all non-ipods. Inherently software is something people can't get their head around. It kinda doesn't exist but does. If 'his' Dell blew up, you can bet he would be on the horn with Dell and mad as hell. If Windows crashes, or Word loses his document, he will be on the phone with Dell.

by Christian Bongiorno on Aug 6, 2007 at 7:08 AM

Ultimately, you will get that guy who says: "but I don't want the Cadillac". And in fact, there is a market for substandard software as there is a market for all non-ipods. Inherently software is something people can't get their head around. It kinda doesn't exist but does. If 'his' Dell blew up, you can bet he would be on the horn with Dell and mad as hell. If Windows crashes, or Word loses his document, he will be on the phone with Dell.

by Christian Bongiorno on Aug 6, 2007 at 7:09 AM

Smart guys who are paid less, simply work two jobs. One for their
employer with their head, and one for themselves with their heart.
Guess which one gets the most awesome design/dev work...

At my company, the java developers are paid around $80K, so they
all have other gigs. The management knows it. The developers know they're
not getting ownership, only a job (just over broke).

I believe if mgmt treated developers as partners, in pay and ownership,
small businesses would accomplish an order of magnitude more than they currently do.

Perl,PHP,Rails and the other script-kiddy languages are
great for cranking out prototypes. I'd think twice about using them
for a real production app of any size and complexity.

Creating clean code means continuously refactoring it to
eliminate redundancy, make it more intuitive, reuseable etc. This is
too much mindless work for lazy-smart programmers without IDEs
to do the refactoring.

by anonymous coward on Aug 6, 2007 at 7:15 AM

"An expert can easily cross over from being a novice in any language in a matter of a few weeks" !
What a load of a rubbish.
Anyone who crosses over to a new language in a few weeks still going to be a novice. And anyone who hasn't used a language in a few years is going to take at least a few months to get back to speed.

An real expert programmer has specialized his/her skills in one language and knows very well which APIs, open source libraries/frameworks and code from their previous projects they can use to build applications well and quickly.

by Paul Gee on Aug 6, 2007 at 7:22 AM

Your points are valid, but sometimes money isn't enough to keep good people. I've worked at places where either the work or the corporate culture has been so poor that holding onto good people was incredibly difficult. I've also seen great places to work go sour due to a few poor recruitment choices.

I guess my point is that you can have good people, but that's only part of it. Just like with hiring exports, it helps to take a holistic view to the workplace.

Which, unfortunately, is beyond the power of many .

by Steve on Aug 6, 2007 at 7:24 AM

"...but sometimes money isn't enough to keep good people."

It doesn't hurt though.
I think that the main point here isn't base salary, but relative salary. That is, how much more do you pay someone that's exceptional vs someone that is just average?

I've worked with people that can do in an afternoon what takes others a month; but, when it comes to compensation, you might see a 10% difference in salary. Paying the exceptional developer more helps a lot. However it's not just because they have more money in their pocket when they go home. It's also important to demonstrate to the employee that they are valued for the work that they do.

by Dave on Aug 6, 2007 at 7:29 AM

I agree with most of this except for the Perl and the apple comments. Perl is so 1999. And apple is so 1989.

by Donovan on Aug 6, 2007 at 7:32 AM

I agree with most of the article, with the exeption of perl, and with the
statement about "experts are lazy". I consider myself an expert with well
earned PhD and, also knowing many other experts" I can tell we are not lazy.
An expert is also the person who stays until the job gets done, you never
become and expert being lazy. Also, kudos to python.

by Paulo Cortes on Aug 6, 2007 at 7:35 AM

I realise this is non-Perl centric. But ask if they own a copy of "Effective C++" by Scott Myers or "Effective Java Programming" by Joshua Bloch. Ask them to bring it to the interview, if its well thumbed with lots of PostIt notes hire them, they probably care about their craft. Also ask left field questions like how would you crash a JVM, an expert can probably think of several ways to do this offhand. I'm sure there are Perl equivalents.

by Tim Armstrong on Aug 6, 2007 at 7:38 AM

apart from being grotesquely wrong about Perl being an adequate language, this article is on the right track ;)

by lofi on Aug 6, 2007 at 7:45 AM

Good programmers are much harder to find then good sales people.
With decent training an average salesman can become quite good.

With decent training an average programmer will be... average.

by David on Aug 6, 2007 at 7:46 AM

yes, the somewhat extrinsic apple plug at the end kind of spoils an otherwise excellent article :(

by wim on Aug 6, 2007 at 7:54 AM

What you describe is not an expert developer -- it is just an intelligent person. The criteria could be applied to any field of work, research, life or polytics -- to life in general. The sad thuth, however, is that more often than not people who hire in industry or make decisions in polytics are *not* that intelligent as a person you describe. However bad the IQ test can be in general, it will do for a simple comparison: you describe a person with an IQ of 140+ (98% percentile) and decision makers have about 100 (average) in many cases. It is simply pointless to try and prove them that it is worthy to pay intelligent people rather than the "qualified-on-paper" ones. Talking of HR it is even worse: you will first have difficulties to prove them that they should consider a Java developer for a Perl role, I do not even say hire him/her. The CV will be sorted out by a robot with IQ of 0 before it will then go to the hands of an HR guy with IQ of 80 and only then it will maybe endup in the hands of a manager with IQ of 100. By that time Perl must be in the CV, otherwise it is out. And why it is so bad?.. Because the overhelming majority of people everywhere on this Globe do not like to think (in general terms), do not know how to think and therefore can not be expected to hire a person who is supposed to think and not simply do a mechanistic job (and this is the one you describe).

by Oleg on Aug 6, 2007 at 8:13 AM

Interesting stuff here...however, I think with the pressure on most companies to maximize profit (and if the company is doing well, throughput), the reality is that it "seems" you can get more out of 10 developers than 5. Try counter-arguing that to a CEO that doesn't know Ctrl-C from Win-L. Also, expert primadonnas (err developers) can be a management nightmare.

So, as with all things in life, it's all about the balance. Just need enough experts to guide the sheep.

by Auggie on Aug 6, 2007 at 8:20 AM

Sometime in the early 80s; Tom DeMarco and Edward Yourdon did some measurements of programmer productivity. They decided the productivity difference was 13:1, expert to novice.

Good post; maybe you should send it to HR departments.

All languages that are described are real; some are just more alive than others and some have better training wheels(static typing).

Would Microsoft really limit the number of buttons in the double digits?

by Bill on Aug 6, 2007 at 8:25 AM

The problem gets worse as you move up. $120-$200k is the upper cap on engineers at any company (not just software). If you do 10x better work, you get 10% better pay. If you're a hotshot, it is simply not economical to go into industry. You do a startup, contract work, or switch fields (entrepreneurship, i-banking, etc.).

by Jimbo on Aug 6, 2007 at 8:26 AM

The article was going great until the last couple lines. I guessing that if I was buying the rest of the article, then I'm supposed to swallow the general attacks of "It is the difference between Apple and Microsoft. It's the difference between the iPod and a 400 disc CD changer with 50+ buttons." as having the same validity?

I mean, sure holding down the play button for an extended duration is akin to turning a something OFF!? What EXPERT thought that up? And PERL ... ahem? Ok, so I walk away with the new understanding that your definition of "expert" may not be the same as someone else and perhaps that is why your experts aren't getting 10x pay.

by Dante Lorenso on Aug 6, 2007 at 8:26 AM

I agree with all of these comments, except for the Perl one ( that could be a whole article unto itself ). Sorry I singled out sales people, I do realize there are amazing expert sales people out there, it is just in my experience most of them seem to be easily interchangable! :)

by Frank Wiles on Aug 6, 2007 at 8:35 AM

"Some of the best developers I know were originally trained as journalists, mathematicians, linguists, and other professions not normally associated with software development."

I'm glad that you've been lucky in that way, but that has not been my experience. I find that people who don't have a computer science background (or worse, self taught) are lacking in certain fundamentals. Like Data Structures or Operating Systems, for example. Maybe this isn't a problem for most perl jobs, but in the embedded software development shops where I work those are killer skills to have.

by Bill on Aug 6, 2007 at 8:38 AM

I suspect one reason why PERL programmers cost more is that PERL is a -much more difficult- language to use effectively.

A core problem with this business is our willingness to use labor-intensive tools that require experts for effective use, both where such tools may be appropriate, and where they are not. Instead, we often use poor or at least labor-intensive tools everywhere, and then throw bodies at the problem.

From my perspective, any time I need to use a debugger, I see this as a personal failure as a programmer, as it's a sign that I don't know what my code is doing. I'd much rather try to get it right when writing the code (using languages like Eiffel or Ada that emphasize design-by-contract, strong type checking, etc) to find or better -prevent- bugs at compile-time, rather than spending painful hours in front of debuggers trying to fix problems.

But such approaches are clearly not in favor these days... sigh...

dave

by David Emery on Aug 6, 2007 at 8:38 AM

"It is the difference between Apple and Microsoft."

Interesting statement.

by Paul Evans on Aug 6, 2007 at 8:40 AM

Fantastic article! Other than strongly disagreeing with the Perl portion which is totally biased....right tools for the right job...don't try to start another flame war (Or at least we leave it for another article ;).

Totally agree with everything else said. I work for an organisation that is totally commited to talent and paying for them. I can see the absolute difference in the work environment, the drive, the quality of people you work with. Its just feel fantastic to work with great people.

compare that to my previous job that has a low pay cap for developers....absolutely sucks....turn over rate was so high...architecture is crappy, code is atrocious...its also full of people resistent about change. Quality of people don't stay consistently bad....it gets worst and worst. All good developers leave soon enuf. Bad underachieving developers are kept for their loyalty (truth is that these people can't find a better job so they have to stay ... and they get rewarded as management thinks its easier to keep these people than some experts who's gonna leave them soon....guess why they leave so soon....). Everybody thinks its a fantastic place to go when you are thinking of retirement.

by Anonymous Coward on Aug 6, 2007 at 8:51 AM

I've watched this evolve for nearly 30 years. The danger in the IT business these days is managers who don't know what they're looking at and HR people who have no idea what they're hiring. As long as larger organizations hand IT job slots-to-be-filled off to HR clerks whose job is to go through resumes and highlight acronyms in green, and as long as managers ask dumbass questions like "where do you see yourself in five years," they'll get mediocre developers who talk a good game but leave wreckage behind. This article brings to mind the legendary Hacker FAQ: http://www.plethora.net/~seebs/faqs/h...

Still a great article with immense truth in it. And when I hire somebody, that's what I keep in mind, mostly because I wish more people had taken it in mind when I was younger.

by Turtle on Aug 6, 2007 at 8:52 AM

I've watched this evolve for nearly 30 years. The danger in the IT business these days is managers who don't know what they're looking at and HR people who have no idea what they're hiring. As long as larger organizations hand IT job slots-to-be-filled off to HR clerks whose job is to go through resumes and highlight acronyms in green, and as long as managers ask dumbass questions like "where do you see yourself in five years," they'll get mediocre developers who talk a good game but leave wreckage behind. This article brings to mind the legendary Hacker FAQ: http://www.plethora.net/~seebs/faqs/h...

Still a great article with immense truth in it. And when I hire somebody, that's what I keep in mind, mostly because I wish more people had taken it in mind when I was younger.

by Turtle on Aug 6, 2007 at 8:53 AM

Interesting article, but you definitely got it wrong about the sales people. Worked pre-sales tech support with half a dozen different sales reps and the good ones could easily out do the poor ones by 5X. Not only do the good ones bring in more revenue but they also do it with fewer resources and time wasting sales calls. The saying around the office, borrowed from real estate, was “ the three most important things in sales is qualify, qualify, qualify.” The good ones did, the bad ones would drag you all over the place without a snowball's chance of closing the deal.

Like the comment about being an Admin in a different life. Have been both a Sys Admin and a DBA and it helps a lot.

One thing you missed is an experts ability to switch between development environments. Once did 5 different development environments in a single day, Informatica, SQL, KSH, C & AWK. All the programs were ones the client's staff could not write. As you might have guessed I'm a hired gun.

by mjr1007 on Aug 6, 2007 at 9:26 AM

I have to agree with most of what is said in this article, having experienced it myself. I managed a developer team consisting of 5 solid members for 6 years and guess what the result was... Happy users, No turnover(zero!!!), frequent releases, team members who were able to cover when others were on vacation, and a group that was more productive then a team of 50 in another part of the company.

And as their manager, I gave them whatever kept them happy. And thus the business grew.... 10 fold that is.. :-)

by Datapimp on Aug 6, 2007 at 9:27 AM

This is a very good article. I think a company needs a range of developer skill levels, but having said that, needs to know where to use them. You let your expert programmers do the initial coding/design or whatever. Then it makes sense to hire the junior ones as tier 2 or 3 tech support as they can write test cases for bugs or even fix bugs (with expert approval of course).

by Jason Pirkey on Aug 6, 2007 at 9:31 AM

"apart from being grotesquely wrong about Perl being an adequate language"

People who consider Perl not adequate have either never managed to go beyond maintaining a few badly written CGI programs - of which there are, like PHP programs, plenty available for free on the Internet - or have been using a hammer when the problem wasn't a nail.

I have been programming Perl professionally as a freelancer for over 12 years, and readable and maintainable programming in ANY LANGUAGE (I master a few besides Perl) require just that the programmer has not only skills to code, but also skills to keep it readable. Following a style guide, or agreeing on one with co-workers is with any language a must.

In short: language can't make you a messy coder, however incompetence will result in both bad and unreadable code. Something that looks like it's written in language /fill in the blanks/ doesn't mean it's good code, nor that it's readable.

I can recommend to grab yourself a copy of "Perl Best Practices" and take two to four weeks to study each piece of advice carefully Steve, unless you're happy with just trolling.

by John Bokma on Aug 6, 2007 at 9:37 AM

You had me on side till the 5th point, at which point you lost credibility.

by Chris on Aug 6, 2007 at 9:37 AM

I'd agree with much of what you say except with your means of determining if a company has expert developers. The only place I've worked where change started at the bottom and worked up was on my own projects. I work at a company with a software development cycle where work begins in the business and works through project managers and analysts and finally to my cohorts and me. Development is more than just developers.

by Peter Schoenster on Aug 6, 2007 at 9:44 AM

Hello
I am new to the field of software development...I just finished school with a BS in software engineering and am going into the job market. I have had a couple of 6 month co-ops, but didn't get great experiences at those companies and now feel like I have fallen behind my peers. I am wondering, how would you recommend that I work towards becoming an expert programmer. Please help...I want to work hard and be successful with a good job, but don't really know where to start. What can you suggest I do in the meantime while looking and what can I do at my first job (when I find it) to become a stronger programmer. I felt this would be a good place to ask because a lot of expert programming managers will probably read this article. Thanks for any assistance and I hope someone can help (even though this may not be an appropriate place to ask).

by newbieSEdev on Aug 6, 2007 at 9:49 AM

I couldn't agree more, especially with that last statement: "It is the difference between Apple and Microsoft." That difference is a philosophy of mine in just about everything that I do be it my job as a Java developer right now, my friendships, how I schedule my days, etc.

by Jim on Aug 6, 2007 at 9:53 AM

Let's say you're right about the cost effectiveness of hiring better programmers.

How is an ordinary company to know the difference between a good and a bad programmer? An HR type who offers 2x pay for 5x the effectiveness is not likely to actually get the 5x. There are plenty of programmers (or "programmers") out there who talk a good interview without delivering much. Being able to tell a genuinely exceptional programmer is a skill in itself, and not a common one.

Also, you are assuming the hiring person is motivated to maximize cost effectiveness, rather than, say, having a large number of people reporting to him so he looks more important. In my experience cost effectiveness is not central to the thinking of manager types.

by Matt C on Aug 6, 2007 at 10:03 AM

We get criticized a lot for having a "haute codeur" attitude to developer hires, but it pays off in the end because of the points stated in this article.

In the world of outsourcing, we are going to see a shift from massive profits to attrition over the next decade. The players that survive will be the ones that have mastered the art of hiring good developers and keeping them engaged through their tenures.

The key to this mastery is identification of aptitude and pedigree. Once you have that, the next challenge is synthesizing this talent into productivity. Once this is achieved, the duration of their tenure in an organization is directly proportional to the degree of expression they are allowed.

Irrespective of geography: very few are good. The great are even rarer. Treasure them.

Shameless Plug: If you are good - click on my name and send us your resume! We'll value you.

by Zubin Wadia on Aug 6, 2007 at 10:06 AM

I don't think you can judge the quality of the developers by the quality of the end product. I've seen excellent code turned into crap products by poor management or outrageous marketing.

by Joshua on Aug 6, 2007 at 10:08 AM

Good article. I can't say I agree with your stance on perl though... and it is ironic because in my opinion one of the marks of a junior developer is that they always have a golden hammer. Perl is great for certain things (text processing) but not for others (<i>truly</i> performance critical tasks). A great developer uses the right tool for the right job, they do not look for excuses to use their favorite technology.

by A on Aug 6, 2007 at 10:16 AM

Nicely done - and well written. Having bounced around five different companies doing sw development (military development in assembly, telecom in C and C++, development tools in Perl) I know that a good developer is worth his weight in gold. If you've got one, pay to keep him. It's easier than trying to replace him.

The thing many hiring managers tend to forget is that the dot com bust is over. There are no more hotshots out there serving coffee waiting for the market to pick backup. The market has picked back up and the good ones are gone. If you need a new developer, be prepared to pay for it - and not just salary; revisit Maslow.

by jmt on Aug 6, 2007 at 10:16 AM

All I see here is more people like me who are new to this industry and who are coming out of college soon being screwed out of good paying jobs. Without 5+ years of experience it seems like I'll be on code monkey status out of college.

by Nick Quaranto on Aug 6, 2007 at 10:18 AM

You underestimate the difficulty of *identifying* expert programmers. I get a lot of resumes across my desk, and it's darned hard to tell the productive ones from the losers. If programmers came with a number tatooed on their forehead that showed how good they were it would be easy to pay more and get a lot more. But unfortunately there are a lot of developers who are skilled at writing resumes and doing interviews and little else.

by Chris on Aug 6, 2007 at 10:18 AM

While I defer to the esteemed Engineer from Kansas on his opinion about Perl programmers ( I call everyone's attention to the "Language Hierarchy" chart: http://blogs.msdn.com/blogfiles/steve... ). So thusly they can become familiarized where Perl's "True" place in the order of things is. But I digress.
On every other point I have to agree not only wholeheartedly, but EMPHATICALLY. Having returned to college, to pursue an MSCS. In part to prove to son and stepson that an old codger like me can do that, and part to prove it to myself ( and yes, to also get ahead, and stay competitive in the job market). The interesting thing is that when I find myself when dealing with mostly younger programmers than myself who've been indoctrinated in the ways of (oh heck, pick your favorite ) TLA, ( OOD, OOP, UML,MVC, MVP, yadda, yadda. Oh sorry that was 5 letters twice.) without the more pragmatic principals we were taught 20+ years ago, I really have to wonder.
Then again, nowadays you've got rampant lunacy running the minds of "CIO'S" (when in the He%L did that mean something other than "Programming manager who couldn't program, so they sent him away to get an MBA" ). Let's outsource this and that to this company that 8000 miles away that uses "Engineers" who received their "Engineering education" in 14 weeks( Excuse me, I just looked up the link, it's 16 weeks ). http://www.australianit.news.com.au/s...
Sure, some of you will say, "The old guy's off his rocker, he's just spreading FUD". Fine, this "Elmer" will clean up all the "wabbit mess" left over in the aftermath of some silly accountants dream of saving beans making him look good to upper management. The bill is in the mail.
Oh, and about the money. I agree, it's often not enough to retain skilled, professionals. Those same folks will tend to gravitate to organizations where the monetary rewards balance with other "non-monetary" benefits. What those "non-monetary" benefits might be, well, that's where your mileage will vary.

by MarceloL on Aug 6, 2007 at 10:22 AM

haha Perl is THe language... you discredited yourself with that nonsense.

by Screw Perl on Aug 6, 2007 at 10:32 AM

100% agree !! Great staff, I wish my bosses would think the same way :-(
About the comments...
As always sales people missing the main point and destructing from the issue to something like... "just a second, let's talk about us". Calm down, guys this time its not about you. So seat back and agree just once for a change.

by DW on Aug 6, 2007 at 10:38 AM

I agree with most of your comments. Expert generalists do better work in any language than narrowly skilled programmers.

by Steve on Aug 6, 2007 at 10:39 AM

I agree with the out of wackness of paying highly skilled devs only 20% more than the average one when they do produce a lot better work. However this article fails in that all the numbers are completely made up. 10x better? Source? Pay 5 perl devs $120k and you'll save money over 15 $60k java programmers? Why those numbers?

Also doesn't everyone think they are a highly skilled programmer and not average?

by BlogReader on Aug 6, 2007 at 10:41 AM

In my experience, a large part of the problem is that non-experts cannot recognize an expert when they see one. So the HR department or even a non-technical manager is unlikely to find the right person. As you allude to above, your chances of success are far higher if you ask your best guy to refer someone who's abilities he respects.

by isomer on Aug 6, 2007 at 10:46 AM

While I can understand the backlash about the sales comments, sales guy wasn't much better. Sales don't generate the revenue of a company, EVERY employee does. And if they aren't then they shouldn't be an employee. Engineering design the product, manufacturing build the product, sales sell the product, finance makes sure bills are paid on time, secretaries keep everyone organised, the cleaners make sure the building is in a state to be worked in... If any one part of this system fails, then the company will fail, or at least lose efficiency.

Overall I thought he article was quite interesting, ignore the obvious bias to Perl and Subversion (great products, but not necessarily the best in my opinion). However it does ignore reality. As mentioned, good programmers are a scarce resource, however if every company didn't settle for less than the best, then most companies just wouldn't get any work done (cause they can't find the people), and the salaries would skyrocket temporarily and then the entire IT industry would collapse.

Reality is that employer's accept what's available. Find the best employees, without artificially inflating wages by demanding no less than the best. At best they can hope to attract a number of excellent programmers and put them in lead positions.

by Ryan Winter on Aug 6, 2007 at 10:47 AM

I can confirm this. I usually come in to clean up jobs that have been outsourced or given to the lowest bidder. The Dot-Bust created a mentality among companies that computer professionals are expendable yay-hoos that fix a problem and then go away.

After working for 13 years in IT and 3+ years in programming, I can see a correlation between people who have a knack for computers with the drive to study and high quality work. The juniors are kinda funny when they ask what "ping" is. I've found the best way to look at computer professionals is to think of the computer like a car. Sure, you can get any old kid to fill the tires with air, or even do it yourself. When you get your oil changed, you want someone with tools and a little intelligence. You might even do it yourself if you enjoy that or are hard-up.

Now suppose you have a Formula one race car at a championship race... Yeah, you want Linus Torvalds... hehe. I suppose the computer equivalent would be a corporate infrastructure for 2000 users. Now you know, if your top IT guy keeps 2000 systems running and you think you can hire an H1B visa holder from India to do the same job for half the price, think again.

As for India, basically a 4 year degree is like a trade school here. A one-two year certificate is like a week/weekend course here. Once you start getting people with PhDs and Masters from India you start getting some qualified people, but don't count on it. Usually the Indian equivalent of an American expert works for Microsoft for $60k/year(in India). I just laugh at companies now because they are destroying themselves from within. Perhaps one day they will wise up.

by JT on Aug 6, 2007 at 10:49 AM

I can confirm this. I usually come in to clean up jobs that have been outsourced or given to the lowest bidder. The Dot-Bust created a mentality among companies that computer professionals are expendable yay-hoos that fix a problem and then go away.

After working for 13 years in IT and 3+ years in programming, I can see a correlation between people who have a knack for computers with the drive to study and high quality work. The juniors are kinda funny when they ask what "ping" is. I've found the best way to look at computer professionals is to think of the computer like a car. Sure, you can get any old kid to fill the tires with air, or even do it yourself. When you get your oil changed, you want someone with tools and a little intelligence. You might even do it yourself if you enjoy that or are hard-up.

Now suppose you have a Formula one race car at a championship race... Yeah, you want Linus Torvalds... hehe. I suppose the computer equivalent would be a corporate infrastructure for 2000 users. Now you know, if your top IT guy keeps 2000 systems running and you think you can hire an H1B visa holder from India to do the same job for half the price, think again.

As for India, basically a 4 year degree is like a trade school here. A one-two year certificate is like a week/weekend course here. Once you start getting people with PhDs and Masters from India you start getting some qualified people, but don't count on it. Usually the Indian equivalent of an American expert works for Microsoft for $60k/year(in India). I just laugh at companies now because they are destroying themselves from within. Perhaps one day they will wise up.

by JT on Aug 6, 2007 at 10:50 AM

I can confirm this. I usually come in to clean up jobs that have been outsourced or given to the lowest bidder. The Dot-Bust created a mentality among companies that computer professionals are expendable yay-hoos that fix a problem and then go away.

After working for 13 years in IT and 3+ years in programming, I can see a correlation between people who have a knack for computers with the drive to study and high quality work. The juniors are kinda funny when they ask what "ping" is. I've found the best way to look at computer professionals is to think of the computer like a car. Sure, you can get any old kid to fill the tires with air, or even do it yourself. When you get your oil changed, you want someone with tools and a little intelligence. You might even do it yourself if you enjoy that or are hard-up.

Now suppose you have a Formula one race car at a championship race... Yeah, you want Linus Torvalds... hehe. I suppose the computer equivalent would be a corporate infrastructure for 2000 users. Now you know, if your top IT guy keeps 2000 systems running and you think you can hire an H1B visa holder from India to do the same job for half the price, think again.

As for India, basically a 4 year degree is like a trade school here. A one-two year certificate is like a week/weekend course here. Once you start getting people with PhDs and Masters from India you start getting some qualified people, but don't count on it. Usually the Indian equivalent of an American expert works for Microsoft for $60k/year(in India). I just laugh at companies now because they are destroying themselves from within. Perhaps one day they will wise up.

by JT on Aug 6, 2007 at 10:50 AM

I can confirm this. I usually come in to clean up jobs that have been outsourced or given to the lowest bidder. The Dot-Bust created a mentality among companies that computer professionals are expendable yay-hoos that fix a problem and then go away.

After working for 13 years in IT and 3+ years in programming, I can see a correlation between people who have a knack for computers with the drive to study and high quality work. The juniors are kinda funny when they ask what "ping" is. I've found the best way to look at computer professionals is to think of the computer like a car. Sure, you can get any old kid to fill the tires with air, or even do it yourself. When you get your oil changed, you want someone with tools and a little intelligence. You might even do it yourself if you enjoy that or are hard-up.

Now suppose you have a Formula one race car at a championship race... Yeah, you want Linus Torvalds... hehe. I suppose the computer equivalent would be a corporate infrastructure for 2000 users. Now you know, if your top IT guy keeps 2000 systems running and you think you can hire an H1B visa holder from India to do the same job for half the price, think again.

As for India, basically a 4 year degree is like a trade school here. A one-two year certificate is like a week/weekend course here. Once you start getting people with PhDs and Masters from India you start getting some qualified people, but don't count on it. Usually the Indian equivalent of an American expert works for Microsoft for $60k/year(in India). I just laugh at companies now because they are destroying themselves from within. Perhaps one day they will wise up.

by JT on Aug 6, 2007 at 10:50 AM

I can confirm this. I usually come in to clean up jobs that have been outsourced or given to the lowest bidder. The Dot-Bust created a mentality among companies that computer professionals are expendable yay-hoos that fix a problem and then go away.

After working for 13 years in IT and 3+ years in programming, I can see a correlation between people who have a knack for computers with the drive to study and high quality work. The juniors are kinda funny when they ask what "ping" is. I've found the best way to look at computer professionals is to think of the computer like a car. Sure, you can get any old kid to fill the tires with air, or even do it yourself. When you get your oil changed, you want someone with tools and a little intelligence. You might even do it yourself if you enjoy that or are hard-up.

Now suppose you have a Formula one race car at a championship race... Yeah, you want Linus Torvalds... hehe. I suppose the computer equivalent would be a corporate infrastructure for 2000 users. Now you know, if your top IT guy keeps 2000 systems running and you think you can hire an H1B visa holder from India to do the same job for half the price, think again.

As for India, basically a 4 year degree is like a trade school here. A one-two year certificate is like a week/weekend course here. Once you start getting people with PhDs and Masters from India you start getting some qualified people, but don't count on it. Usually the Indian equivalent of an American expert works for Microsoft for $60k/year(in India). I just laugh at companies now because they are destroying themselves from within. Perhaps one day they will wise up.

by JT on Aug 6, 2007 at 10:50 AM

Very well said, thanks for taking the time to illustrate this. Echo to lofi's comment!

by Craig on Aug 6, 2007 at 10:53 AM

I found this informative and potentially useful until I read your childish comparison of Apple to Microsoft.
Congratulations on discrediting yourself.

by Jarvis on Aug 6, 2007 at 11:16 AM

I completely agree with a lot of this article. As a programmer myself, I just got into the management aspect in the past year and I have had a horrible time finding developers.

One thing I've noticed is having a kickass resume of several large companies is not NEARLY as valuable as actually being interested in what you do. I've had several developers who came from major companies with good credentials, but had little interest in programming or learning. Seems a lot of people choose the occupation because it pays well (or rather, people think it does :)) not because they have any interest in it as a job.

And location doesn't matter. While there are many more people on the highly skilled end, the vast majority of programmers in Silicon Valley are no better than programmers from the midwest, and they get paid a LOT more.

by Patrick on Aug 6, 2007 at 11:25 AM

Interesting especially because of the several (not many, but several) Apple API's I've worked with have universally sucked, with poor documentation and weird bugs. That's not to say Microsoft hasn't produced some pretty crappy libraries in their time, but if we're talking about developers here, putting Apple on a pedestal isn't the right way to go.

Perhaps expanding your examples to designers in general :-)

by Malderi on Aug 6, 2007 at 11:27 AM

I have to agree with this article. I'm a 5th (or maybe 6th) year Mechanical Engineering student. I'm working as a C/C++ programmer, writing engineering software. I took over a project that my boss started, and then another guy did for a while. It was amazing. Those two guys were the best programmers where I work (yes, we are almost all interns). The code was impossible.

I'm a good programmer for a CS student, and a very good one for an ME student. It took me 2 weeks to rewrite the code to a point where it was even workable. It was amazing. I had to remove about 6 global variables (that never should have been there), move almost all of the code somewhere else, etc. But now it is good enough that you could at least figure it out without too much of a heartattack. Still, I'm sure that a good programmer would look at my work, and see about 1,000,000 to change, but I'll get there with experience.

by Diltsman on Aug 6, 2007 at 11:40 AM

Nice to learn that other people also have thoughts like in the article.
Now I just need to find a company with such people...:-)

by CodeSlave on Aug 6, 2007 at 11:43 AM

I've walked into a number of failing projects that were started in the latest fad language and witnessed the use of PERL save everyones hides. There aren't many other languages that have the resources of CPAN modules where much of the time, someone's already done the heavy-lifting for you.

PERL/CPAN is a great example of a community sharing their work so other members can turn around and leverage that work in their code. PERL may be "inferior" in a number of ways but at the end of the day, the stuff works, which, I'm finding, bosses seem to like.

by Dr. Alabaster on Aug 6, 2007 at 11:46 AM

To the author:
I don't know if you want to post this (you might consider it an attempt to advertise), but you might be interested in what we are doing. We've created an entire business around the principles you mention, with a multi-skilled, multi-technology, multi-country group of developers. Quality leads to a cheap development, whereas cheap developers do not. There's no substitute to getting it done right the first time. We're a relatively small group, but we get to work on great projects, travel to interesting countries, and have a huge amount of flexibility in our non-working lives - http://www.virtualsoftwarecompany.com. Cheers, Phil Callender (co-founder).

by Philip Callender on Aug 6, 2007 at 11:50 AM

I think almost all languages of a given paradigm are similar. Imperative ones like java, c, c++ or perl, functional languages like haskell, ML or logical languages like ProLog. One only needs to know a langauge from each of the paradigms well and she/he would be able to learn any existing language in no time.

by Nilay Vaish on Aug 6, 2007 at 11:51 AM

Jimbo, you're right on track.
If an expert is 10x more productive, why can't companies pay at least 5x for them? (e.g. 500k instead of 100k) Because they are not close enough to the decision-making. I have found that the closer you are to making decisions that have a direct impact to revenue (e.g. sales, biz dev), the more "unreasonable" pay you can make.

Employment truly is a bad deal for someone who really is a hotshot: you are purchasing stability, at the cost of little risk for a much larger potential (even doing contracts you can work 6 months and make the same money, how much stability do you really need?).

by Martin on Aug 6, 2007 at 11:55 AM

"It is the difference between Apple and Microsoft."

You just discredited your entire article right there. You did yourself a disservice. You are not big enough to express you own independent thoughts, which are admittedly above some marginal level, without bringing yourself down to some idiotic level.

The big time developers such as myself--getting paid $200k+ a year--know that "Microsoft vs. Apple" has NOTHING to do with real results on mission critical, high consequence projects.

by Mike on Aug 6, 2007 at 11:59 AM

I agree with most of what you have said here, but the main counter issue in my experience is that the people making the decisions can't tell average programmers from expert programmers (or average sales guys from expert sales guys, for that matter). Nor do they care to put the effort into being able to do so - after all it doesn't benefit them.

In a lot of cases, managers do _not_ want great programmers on their team, they are seen as prima donnas, and their technical leadership can encroach upon the authority of the team manager. Great engineers care most about making great products - managers care most about promotions.

The decision makers make decisions mainly based on who they know, not what they know. Getting a team of 5 or 10 average programmers together and delivering something successfully is good for the manager, even though it is not necessarily good for the company. Just like getting a team of 5 to 10 average sales guys together and making 1M in sales is good for the sales manager (whereas maybe the expert sales guy can pull in 5M in one deal).

by Gunther Hust on Aug 6, 2007 at 12:58 PM

I have found that in many industrial positions (banks, telecom, etc) software developers are judged by image and politics rather than the quality of their product. This leads to a "race to the bottom" condition where good developers leave for true high-tech companies while mediocre but "social" developers remain and get their promotions. This ensures that poor technologists get into every level of the organization.

by AF on Aug 7, 2007 at 1:01 AM

fantastic observations. I find it really close to heart. Good developers/programmers are those who do it with few words and inarguably with smart code. And even smarter are those who do it in just one go...nay, their very first one.
To be an expert, it simply takes your willingness to learn the lang X...and it determines(may not be solely) your pick-up speed...and good developers are definitely lazy.

by thekornsk on Aug 7, 2007 at 1:15 AM

How can this entire article not mention http://www.joelonsoftware.com/ ? Joel has written many articles on this general class of problem, heck, his entire company seems to be a sort of vanity project to build an environment for him to work in!

BTW, I think this is a deep problem. It's really easy to manage this (or at least argue it) at companies living in the tech industry. In fact, at a tech company, not aiming in this direction is unforgivable. But technology is becoming so deeply embedded everywhere, and often it's hard for, say, an insurance company to put up the kind of environment which is conducive to retaining expert programmers. It becomes a big political problem, and all of the motives involved are _not_ questionable (though many motives are). You can end up having resentment from too much variance between people's working environment - sort of like how CEO pay has gotten so out of whack WRT lower-level workers. The "obvious" solution is to pull everyone's environment up, but that's not always workable, either.

by Scott Hess on Aug 7, 2007 at 1:15 AM

fantastic observations. I find it really close to heart. Good developers/programmers are those who do it with few words and inarguably with smart code. And even smarter are those who do it in just one go...nay, their very first one.
To be an expert, it simply takes your willingness to learn the lang X...and it determines(may not be solely) your pick-up speed...and good developers are definitely lazy.

by thekornsk on Aug 7, 2007 at 1:15 AM

Nice writeup.

I agree with a lot of the items - can even relate to the Perl one (been there myself).

What you somewhat miss in the article, is that most experts are not tempted by money. They are looking for peer recognition. Having multiple experts helps a lot - also allowing the experts to participate in other activities (writing books, seminars, speak at conferences etc.) So its not all about salary basically.

From personal experience managing software developers for 15 years, I would MUCH rather have a crack team of 3 experts than 30 mediocre n00bs. They are way more productive, code works and way less HR problems - sort of.

Sort of - because the experts are usually super weird personalities. This can cause as much grief to other employees than having a bunch of green guys asking questions all the time. So be wary of the lone coder that can do it all himself.

by Thomas Lund on Aug 7, 2007 at 1:30 AM

I totally agree with everything you said. And if you need a programmer that matches your points, I will telecommute. :) I've always wanted to use perl... it looks fascinating (I was raised on C++ and love it.) Any language, and a database to connect to, and an expert programmer is in heaven.

by Derrick on Aug 7, 2007 at 1:39 AM

@sales guy

You got it the wrong way around. We make the product that is worth selling. Without us you could go look for another job.

Erhm - but without you I would need to go look for another job. Guess we need each other then, don't we.

But HR? No

by programmer guy on Aug 7, 2007 at 1:43 AM

"It is the difference between Apple and Microsoft."

Uh, no, I completely disagree. To even compare Apple and Microsoft in such a way is completely wrong. I'm not going to attempt to enumerate why, but a lot of it is the same old boring rhetoric. Throwing in that argument doesn't make any sense to the rest of your excellent article - it just seems like pandering to the Anti-Microsoft crowd.

by Dave on Aug 7, 2007 at 1:48 AM

Oh, and while I personally think that you are spot on, being one of those "generalists" and also having to run a decent sized software development department, this type of article doesn't fly with un-enlightened executives. I've said the same thing until I was blue in the face, and many executives still think of developers as interchangeable gears.

What I'd like to see is real efficiency studies that have analyzed some real world data to give a quantitative comparison. Just pointing to the Mozilla project isn't going to cut it when you are going to your boss to ask for another 10k for a really good candidate. It sounds like the sort of thing some MBA program somewhere would be studying...

by Dave on Aug 7, 2007 at 1:54 AM

"We love Perl and think it's a great language that you graduate to after you have been forced to use less agile languages such as Java, C/C++/C#, etc."

HAHAHAHAHAHHAHAHAHHAHHAHAHAHHAHAHHAHAHHAHAHHAHAHHAHAHHAHAHAHHAHAHHAHAHAHHAHAHHAHAHAHHAHAHHA !

by Bob on Aug 7, 2007 at 1:56 AM

"We love Perl and think it's a great language that you graduate to after you have been forced to use less agile languages such as Java, C/C++/C#, etc."

HAHAHAHAHAHHAHAHAHHAHHAHAHAHHAHAHHAHAHHAHAHHAHAHHAHAHHAHAHAHHAHAHHAHAHAHHAHAHHAHAHAHHAHAHHA !

by Bob on Aug 7, 2007 at 1:57 AM

In my opinion exists another important point: how we train HR department to recognize an expert programmer or a bad programmer (more important to save our time in one more uninteresting interview)? Most of times, only expert programmer recognize an other expert programmer!

by Joao Richiard on Aug 7, 2007 at 1:57 AM

The problem is..most people think they are expert programmers.
Like me, i'm pretty sure I suck, but I also think i'm an expert & should be getting paid more

by king on Aug 7, 2007 at 2:03 AM

Great post. I agree with most of what you say, except the whole Pearl better than C# statement - pfft, yea right. ;o)

I agree that $$$ is mucho importante. I also believe that corporate culture plays a big role in determining whether anyone feels "at home" or not. I've been cranking out software for more than 20 years, over 10 years as a consultant, and I've learned the hard way that most companies fail to understand how important it is for developers to be able to focus on crafting fine software without constant distractions or meaningless restrictions.

The best project I was ever involved with paid me almost nothing but I enjoyed the environment so much that I stayed 'till the sofware rolled out the door. I was a consultant but I outlived several employees - including the manager who hired me! I can contrast that with several projects where I left early, even though the $$$ was good.

Thanks again for a great post.

by CodeGator on Aug 7, 2007 at 2:03 AM

Why do perl when you have ruby? Perl is dying....

by phil on Aug 7, 2007 at 2:10 AM

nice article, would it also not help to pair Experts with Averages so that they can also catch up on the Best practices and Techniques. This would grow the Talent Pool and benifit all. I also completed agree on the corporte culture part.

by Ranganath on Aug 7, 2007 at 2:10 AM

I agree whole-heartedly about finding a good programmer. The few that I have run across like to make enough to pay their bills plus a little, but enjoy having challenging projects.They would prefer to work at a lower pay to do better work.

A good programmer should have no problem picking up another language.

We would rather leave the seat empty than put someone in it that doesn't have a clue.

Perl? 1999 called, they want their language back.

by Rob Kallmeyer on Aug 7, 2007 at 2:11 AM

Thank you for your insightful article. I work as a photographer and a writer and find that your insights can cross into my chosen field. In professional photography the problem is often what I call the "democratization of the camera," where a bride might "hire" that friend who will shoot the wedding for free--and oops, the images are underexposed or with a harsh head-on flash or even worse, lost or destroyed because the film or digital data files weren't handled well in post-production.

The problem with "under-hiring" (my new word on the fly) is that products and services are degraded _and_ the industry as a whole earns a bad reputation for it. My basic principle for the topic is that it is best to invest time and resources into those ambitions that are important aspects of your business--otherwise, you just waste time, as if you were writing at 2 in the morning and have to spend the whole next day redoing what you've done (but that's another topic...).

Thanks for sharing!

by Dena Rosko on Aug 7, 2007 at 2:12 AM

I do think you make completely valid points, but surely you can't be advocating hiring experts over juniors 100% of the time? I'd consider myself a junior, or nowhere near an 'expert', programmer, but if no one hires me and allows me to work to better my skills, how am I ever going to become an expert in the first place? Grad school? Just practising programming in my spare time to improve my skills? All fair enough, but if the expert status comes from industry experience and knowledge of how to work in an IT company, I won't ever get there if the companies won't hire me to let me get all that.

by Dache on Aug 7, 2007 at 2:15 AM

I'd like to see your comments on why you think Perl is as good as Python. I find well-written Python to be much more readable/usable/maintainable than Perl.

by Bruce Blodgett on Aug 7, 2007 at 2:19 AM

As someone currently looking to telecommute, I thoroughly agree. I don't want to relocate or commute to work every day, even if the salary is $120k (I'm in the UK, £60k for a technical architect is about average for a London or City worker). As a developer I don't always work a 9-5, as you can't always tell when your next thought is going to strike - you may have that coding breakthrough at 3am, and need to work there and then!

Nobody would ask an artist to relocate from their home-based studio to work in the stark surroundings of an office now, would they?

by Chris Lewis on Aug 7, 2007 at 2:27 AM

Nice article ! I agree completely with you.
In other countries situation is worse, i live in Italy where a senior developer gets maximum $40k/year , at the top of his career ..

by nicola on Aug 7, 2007 at 2:28 AM

"It is the difference between Apple and Microsoft."

This statement ruined the whole article.

by Alex on Aug 7, 2007 at 2:42 AM

I agree that an expert programmer is an ASSET to the company and companies need ASSETS not just ASSES :) so if you want ASSETS...let me know..I can give you some... yes.. we do offer high quality offshore programmers...but we are expensive. We charge higher than standard offshore rates..because we provide quality programmers with brains! and that goes for designers too :)

email me at mamoon@hashe.com if anyone needs a top quality programming or designing person or a full team for that matter.

by Mamoon Rashid on Aug 7, 2007 at 2:55 AM

This post is a good read. And I agree with you, with one qualification. If you're capable of doing it, then by all means do it. You'd be nuts not to. But in my experience very few people have the skills required to build a team of experts. Many hiring managers, or perhaps mostly their superiors farther up the ladder, view IT people as all the same, so it's really only a matter of hourly rate. Hence the offshoring craze. I agree with this statement: "Companies need to stop thinking about their developers as cogs in the machine. They are more akin to artists, authors, designers, architects, scientists, or CEOs." But the reality is that's how they think. I have frequently heard project management folks talk about how building software is just like building a house with the blueprints and the schedules and so on. So these are people working in this field who don't have the ability to see how building software is essentially nothing like building a house. Further, their description of what it's like to build a house isn't even accurate unless maybe they build their house in fantasy land. Most of the people I run into in the business world are not particularly good at their jobs. Lets face it. Most people are average. That's the definition of average. So if there's a top notch shop like the one you describe, I haven't seen it yet. At least not in Milwaukee. Man, it's dismal out there. If I ever found the right place, a place that could properly challenge me and reward me, I think I could make huge contributions. Until that happens (which seems less likely all the time), I choose to consult. As a consultant I get paid well, keep a "professional distance" from my clients, and move on when the project is finished. That's the best part - leaving.

by Ron on Aug 7, 2007 at 2:55 AM

??
The problem gets worse as you move up. $120-$200k is the upper cap on engineers at any company (not just software). If you do 10x better work, you get 10% better pay.

by viktor on Aug 7, 2007 at 3:05 AM

I guess in the non software-development industries, sometimes it suffices to have average developers, but superior business analysts. Great business analysts know their domain and write down the requrements fast. At least they do not let themselfs to be used as a showstopper to let projects just to hang around forever.

by Diego on Aug 7, 2007 at 3:05 AM

High cost isn't always a bad thing. Many large consulting firms follow a "warm body hire" model. In this model it's all about keeping a high burn rate while throwing product quality and employee satisfaction out the window. This allows large system integrators to remain on lucrative fixed price contracts that are poorly managed by the client and whose products are built to throw away.

by Alex on Aug 7, 2007 at 3:05 AM

I can see paying a premium to get an expert Java developer instead of an average one for a Java position. Paying a premium to get an expert Perl developer for a Java position, as in your example, seems to be stretching things a bit far.

Can't they go to a camp or something and learn Java first?

by Herr Ziffer on Aug 7, 2007 at 3:12 AM

Excellent comments above regarding social networking and corporate culture. I want to tie those together.

Bad programmers cam destroy a team culture. Bad programmers are frustrating to work with. Especially in a large corporate environment, they seem to be the quickest to pick up on the political nuances, and wedge themselves into a position of limited authority where they can impede and sabotage talented programmers from doing good work.

So in addition to the social networking aspects of building a quality team, there is the flip side that even a couple of bad programmers on a team can drive all the good people out.

I've seen the situation where a lot of the systems I'm working on are really clever, sophisticated and well-written. When I ask, "This is nice, who wrote it?" The answer is, invariably, "Oh, they left about a year ago."

Good programmers can be annoying, too. But once you get in an "inverted pyramid" situation where the weakest, most inflexible programmers have the most seniority or authority, it severely limits a team's ability to work at all, or to hire good people who can break the cycle.

by Rob Young on Aug 7, 2007 at 3:16 AM

You seem to be contradicting yourself. If you can find lots of Java developers, but no Perl developers, why not hire an expert Java developer, and have him/her learn PERL? By your own statements, an expert in one language should end up being a superior developer in a new language.

by pgmer6809 on Aug 7, 2007 at 3:18 AM

Great article. I'm not going to touch the Perl bit since I think chosing the right language (and framework/platform) for the right task is at the root of development wisdom itself.

What I will touch is the bit that states an expert is 10x more efficient. I think this is understated. Sometimes only an expert can do the job, this is twofolds in terms of risk and time.
Risk - take an average architect to design a bridge and risk it collapsing - take an average surgeon performing heart and brain surgeries and risk a flatline; an average programmer can endanger a critical system.
Time - a scientist or mathematician that does not have the IQ to solve a problem will never be able to solve it no matter how long they work at it; an average programmer may well be unable to deliver on complexity even given infinite time.

In this sense, an expert is not just more efficient times 10, but times infinity, and therefore we are not talking of a cost/efficiency ratio, but the true cost of failure vs success.

A good case study is a startup I work at. They received 7 million to develop their software. First outsourcing in the USA, then outsourcing to India, then to a team of cheap developers brought in through an acquisition. After 12 months, 3 failures/rewrites, and 6 million down the drain, they hired 3 experts that delivered the project in 3 months, costing below 500k flights and hotels included.

In summary, 12 months and 6 million wasted. In this sense, I feel the 3 experts on this team were effectively worth over a million dollars each.

It seems the finance industry has understood this, with contracts over $1000/day going out. Perhaps the rest of the business world will follow.

by Alex Grenier on Aug 7, 2007 at 3:18 AM

"sometimes money isn't enough to keep good people"

The corollary is also true – You don’t always have to pay big money to keep a good developer.

Before everyone flames, yes you have to pay good money attract them, but what’s wrong with developing them from within? This HR model has been almost completely lost, but if the company is committed to its employees, this can happen.
No one WANTS to fail, and good management can provide the opportunity for growth and learning and the challenging environment that makes peoples want to learn and grow.

I know of a development shop that NEVER hired above the entry level developer position. Even the manager had started at the bottom.
Because people knew that if someone left, their position would be filled by one of them, they would take opportunities to improve because they could look for a promotion. Not all new hires stayed on for ever, but the good ones did.
The turnover rate was less than 10% per year for over twenty years, even though the pay rate was actually slightly BELOW the norm. And they were a fantastic TEAM of developers. (Why do we so rarely use that word today when talking about developers?)

Nowadays, the only way to get a promotion is to move to a new employer. Is this progress?

by DavidGB on Aug 7, 2007 at 3:20 AM

I generally agree with the article, except for a few key flaws. I think these flaws are probably due to the author being a programmer and not an HR expert.

"Suppose you instead took the time to find 5 expert,..."

Typically there is no time to find experts at all. Many projects are started even before the programmers have been hired. If an expert happens to walk through your door you lock the door behind him as fast as you can. Unfortunately, that's not the normal scenario. Looking at your example, you say 10 average developers might cost 900k/year vs. 600k/year for the experts. However, this is an over-simplification that does not take time into account. Let's say that your project started yesterday, and costs 10k/day in overhead expenses. If it takes us only 3 months to hire 10 average developers vs. 12 months to hire 5 experts then the actual costs are 1.8M vs. 4.25M respectively. Not to mention the costs associated with not having a product during those interim months! (btw, this is a real-life scenario I had to deal with).

Now, you might think that this was a silly management gaff, but in reality this is how the industry works. Many projects get started after being sold, or they are started in response to a move by a competitor. It's not always possible to plan out a project and have all the resources ready in advance.

However, management isn't stupid. Over time the poor and average programmers, that don't improve their skills, will get replaced by experts. As a project matures, so do the resources allocated to it.

Of course, if you're a small-time start up that has time, but not resources, then it may be in your best interest wait for that 'perfect' programmer.

"...it's very easy to tell whether or not a company has expert developers. ... Do the users find it difficult to use?..."

Interface problems are rarely made better by hiring expert programmers. If you want a good UI hire a UI designer/company and do usability testing. Programmers are poor UI designers regardless of programming expertise.

by Scott on Aug 7, 2007 at 3:25 AM

You are so on with this artical,
There are a few good developers where i am (including myself in that bucket),
could be paid more, but isnt that always the case?
We also have some dire developers, and we loose so much time / resource through them,
but its difficult, ive had to interview people, and without setting up literally an exam system, it can be very hard to tell what type of person you are going to get, ive had some suprise me, and ones i thought would be perfect turn out to be butcher developers.

Thankfully i have a small but good team on my latest project. all are very experienced and i have worked with all of them for well over 5 years so we know each other very well, the only issue we have at the moment,
is a lazy, cant be arsed to get to know the software sales team at the moment.
it makes me laugh you often have one or the other but not both, how many times have you seen a great salesman and bad developers....?

Good hunting (K, Developer / Architect / Ex Sys Admin / & previous non IT career..... lol)
(have you been spying on me ED?)

by Kristian on Aug 7, 2007 at 3:27 AM

This is an excellent article, and might do a world of good if only executives would read it, because it is so very apropos to the topic of outsourcing to India. All of the shops with which I've had experience focus entirely on "bodies in seats" and the resultant quality (or complete lack thereof) of the "finished" project shows through in spades to everyone except to the executives, who are busy counting the money they "saved" and cashing in their stock options.

by Paul on Aug 7, 2007 at 3:28 AM

Jimbo, most expert developers (Nerds for short :) aren't really motivated by money. It's what business types call a "Hygiene Factor" for Nerds. Salary is a way to measure how much you're valued relative to your colleages. Alpha Nerd gets the most with the rest falling in below.

IMPORTANT - if an Alpha-Nerd ever finds out he's not the top paid Nerd God help you. Even if it means an unscheduled and unrequested pay rise you make sure that doesn't happen.

by Michael on Aug 7, 2007 at 3:37 AM

Brilliant! I enjoyed reading every word, and event sent the link to friends (And my CTO).

Your description of the expert developer is exactly how I strive to do my work. In my last job I had to take over code written by, let's say a 'below below average' programmer, and was exposed to the 'other' style of coding. I had to waste so much times just figuring out bad code with no comments (And terrible variable and function names didn't help), that it always seemed like it would be just simpler to start from scratch.
What good is work that has to be redone from the base each time, I ask? How economical is that?

"Smart people find boring things even more boring than the masses." - Loved it.
"It is the difference between Apple and Microsoft. It's the difference between the iPod and a 400 disc CD changer with 50+ buttons" - That's gold :)

by Erick S on Aug 7, 2007 at 3:38 AM

Smart programmers are getting more share of businesses these days, why be a rent a coder when you can actually get a piece of the action that you are instrumental at building? If businesses are not offering part ownership, outrageous salaries, or stock options then they will ultimately lose far more in wasted opportunity and missed business goals than they will ever save by hiring novices.

by Java Guy on Aug 7, 2007 at 3:42 AM

But if there no experts on the market or the technology is to young - hire young people who are willing to learn! Every expert has startet without the good skills. And if I am looking for a JavaScript GUI expert, I am not interessted in a cobol backend expert ...

by Wingi on Aug 7, 2007 at 3:44 AM

You've said it right. One thing though. I've been using all kinds of languages in my life as a CS researcher and SW developer. Even created a couple of my own. Yes, I did program in Perl. Try Python, really.

by Sergey on Aug 7, 2007 at 4:02 AM

Finding good programmers is hard in any language. Finding good Perl programmers is extra hard. Smart programmers use Python.

by Jacob on Aug 7, 2007 at 4:05 AM

Excellent article, I couldn't agree more.

by Chris on Aug 7, 2007 at 4:12 AM

Do we have at Microsoft a several thousands of lower-than-average programmers, that implies (in the above sense) poor software quality (both applied and system)?.. Or may we excuse these really good guys by pressure from marketing dudes, condensed timelines (you know, buggy but first at market), and overall corporate guidelines?..

And may we say that Apple (to mention Google too) is something closer to arts workshop or an agile scientific lab?..

by pavel on Aug 7, 2007 at 4:14 AM

This article has been Dugg!

It's sad that the audience who should hear this -- wont. They read kiss-up rags like Computerworld. Programmers are dirt under their shoes. And gee wiz! Six digit salaries for writing systems? Please tell me where. I've got two and half degrees, been building successful systems for 35 years, since I was a kid, as my Dad did before me, and the best I can raise around this country (at least this part of it, Ventura CA.) these days is $75K. Maybe I should go help build Total Information Awareness. Does anyone have any job leads in New Zealand?

by Mark B on Aug 7, 2007 at 4:15 AM

Thanks for pointing out the "Better developers -> need fewer developers -> reduced communication overhead -> even more productivity" connection.

Despite having read MMM, most of it twice, that conclusion had somehow escaped me despite it being pretty obvious once you see it!

by Rob on Aug 7, 2007 at 4:49 AM

great analysis.

by srini on Aug 7, 2007 at 4:50 AM

You left out one important strength. Have good code written by experts, also teaches junior developers good coding style and "better" ways to approach problems.

When I started working commercially (long ago) I was hacking a lot of Open Source projects (30 or so projects). I was exposed to coding styles that were really, really, bad and other coding styles that were more of a piece of art (Apache comes first to mind). If I hadn't had that experience, my coding style wouldn't be half as good as it is now.

As for perl being a "real" language: Well, it's real powerful =) (but the Perl 5 implementation of objects and namespaces was a joke - I haven't looked at Perl 6).

Hey, are you still hiring? ;)

by RKraay on Aug 7, 2007 at 5:10 AM

Didn't Perl die some time ago?

by Alper on Aug 7, 2007 at 5:10 AM

I think this article is well thought out but it's obviously written from the perspective of what frustrates programmers about their jobs. We all think that if we work with people who are average or below that replacing them with experts would make our lives easier. It's a great idea, but obviously a bit narrow minded because of these truths:

1) Everyone believes that the people above them in an organisation are idiots: Believe it or not, most people who run companies at the end of the day want to keep growing profits to satisfy the shareholders. They have two ways of doing this, sell more stuff or make stuff at a lower cost. Most try and do both. While it may make your life easier on the ground to have expert programmers in every position, it makes far more sense to have a mix of skill and experience to the people paying the bills. A product that is 80 or 90% good enough that cost 80 or 90% less to make is a win for them, especially if most customers don't notice that small drop in quality. They don't care if your life is easier, or that you bitch about your incompetent co-workers if their goals are met.

2) Your job is the most important in the company: everyone believes this, no matter what their job happens to be. Everyone can see things around them at a micro level and think that managment are idiots because they can't see what's going on at a low level. We all know how to make our own lives easier and more productive but at the end of the day someone above you has to decide if hiring experts helps their bottom line. In most cases, problem 3 puts them off...

3) you can't have five smartest guys in the room: experts, by and large, have an ego. Someone is going to have to do the crap that juniors would normally do. Everyone wants to be the recognised genius in a group, and experts are more likely to have been in this position before. Working in a room full of experts is going to cause more tension than it eases.

What you really want is to hire a good mix of experts of varying levels and especially keen and bright grads. I think there are experts out there who get a kick out of showing people how smart they are and don't mind teaching people who actually comprehend and build on what they learn. Grads are cheaper and a high turnover at the low end isn't going to be as big a problem if you have a chance to pick out the diamonds in the rough and promote them before anyone else gets to them.

by sj on Aug 7, 2007 at 5:14 AM

+1 to Jimbo comment

Sales people must be experts as well. However, I find no phrase in article that states the opposite :)

To, paraphrase, expert sales department is what make difference between IBM and Sun. IBM is known to successfully sell most of software they produce, regardless of how crappy it is. Sun frequently (if not always) fails with their commercial software.

by Valery on Aug 7, 2007 at 5:16 AM

I agree in parts, I'm just intrigued by your bias towards perl. Where I work (finance) its about getting the right tool for the job and we higher people with the ability/aptitude to apply those tools at the right time. The last thing you want is someone writting a server that has to manipulate prices and provide analytics in a scripted language (equally we don't want general housekeeping tasks to be done in a compiled language).

by rawpaw on Aug 7, 2007 at 5:20 AM

"Companies need to stop thinking about their developers as cogs in the machine."

This is the show stopper. I agree with pretty much everything written above, and it's not the first time I've seen this. Peopleware by DeMarco and Lister, for instance, covers a lot of this and more. But DeMarco suffered a significant drop in his consulting business after writing Peopleware. Never mind that he was right, and had respectable studies to back it up, and showed how the numbers work.

The very concept that developers are not cogs is anathema to many in management.

by Darrin Chandler on Aug 7, 2007 at 5:37 AM

>I do realize there are amazing expert sales people out there,
> it is just in my experience most of them seem to be easily interchangable! :)

I'm a professional software developer, and I work with other professionals. Comparing a professional salesman to someone who works at a mall is as absurd. So I have to disagree, hiring a good salesman is just as important as a good graphic designer or software developer.

developers make it work.
designers make it look good.
and salesman make people want it.

by Kyle on Aug 7, 2007 at 5:38 AM

"We agreed on the following"

Then you were all average programmers. Above average programmers are argumentative and don't work well with others. They tend to disagree - very loudly - about little things like "vi" or "emacs" and refresh the /. site constantly. They'll work like the dickens to complete a project and create some incredibility powerful programs and there won't be a single comment in the 28,000 lines it took them to write the program in.

by David on Aug 7, 2007 at 5:47 AM

I agree with you that good programmers are 10+ more productive that average ones, but why should they necessarily be Perl programmers ? The good programmers I know are good in *any* language, and they use half a dozen of them on a regular basis. Each programming language has its pros and cons: you won't use C++ for scripting or Python for high-performance numerical computations. Very often in large projects you don't have the choice of the programming languages of the various components because of existing code basis or third-party libraries. So I'm highly skeptical about sentences like "XZK language is for noobs, the real programmers use YBT".

by Patrick on Aug 7, 2007 at 6:02 AM

Great article and I cannot agree more.

Just a pity about how Perl were highlighted as somehow superior.

I have development experience in Perl, C, C++, Java, Delphi, VB, PHP, Ruby, XSLT, Assembler, etc. Each language has its strengths and weaknesses. As you mentioned, it takes very little time for a professional programmer to switch languages. Technology requirements, existing platforms, corporate culture and familiarity all plays a role in deciding what language to use. If I would develop an app in Perl for one of my customers, they may have serious problems few years from now maintaining the source. If I do it in .NET or JAVA, then it may be less of an issue to maintain. These are all things that professional programmers think of before making a choice as to what language to use.

BTW: I drive a 4x4 truck, not because I think it is a better vehicle than a sedan, but because I can maintain it easier and it can handle the South African roads...

by Francois on Aug 7, 2007 at 6:05 AM

I agree with your comments on hiring good programmers; however, you still seem to follow the bias that experienced programmers are better, through labelling junior programmers as not being good. I don't think it's right to say that hiring a good programmer is just as good as 5-10 junior programmers since, according to your theory, junior programmers could also be good, right?

As a junior programmer myself, I find it very difficult to show that my skills are good because, as you say, the recruitment teams focus massively on experience. I suppose the question I should pose is: if I were to have an interview with you, how could I show that I am a good programmer?

by Jon Black on Aug 7, 2007 at 6:06 AM

I wonder why sysadmins make good coders. Dealing with stress and solving problems perhaps? I can't remember, it was so long ago ... but then i've been burnt out a bit by my last job, or maybe it's age.

Quite funny the perl and readable code claim in the same article. I find the 'smarter' perl coders write the least readable :)

by Michael on Aug 7, 2007 at 6:06 AM

If perl was so great, perl programmers wouldn't be the minority would they eh ;)

by Tom on Aug 7, 2007 at 6:10 AM

There seems to be a contradiction between "You don't need to hire an expert in language X, you can and should look for expert programmers that are willing to learn language X. An expert can easily cross over from being a novice in any language in a matter of a few weeks." and your advice to look for Perl programmers instead of taking the Java developers you can swing cats at. What about taking on some good Java developers and getting them to learn Perl? The first one is an excellent point. Job descriptions are so narrowminded sometimes. Who cares if you've done language x when it takes 3 weeks to transfer all your skills in language y.

by Giles Hogben on Aug 7, 2007 at 6:17 AM

more comments on http://developers.slashdot.org/articl...

by Xavid on Aug 7, 2007 at 6:29 AM

sed s/[pP]erl/Ada/g and I totally agree. Having said that, there is a place in any organization for a few average programmers...Just not as many as appear to be present in most organizations.

by Jeff C. on Aug 7, 2007 at 6:31 AM

The sales team thing is correct. If one salesman performs badly, it doesn't (in any significant way) impede the ability of another salesperson to do well. A poor software developer however can degrade a valuable codebase, and introduce complexities that greatly inhibit other developers on the team from achieving. Like building a house, if the foundations are weak it can take a lot of work keeping the rest to stay up. Its all about the nature of the interdependency between roles.

by S. Morley on Aug 7, 2007 at 6:32 AM

Having been a programmer myself, and having since moved on to spend my time designing the system and writing specs rather then actually implementing them in code, i have to say i disagree with what you wrote.

First of all, many of the arguments you use can be applied to any seat in the house. You'll find that if you do a search/replace on your text replacing "programmers" by "marketing managers" or "production planners", most of your arguments will still be valid. If a marketeer or planner read the text they would agree completely with the result. Your basic message is "you need good people to do the job rather then bad people, because otherwise the other good guys need to clean up the mess of the bad guys". That's valid in any organisation from a football team to a team of rocket scientists.

Secondly, i get the impression that you greatly overestimate the importance of your own position in the company and of the decisions that you make, as evidenced by the sentence "They are more akin to artists, authors, designers, architects, scientists, or CEOs". In my opinion, this is nonsense.

Programmers are as important to the company as the people operating the production line and the people handling the sales orders: you need smart people who do their job reliably, you need them to follow procedures and stick to guidelines, and a good guy is easily worth 3 mediocre guys; but they are replaceable and interchangeable, even though they are all convinced that they're absolutely essential to keep the company running. Line operators have the same delusion regarding their own importance in the big picture, but i can assure you that the line won't stop if they leave or are. In the same manner, the business won't stop because a spec was implemented by the guy sitting next to you when you're on holiday. Chances are, the guy who wrote it will barely see the difference.

by Mark on Aug 7, 2007 at 6:36 AM

I agree with:
- the article.

Do not agree with:
- language and company comparisons, they are just diferent worlds!

Comments:
You are direct and objective, but from the business prespective, unless you are a product software house you cannot justify the very good programmers, when you are in a services context, clients pay your company by the market price, the problem starts here...

Want to continue the discussion? :))

by Bernardo on Aug 7, 2007 at 6:37 AM

I'd rather have a few juniors around than one toxic pseudo-expert (TPE). A TPE can put on a good show, but produces code at least as bad as a junior programmer and drains company time and resources. They are strong starters but always sink the ship in the end. To help limit the chances that you hire a TPE, try to include other programmers you have in the hiring process, and make sure they don't notice anything fishy about the new candidate. Take their advice seriously, and if they raise a red flag at some point in this new person's first few months, take notice and jettison the TPE before they waste any more time or money.

by devin the artist on Aug 7, 2007 at 6:40 AM

Good article. I agree with much of what you say and hear hear to your telecommuting comment. I am helping Australian companies setup satellite development teams (not outsourcing) in Malaysia because competition is tough in Australia for good programmers not to mention the cost savings. One of my clients has hired 4 excellent programmers, paid for the office accomdation including a corporate flat for visitors from Australia for less than 1 expert programmer in Sydney. But I disagree with your suggestion that a loss of corporate knowledge when a programmer walks out the door is a problem of perception. It is one thing to be "technically" a good programmer. Its another thing to know and understand intimately a particular business application especially if it involves the complicated world of say integrated ERP systems. Yes...in a perfect world....blah blah blah.

by Unblinkered on Aug 7, 2007 at 6:41 AM

I guess the real problem is that if no businesses hire junior developers, in a few years from now there will no even fewer good programmers. I seriously doubt there are enough hacker type people (who will spend years of spare time programming for the joy of it) to fill the needs...

by Joe on Aug 7, 2007 at 6:42 AM

author, i do believe you are flying by the seat of your pants on most of these claims. any evidence, even anecdotal, to back any of this up?

by matt on Aug 7, 2007 at 6:44 AM

Your article is commonsense for most people who have been doing development management. The real challenge is finding these unique people. For instance, after looking for 4 months for the right person, you do often end up satisficing.

PC.

by Patrick Collins on Aug 7, 2007 at 6:50 AM

I do agree with this article to some extent. Although,if every software company/IT department were to hire expert developers, where in the world would the college graduates find jobs? Senior developers don't grow on trees, nor do you find them right out of college. Eventually all the experts would retire, and then you would have to hire average developers. I think the proper mix is to have a few senior developers, a few average developers, and a few college recruits. Even if all the expert developers do is train, walkthrough junior developers code, and advise. After a few years, you may end up with your own versions of expert developers, that your company trained. The trick isn't to hire expert developers, it is to hire teachable developers, and a few experts. I do agree with the variety of programming languages being a big plus. I myself am still in college, but I have had work experience with Perl, PHP, COBOL, C#, and VB. From college I have picked up C++ and Java. I'm not a pro at any of them, but stick any language in front of me, and I will be able to program somewhat efficiently in a week. Hire me and my two college buddies, and have an expert review our code, and you will get three times the productivity, quality products, and a future.

by James on Aug 7, 2007 at 7:06 AM

This is a very interesting article. This has a correlation well outside of just programming. I found your "sales" comment interesting as I work in a technical role at a sales heavy company. There are "expert" sales people and there are poor sales people. The analogy to programmers is exactly the same with sales people, or really any profession. combin

An expert will end up making you money and have lower support overhead then the average (or even poor) group of people. The extra support time needed for those people to close a deal is often not taken into account by what additional (often larger) deals could have been closed by the experts with just a little extra help.

by Kent on Aug 7, 2007 at 7:10 AM

I agree as well as disagree with you on some points. Some one mentioned that money doesnt always get an expert programmer its true. You have undermined "sales" folks in your artilce. The fact is technology doesnt drive the business, its the business that drives the technology.

if the sales guys doesnt make their number no matter how expert you are your company will incur losses and eventually you will be ousted no matter if you are Einstein.

Every job has its own sgnificance. I agree with you a low quality programmer can ruin the system but a low quality manager can ruin the whole group, so its very important to keep the balance betwen all entities.

Well most of the expert programmers are not that good team players and they are very stubborn to deal with. Any genius is a human being after all and can make mistakes and in my career i have worked with awesome programmers but they are not right all the times.

Ex. An expert some times ignores everyone and develops what he thinks is right but from a functional stand point its a total useless product do you still want to hire this kind of a guy ? may be not.

Some genius are not born they are made, Like wise some times its good to have an expert programmer who can bring alot to the table and sometimes its better to reject an expert programmer as he/she might simply kill the morale of the team.

Management teams have a bigger role to play they need to cultivate the culture. If you keep a horse among a bunch of donkeys, after some time horse starts thinking like donkey and vice versa.

Not every time an expert will save the money and bring great things on to the table at the same i totally agree with on hiring one good senior developer instead of 10 poor programmers.

by Prabhakar on Aug 7, 2007 at 7:13 AM

It is simple, you get what you pay for. The older gentelmen programmers that knpw computers from the ground up are largely gone. You "can not afford" to pay them. The young people that you do hire lack that total experience and you throw too much at them at once, set unreasonable deadlines for the product to be done and often you just hire the "well rounded" programmers that "fit in"---sorry folks, that is the way you end up with such failures as Vista will be.

What makes things like Linux so great?----fantastic programmers that are under little pressure to
"produce, produce, produce" by xxx time. They do not have to know every programming language on the face of the planet either. Some of these programmers (sorry "T"--but you do fantastic programming) even have to be reminded to put on thier coats in winter, escorted to the bathroom,
taken home and picked up from home, have few friends outside of work. Bit give them an outline of what needs to be programmed properly broken down, give them good working conditions, quit beating them over the heads to produce, let them experiment a little. You will have a great product with little need for "updates" (alright, you do have to filter out some of the "play code", big deal).

Yes, some of these people are borderline with being classed "idiot savant", "queer (as in odd)",
"odd", or "bookish", or "anti-social"--but they are really some of the best programmers ever.

Sorry to burst you bubble oh Administrators, Human Resources, Presidents and thier underlings and CEO's. Most of the people that changed you world for the better you would have called such names as "tardo", retard", "stupid", "idiot", "maynard","brad", "weirdo"--which also applies to I. Newton, E. Einstein, J. S. Bach,W. A. Mozart, Francis Bacon, Issac Asimov,J. Smith,..............
.............

Is my point beginning to come across to you? Are you sure or are you just saying "Yes" to make reality easier to swallow?

by John Wiley on Aug 7, 2007 at 7:17 AM

I agree as well as disagree with you on some points. Some one mentioned that money doesnt always get an expert programmer its true. You have undermined "sales" folks in your artilce. The fact is technology doesnt drive the business, its the business that drives the technology.

if the sales guys doesnt make their number no matter how expert you are your company will incur losses and eventually you will be ousted no matter if you are Einstein.

Every job has its own sgnificance. I agree with you a low quality programmer can ruin the system but a low quality manager can ruin the whole group, so its very important to keep the balance betwen all entities.

Well most of the expert programmers are not that good team players and they are very stubborn to deal with. Any genius is a human being after all and can make mistakes and in my career i have worked with awesome programmers but they are not right all the times.

Ex. An expert some times ignores everyone and develops what he thinks is right but from a functional stand point its a total useless product do you still want to hire this kind of a guy ? may be not.

Some genius are not born they are made, Like wise some times its good to have an expert programmer who can bring alot to the table and sometimes its better to reject an expert programmer as he/she might simply kill the morale of the team.

Management teams have a bigger role to play they need to cultivate the culture. If you keep a horse among a bunch of donkeys, after some time horse starts thinking like donkey and vice versa.

Not every time an expert will save the money and bring great things on to the table at the same i totally agree with on hiring one good senior developer instead of 10 poor programmers.

by Prabhakar on Aug 7, 2007 at 7:18 AM

Speaking as someone who is currently a software developer and has been a sysadmin, pre-sales tech, post-sales tech, phone support jockey, bartender, and everything that overlaps the above, I have to say that the arguments presented above are applicable to many professions, not simply software development. Simply put, experts in any field are many times more productive than their compatriots, regardless of whether the job is electrician, hair stylist, butcher, bartender, or programmer. An expert salesperson is worth his weight in gold just as much as an expert programmer or (god forbid) an expert manager, or even a top-notch phone support person. All of these roles, poorly performed, have a negative effect on a company, just as much as they have a positive effect when they are expertly done. Any CEO would do well to remember this. Rewarding excellence makes good business sense regardless of the role.

--PLB

by Peter Buschman on Aug 7, 2007 at 7:36 AM

Thanks for the post! I wish my company would take this advice. They increasingly hire vats of fresh college grads and suffer from low quality and high turnover as a result.

by Michael Braun on Aug 7, 2007 at 7:47 AM

Thanks for the post! I wish my company would take this advice. They increasingly hire vats of fresh college grads and suffer from low quality and high turnover as a result.

by Michael Braun on Aug 7, 2007 at 7:48 AM

In addition: bad programmers not only are slower and less useful than an expert programmer, but can end up causing your expert programmers to lose efficiency.

An example of this: one of the people I work with has recently gotten a new job. During the time he was with us, he achieved the least amount of work I've ever witnessed.
There were vast amounts of time and effort applying "what I've read in books" or "this is how its done in rails" to our inhouse PHP application.
Every 9 times out of 10, the exact wrong answer or solution was chosen, argued for, tooth and nail; implemented; and then dumped onto someone else...
This consumed the time, anger, resources and patience of all of our other programmers, managers, users...
It seemed the more expert the individual, the more ire was experienced in day to day dealings.

In the end, we opted to give him a great reference to be sure his new place of employment wanted him.

by Daniel O on Aug 7, 2007 at 7:51 AM

I am new to your blog. I enjoyed this post, though, and I agree with your overall theme and ideas. Have you written about the interview/hiring process as a way of making these types of determinations between expert and average? In your lists of pros and cons it seems to assume that an HR department or Engineering Manager can recognize a great developer when it sees one if it only takes more time. I do not think this is necessarily the case.

by Michael Felberbaum on Aug 7, 2007 at 7:53 AM

>> It is the difference between Apple and Microsoft <<

Good article - but you just had to slip in the standard jab at Microsoft, didn't you? Not everyone agrees that Apple represents better quality than Microsoft. Too bad ... otherwise a good article.

-MP

by Max Peck on Aug 7, 2007 at 7:55 AM

Friends have been asking me why I work myself to death instead of finding and paying some people relatively cheap, and I keep explaining that I'd rather have experts which I can't afford right now.

They don't understand.

You have nailed the problem right on the head; it's exactly the way you describe, everything.

I've always argued that it's better to pay premium or even above-premium wages to a handful of experts than to have an entire army of average or less than average people working for you.

In fact, I'm so convinced that you are right, that I'm only going to hire experts, ever. Else, I'll rather leave the position unfilled.

by ux-admin on Aug 7, 2007 at 8:02 AM

I am a great programmer with 30 years of experience in multiple industries in multiple languages on multiple platforms currently working for the Federal government in a boring job in Baltimore. Please make my a job offer. -Marcus 410-786-5309

by Marcus on Aug 7, 2007 at 8:08 AM

Thats so damn right. I've been seeing so much 'software crisis'
and poor/average developers often reinventing the wheel and losing
their head in complicated frameworks and extremely spaghetized code
in anti-projects managed by software anti-managers completely lost
in the intergalactic void without a compass.

by Marius on Aug 7, 2007 at 8:22 AM

It's about the money and the hours! Although high-level programmers can command over 100K easily, the workload these days is much more demanding than in the 1980's and 1990's. Longer hours, more projects, endless and sometimes useless conference calls not to mention tougher metrics.

Add to that the fact that 100K does not buy what 100K bought 10-20 years ago and you have a environment where a senior programmer is better off starting another career. The solution for hiring managers is to increase pay and decrease workload. 300K is the new 100K! Wake up HR!

300K is the new 100K!

by Bobby Mak 8-8008 on Aug 7, 2007 at 8:24 AM

Very timely article ... dealing with this issue and about to put out a job posting. Especially like the contrast of the iPod and the Megapod-400-CD-Changer as a summary. And I got your back on the Perl issue! ;)

by J W on Aug 7, 2007 at 8:25 AM

There is one point that you glanced off that I thought could use more comment. You stated that when you work in an environment with experts things simply work.
Sometimes, when an expert develops something that just works, management doesn’t recognize the value of the accomplishment. Therefore, they don’t recognize the expert at all. I have seen so many companies reward the fire fighters who royally screw things up, and then bail themselves out at the last minute by burning the midnight oil. They ignore the developer who gets his projects done without bugs or incidents. Sometimes it’s said, that if it works the first time, it probably wasn’t that difficult to do in the first place. This is a huge problem with managers that don’t understand the job their employees do, or are not experts themselves. It will actually repel experts.

by john on Aug 7, 2007 at 8:31 AM

Any naysayers here should check out Paul Grahams talk at OSCON '04, I bet he has more experience than anyone bagging this article, which admittedly isn't perfect, but people are missing the point.

http://www.itconversations.com/shows/...

by Dave Novakovic on Aug 7, 2007 at 8:31 AM

couldn't have said it better myself.

by DMReference on Aug 7, 2007 at 8:31 AM

In general, this is a good article - although it is heavily biased towards the Perl developer community. The points can be related to anyone in a programming position and a lot of them hit home -- especially each and every point made about average/below average programmers. In my job I have been on a constant mission of fixing literally tons of bugs and flat-out programming stupidities. It is ever painful to scour 1600 lines of code and see maybe 3 lines of comments, each one increasingly useless with remarks such as "Set up variables" ... or just seeing a never ending else if loop that takes 140 lines for no apparent reason.

If I am to follow the logic of this article, then that would make me very close to expert level. While I don't quote know every programming shortcut, comparing my work to my predecessor's is certainly encouraging.

In a field where salary is merely "competitive" - a good programmer needs every bargaining tool in the shed in order to squeeze that last few K/year out of their employers.

Thanks to "competitive" pay - good programmers like myself are making as much or even less than the newbies since "competitive" merely refers to "How much everyone else is making in town X"

by nf on Aug 7, 2007 at 8:34 AM

In general, this is a good article - although it is heavily biased towards the Perl developer community. The points can be related to anyone in a programming position and a lot of them hit home -- especially each and every point made about average/below average programmers. In my job I have been on a constant mission of fixing literally tons of bugs and flat-out programming stupidities. It is ever painful to scour 1600 lines of code and see maybe 3 lines of comments, each one increasingly useless with remarks such as "Set up variables" ... or just seeing a never ending else if loop that takes 140 lines for no apparent reason.

If I am to follow the logic of this article, then that would make me very close to expert level. While I don't quote know every programming shortcut, comparing my work to my predecessor's is certainly encouraging.

In a field where salary is merely "competitive" - a good programmer needs every bargaining tool in the shed in order to squeeze that last few K/year out of their employers.

Thanks to "competitive" pay - good programmers like myself are making as much or even less than the newbies since "competitive" merely refers to "How much everyone else is making in town X"

by nf on Aug 7, 2007 at 8:35 AM

Even good sales people will get the cold shoulder for being successful. There was a guy in one company that I worked for, who was on a 2.5% commission. The company sold network hardware, and he was pushing to sell a couple of boxes for a pilot project. He was so successful in his sales technique, than the customer decided to award our company the entire 150 million pounds project. For being that successful, you would think the company would have been more than happy to give him the commission. Instead they asked if he would like to "donate" the commission into the R&D budget. He refused, and they fired him.

by Michael on Aug 7, 2007 at 8:35 AM

Even good sales people will get the cold shoulder for being successful. There was a guy in one company that I worked for, who was on a 2.5% commission. The company sold network hardware, and he was pushing to sell a couple of boxes for a pilot project. He was so successful in his sales technique, than the customer decided to award our company the entire 150 million pounds project. For being that successful, you would think the company would have been more than happy to give him the commission. Instead they asked if he would like to "donate" the commission into the R&D budget. He refused, and they fired him.

by Michael on Aug 7, 2007 at 8:35 AM

News flash: Expert programmers think expert programmers should be paid more.

by Chris Hehman on Aug 7, 2007 at 8:37 AM

I must disagree that an "expert" can pick up any language in a matter of weeks. With C++/MFC/STL, Java and its frameworks, C# and its class libraries, etc., the learning curve is quite steep, and even an "expert" won't attain 90% of maximum productivity for a matter of months at least.

by Grant Schultz on Aug 7, 2007 at 8:39 AM

I feel that it is difficult to hire good programmers:
1] There are no problem solvers since the massive advent of the internet when people can download code and without understanding use it.
2] After the bust the people who are left in the industry are the good programmers and they do not plan to change their vocation when the next bust happens.
3] The 'laundry list' is getting longer of computer skills needed. When even the interviewers do not read resumes and ask language syntax questions then you know you do not want to work there. Education is needed among the interviewers, HR and even IT folks.

by Longtime Developer on Aug 7, 2007 at 8:40 AM

I would like to point out that even though an expert can learn multiple langauages, they are not going to be an expert in a new language, new platform, for a long time. Experiance and time are the only things that can teach a person a best practice. For the lighter things like, writting files, popping up a form, sure anyone can learn that. But what if the OS, and the language of choice does funky things with that popup before it is displaye, things no docs cover. I have seen this over and over again, and the only one that new about it, was truly the expert who had years of experiance in both the language and the platform.
Thanks for the article.

by Dan on Aug 7, 2007 at 8:46 AM

Just to add to the few folk that talk about corporate environment and culture being just as in important.

If your company thinks spending top notch to get a small team of expert developers is good business sense then you have a good culture.

If it doesn't you haven't got one

Simple.

And that means *shock horror* sometimes the developers earn more than the managers! Live with it

by mark hewis on Aug 7, 2007 at 8:47 AM

I thought this was a good article. Sales guys are a little touchy aren't they?
Another problem is when the company promotes "programmers" into management positions just because there wasn't anyone else around to fill the spot. There is nothing worse than having a manager who thinks they are a programmer or think they are technical trying to tell you what to do and how to do it.
The old mentality of work and human resources, all that rot, needs to go away and be replaced by the new paradigms that have blossomed (telecommuting, etc.).
Software version control in our group is non-existent because we "aren't a software company".
BLAH.

by Tom on Aug 7, 2007 at 9:02 AM

"You don't need to hire an expert in language X, you can and should look for expert programmers that are willing to learn language X. An expert can easily cross over from being a novice in any language in a matter of a few weeks."

I've heard this statement made on a few occasions and seen it not work on just as many. While, I agree that in principle an expert can pick up a new language quickly, there has to be a very good reason for it. If he/she was interested in learning it, he'd already know it. It's even harder for experts to cross platforms, if someone has never used Windows and religiously runs Linux on his/her home machine, good luck convincing them to start using .NET on your project and using Visual Studio. Even if you're already a Microsoft junkie and using one of the .NET flavors (e.g. C#), it's tough to come up with a good reason to pick up another (say VB.NET). There are exceptions to the rule, peope who enjoy learning new programming languages, but even in those cases, I usually notice an OS bias that supercedes any language choices.

Good article overall, but next time I would stay away from broad generalisms. An iPod is not the only mp3 player (to compare with the 400 disk CD changer) and if lack of functionality makes for a good product then I suppose an abacus trumps all calculators.

by Peter Voutov on Aug 7, 2007 at 9:05 AM

You have completely omitted the most important factors:
1) The people who do the hiring are completely incapable of differentiating a superior talent from a fencepost. The process accommodates mediocrity rather than excellence.
2) Management typically is incapable of understanding that individual contributors are more than foot soldiers.

by Eric Houston on Aug 7, 2007 at 9:08 AM

After 20 years developing different types of solutions, managing developers, IT departments and so on, working in 5 different company cultures, Portuguese, Dutch, American, French and Brazilian, I fully agree with everything in the text.

I've seen all the situations in real life. Isn't only money that keep expert developers in place, but the article mentions it too, benefits, good work environment and interesting points are some of the key motivations for expert programmers.

The article talk about "Experts travel in the same social circles". This is absolutely true. I remember an expert developer, friend of mine, talking about the dinners in my home with other developers, something that use to happed 3 to 4 times a week. And he use to say "The IT quality that seats on your table at dinner isn't obscene, it's pornographic!".

We worked with many developers too, but just the best ones use to get together. They actually help one each other and often they are working for the same company. These developers from many nationalities that use to work for the same company in Lisbon are almost everyone working for the same company in Amsterdam now. So they do travel in the same circles (By the way, those are Perl Mongers).

I have to agree with the article in all it's points, well done and explained.

Congratulations.

by Joao Ribeiro da Silva on Aug 7, 2007 at 9:12 AM

this is naive. In your egocentric illusion of how corporations function,
you assume programmers and their skills are vital to the companies needs.
even though you lay out the reasons for this, you will not get HR or senior
leadership to take action to support your claim.
And EVERY skill set in the corporation makes the same claims about their
skills that you make about yours.
HR managers can give you 20+ reasons why good HR professionals drive up quality and drive down cost.
Help desk managers can give you 20+ reasons why good help desk professionals keep customers satisfied and drive down cost.
Middle managers can give you 20+ reasons why ...

Instead of putting forth your arguments for change, you should look deeper at the problem, and discover why the corporations make decisions the way they do.
You're like a bad programmer who thinks the compiler is buggy. It's not the compiler. Look at what your doing closer. If you really want to understand 'why', you need to change the approach you're using. You're approaching the situation from the wrong angle.

by billybob on Aug 7, 2007 at 9:17 AM

With the exception of the perl bias (I have a python bias, so who am I to complain?), I agree with your points 100%. Your sentiments echo with statements I have made to my clients for several years now (I'm one of the evil money-grubbing contractor "experts" who realized he didn't like the cap on engineering salaries). Thanks for putting it so well into writing.

by James W on Aug 7, 2007 at 9:23 AM

One more remark about the sales people remark... This is exactly the same thinking that leads HR and Sales people to regard developers as interchangeable. The fact is that virtually every role in a company has experts that are astoundingly more productive than others, but because as developers we don't know what they do, we can't necessarily tell them apart. That is, until we spend some time with a really good one and see what they do differently. I think the frustration in general comes from dealing with the 90% of people in any function who aren't in the top 10%. That's the same frustration the sales guy feels when he gets an estimate for 2000 hours effort for 12 mildly interactive web pages. He *knows* there is a developer out there that could put that together with good quality in a quarter of the time. He's heard of these mythical "10x" developers, but he knows that the knuckleheads in his engineering group will never pull it off.

by MorganAtlanta on Aug 7, 2007 at 9:38 AM

Not necessarily a first language you get your feet wet with and then move onto a *cough* "real" language.

I guess I'm just royally screwed up then... cause I started with Perl and then branched from there.

Very good article... it's interesting how much of this I've been saying for years. The unfortunate part is that more people don't think this way, especially "technical" recruiters. They don't seem to have an appreciation for the technical/programming generalist. They want a Java or .NET developer. Not a good developer who has been around the block and makes stuff work. They just look for buzz words and then get what they pay for.

I was taught that if a good developer goes onto a project that uses a language they don't know, they should be able to pick up the new language and be contributing to the project within 3 days. Sure the syntax is different and each language has idiosyncrasies but the logic and problem solving skills are the same from one to the next.

by Derek on Aug 7, 2007 at 9:40 AM

I have many of the same feelings about Perl. I don't get to use it professionally very often, but I just love the language. It lets me do more with less... or if I am feeling verbose, I can be noisy too.

However -- great article. Sort of fits will with mine on writing job posts for programmers: http://agentultra.com/?p=72

As a programmer, there are a few key tips for managers and HR people to understand about us:

1. It is an essential perk at a certain level in a programmers career to work with other talented experts. When the most inane problems are persistently bogging down a project; it becomes insanely boring and mind-numbing. Especially in web development -- if you hire a good programmer, listen to them and let them take the lead; it can alleviate boredom.

2. Hire good management. If you are building software, even websites, make sure your managers are either former programmers or know to leave the technical reigns to a lead programmer.

I don't know how many times my suggestions or work has been over-ridden by poor management choices. It is infinitely frustrating to be treated like a monkey with a typewriter. Make sure your managers are careful not to step on the toes of your expert programmers.

Bad management is equally destructive to software development as bad programming.

3. Do the Google thing -- let your developers have a little time to be creative and work on things that interest them during work time. We all store away lists of cool ideas that sometimes never see the light of day. Given the opportunity and incentive, it might be worthwhile for an expert developer to divulge one of said ideas to an employer. Both can benefit from the effort in some way. Ultimately it may encourage your hard-earned expert programmer to stay with your company for a long time.

by James on Aug 7, 2007 at 9:42 AM

I very much agree that one programmer can be worth 5 - 10 average ones. A good programmer is guaranteed to be worth 1000 mediocre ones (because a project with that many bad programmers will never ship anyway.)

However, even expert programmers might require more than several weeks to become expert in another language especially if the language is C++.

The 400 disk CD changer has 37 buttons, not 50, but the analogy is accurate nonetheless.

by christopher miller on Aug 7, 2007 at 9:48 AM

Great article - the general point is definitely valid. I will say it is easy to find bad java developers, very difficult to find good/great ones. It is not limited to Perl by any means. I can also tell you that salary/rate definitely does not always correspond to quality. The trouble is that 'bean counters' rarely see the value in getting one good guy for $125 - $150K when they can have 2 lesser ones at $75K. The sales comment was also way off. Every profession has superstars and duds. High end sales is even more pronounced in this than programming. For example, a good salesman might produce 2M$+ in revenue and allows quite a few programmers to be hired for the work where a bad one produces absolutely nothing and is soon out the door. 20% of the people do produce 80% of the results in just about every profession. One could argue it was more like 10/90.

by IT recruiter / ex developer on Aug 7, 2007 at 10:06 AM

"<i> it is just in my experience most of them seem to be easily interchangable! :)</i>"

Jimbo: Perhaps this is the essence of the issue. Everyone sees what they do as essential. However, one has to ask, how would you go about paying for an expert developer without an expert salesman? Or, how would an expert salesman be expect to sell crap software made by sub-par developers?

Excellence and expertise is an enterprise wide thing.

by TheFuzz on Aug 7, 2007 at 10:08 AM

I completely agree with these comments but I'm curious how someone develops into an expert programmer from a junior programmer. I'm a recent graduate (BS in CS) fighting the notion that 3-5 years of experience is "golden" to many companies. I certainly don't consider myself an expert in anything except sleeping; however I do have a certain passion for programming similar to what an artist feels for his or her work. In the end, (and as a recent graduate) it's quite difficult to prove everything that you have said to hiring managers - especially if the person does not know programming - so how do you prove to someone you're not just a junior?

by Mat Anderson on Aug 7, 2007 at 10:13 AM

I just hope that your kidding about Perl. I am an expert bottom line developer and I know it. I use Microsoft and hate all corporations. I use Microsoft (C#) because their tools allow me to create better products in less time than any other language; ie: more money for me. However, this article was amusing - especially about experts being lazy....ding ding - we have a winner.

by SpoonStomper on Aug 7, 2007 at 10:14 AM

This article frequently gets arrogant in tone when it's making sense at all (the difference between an iPod and a 400-disc CD changer? engineers may have implemented those products, but they were the fruits of corporate-level decisions, and don't necessarily reflect how good or bad their engineers were). When you work with experts stuff just works? I agree that an army of mediocre programmers/developers are probably going to create a mediocre, bug-riddle product or service, but it doesn't follow that "experts" necessarily make things better. They're usually so busy being cute and coming up with ever-stranger ways of implementing functionality that the core requirements get left behind in the dust. Sure, it's stupid and annoying when an inexperienced or just bad programmer comes up with a hard-to-read, 500-line procedure to do something that a 50-line class object could have handled much more easily, but it's also pretty annoying when a product ships late because the resident genius spent more time pondering the marvels of his core architecture rather than, uh, getting the work done.

by Jeff on Aug 7, 2007 at 10:14 AM

Your writeup was excellent until you made the rather sophomoric
statement about Apple and Microsoft. It exemplifies the "High Cost
of Low Quality Bloggers".

by Jim White on Aug 7, 2007 at 10:21 AM

Just my own 2 cents: Salary is definitely a few notches down on the importance chart for me. Flexible hours is most important; if a great idea comes at midnight and I have to be in an office at 9 am, that great idea will just be lost. Telecommuting is becoming more and more of an issue for me. Why am I wasting almost 3 hours a day traveling to a place where the chattering and noise interfere with concentration non-stop? My company isn't very good in this area and it continues to amaze me. I'd take less money to work from home more often in a heartbeat, and get more done in the process.

by BW on Aug 7, 2007 at 10:23 AM

I have been in this business for forty years now. Software, hardware, operations, and management across many different industries. I have worked hard to be a generalist! While I agree with you completely about experts, I find that the industry rarely works this way any more. For instance, I have programmed over the years in literally hundreds of different languages on hundreds of different computer platforms and operating systems. Yet, I find it almost impossible to get a programming job today, because I don't specialize in one single language!

Today, it seems that if you are not a person that can be fit into a small pidgeon hole and be happy to stay there, then you don't work. HR departments have no vision. They don't really care because they are filling holes. Managers don't seem to care, because they are too busy managing and advancing their own careers. Being a manager does not make you a leader, and it seems that most managers don't understand the difference! So unfortunately, many companies suffer with poor IT departments and don't seem to understand why!

I also like your comments on telecommuting. This is an area that needs to be greatly expanded and exploited in the workplace. But here's something often overlooked! Discrimination! By this, I mean discrimination by age and dissability. Let's face it. Older workers are being forced out at an alarming rate. And who wants to have someone with dissabilities working for them when they can have someone without dissabilities. It's hard to prove, but it's happening at an ever increasing rate. Telecommuting could help solve this problem if more employers would jump on the bandwagon!

I have narcolepsy! What company wants to hire someone that falls asleep on the job! Even if there is no danger involved, companies don't want you because you're a bad example to the other employees! I can no longer even find meaningful work in my career field of forty years! I have tried to find a good telecommuting job, but as of yet I haven't found one!

So yes, I agree with you completely on experts! But, the entire industry needs to re-think things and go through a complete change of attitude towards employees!

by Philip Evans on Aug 7, 2007 at 10:40 AM

I have been in this business for forty years now. Software, hardware, operations, and management across many different industries. I have worked hard to be a generalist! While I agree with you completely about experts, I find that the industry rarely works this way any more. For instance, I have programmed over the years in literally hundreds of different languages on hundreds of different computer platforms and operating systems. Yet, I find it almost impossible to get a programming job today, because I don't specialize in one single language!

Today, it seems that if you are not a person that can be fit into a small pidgeon hole and be happy to stay there, then you don't work. HR departments have no vision. They don't really care because they are filling holes. Managers don't seem to care, because they are too busy managing and advancing their own careers. Being a manager does not make you a leader, and it seems that most managers don't understand the difference! So unfortunately, many companies suffer with poor IT departments and don't seem to understand why!

I also like your comments on telecommuting. This is an area that needs to be greatly expanded and exploited in the workplace. But here's something often overlooked! Discrimination! By this, I mean discrimination by age and dissability. Let's face it. Older workers are being forced out at an alarming rate. And who wants to have someone with dissabilities working for them when they can have someone without dissabilities. It's hard to prove, but it's happening at an ever increasing rate. Telecommuting could help solve this problem if more employers would jump on the bandwagon!

I have narcolepsy! What company wants to hire someone that falls asleep on the job! Even if there is no danger involved, companies don't want you because you're a bad example to the other employees! I can no longer even find meaningful work in my career field of forty years! I have tried to find a good telecommuting job, but as of yet I haven't found one!

So yes, I agree with you completely on experts! But, the entire industry needs to re-think things and go through a complete change of attitude towards employees!

by Philip Evans on Aug 7, 2007 at 10:40 AM

This is quite true. But bigger companies need employees who can be replaced easily. Their policy is simple we don't need star programmers(that is what hollywood needs one good star for the movie). They need a bunch of monkeys whose work can be replaced if outsourced. Again depends on what kind of work is done for e.g developing a new language, yes we need larry wall or guido van rosum. Consider writing GUI app which is always easier done with a RAD will be better when done with notepad and demoed to a customer(pretend to paint the wall with toothbrush before customer, they might pay more if they are dumb). Bigger companies don't need good programmers alone they need people who can come up with patents and squat on patents to sue competitors.

by kenthil sumar on Aug 7, 2007 at 10:43 AM

Great points. But how do you decide who is good enough? I'd like to think I'm good enough but there are programmers better than me in the world. Then again, I've worked with other programmers who I don't think are quite as good, but they were still worth having on my team. That's not to say I haven't met some bad programmers, too.

I agree that good programmers probably have longer tenures and that's why they seem so hard to find.

by Marc on Aug 7, 2007 at 10:45 AM

This is a very insightful post. The only question I'm left with is, what about the average or below average developers? How do they then become experts?

I recently graduated with a CS degree from a second tier university. I've quickly learned that my degree really does not mean much and the best way for me to hone my skills is from practice and projects in the work environment. I realize this does not always benefit the company I work for, but without the opportunity to learn on the job, how would I gain the necessary experience to become an expert?

I realize the easy solution would be for me to contribute to open source projects and devote a large amount of my outside work time learning, but in the mean time I need to make enough income to eat and pay the bills.

So where do I and others like me fit in?

by John Comberiate on Aug 7, 2007 at 10:49 AM

But the problem with this idea is that it can only be practiced by a few people for the same reason that only a few people will ever be able to marry supermodels. On average, the average programmer is average. I agree that there is such a thing as an expert programmer and such programmers are far better than regular programmers but it's by definition impossible for every company to hire nothing but above average programmers.

Recognizing great programmers is important to a successful business but trying to hire nothing but great programmers is impossible. A better question to address may be that of how to most effectively use the few great programmers a company has and how to cope with the lower skill level of average and even below average programmers.

by jason wong on Aug 7, 2007 at 11:06 AM

I became a contractor because I got tired of being mistreated by company
managers. As an employee I considered myself a hotshot programmer. The
result was any serious problems were sent to me for solution. And when
some problem took longer than they expected, they got very touchy.
Managers not appreciating expert developers is also a serious problem.
When you get lots of tough problems, there will inevitably arise problems
that take much longer than anyone would like.

So, as a contractor I got paid by the hour and got to laugh at incompetent
companies. I just took advantage of them.

The downside of contracting would the companies I worked in. They tended to
be broken one that attract only low-level programmers. While the pay was
good, the work was yucky. And the managers were just as third class as the
developers.

But, I treated people well and tended to stay in contracting jobs 2-3 years.
I simply did it for more money.

by cecil on Aug 7, 2007 at 11:12 AM

I respect your views and hope I’ll be called an expert in few years time, but you make beginners look stupid. All expert start from one place (including you, that’s if you are an expert). If beginners do not work with expert, were would they get this knowledge from. By the time experts retire, who would take their place. I'm sure you are aware that expert’s knowledge is not thought in colleges.

by Yemi on Aug 7, 2007 at 11:19 AM

Salaries should go up as you related. They will not because you are discussing exceeding the compensation of the management, maybe for be several layers up, Also IT and code development is seen as an expense or overhead. Few businesses see any processing or software development as 'core' unless it says so on the letterhead, and even then maybe not.

We need to shift the culture of the US so that good technical folks can be compensated well. While we are at it, maybe a voice in a couple of decisions wouldn't be so bad either.

by Jeff on Aug 7, 2007 at 11:24 AM

I think a lot of what makes a programmer succssful in an organization is beyond their control, or responsibility. The biggest job in most projects is discovery, data definitions, and modeling of the business rules, assuming they remain still long enough to do this. Good development and specs lead to good code, usually. Poor specs rarely ever lead to good products, or products in budget. In a nutshell, SW dev is a big organizational PROCESS, not just something an expert does in a cube w his iPod on.

by MisterKev on Aug 7, 2007 at 11:40 AM

I worked for a great tech company with great people. We had everything going for us that has been described here as the way to get and keep highly-skilled developers. We went belly-up for lack of sales.

But it wasn't the sales staff that did it. Management started a revolving-door approach to sales, and if your sales weren't up to snuff you were out in six months. This is not enough time to develop the client relationships necessary to sell high-tech software in the $150k-$400k price range.

Like software development, sales, especially at such a high level, is an art unto itself, and qualified sales people are *NOT* plug-and-play.

by James Darlington on Aug 7, 2007 at 11:40 AM

I'd take it a step further, however. Experienced developers jointly develop software with stakeholders who provide requirements, specifications and use cases that the developer can use to build an intentionally-structured design, complete with design documentation that allows him to trace software artifacts to requirements - through all the levels from the concrete to the totally abstract - which you find in the minds of the stakeholders.

That is, a good developer can show how and why a software solution works - as well as how to maintain it. Unfortunately, we are often "ruled" by a kakistocracy, a collection of the most dim, corrupt or evil - who are unable to deal in the currency of reason - because they only understand power. This is doing software development by fiat, where management is completely unqualified to even pick the right development management - who would then hire "good programmers". Fortunately in this instance, natural selection does prevail, it being based in natural law.

by Frederick P. Blume Jr. on Aug 7, 2007 at 11:42 AM

Salespeople possibly have an even greater differentiation between great and average than programmers.

I believe I have seen some statistics that show that something like 80% of sales are made by 20% of the salespeople.

Its definately true that a good programmer is 2-10x more productive than an average one. - wish we got paid something like that kind of multiple too though :D

by Diarmuid on Aug 7, 2007 at 11:57 AM

This article is on the right track. But why waste time talking about the sales guys. The problem isn't with them. It is with upper management not understanding their own business. If your business depends on first-rate technology, it is only good sense to try to positively support technology and technologists. Too many CEOs think they can low-ball technology costs and get away with it. Hiring average staff or paying below average salaries is a formula for increasing risk significantly.

As to Perl, I'm sure it is a valuable tool in very restricted contexts. But anyone who knows programming languages and software engineering will tell you that it and other "write-only" languages are a danger to long-term maintainability of code. If I were a CIO or CTO who fired his staff of experienced Java programmers to replace them with 5 Perl experts, I'd expect to be certified after being fired.

by Sid Kitchel on Aug 7, 2007 at 12:04 PM

I am also someone who has experience searching for talented developers. Over the last 5 years I have seen hundreds of resumes, pre-screen questionnaires, and sat in on more interviews than I would have liked. The simple fact is that a great deal of effort is required to hire "A" grade talent. However, as tiresome and expensive as the search seems it is always more expensive to hire low-grade talent.

A good book I read several years ago on this subject is called "Topgrading". I would recommend it for anyone who is in a position to hire as this philosophy does not just apply to software engineers. Here is a link to the book on Amazon: <http://www.amazon.com/Topgrading-Lead...>

I also agree with your assertion that software engineers with broader backgrounds tend to be the most capable of handling any situation. This is because they have often been tested by fire and forced to learn new technologies rapidly and adjust to new business domains. It is the natural tendency of the mind to become lazy when it is not challenged (just as the body does when not exercised), so perhaps this is a factor in the ability of engineers who work in the same role for many years. I believe there is also a great deal to be learned by a fresh set of peers. Every time I have switched companies/departments I have learned new things from the talented people around me. Maybe the lack of this immersion early in their career hurts some engineers. Finally, I would say starting your career being influenced by low-grade talent could hurt your career. Bad habits picked up in your first few years as a developer can be very hard to break, particularly because pride does not allow us to think we may have been doing our job poorly for years.

I am going to give you the benefit of the doubt with the Perl thing and assume you are joking. You showed wisdom early in the article when you said, “You don't need to hire an expert in language X, you can and should look for expert programmers that are willing to learn language X. An expert can easily cross over from being a novice in any language in a matter of a few weeks.” To then turn around and assert that language X (Perl in this case) is the definitive language of choice for experts (refined developers, those in the know), would be a bit inconsistent. While I would agree that certain languages are easier to use for certain tasks. And further I could agree that for individual developers certain languages are easier to use based on their prior exposure to the language or a very similar language. However, as I matured from being a good developer to a great developer one of the many important lessons I learned is that programming languages are just tools for accomplishing tasks. The Dark Lord did not forge one language in the fires of Mount Doom to “rule them all”. You evaluate the needs of your company and your product such as: platform, performance, time to market, skill of your engineers, etc… and you pick a tool. I have left the “religious wars” over OS and language choices to the rookies who have not yet learned that those things don’t really matter to a great engineer. It is a poor craftsman who blames his tools.

Thanks for the article, it is a message that can't be repeated often enough.

by ProfessionalEngineer on Aug 7, 2007 at 12:07 PM

You exactly described the chaotic state of my previous job: low salaries and lots of incompetent programmers posing as "experts".
At first you may not care, but staying in a constant situation like this gets one worn out really soon.

by gonchuki on Aug 7, 2007 at 12:16 PM

This was an interesting read and I think the same principles would also apply to system administrators (or most other tech fields). I particularly liked the "experts are lazy" comment. I am lazy because I have spent the last twelve years figuring out how NOT to do things. :) A big howdy from William@Scottrade.

by William on Aug 7, 2007 at 12:20 PM

Very interesting and infomatic. Realy usefull. Thank you...

by sandeep on Aug 7, 2007 at 12:25 PM

"Companies need to stop thinking about their developers as cogs in the machine. They are more akin to artists, authors, designers, architects, scientists, or CEOs."

Well said, all your points are valid and I totally agree with your thoughts.

Regards
Sudar

by Sudar on Aug 7, 2007 at 12:29 PM

Sales people, hr, programmers and even the cleaners of the office - should all work in tandem. And this tandem is organized by the managers.
Managers - are the core. I hate managers ;)

by sanjar on Aug 7, 2007 at 12:30 PM

Lovely article. I've been on all sides of the issue. I'm now stuck on the worst side -- I've spent three years trying to hire a semi-decent Perl & JavaScript programmer to assist & take-over my programming so I can better run my small business. I've gone through five individuals who have made it through an increasingly rugged interview only to wind up producing code that is more of a liability than an asset. The type of code that may work perfectly, but cannot have any features added without blowing chunks. Granted, I can't manage humans for my life, but I'm still hoping to find someone with the passion to be capable of seeing the future without being told what it is -- something that I do each and every day.

by Bryan on Aug 7, 2007 at 12:32 PM

Can't agree with you about Perl. I love to hack and have for nearly three decades now but perl still turns my stomach. Python, Ruby and especially Lisp are much more to my liking.

That aside the big difference to a hacker is how much BS is in the way. The BS takes many forms, endless meetings, office politics, getting the approval of a manager who think he (usually) know enough to vet the hacker's ideas, BS that says all success is due to "the team" and only gives individual recognition for insane number of hours worked or throwing chunks of your life away whenever a "higher up" gets nervous and wants to see extra sweat inequity to feel better.

What is up with the telecommuting thing? I couldn't agree more. Except for meetings (mostly worthless) and bonding with a few folks I don't get the point of moving the flesh to a particular locale before I can use my mind. It is utterly stupid. Drive an hour each way in prime hacking and idea time to arrive sour and a bit teed off to the first meeting of the day leading into questions that somehow get answered otherwise if you aren't there leading to a bit of monkey interaction and then lunch, a few more hours of countless interruptions of real work and then back into the tin box for the mind numbing right home. Is it any wonder that most corporate software is a piece of krap unless someone laid up or using all their non-office time designed and largely developed it?

by Joan Jett on Aug 7, 2007 at 12:34 PM

I disagree that simply having experts makes software more stable. With experts or not, business needs drive software to stability. If a 50% risk of non-critical -- albeit ugly -- bugs is acceptable to get a first version out the door, then an expert will make it clear what corners were cut and will get the software live and making money.

Experts don't make perfect software -- no one makes perfect software -- rather, experts tell you exactly how their software will perform given the business needs and time required to do the job.

by Owen on Aug 7, 2007 at 12:35 PM

I disagree that simply having experts makes software more stable. With experts or not, business needs drive software to stability. If a 50% risk of non-critical -- albeit ugly -- bugs is acceptable to get a first version out the door, then an expert will make it clear what corners were cut and will get the software live and making money.

Experts don't make perfect software -- no one makes perfect software -- rather, experts tell you exactly how their software will perform given the business needs and time required to do the job.

by Owen on Aug 7, 2007 at 12:35 PM

I disagree that simply having experts makes software more stable. With experts or not, business needs drive software to stability. If a 50% risk of non-critical -- albeit ugly -- bugs is acceptable to get a first version out the door, then an expert will make it clear what corners were cut and will get the software live and making money.

Experts don't make perfect software -- no one makes perfect software -- rather, experts tell you exactly how their software will perform given the business needs and time required to do the job.

by Owen on Aug 7, 2007 at 12:35 PM

Click on the 400-disc CD changer with 50 buttons link... then on the one 'view larger image'

:-)

Off-topic, sorry...

Cheers,

Colin

by Colin 't Hart on Aug 7, 2007 at 12:36 PM

I agree with you, except for any time you mention Perl.

by Al koholic on Aug 7, 2007 at 12:37 PM

I read through this, and I found that for much of what was written, you could substitute any job description that has journeyman-level professionals doing that job - salesman, technical writers, artists or animators, cabinet makers, tailors, you name it. One could easily go the other way, though, and hire nothing but chiefs and have no indians, and you'd have the converse problem - lots of great ideas on how to do things, but not enough discrete sets of brains and hands to do the work. Rise above the details in this article - what counts is the key concepts!

by Gene Turnbow on Aug 7, 2007 at 12:47 PM

Right on, although I think you are wrong to suggest that Microsoft has no experts. They certainly do, they just work on Visual Studio and Excel. :) Apple also has its share of non-experts. They may sweat the details when it comes to UI, but it's a mess under the hood.

by Randall Cook on Aug 7, 2007 at 12:49 PM

I agree a long way. There are way too many people in the IT business (not only development), who has way too little understanding about how computers work, who do not have the gene that makes them want to explore things, like why the code does not work.

If we look at web developers (both the coding guys, and the graphics guys), then at least half of them are so under qualified or under skilled that they should never have been given the job they have.

I see young people from short IT educations who also understands zip about computers, but have a little knowledge. This is not enough. There isn't many experts out there, and I have the feeling that those that are there stay pretty long time with the companies smart enough to hire them. The job shoppers usually are the underperformers.

by Danish Guy on Aug 7, 2007 at 12:53 PM

I would disagree with this statement:
"You don't need to hire an expert in language X, you can and should look for expert programmers that are willing to learn language X. An expert can easily cross over from being a novice in any language in a matter of a few weeks."

Regardless of prior skill, it takes a good half a year to pick up "Best Practices" and be fluent in the toolkit API. That is under a language specialist supervision / mentorship. So, either formal education in the language or solid experience is absolutely needed PRIOR to hiring. Otherwise it turns into company subsidized education camp at $60.000/per head per 1/2 year.

Sometimes, learning a language actually makes you realize you really cannot stand it, or the specific API and would rather be a shipmate on crab catches in Alaska than program in it.

You are an expert in Perl, great! You want an expert pay in different language, learn it and really work in it before seeing the HR person.

by Dan on Aug 7, 2007 at 12:58 PM

I overwhelming agree. There is a clear difference in code quality between someone who is a strong generalist programmer and someone who isn't. The approaches to problems are different, the quality of the code overall is markedly different, and the code is far more maintainable. It never ceases to amaze me that so many companies think that they have to throw bodies at a problem when in reality you need the right people, not more people. Simply adding more people increases defects and the amount of problematic code produced. You cannot makeup for expertise with more people - you either have it or you don't. The example cited in the mythical man month with the OS/360 project is true time and time again - a few very competent people can far outproduce a horde.

by David L on Aug 7, 2007 at 12:58 PM

May be recruiters should look for furture experts or candidates that display characteristics of an expert, because in reality there aren't enough "Expert" programmers. Having said that, an "Expert" is defined only by the environment and is very relative.

by Vaithee on Aug 8, 2007 at 1:31 AM

Expert programmers started off novice and average. I think its the attitude that matters. Of course, the flair of grabbing new concepts and languages fast is a key factor as well.

by Leoin on Aug 8, 2007 at 1:34 AM

Wow, this article really seems to hit the mark. I've been job hunting for the past few months now looking to land a web developer position. Skilled in HTML, CSS, JavaScript, Flash, ActionScript, Java, C#, ASP.NET, phososhop...etc...pretty much everything you need to make a fully developed website. Unforunately, even with my skillset and time I've spent amassing these skills on my own, I still can't find a place willing to hire because I don't have that obligatory "5 years experience" requirement for the industry. That has most certainly proven to be a major stumbling block for getting a foot in the door.

Hmm...now that I think about it, I guess that's kind of the opposite of what this article says, actually. Either way, it always seems like the less-talented programmers/developers land the position that *do* have that 5 years experience, rather than that budding developer eager to soak this stuff up like a sponge.

by Adam on Aug 8, 2007 at 1:34 AM

The problem is not hiring good programmers. But rather making good programmers do their work effectively and also letting them get their deserved rewards.

by fornls on Aug 8, 2007 at 2:06 AM

HR departments have a particularly tough time evaluating candidates. Some times they don't know enough because they don't involve the technical side, and sometimes the technical side doesn't really know enough to properly interview a candidate. Besides that, the company's HR and tech people are emotionally invested in the process, which affects how they go about the interview.

As such, Stonehenge has done some HR consulting for its clients. We're certainly experts in Perl (and a few other things), and we don't have the big emotional investment. We can help with the interview, tell the hiring people what they think, and leave it at that. Since we're not part of the company, we can also play the bad guy and ask the really tough stuff that the tech people might not want to bring up.

Despite that, the best way to hire experts is to know other experts. It's one of the reasons I started Perl Mongers; now I know people all over the world. If you're working extensively in Perl (or whatever), get involved in the community somehow even if you don't have a short-term goal. Later on those people will know who you are already. :)

by brian d foy on Aug 8, 2007 at 2:14 AM

Looking for a Java and Perl computing 'experts' is an oxymoron.

Watch the crash.

by A on Aug 8, 2007 at 4:41 AM

Focus on Perl just shows you what sad state this industry is in.

Same goes for bloated Java frameworks.

by huma on Aug 8, 2007 at 4:49 AM

Really good post ... I liked it..

by Kvins on Aug 8, 2007 at 5:19 AM

Conclusiones en español (in spanish):

<a href="http://www.elholgazan.com/2007/08/men...">Más desarrolladores pero más expertos</a>

by elholgazán on Aug 8, 2007 at 7:02 AM

As Theodore Sturgeon once pointed out - "90% of everything is crap". This is especially true in the software industry. This statement basically means that the only truly meaningful advances are made by the top 10% of the people. I would like expand that observation some. While the envelope is pushed by the top 10%, only the top 25-40% understands the ramifications and problems associated with this new technology. The rest take a lot longer to catch up.

Yet, so many companies are trying to create the next great technology. In order to do create something new, they need a staff that has that capability, creativity, technical savvy, and expertise to do that creation. If that company does not have the talent, the project is doomed. Thus, the “expert” does not provide a 2X, 5X, or 10X in improvement. Instead, it is the difference between success and failure. (I will note that even having the proper technological staff is no guarantee to success. Projects are subjected to a number of business and internal political factors that are beyond the control of engineering.) Therefore, hiring the proper expertise is not just an enhancement to efficiency; it is a requirement for success.

On the other hand, a company that only has experts is inefficient and expensive. While a project needs expertise, there are lots of lower-level tasks that need to be completed as well. It is not cost effective to have your group of experts code up those segments that the recent grads can code. Since most of the work does not, in fact, need serious expertise, it is important to have a well balanced team.

An engineering team can be likened to any professional trade, with the artisan, craftsmen, journeymen, and apprentices. Any team of 10-15 engineers should consist of 1-2 “artisans” – those that are in the top 10% and have significant and wide ranging skills and vision. It then needs 2-4 “craftsmen” – those with significant experience and technical know how (10+ years of experience). The project needs 3-5 “journeymen” engineers – those with excellent skills, but not a whole lot of experience (5-10 years of experience). Finally, the project needs 4-5 “apprentices” – those with 0-5 years of experience.

Having the people is not the only key for success. The engineering staff needs a proper infrastructure that encourages success. Information and skill must flow from the top down. There must be a culture where proper comments and documentation are not only encouraged, but required. The top levels can not be prima donnas who refuse to share there skill and knowledge, or their value is diminished or even non-existent. This environment then allows movement up the ranks from apprentice to craftsmen, bringing the associated benefit to the company.

Note that an artisan is not someone who can be trained. He or she can not gain the knowledge in school. Instead, its must be a person with both the right innate skills that also has had the correct training and experience. While this person is extremely valuable, the value is not always noticed, nor properly used by the company. Therefore, there is no salary compensation that can work correctly – the only method is stock options. Bill Gates did not amass his fortune on a $400K/year salary – he amassed it with stocks. Same holds true for Steve Jobs, George Lucas, and any number of other artisans out there. The salary gets them in the door, but a true artisan will so help the company’s bottom line that the only real way to compensate that person is via equity.

Unfortunately, finding people in the top 20% requires a similar level of expertise in management, HR, and the interviewing staff. The probability is the company does not have the expertise to find and hire the right people. In that case, they need to get lucky – and since 90% of your respondents will not be in the top 10%, the odds are not good.

by Skip on Aug 8, 2007 at 10:01 AM

So, how do you hire a really good programmer?

Unfortunately, the resume doesn't tell you. The resume should be a conversation starter.

Unfortunately, you can't pass out a multiple choice test.

You have to be able to ask the right questions, and know when you've gotten a good response. You need to have a really good programmer do the interview. Good questions are not "What is the definition of denormalized?"

One of the best questions i've been asked: What are your favorite tools?

It was a good question because good programmers don't generally limit themselves to one or two tools. They evaluate tools, and often find really good ones. But you can't judge the answer by how many, or which tools they were. You need to listen to why any of the answers makes it to the list. So, "Oh, i like Subversion", might be a good answer, but the explanation of why will give you what you need to know. It's not multiple choice. It's the Turing test. And both sides of the conversation need to pass the test.

Unfortunately, good programmers are often not good sales people, good managers, etc. There are few people who can be said to be good at all possible skills. It's still an open question on what skills a good programmer has.

Most companies i've worked for have no way to identify good programmers. Even years after the hire, they can't tell the difference.

It's a myth that the top programmers don't get along with each other. In fact, having a true peer to talk to is a bonus for most. And even the grumpiest of the truely skilled will get respect from everyone else.

by Stephen Uitti on Aug 8, 2007 at 11:03 AM

I'll echo what many others have said in the comments: your ideas are appropriate for
a new organization or a whole new team/project. However, more typical situations involve
a team that is already in place and is dragged down. You cannot start over wholesale, so
the process of trying to hire those kickass people happens piecemeal and slowly. So
the management approach you cite does not fully address what to do in that situation.
I would be interested in a followup article that seriously attempts to discuss that,
but I sense that since the original text grew out of dinner party talk amongst angst-riddled
underpaid perl hackers, that is not in the pipeline. Echoed above by others, also,
is what advice this article really has for inexperienced programmers who really want a
chance to become wizards. From the organizational management point of view, you don't
seem to believe there much the company can do to foster that, other than hiring enough
wizards to make the people around them better (some kind of Michael Jordan effect). Rather,
you bluntly indicate a preference to avoid junior programmers whenever there is a choice.
At the same time you seem fit to say that training is "easy" and "cheap" and so I wonder
if you could post a followup that explains best management practices for getting promising
1-yr programmers over the hump to the point of becoming wizards. The way you write, you
almost imply this is independent of anything the company does or doesn't do! I doubt you
mean to imply this. Obviously there are things to be done to avoid 1-yr programmers from
falling into bad habits that will make it impossible (or nearly) for them to become
wizards. But surely you can be more articulate about how companies CAN hire junior
programmers and five years down the road turn a higher fraction of them into wizards.

by dsm on Aug 8, 2007 at 11:37 AM

A major problem in attracting grade A developers is the interview process. Nowadays there seems to be the "puzzle & hit-em-hard" process of deciding is an individual developer meets the grade.

In particular an interviewee has to produce specific trick programs in the interview. These include C questions of reversing a list in place, traversing a binary tree in multiple ways, throwing a stone off a boat and deciding if the water level rises or lowers, throwing an egg or two off a building, fox and geese crossing a river, ... ad nauseum! And this is for a C++ job!

When I ask people why they ask the puzzles, I get something like "so we can see how you think" as answer. When I ask them how they will evaluate the answers I give, I get blank stares: "we will know". They obviously have no criteria - just some kind of "feel good".

And they use C to evaluate a C++ job. That's like interviewing an auto machanic on a Model T Ford to work on your modern auto.

How many really good developers have you passed by because you have no real methodology to identify these people?

by cecil on Aug 8, 2007 at 12:14 PM

"And that means *shock horror* sometimes the developers earn more than the managers! Live with it" --

And some McDonalds/KFC/Burger King workers get paid more than I do, and I develop mission critical software...

It all depends on the company. Some companies still believe "expert" software engineers are replaceable like McDonalds workers are...it's sad, I know, but you gotta live with it.

Life is hard, and by the way, I totally agree with the article.

by MezZzeR on Aug 8, 2007 at 12:48 PM

Just reading your article makes me wonder if you have put any thought in it? You talk as though experts are born experts, which gives an indication to your inability to think further. If you remember the early history of computing particularly programming, the relationship was a mentor-apprentice kind implying that there was a novice(Average relative to expert) and an expert (who was once a novice).

The relationship in a working environment is much more complex than you think, you get people with different programming skills meaning that expert/novice is relative to an individual that is why a company is called a company due to the diversity therof.

The logic in your article is multi-channeled and gives the impression that average Programmers should not be considered; that is robbing other people of the chance of becoming experts. Who is going to be an expert once you die or loose your hands? Have you actually thought of that? You need illumination buddy. Being an expert is a journey that everyone has the right to partake in.

You might as well just annihalate the whole concept of studying computing because once graduates enter the industry, their initial phase(relative to the industry) is average. Are you saying that they should not be considered?

by neural_net on Aug 8, 2007 at 12:49 PM

"Smart programmers use Python"

Wrong. Smart programmers use what works best and don't let themselves be dragged down into language pissing contests. I use mostly Perl, but I am learning Python right now. Not to switch, or to *appear* smart - my work speaks for itself according to my customers - but just to have another tool in my toolbox (besides some other languages).

by John Bokma on Aug 8, 2007 at 12:55 PM

This is seriously great.
While C++ is my primary language, I am usually able to transfer my skill to other languages too. More than your skills in a language what matters is your ability to understand and solve the problem.
Language can be learned but programming skills can only be developed and that needs time :)

by Rohit Gupta on Aug 9, 2007 at 5:07 AM

This is an awesome article!

I'm seeing this whole situation happen at the company I work for. They're trying to fill a few developer positions that pertain to some oddball programming languages. Instead of hiring someone local who has a lot of experience and diversity (someone who could pick up a new language easily), they'll search and search for someone with experience in that particular language.

The last person they found was a POWER BUILDER developer in Canada, who they hired and moved to Omaha, Nebraska. Nice.

by Kevin Zink on Aug 12, 2007 at 6:04 AM

Interesting article. For the most part its totally true but I definitely think its not totally true. Even though experts in one language can learn a new language I think that any average programmer could learn the same language just as well as the expert.

by C Tutorials on Aug 12, 2007 at 8:31 AM

What's interesting is that even when you interview university student with no former experience applying for an internship, with high probability you can judge whether this one turns into an expert one day. This is a matter of mindset, that formal education and experience can only enhance. This effect is not unique to software development, I'm sure experts can be found in every creative field. Maybe the main problem is that some people can't believe that software development is indeed a creative activity, not manufacturing.

by Adam Byrtek on Aug 12, 2007 at 11:58 AM

Real Programmers program in 'C'... Sometimes we'll resort to C++, but only when we're forced to write MS-Windoze user interface code... Java and Perl or for quiche eaters... They're fads, nothing more... A Real Programmer can write object oriented code in straight K&R 'C'... In fact, no Real Programmer writes anything other than shell scripts in an interpreted language...

by Mike on Aug 13, 2007 at 5:05 AM

Nice article indeed! I have read through quite a number of discussions on how to hire/be a good programmer and I learned a lot and I really wish that I can contribute back to the company as a good programmer if not an expert programmer.

I'm working for a local start-up, a software house if you would like to call it so. I'm employed as an analyst programmer, which not only working as a programmer but also touch a bit on analyst side, but I find it fair enough since a good programmer should be able to analyze the problem and give a solution to cater for the problems raised.

I constantly want to write simple and easy-to-understand codes in the company and try to comment when a complicated logic is used. But that's not always the case as I always try to use simple codes. But not for the seniors here. Whenever I'm tasked to help in debugging and enhancements for a particular projects I'm totally unfamiliar of, I always found myself having hard time reading their codes and how to continue from there. It usually ends up in two different coding styles found in the same piece of codes though they work perfectly well with each other.

I always try to value add to the company, however, due to the fact that they only work with Microsoft and they are Gold Partner of MS, thus they are only dealing with projects on MS technologies. When talking about the possibilities of using languages like Java and PHP, or even Perl. They won't give a thought on it and would tell customers that we are only strong in MS technologies and we don't want to risk working on a new area which is unfamiliar to us and to prevent the risk of interoperability issues.

I'm speechless when I know that they won't even consider other solutions except MS solutions. Thus, I have no choice but only code in Visual Studio, what makes it worse is they only code in VB or VB.NET. For database, they also stick closely to MS products, Access, or SQL, won't even consider using MySQL, Oracle, etc. I'm all the while opening to other languages and technologies, I even pick up Java and PHP, MySQL, Apache, etc on my own. But all do not apply to my work. What a waste. But I'm still waiting for chance to learn as much as I can and probably find a better one next time or contribute when they can see the unstoppable trend to integrate different platforms and languages when implementing a system. Or perhaps the trend of having to code in different languages and platform due to market demands :)

That's my two cents. I can't say I'm good or expert programmer, I can only say I want to be good or expert programmer and I believe I can be one. Now I'm considering myself a programmer in probation ;)

Regards,
Jenson

by Jenson Chew on Aug 27, 2007 at 12:39 PM

Praise the ....! This is what I try to convey to my clients. I have a 15 year background in Accounting and finance therfore this make perfect sence to me. However it's easier said than done. The biggest challange I face is tring to prove this point to others. So seem to think more is better. My view is that it gives you more to argue about, thus costing more money.

by email@myrececruitatlanta.com on Sep 24, 2007 at 12:19 PM

I too strongly believe in hiring the best programmer. Sometimes however you may not find what you need, then you need to hire fresh developers and train them to the task as the next best alternative.

BTW: Very occasionally your great programmer may also turn out to be very high maintenance and have their own agenda. In a company I worked for, we had one such case. It can be a royal pain.

by Angsuman Chakraborty on Dec 24, 2007 at 6:12 AM

I have had this conversation with my management and members of HR for years. I can say that you have articulated the message far better than I have in the past. I will shamelessly steal (I mean leverage) your thoughts in my next discussion.

by Matt on Aug 5, 2008 at 8:52 AM

I think people misunderstand the role of education in the hiring process, for purposes of hiring a software engineer. Education doesn't enable or impart productivity. It merely, allows the hiring manager to recognize talent whether it be innate or just plain diligence.

Real productivity comes after the employee has been at the job for a while. There's no better training for the job other then the job itself: this is especially true to the software field.

by Bob on Aug 27, 2008 at 8:01 AM

We had a perl "programmer", he was very fond of the game perl "golf" and tried to play it with our valuable assignments. We got rid of his stupid ass before he had much chance to cause trouble, and from now on we only hire programmers skilled in the dot net toolset, and its served us very well. <a href="http://www.teomandogan.com">estetik</a>

by Estetik on Nov 28, 2008 at 11:21 AM

I think a balanced team of expert and average programmers, say 1 expert programmers out of every 3 average programmers, can drive excellent results and can also maintain the budget of the projects.

by Offshore .Net and Java Programmers on Dec 11, 2008 at 5:46 AM

Interesting.. high cost doesn't always mean high quality! I think that the best way of finding a good web designer/ programmer is to arrange a meeting with them directly. After 1 hour you will know if the company / person is professional or not. Rates ... they are always negotiable.

by website design Ireland on Mar 14, 2009 at 9:16 AM

Comments on this post are now closed. If you want to get in touch with us to discuss this post please send us something on Twitter @revsys or use our contact form. Thanks!