Stack OverflowHow to define code quality in a contract?
[+70] [20] Daniel Rikowski
[2010-08-20 11:29:05]
[ requirements code-quality outsourcing ]

At my workplace we want to outsource some parts of our software development project to an external contractor (another company with multiple developers). Our greatest fear is that although all functional requirements will be met, the code may be very bad, as it is often the case with outsourced software projects.

We can't use any of the conventional methods to ensure code qualitity, like peer review or pair programming, so our first idea was: "Hey, make code quality part of the requirements."

But how can we actually do that?

Even if we review the code, we must be able to say: This is not what we agreed. So we need something which can be quantified, something that isn't open for debate.

On the other hand we don't want to scare off every contractor, because the definitions are impractically strict. We don't want perfect code, but at least a minimum of code qualitity should be ensured.

(2) +1 - Really a tough question to answer given that there isn't an "industry standard" way of measuring code quality. We have a build server that generates a % quality based on several tools (CPD, PMD, FindBugs, etc.) However, anything that produces some "metric" can (and will) be manipulated by the developers. - Tim Reddy
(2) Wow, code quality metrics are not related to programming and software development. Who would have guessed? - nikie
(2) what an awesome question - Epaga
[+36] [2010-08-20 13:07:11] Julien Hoarau [ACCEPTED]

Controlling an outsourced project quality

If you want a particular project quality, you will have to :

  • Integrate clear quality objectives in your contract
  • Have a clear vision of the project's quality during the development
  • Use an independent system that acts as a trusted third party to validate the level of quality

Clear quality objectives

To help your contractor to comply with your objective, translate your exigencies into concrete technical requirements tailored to your issues.

  • Selects the best practices that must be follow based on the technologies used (Fxcop, StyleCop rules for .NET / Findbugs, PMD, Checkstyle for Java)
  • Sets appropriate tolerance thresholds to achieve the optimal level of quality :
    • Cyclomatic Complexity must be below 20
    • Method size must be below 50 lines
    • No duplicated block with a size superior to 10 lines
    • Test coverage must be superior to 80%

(Indicative threshold, you should adapt the value based on the expected quality. Overquality isn't good either for your project $$$)

Clear vision of the project's quality

In your contract, ask for regular delivery. Evaluate the delivery from your contractors to ensure that they are following your requirements and that they will produce the level of quality necessary for the success of the project. If not pull them back on track.

How to ?

Manually using a bunch of tools

You could use a bunch of tools (For Java : Findbugs, JavaNCSS, Simian...) and manually analyze the project with the right configuration (the specified one). Your contractor will do the same to ensure the quality of the project before delivery.

Using a platform to manage quality

The easiest way to do all that, is to use an existing quality managing software. This software must be available to you and to your contractor. It'll be used as an independent system that acts as a trusted third party to validate the level of quality.

Examples :


(8) +1 for taking the time to correctly format a long answer. :) - Mizipzor
(3) "No duplicated block with a size superior to 10 lines": this is absolutely crucial. I've seen seventeen separate copies of a "formatZipCode" method that all did the same thing in different ways. The one bit I think needs more detail here: "Coverage must be superior to 80%"; define coverage. We also had one developer - out of many - who simply wrote tests that returned true. "Covered!" - Dean J
(3) @Dean J: Now, define "duplicated block". Character by character duplication? Duplication if you normalize whitespace and character names? What about slightly different function use? Copy-paste coding is bad. Doing much the same thing without using copy-paste can be worse. - David Thornley
Detecting "doing much the same thing without using copy-paste" could be difficult and could result to a lot of false positive masking the real duplicate (copy-paste one). - Julien Hoarau
@David Thornley: Definitely character by character and normalizing whitespace and variable names. It's harder to automatically spot slight variations, but you're absolutely right; rewriting code without copy and paste is worse, but unfortunately harder to catch. - Dean J
Good answer, but I still don't think it will work. All metrics can be gamed and outsourcing companies are great at this. A metrics is only a proxy for what you really want, which is ease of maintenance. To use the "duplicated code" example again: what if they copy/paste a method and then make small changes to the logic (eg. subtract instead of add a number)? What's worse is that in some cases this approach is actually justified, because it's more maintainable than the alternative - but that kind of decision is pretty much impossible to put into a metric. - EMP
Metrics should be a guideline / preference, not a hard requirement - else you'll get funky tricks to, for example, reduce a method length to less than 50, or different formatting in copy/pasted code. Of course, that's based on only checking that using the tools - if you find that out from your own code reviews, you can still refuse the code. - fwielstra
[+14] [2010-08-20 11:58:43] Dirk Vollmar

Use a test balloon: Let them start with a small package first containing only a small feature set (equivalent to a very first milestone). Indicate clearly in the contract, that you expect a high quality of the code.

They might possibly get that a successful delivery will lead to a new contract and deliver high quality. If you are not satisfied you might decide to do the work yourself or find somebody else without having wasted too much money yet.

In addition, it might be useful if you have someone regularly reviewing the code. It's better to define smaller work packages that can easily be reviewed than having one big milestone where it would be too late to ask for any changes.

I like this idea - make them work for the contract. If they can deliver on a small portion, then you can give them a bigger assignment. Of course, this depends on the size of the eventual project and its costs. - fwielstra
[+13] [2010-08-20 11:33:29] David Christiansen

Have you checked out FxCop [1] or StyleCop [2]?

They are good at sniffing out 'bad smells' and can be integrated into your build.


(1) and can be customized to enforce any rule as lonag as noncompliance can be found using static analysis - Rune FS
[+8] [2010-08-20 11:34:42] Lazarus

If you want particular code quality then you'll have to provide your outsourcer with your coding guidelines and ensure your contract includes a clause stating that these must be followed or code may not be accepted.

(9) If it takes you five years to become a senior dev, and four years of college before that, "writing code quality into a single document" is the equivalent of "writing down every bit of important experience you've gained over the last decade into one document. you have a week." It's really, really easy to say, and almost impossible to do well on a deadline. - Dean J
(1) @Dean-J : You are over-engineering a coding guideline document, it doesn't attempt to distil the experience and knowledge of a senior dev, that's a ludicrous idea. It set out a framework of guidelines within which the 'supposedly' experienced outsourced dev can work. I work for a global enterprise, this is what they do. - Lazarus
[+6] [2010-08-20 13:23:38] Saul

This is not a technical issue. It is about trust.

I suggest concentrating your efforts on finding the right partner. Attempting to manage risks by focusing on effects instead of causes is not a very good approach.

Being open about your concerns and speaking frankly with your potential subcontractor helps also.

Take a couple a days to brainstorm with your team and come up with, say, 10 useful guidelines for code quality as you see it and present those to the other side. Take note of their reactions.

Find the right people. When you are not sure, accept the risks or walk away. Communicate. The is no silver bullet on the technical side for ensuring code quality.

(2) +1, As a contractor, I would be hesitant to bid on a project with an amorphous concept of 'code quality' as one of the requirements. I always work to write quality code, as I put my professional reputation on the line whenever I deliver code to a client. Interview a contractor like you would interview a potential hire and remember that software consulting should be a collaborative process; when it becomes confrontational, both sides clamp down in an attempt to reduce risk and the process becomes much less pleasant and productive. - Dan Bryant
+1 Communication and having the right people who are of the same viewpoint as you goes a long way - InSane
[+5] [2010-08-20 15:38:18] InSane

EDIT : I realise that i havent directly answered the question at all but i think that if you are more interested in making your sub-contracting effort work and get your work done on time and well, the below points are probably more important than any contract you can sign.

As a person who has worked on both sides of the fence, i think that the most important thing is to make sure

1. Communicate the architectural decisions made and their impact

  • Communication is one of the biggest reasons i have seen especially when it comes to remote teams. If the architecture of the project and the core high level design decisions are clearly understood by the sub-contracting team.
  • In a lot of cases, the sub-contracting staff is not a part of the architecture decisions and not knowing the rationale behind decisions goes a long way towards making wrong choices even by well-intentioned developers.
  • This kind of communication of key project architecture / design practices is unfortunately not something that works well remotely, im my experience.
  • Eg: We actually had people from our core architecture team spend 1-2 weeks with the sub-contracting dev team to make sure that key points were understood and documented.

2. Ensure that you have people who have the same mindset as you when it comes to the code on the other end

  • The only way to do is to interview people. It is not at all uncommon to interview sub-contractors before they join the team. Interview them not just on technical questions(that is not enough) but by giving them practical real life examples involving code cleanliness, patterns, refactoring etc and gauging their response to it. This goes a long way towards getting people with a matching mindset on your team.

  • Most important is to get the right person leading the sub-contracting team. You dont necessarily have to get all developers who are mirror - images of you on the subcontracting team. If you can get the LEAD there though who is atleast as close as you would like to someone you actually work with / yourself and make sure he is accountable for the rest of the team, most times, he will automatically ensure that he gets the right people for his team. I would not ever EVER recommend being ok with a NON-TECHNICAL person as your outsourcing team lead. He will never be able to staff / control the team correctly especially in code quality and related issues.

[+4] [2010-08-20 11:37:16] rmx

You could use some form of metrics to measure cyclomatic complexity and coupling and give an overall target, say, 60/100 (in the VS default metrics tool). Its not foolproof however and the only way to really know the quality is to review it.

edit: Actually, on second thoughts you might be better off taking a different approach. Try to gauge the quality of other code that company has produced and estimate what your product will be like from that. I read something yesterday about outsourcing to India and how it generally produces poor code. Go for a company with a good reputation - youll get what you pay for. I think it's too difficult to measure otherwise.

Do you have a link/reference to that article? We had bad experiences with India a few years ago, too, but my boss still thinks all problems we had were one-time problems because of bad luck. (And not because outsourcing software projects to the lowest bidder is risky in general) - Daniel Rikowski
I will look around - it might have even been on SO. - rmx
There are some very good programmers in India. There are some very cheap programmers in India. I haven't seen evidence that those sets have any common members. Since outsourcing is generally driven by short-term cost measures, outsourcers tend to wind up using the cheapest, and hence worst, Indians available. What would happen if the company hired the cheapest programmers available in the US? - David Thornley
Ah I didn't mean to come off as discriminatory. I'm sure India has both good and bad but it's just common that when you hear 'outsourcing', you think of India. - rmx
@David: 100% ACK. The given article (written by an Indian) just states the same thing. Nevertheless the association exists, just like Germany and lederhosen or France and champagne. - Daniel Rikowski
@rmx: I wasn't implying discrimination, but rather explaining why outsourcing to India generally produces poor code. It's unfortunate that this gives some people the idea that Indian programmers are bad, of course, because that certainly isn't the case. - David Thornley
[+3] [2010-08-20 11:55:57] Chuck

Require a minimum of 100% unit test coverage (or as close as is reasonable for your situation) and establish a baseline cyclomatic complexity score. Class/Method/Variable naming standards should also be established and adhered to and all methods and classes should be documented with standard comments (not normally a fan but in this case it will help ensure they're at least reading what they're copying/pasting from expertsexchange and here)

All those things can be quickly checked and, while not perfect, can at least establish a baseline. I would also require real-time access to the work, daily check-ins, and do a weekly code review of one or two random files. That way you have some chance of catching stuff before it goes horribly wrong.

(1) though I agree that test coverage should be part of the deal it's very shaking. Depending on the culture/people developing. I've used that approach and gottan tests that only executed code and had no assertions but only comments like: "This test is to increase coverage" if only it was possible to measure test coverage and not just execution coverage - Rune FS
Good point. I'd rather have no test than a test that is there simply to "increase coverage". At least with no tests you aren't misled. - Chuck
100% test unit coverage is almost impossible, and in most cases, 30-40% of those tests will be pointless. 100% coverage of /important/ code I can live with. Of course, then you have to set up guidelines to what is considered important code. - fwielstra
[+2] [2010-08-20 11:32:00] Federico Culloca

To solve the copy-paste-pasta problem you could ask for specific code-formatting standard, variable naming and such. At least they'll have to re-read what they pasted and - who knows? - maybe they'll think twice before doing this.

[+2] [2010-08-20 11:49:11] Jon Smock

Code quality is so subjective, especially if all the functional requirements are actually met. The best you can do is request certain concrete things like using an MVC pattern.

Perhaps you can require checkpoints for code review.

But a large part depends on hiring the right people to do the job.

Edit: I tried to find quantifiable items/checklists for code quality, and the best I could do is Wikipedia's list on code quality factors [1]. If the developers still don't want to fix they're code, these factors might not hold up, but they're at least more specific than just saying "quality".


[+2] [2010-08-20 13:46:46] Dean J

I just went through this [1], and it's a crapshoot. As the "example that worked" to explain this to management, management definitely requires a four year bachelors degree to write code here. They hire senior developers with 5-7 years experience as a minimum for a technical lead.

So, if you have your best developer, who has a minimum of nine years of experience doing just this, you're asking that developer to distill everything they've learned into a single document that can be easily used to verify other developer's code.

It's not impossible, but it's a really, really hard thing to do well.

FindBugs [2] and PMD [3] are code checkers that report back possible problems. Make absolutely sure, above all else, to run something like PMD-CPD, a copy and paste detector. Have a cyclomatic complexity [4] cap.

If it's an option, split requirements, coding, and testing to three separate contracts and companies; do not let the testers write the code or requirements, or have close contact with those that do.

Require enough comments in the code that someone can follow what's going on. Make incorrect comments - those copied from another place without updating them - be something that will slow down payment to the contractor.

In general, make quality a requirement of them getting paid, and make that very clear up front. Review potential contractor's sample code for quality ahead of hiring them, if that's an option. Largely, we had to teach our outsourced team quality from scratch, as they'd really, really, really never written any maintainable code before; each new version that they were paid for was previously largely a rewrite, so paying them to improve the software was billed at their cost for redoing everything.

That works fine... on simple applications; the problem is several versions in, when they didn't quite plan for any future of your project.


[+1] [2010-08-20 12:21:50] Peter M

I'll suggest another carrot/stick approach.

First of all you have to define what is good code quality. If you can't do that then you can't measure it. (There have been some other replies that have mentioned various tools for measuring complexity etc.) This is probably the hardest thing to actually do.

Then you have to define your contract. Not a code contract, but a business-to-business-real-life-in-the-flesh contract that is physically signed by both parties. This contract needs to state that the outsourced code has to meet you minimum metrics for quality or you will either apply a payment penalty or withhold payments until they correct their work and bring it up to your standards.

The carrot part of this contract would have clauses like bonuses for meeting quality requirements in under the agreed time frame.

If you don't have a physical contract with penalties then there is no incentive for the outsourcing company to supply code to meet your standards.

(1) "First of all you have to define what is good code quality" - The question was: how do you define it? As far as I can see, you don't answer that question, do you? - nikie
@nikie - I can't define code quality for someone else as it is entirely subjective. Only the OP knows what his quality requirements actually are, and his question was framed as how does he manage/enforce those requirements. - Peter M
If it takes you five years to become a senior dev, and four years of college before that, "writing code quality into a single document" is the equivalent of "writing down every bit of important experience you've gained over the last decade into one document. you have a week." It's really, really easy to say, and almost impossible to do well on a deadline. It's why outsourcing is often incredibly flawed. - Dean J
@Dean - I don't think that outsourcing is flawed per se as after all manufacturing has been outsourcing for years. Also you effectively outsource when ever you go to the supermarket and buy food rather than grow it in house. However software outsourcing industry seems to think that quality is not an issue - something that all other industries that outsource have long since dealt with. - Peter M
@Peter M: Manufacturing and programming are entirely different things. Manufacturing consists of "Make me two million copies of this", and the design should suggest quality tests. With software, making copies is trivial; instead, you need new things that are properly designed. It's much harder to outsource design (even fairly routine design) than duplication. - David Thornley
@David - I agree but I believe that what you are highlighting is that the outsourcing industry is saying "software quality is hard, so we won't even attempt it". That position wouldn't fly in any other industry. - Peter M
(1) @Peter M: it's very easy to specify an LED, or a physical part of any kind. I want a diode light 5mm wide, with a 9mm and 10mm pin of 22 gauge solid core wire, that produces a 5400k light in reference blue, lasts 100,000 on/off cycles, and takes .03 amp. Done. It's harder to specify an entire car. It's impossible to specify software with that level of precision, without it being cheaper for the people doing specifications to write it in-house. - Dean J
[+1] [2010-08-20 12:33:30] dwarFish

+1 for Lazarus.

Provide coding & development standards and make them sign up to follow those. Allow them to review those standards in advance. What you are trying to achieve is cheaper coding to the same standard that you would deliver yourself. Make sure that those standards tie up with what you do and don't go easy on them or try and make life harder for them than it is for you. If you feel comfortable with it, show them evidence that you review compliance yourselves but you don't have to do that because you are the customer.

The standards can include test case coverage, review coverage, pass rates, or indeed anything that is right for you in your organisation.

Reserve the right to audit compliance with your standards where there are issues. If they deliver quality on time, leave them to it. Where they deliver defects, audit the code, the code review and their standards compliance. Reserve the right to penalties or termination once they fall below a compliance threshold. The same applies if they are late but take care not to put delivery ahead of quality.

If they wont sign up to that, walk away. Outsourcing companies are ten a penny.

[+1] [2010-08-20 13:25:02] nikie

I'd argue that if you put a quality metric in a contract, it's always possible to write low-quality code that formally fits these metrics. Simply because a quality metric in a contract has to be a formal criterion, and quality isn't a formal criterion.

My advice would be: Split the projects into small parts. Review each part when it is completed. If the part is up to your coding standards, fine, the contractor can do the next part too. If it's not, give the contractor two choices: Either (a) she improves the code or (b) she won't get a contract for the next part. That way, you can say "this isn't good enough, please improve it", even if there's no contract-ready quality criterion.

[+1] [2010-08-20 16:20:50] Denis Kniazhev

Whatever tools one uses, the following will always be the problem. He will never be able to distinguish the code which was designed well and the code which seem to be OK but which is wrong from architectural point of view without the proper analysis. Proper analysis means involving tech experts, and this in fact means more money and time spent. I can't see the other way if one wants flexible, reliable and maintainable code from outsourcing.

[0] [2010-08-20 11:32:45] Nikolaus Gradwohl

you could define a maximum error/warning rate for tools like checkstyle [1] or pmd [2]


how would that fix poorly designed functions? - user195488
it would prevent several common errors, or very complex functions - i don't think there is a method to guarantee 100% cool code - Nikolaus Gradwohl
Agree. But that is his question. - user195488
i know. you can measure some quality aspects using pmd or checkstyle and that is what you can define in a contract very easy. you can't get a 100% certainty but measurring the quality aspects is better than nothing and you can at least enforce some codestyle rules or quality measures - Nikolaus Gradwohl
[0] [2010-08-20 11:33:39] wadesworld

If it's not going to be reviewed, it's not possible. There's no automated program that can determine code quality.

Defining something like maximum errors in a static analysis tool doesn't buy you a thing to determine code quality.

(1) Of course we would review it, the problem arises when we say: "this is bad" and they say: "please show us the page of the contract where it says 'this is bad'" - Daniel Rikowski
Write into the contract that your lead developer must accept the code, and is willing to work with them on a fluid description of acceptable code. - Dean J
[0] [2010-08-20 15:14:23] Dan Bryant

The problem with quality metrics as a criterion for acceptance is that a low quality developer will simply write the code in the way that they know best to write it, then muck with it until it passes all of the metrics. Metrics also can't capture some of the most important aspects of code quality, such as whether the names of symbols are meaningful and lead to easy comprehension and maintenance of the code.

It's reasonable to expect a certain standard of quality and even to include it in the contract, but the consulting relationship requires a certain degree of trust. If you try to reduce risk by giving yourself 'escape hatches' to avoid low quality code, this will likely discourage higher-quality developers from bidding on your project, as they are also interested in managing risk.

[0] [2010-08-25 12:46:16] Matthew Sposato

NASA Documents Online [1] lists some of the industry/academic articles on software quality. Scroll down to the Software section.


[0] [2010-10-19 10:39:47] Patrick from NDepend team

Use NDepend and its Code Query language (CQL) [1]. CQL is dedicated to write code quality rules that can be verified live in Visual Studio, or that can be verified during build process and through a report. Here are a few samples of CQL rules:

I don’t want that my User Interface layer to depend directly on the DataBase layer:

WARN IF Count > 0 IN SELECT METHODS WHERE CyclomaticComplexity > 15 AND PercentageComment < 10

I don’t want that my User Interface layer to depend directly on the DataBase layer:

WARN IF Count > 0 IN SELECT NAMESPACES WHERE IsDirectlyUsing "DataLayer" AND NameIs "UILayer"

Static fields should not be named m_XXX (Custom naming conventions):

WARN IF Count > 0 IN SELECT FIELDS WHERE NameLike "^m_" AND IsStatic

Methods out of MyAssembly and MyAssembly2 should not have more than 30 lines of code:

WARN IF Count > 0 IN SELECT METHODS OUT OF ASSEMBLIES "MyAssembly1", "MyAssembly2" WHERE NbLinesOfCode > 30

Public methods should not be removed to avoid API breaking changes:

WARN IF Count > 0 IN SELECT METHODS WHERE IsPublic AND IsInOlderBuild AND WasRemoved

Types tagged with the attribute MyNamespace.FullCoveredAttribute must be thoroughly covered by tests:

WARN IF Count > 0 IN SELECT TYPES WHERE HasAttribute "MyNamespace.FullCoveredAttribute" AND PercentageCoverage < 100