share
Stack OverflowPunishment for breaking the build
[+72] [40] Andreas Magnusson
[2008-10-17 08:38:06]
[ build-process management fun ]
[ http://stackoverflow.com/questions/211426] [DELETED]

I'm sitting here frustrated that a team member broke the build and then goes on a long-weekend vacation. What happens on other teams when someone breaks the build? What would you consider a fair punishment?

Just as an update: Of all the great responses it just feels plain wrong to select one as the answer, but in the end our team member paid his penance with some yummy Danish (unfortunately before we had time to get our nerf-guns to shoot him with). - Andreas Magnusson
(3) How does it always happen that people break the build right before they leave the office? I've never seen anyone break it in the middle of the day when you can actually walk over there right away to talk to them about it. - Jesse Taber
(4) Whoah - victimization isn't going to help that programmers' morale. A quiet word from the team lead is more appropriate. - JBRWilkinson
Hehe, it depends on the team. Our team is very small, friendly and we are all quite autonomous. I've broken the build countless times but I never went on a four day long vacation to Barcelona (or wherever he went) with my girl friend after doing it. That's what sets this apart, that's what earned him a punishment; he's having a great time while we're having a not so great time cleaning up his mess. Sorry if I was a bit vague in my question about that. - Andreas Magnusson
chop his head off and mount it on a pole outsite your office :) Or at least the head of an effigy of him. - DRL
(2) One place I worked, our lead PhD was going on a three week vacation. His last check-in comment on the Friday night was, "Does this work for anyone else?". Short answer: no. - Eddie Parker
I'm assuming this was the main branch and not his local branch? Because you are allowed to break your own branch. And that should be the default place you commit to (otherwise, you need source-control which takes branching into account--this supersedes continuous build systems). - TamusJRoyce
[+103] [2008-10-20 22:45:14] Tanktalus

My experience with build breaks has been that the guys checking in the most breaks are often the ones doing the most work.

Yes, there are sloppy developers checking in crap. There are also some geniuses that break on one platform or another. When the average developer breaks the build in 1% of their check-ins, and the geniuses break the build in 0.1% of their checkins, the geniuses still break the build more.

Sometimes, you need to know why. Is there a cultural or process reason? I know that in our case, those "sweeping changes" alluded to above were part of the problem. If you need to touch 120 files once in a while (yes, I've had a delivery in clearcase within the last year with ~120 elements in it, and, yes, it almost broke the build, but I managed to catch the files I missed before the next build - this time), you need to either expect build breaks or find a procedural improvement that can catch them faster/easier (continuous integration, for example, probably would have been sufficient). Ostracising your star developers who are the only ones gutsy enough to make sweeping changes probably isn't the right answer.


(6) I love this answer! - Wayne Koorts
(1) I agree with this. When I started my development career, I assumed that breaking the build was a cardinal sin. But everybody does it now and then, me included. It's not the end of the world. In many cases you can see how to fix it yourself when you discover that someone else broke it. If not, go talk to that person and find out how to fix it. There's no point in getting adversarial over something that happens every so often. I'm working in a place where one guy is a real stickler for this, and all it does is create workplace tension for no good reason. - Kyralessa
1
[+53] [2008-10-17 08:40:18] Greg Beech

We have a bee outfit, consisting of springy antenna, wings, and a wand.

This is an upgrade from our previous rabbit outfit :)

You have to wear it until the end of the day (including when going out to get lunch, if you break the build in the morning).


(1) Hehe, I'd like to see that outfit... - Andreas Magnusson
I'll have to see if I can dig out a photo of me the last time I broke the build (did I mention we take photos too...?) - Greg Beech
This is just getting better and better! - Andreas Magnusson
I say we all vote this down until we get a picture! - Aardvark
(17) I don't think embarrassing people is a good strategy. - David G
(12) One problem with trying to embarrass people is that not everyone would be embarrassed by that. The last thing you want is someone breaking the build so he can pick up pizza dressed as a bee. - Dave DuPlantis
(2) It's not really that embarrassing in our environment (it might be in others but we're very informal and casual). It's just intended to be a bit of fun, which is how everyone takes it. - Greg Beech
Hehe, cool. Where can we get one? - Andreas Magnusson
(5) To protect the innocent? But he's guilty! - Germán
(1) I see no picture or link? - Oorang
2
[+49] [2008-10-17 09:00:53] keparo

Take No Prisoners

I maintain a nerf arsenal.

Revert the check-in and aim for the face.


(8) Hate check-in reverters :) Give me a moment to fix it, will you! It's a different thing if I'm away, but if you just need to get something to build while I'm fixing it then for pete's sake just leave your change in your workspace. - romkyns
(1) +1 for "aim for the face" - iandisme
git blame file and shoot the target in the middle of the eyes. - karlphillip
Or do a Shelveset while waiting for it to be fixed, then commit the shelveset once its done - danderson
3
[+29] [2008-10-17 11:47:11] Penguinix

You Broke the Build

from YBTB [1]

[1] http://www.youbrokethebuild.com/

(1) I am laughing so hard right now ;-) - matt_dev
+1 for old woman giving the finger - Martin
4
[+28] [2008-10-24 11:12:31] kenny

$1 in a bucket that is saved for release celebration, beers, etc...


(2) I like it, contribute to group morale :) - JettGeek
That's what we used on the Blacksite project. - 280Z28
5
[+26] [2008-11-05 01:02:00] vfilby

None. Encourage programmers not to be scared. It is when they break but not fix the build, then may {insert choice higher being} have mercy on their soul.


(1) I worked on an enormous project that took nearly 24 hours to build, so breaking it meant a lot of time lost. 2000 people worked on that project, and many were waiting for a good build to come out, so that made it all the more expensive. Don't break the build. - Jay Bazuzi
(12) Programmers scared of breaking builds will be scared of making changes, and therefore scared of working on the code. Not productive in the least. Revisit program without fear. Sounds like the project was a little too monolithic and poorly architected. 2000 people shouldn't be responsible for or dependent on a single build. Individual components with stable version would have been a better choice. Alternatively implement some measures to detect build break changes before they get checked into source control. - vfilby
imho, continuous integration that notifies everyone of build-breaking makes breaking the build no big deal at all because you'll fix it. You lose like 2 seconds of time. So CI makes punishment obsolete in my experience. - apollodude217
6
[+25] [2008-10-17 08:42:07] thatismatt

USB foam dart gun [1].

alt text
It is the geek way!

[1] http://www.engadget.com/2005/12/02/target-your-co-workers-with-usb-air-darts/

If I had such thing on my desk, I couldn't wait for a broken build to use it. +1 for the suggestion though. - OregonGhost
(5) could could probably code something watch for broken builds and aim and fire it at the person who broke it automatically. - Scott Cowan
I got one that didn't work as well as described on the web site. - stephenbayer
(1) Yes; scott, that's exactly what I wanted to implement here. We didn't get around to it though. Definitely the most awesome solution. - Noon Silk
7
[+24] [2008-11-14 02:23:46] Kyralessa

A guy I worked with, in a past job, worked in a place where if you broke the build, you got to have, all to yourself, in your very own cubicle:

A giant cardboard cutout of Jar-Jar Binks.


Oh, the horror!! - Evan
8
[+20] [2008-11-30 07:31:52] Dustin

We have a CI system that will send an email and let people know that the build has broken (tests are failing or whatever). It saves us a lot of time. But seriously...

It's just not a big deal.

Why? We use git, so one of two things happens:

  1. The rest of us continue to work without those newest changes until the guy who pushed bad code fixes his code.
  2. One of us throws away his changes and resets the main branch and lets him rework it and try it again.

The biggest problem I've run into is having people who don't check in changes. I'd rather have pushed changes and build an after-the-fact branch to stash stuff away while it gets fixed than to have people who sit on changes for months to hide stuff.


9
[+15] [2008-10-17 10:39:38] philippe

We sometimes use fingerpointing:

alt text

(from codesmack [1])

[1] http://codesmack.com/tshirts

10
[+14] [2008-10-25 18:04:57] Tim Tonnesen

Yes, breaking the build and going on vacation (or even home for the night) is bad. I'm talking about the occasional break that gets fixed right away, which is what most of the replies in this thread seem to be referring to.

On our team we have no such "punishment policy" no matter how unwritten or informal. I see no need, because we all know that breaking the build keeps the others on the team from updating or checking in (and that's all it does). That's plenty of motivation for us to be careful.

So where is the need for punishment? To me it smells of 8 year-olds on a playground pointing at each other singing: "nyaa, nyaa, nyaa-nyaa, nyaa... you broke the bui--ild."


(2) I agree. unprofessional. - Tim
(2) I once worked on a team where one developer always broke the build. EVERY check in. And he was a senior developer. Because there was no policy, there was no discipline. And discipline was most certainly needed. - salt.racer
I think while I agree about it potentially being unprofessional, in a good team the 'punishments' are known to be there to raise morale rather than point and laugh. Everyone will break the build at some point, so it's not personal. If someone gets upset about it just back off and don't 'punish' them next time. - Kurucu
11
[+13] [2008-10-17 12:55:25] Greg Whitfield

Is death a bit harsh? :)


(24) depends on the proximity to the launch date. - Rob Allen
we should be able to give +1 to comments lol great one - fmsf
+1 for Rob's comment. - Bill Williams
12
[+12] [2009-02-06 18:45:32] Jeffrey Fredrick

Revert the offending commit.

Having to merge changes when they return is usually punishment enough.


I like it! It's punishment, but also doesn't upset the more politically correct people :) - Kurucu
13
[+11] [2008-10-17 08:39:53] jalbert

We have a rubber chicken that hangs outside the cubicle of the person who most recently broke the build.


14
[+9] [2008-10-17 08:41:54] harriyott

The person breaking the build becomes the build administrator


(2) What? You're not using automated builds? - Kelly French
15
[+9] [2008-11-30 07:23:34] J.T. Hurley

It's my understanding that even "fun" broke-the-build markers are considered to be less than optimal because they discourage frequent check-ins, which prevents problems from being caught earlier while they're smaller, not to mention increasing the difficulties of merging.

On the other hand, I did like the dollar into the beer fund suggestion.


16
[+9] [2009-02-05 10:18:06] Evan

Have the build server automatically start playing country western music over a loud speaker which cannot be shut off until the build is fixed...


Collective punishment! :) This reminds me of tales I heard from Romania, where if a few people didn't pay their hot water bill, they'd shut off hot water for the whole apartment building. - Kyralessa
17
[+8] [2008-10-17 08:46:31] Nicolai Reuschling

I "suggest" donations to charity organisations. Works great for people coming late for meetings, too.


Shouldn't it be, I suggest "donations"? I find the quotations around the word suggest to be alittle odd, and maybe a little... insidious? It could just be too early in the morning however. - James McMahon
I guess it is 'suggest' spelled 'o-r-d-e-r'. - Remou
Well, I am not a native speaker. Please be gentle with my wording here. - Nicolai Reuschling
(1) He makes donation offers they can't refuse ;-) - ypnos
I would be more hesitant to be late for meetings if the punishment was removing money from a pot which would otherwise be going to charity. For instance, if there was a budget for $1000 to go to charity out of the company's funds - and $5 removed from it if you're late. - Smashery
+1 for the charity for coming late to meetings ... You have Outlook, it reminds you TWICE ... come on time! - Martin
18
[+7] [2008-10-17 10:35:30] Daniel Earwicker

About five years ago things got so bad that I got very "librarian" about it, and started a spreadsheet with WALL OF SHAME in large letters at the top and entered every breakage into it. Each row had the name of the person responsible, the reason why it occurred, and an explanation of the steps taken to ensure it didn't happen again. It was surprisingly effective, but excruciating to have to administer. And people complained about it. So I told them "This is an optional scheme - if you don't want to take part, just never break the build."

Now we have continuous integration, the full nightly build only breaks extremely rarely - if there's a powercut or a disk goes pop. Though the old timers still speak in hushed tones of the dark days when we had THE WALL.


19
[+7] [2008-12-23 15:54:00] Martin Brown

We have just started throwing shoes [1] at the offender.

[1] http://www.youtube.com/watch?v=9uIj0YvDBKE

ROFLMFAO its good even a week later. - Unkwntech
Good one. Given the fact throwing shoes started again .. :) - Guru
20
[+6] [2008-10-17 10:43:46] philippe

What about punishment ?

alt text

(from codesmack [1])

[1] http://codesmack.com/tshirts

(4) for (int i = 0; i < 1000; i++) Console.WriteLine("I will not break the build."); - Martin
21
[+5] [2008-10-17 08:46:53] John Mulder

The person who broke the build is now responsible for testing the build.

(with the benefit that that programmer will be incentivized to catch the next offender)


22
[+4] [2008-10-17 08:39:48] Vyas Bharghava

You seem to practice agile, you better check it out and fix it, NOW!!! :)

Update: Punishment for the guy who broke the build, fix the next broken build ;)


23
[+3] [2008-10-17 09:22:24] bmatthews68

I know one team that made the developer who broke the build wear a dress for a day.


(1) I know developers that wear dresses on regular days... do you have any women at your company? - emddudley
Actually on that project which was split over 3 sites (2 US, 1 UK) there was not one woman working as a developer. - bmatthews68
24
[+3] [2008-10-17 10:17:05] andygeers

"He whose release doth cause offence shall bake a cake in recompense"

Serves us very well, and we get some great cakes out of it! Helps people take responsibility but without feeling beat up about it.


(3) Problem is of course whether the person's cake-baking skills is on pair with their build-breaking skills... - Andreas Magnusson
So change "bake" to "buy" ... - Dave DuPlantis
"Four large eggs. One cup semi-sweet chocolate chips. 3/4 cups butter or margarine. Fish shaped crackers. Fish shaped solid waste. Alpha resins. Unsaturated polyester resin. 2/3 cups granulated rhubarb. One tablespoon all-purpose rhubarb. One cross borehole electro-magnetic imaging rhubarb." -GLaDOS - Erik Forbes
Be careful of disgruntled workers -- who might spike the cake with say.. x-lax or other more natural ways of punishing people! - torial
25
[+3] [2009-02-06 18:55:23] community_owned

A small, stuffed Cthulhu in a coffee cup that sits on the culprit's desk, hight "Works". The developers always say "Works on my machine", so we made it so!


26
[+3] [2008-10-21 10:07:13] Suma

The best way how to deal with this is to create automated checkin policy, which prevents committing (Checkin in) broken builds, or at least reports the broken build very soon after checked in.

One possible implementation for this can be:

  • have a dedicated build server
  • have a set of automated functionality tests (executed on the build server) which check the build can successfully built, can be run and performs basic functionality OK

27
[+3] [2010-03-12 09:58:03] David

I'm strongly disagree with any punishment. A simple friendly and private note from the others is enough.

Punishing others could damage moral. Maybe it sound fun, but could hurt someone feeling and to avoid mistakes in the future it minimize his development efforts (loosing creativity).

For me CI means to realize mistakes as soon as possible and not, developing without mistakes. Every human makes mistake. Any kind of solution, which based on: "that this is so simple, no one ever going to mess up" is going to fail.

As always if the build can be broken than it is architectural mistake. Saying don't break the build is no different as the note: Please use large capitals in this text box or we cannot parse your order and can seriously damage our servers. Or with other words. How do you going to develop user friendly stable applications, if the tools and processes what you use and develop for your self aren't satisfying these criteria.

If you use any kind of script or build tool you can create a process, where the check-in procedure involves code update, build, QA and check-in.

However this could only lower the number of broken builds. As for example special file extensions wasn't checked in into the last step could affect the build.

The solution is double QA. When you fix someone bike, you first test it then the customer test it as well before accepting it.

If you use transactional SCM, than you can maintain two branch. One is where everyone checks in the code. On that branch a build tool runs, which builds and QA against every transaction (check ins). If it succeed it pass the transaction to the main branch. If not it throws away that transaction. (Also notify the checker :-) ).

With this the developers will have a branch, where they always have stable code. And a branch where they can check-in without risk.

But have in mind the effort and benefits. If you are working locally with 2 - 5 person, spending to much time creating this environment maybe not worth the effort. A simple please fix it could be enough. On larger teams working in different location could be handy.

Also this won't answer code QA issues on some level, recognize integration issues on early level and point out invalid business logic.


First off, as I said in a comment to the original question it was all done in jest between equals. I'm not his manager. The key point was not breaking the build but taking vacation after doing so. This is a highly skilled dev who should know better we're talking about. Secondly, in work life you may not always get to choose the tools nor the way code gets developed. If you're working on a 10+ yrs old code base, chances are you're stuck with a great deal of old tools and practices. - Andreas Magnusson
Everyone makes mistake. But with punishment we are only damage the moral, but the problem will exists. I know what you are talking about. For e.g.: currently the whole tech team went for vacation after bringing the new system alive. But before that they left me a note that please undo the build fix what I introduced because they not planed it. Not because it is not working. It is not yet decided. Now these guys is the not now and not ever types. They don't have the skill to fix or undo this. So it will stay there and they soon find something else to manage. Sometimes pro-activity is the key - David
28
[+2] [2008-10-17 13:33:08] stephenbayer

I guess someone could have to write "I will not break the build" 1000 times, but then again, programmers would just do:

for (int i=0; i<1000; i++) System.Console.WriteLine("I will not break the build");

but in the projects I've use continuous integration on, we used CCTray [1], and it popped up with a message to all developers on the project within a minute of checking in broken code. All the programmers would just convene at the desk of the culprit shaking their fists and suggesting that it be rolled back or fixed code be submitted. It usually wasn't an issue.

[1] http://confluence.public.thoughtworks.org/display/CCNET/CCTray

29
[+1] [2008-10-17 08:44:43] Greg Hewgill

One placed I worked, we had a broom. Its use came from checking in "sweeping changes" which usually broke something or other. So we appropriated a broom and it eventually changed into an I-broke-the-build indicator.


30
[+1] [2009-02-05 10:19:55] Evan

I did hear of one place that had a big green lava lamp, and a big red lava lamp. When the build was okay, the green one was lit. When the build broke, the red one would light up.

Not really a punishment, but a pretty cool I thought.


31
[0] [2009-02-05 10:46:24] Bhushan

We have a big red color foam hand pointing index finger. Whoever breaks a build or does something nasty that hand is kept at his/her desk with finger pointing towards them.


32
[0] [2010-06-17 00:34:52] jpierson

At my current company I had an svn precommit hook that would build the project with the new changes on the build server before it would allow the commit to be accepted. All was well until the project grew and build started to take longer and handling special cases of commits happening while a build was in progress due to a prior commit by someone else. While it was in place it was nice though because no build breaks ever occurred and it gave us a place to run any future automated tests or code analysis as well in order to also reject the commit on that basis. At the end the developer would get a little message back from SVN saying commit accepted or commit rejected and the reason.

Now since I've disabled this due to the slow performance of the build we just tend to declare the usual shenanigans and rally the villagers until we get an apology.


33
[0] [2011-06-15 13:26:47] prostynick

In my previous work the person who broke the built was obligated to buy pączek [1] (think of it as an equivalent to doughnut) for every project member (about 30 pieces), so the person needed to spend payment for 1-1.5 hours of work - not so much, but enough to be careful.

[1] http://en.wikipedia.org/wiki/P%C4%85czki

34
[0] [2008-10-17 08:40:41] Simon

We all do it from time to time, but some people are definitely worse than others.

A publicly visible count of build breaks with blame attached ought to do it. The common culprits will become obvious.

Do it in a light hearted way on one of your common servers.


35
[0] [2008-10-17 09:06:58] redsquare

this is more a training issue, punish the team lead I say


Then Sun Tzu said, “If the instructions and words of command are not clear and distinct, if orders are not thoroughly understood, the general is to blame. But if the commands are clear and the soldiers disobey, then it is the fault of the officers.” - Andreas Magnusson
He [Sun Tzu] immediately ordered the women who were at the head of the two troops to be beheaded. - Andreas Magnusson
36
[0] [2008-10-25 16:08:19] Joshua McKinnon

We have a whiteboard on the wall in the hallway. For any break that should have reasonably been detected, if your breakage makes it past the 5-10 min build and on to the 2 hour build, your name gets added to the board.

We also use the continuous integration game with Hudson, so people want to be the one with the highest score.


37
[0] [2008-10-17 13:11:30] codeLes

we use a Mr. Potato Head. When the build is broken then the Potato Head gets set on the edge of the breaker's cube for all to see.


38
[0] [2009-08-13 01:43:48] banister

spank them with a mechanical corset


39
[0] [2009-12-20 11:14:00] fastcodejava

No punishment is severe enough!


40