share
Meta Stack OverflowWhy do incorrect answers keep getting "accepted"?
[+112] [22] Sherm Pendley
[2008-11-12 17:26:11]
[ discussion ]
[ http://meta.stackoverflow.com/questions/7572/why-do-incorrect-answers-keep-getting-accepted ] [DELETED]

Seeing incorrect answers get "accepted" makes me seriously doubt the value of Stack Overflow. For example, this answer [1] about Perl's symbolic references was accepted - but any competent Perl programmer knows that it's a horribly poor approach, and the answer that says as much outvoted the accepted one by 15 to 0 (at the moment).

Is this a technical site that values accuracy, or just a popularity contest? I was hoping for the former, but examples like the above make it look more like latter in my opinion.

[Edit] A related question: Maybe a delay of 24 hours or so before any answer could be accepted would be helpful. I get the impression that many of these cases aren't "gaming" the system, they're simply newbies accepting the first answer they get that seems to help. A waiting period would allow them to see what everyone else thinks of the answer - maybe they'd be more cautious about accepting an answer that's voted down to oblivion and loaded with comments that say "no, that's wrong."

Related question: Etiquette for correcting old questions with incorrect answers [2]

EDIT

Here is a list of problem questions [3] I compiled.

(6) The question author is in no way obligated to be smart. On more serious note - the answer could be accepted as a good-enough before the community had the chance to vote on the best one yet. What the community can do is to downvote the accepted answer, so people can see it's wrong. - Franci Penov
(3) I've seen this happen too. Sometimes others trust the accepted answer and vote it up more - Johannes Schaub - litb
(8) I'm really not convinced that answers should get "accepted" in the first place. Why not simply rate them by # of votes, and let the one people vote for float to the top? - jalf
(5) About the delay of 24hrs: I see already enough good answers not accepted because the author of the question forget to do that (but sometime comment "it works!"). With such delay, it might worsen... - PhiLho
(1) Not to mention answers with negative score being accepted, like on this one: stackoverflow.com/questions/277991/… where the best answer vs the accepted answer is currently 23 to -9 (IIRC, it however had a score of 0 when it was accepted). - CesarB
(1) I would agree - 'accepted answers' seems to be more of a scorecard than anything with true value. I would vote to remove it. - Huntrods
@Steven A. Lowe - Looks like a feature request to me.... - Robert S.
It's not a feature - it's a bug... or is it the other way around... - HBoss
@[Rob S.]: it wasn't a feature request when the discussion started, but it seems to have turned into one - had it been posted to uservoice, it would have gone nowhere, now at least there may be a rational proposal to make on uservoice - Steven A. Lowe
@Steven: It wasn't intended as a feature request; the "accepted" flag is just the mechanism on this particular site. I'm more curious about the phenomenon in general, i.e. why newbies tend to accept answers that simply reinforce their preconceptions. - Sherm Pendley
<i>Is this a technical site that values accuracy, or just a popularity contest? I was hoping for the former, but examples like the above make it look more like latter in my opinion.</i> Actually, if this were a popularity contest then the correct answer would be on top. - Whaledawg
In any instance where voting is used, the most common determining factor is popularity. Truth / correctness really doesn't matter as much as how good the answer makes the voter feel. - Chris Lively
(1) @Whaledawg: only if the upvoting masses recognized the correct answer, which they don't always do - Steven A. Lowe
I think a big issue is the order of events, often later answers are better but they don't get noticed, not only by the asker but also by other voters. It's something we need to fix somehow. - Ray Hidayat
I think that, in many cases, people vote for the answer they hope is true. Often because they are personally invested in the answer. Which just reinforces the appeal of "conventional wisdom". - le dorfier
(17) Atwood has stated that he doesn't understand why so many users have a problem with getting answers so quickly. Apparently, he also doesn't understand how bad those answers often are. Nor does he understand that those quick, bad answers deter those who have the correct answer from providing it. - raven
(1) This is a great idea, really. What's the rush in accepting an answer, especially since there is no limit to reputation earned on any given day by accepted questions. This would also have the effect of encouraging reflection, rather than rewarding the "quick fix" mentality. - EnocNRoll
(5) I hate the quick answers. Makes me not want to even bother giving the right answer. - Nosredna
The PHP link still works but the other two get page-not-found. - Kelly S. French
the accepted answer on this question is frustrating: stackoverflow.com/questions/1645161/… - Arian
[+85] [2008-11-12 17:30:36] Joel Coehoorn

The very act of asking the question means the individual is likely to be the least knowledgeable person in the thread for that particular narrow topic. This same person accepts the answer, even though it's likely they are the least qualified to do so.

But it's okay. It should still (most of the time) indicate that the accepted answer was able to solve the problem at some level for at least one person. The voting system also ensures that any caveats not included with the answer can be readily visible.


(3) an excellent point! - Steven A. Lowe
(14) Just as an aside, this syndrome (person who needs information most is in the worst position to evaluate it) is also relevant when it comes to book reviews, IMO. On the other hand, those who do know the topic of a book can't easily evaluate how well it imparts information to those who don't... - Jon Skeet
Perhaps it was able to solve the problem, perhaps it just transformed it into a harder, more subtle one. I think it's a problem, both for the person asking the question and for anyone else reading it later. - jalf
For the questions/answers that I read and review, around half the time the answer is concrete enough that it can be directly verified. But, I've had my share of good answers voted down without explanation... - florin
@jalf: read the last sentence again - Joel Coehoorn
(1) Exactly : SO is not about getting the best answser, but about solving your problem. Getting the best answer is just sugar on top of it. - e-satis
I'm not sure. Why have the editability. Why have the long-term feeling of this stuff. SO, in a way, feels like it wants to be a really cool Wiki, really quickly. And darn, it's got the style to do that. I'm fascinated by the popularity (though a lot of that is due to the fantastic UI, I'm sure), the attractiveness (especially to those of us who answer&mdash;I want points!) and the love. - Curt Sampson
Though the lack of consistency in markup is a bit annoying.... - Curt Sampson
@jalf - If it is a problem, then the user will come back to SO knowing that SO gives good answers, and ask a harder question. And this time there will be no "easy" answer for them to accept, because their question is harder, and all they'll get will be the "best" answers. And then they'll learn from experience. - Super Long Names are Hilarious
Would adding "unaccept" answers help? - I've been guilty of this myself quite often. Accepting an answer that looks at the first glance correct. However than days later a better answers comes along, and after evaluating I wish I could unaccept and reaccept. I've stopped accepting answers for now unless I notice it really is the best. - paul23
@paul23 you can already do that. - Joel Coehoorn
1
[+35] [2008-11-12 17:46:23] FlySwat

Jon Skeet can't possibly answer every question, so the chance of an incorrect answer slipping in could happen.


(1) lol. Nice one Jonathan - Joel Coehoorn
(2) This should at least get an upvote from Jon Skeet. :) - Bill the Lizard
(1) we expect you to pick up the slack! - Steven A. Lowe
(1) how many books has he gotten from him for this? hehe - Johannes Schaub - litb
(8) Only just spotted this. There ought to be a way of doing an ego search on SO without picking up your own questions/answers/comments :) - Jon Skeet
(1) Hahahahahahaha!! :P - Thomas Hansen
(3) Jon Skeet answers every question correctly but Newbies does not know to always accept his answers. - Gerhard
(3) Jon Skeet CAN answer every question. He just kind think it's more beautiful to let the humanity a part of dream. - e-satis
2
[+28] [2008-11-12 17:49:55] Steven A. Lowe

This is probably something for which there is no good solution.

One of the early questions that I saw on Stack Overflow was phrased as "which-is-better-this-or-that" and the simple, obvious answer "this" was massively upvoted while a different non-answer with no votes was accepted. Yet the technically correct answer was "neither of those, here is a third way"

This illustrates Joel's point and the limitations of any site like Stack Overflow:

  • the asker did not understand the problem well enough, and asked the question in a manner that made bad assumptions about the answer.
  • the "wisdom of the masses" is not always so wise, as the incorrect answers received lots of votes and the correct answer received only one
  • the asker then accepted a non-answer with no votes
  • the asker later added a comment saying 'I guess there is no correct answer but thanks for the discussion', which is funny because actually implementing any of the proposed solutions (or reading the background material) would have clearly demonstrated the correct approach (correct in the sense that "it actually works and is robust")

So people are not perfect, thus the sites they collectively create and moderate will not be either.

There is a solution though: get rid of the 'accepted answer' designation (though I kind of like it, it provides closure without closing the question.)


(2) excellent post. i agree with you. some people may also vote up questions just because it's already voted up above other answers. - Johannes Schaub - litb
(1) "piling on" does seem to happen - popular answers get more votes, downvoted answers get downvoted more. it seems there is some tendency to groupthink even when the group is virtual! - Steven A. Lowe
(1) What if answers were limited to, say, 5 votes per day. If it's already had 5 votes dahy You could still upvote it to give the answerer rep, but your vote wouldn't add to the answer's total. - Joel Coehoorn
or perhaps the value of votes could be discounted at higher levels... - Steven A. Lowe
My thought was, we need to be able rate answers against others. - Curt Sampson
3
[+16] [2009-01-31 01:12:44] waffles

I think the whole "accepted answer" needs to be abolished and replaced with a "thank you" scheme.

Every person asking a question should be able to distribute up to 30 points (in total) to people who assist them. SO should display a thank you section at the top of the question to all the people who helped. Note you should also be able to thank people who commented and did not reply to your question.


(1) I think this is a much better schema. - Stefano Borini
I'm not sure that this is the entirely correct way to go, but it's really the sort of way we should be thinking. But you also need to find a way to include the useful votes of those who really know about the topic/problem/whatever. - Curt Sampson
I second that request. Accepted has all too often too little meaning, and I think a "thank you" is in the spirit of SO. - gimpf
(1) I've got a question open with no accepted answer, because none of them fixed my problem. I'd have no problems with thanking the people who tried to help, and I upvoted the ones I thought deserved it. - David Thornley
(2) David points out that you can thank them, with upvotes, so the rest of this suggestion just seems to be how to "officially recognized by the OP as helpful with a rep bonus". Well, the OP can leave comments on specific answers, the "official rep bonus" is just complication without benefit, and do we really want a system that implies everyone else wasn't helpful? - Gnome
See stackoverflow.com/questions/2523281/… for example, where I really wanted to accept two answers. - TRiG is Timothy Richard Green
4
[+12] [2008-11-12 17:47:39] Johannes Schaub - litb

I really vote for a 24 hours delay before answers can be accepted. Great idea


I think this is more likely to produce a lot of questions that don't have an accepted answer at all. But that's probably not such a bad thing. - Joel Coehoorn
(3) I think that no accepted answer would be better than an accepted answer that's wrong. - Bill the Lizard
ack, Bill teh Lizard - Johannes Schaub - litb
(1) Naw. Maybe accepted answer should be a wiki-like function for high rep users instead. - Daddy Warbox
5
[+11] [2009-01-31 01:48:03] le dorfier

Here is my completely subjective view of how the voting system works. Roughly speaking (correct me if you aren't seeing the same behavior - and note that advantage is cumulative - e.g. you can be both fast and pithy, etc.):

  1. The fastest answer has the biggest advantage.
  2. The first answer that comprises a pithy statement of conventional wisdom, without reservation or offsetting factors, gets the next biggest advantage (which is what you'd expect, given the definition of "conventional wisdom").
  3. Third advantage (compensating for late arrival) is pointing out an obvious flaw in a type #1 or type #2 answer, with a correction.
  4. An answer that references a canonical source, addressing the question, comes next.
  5. Then comes an answer referencing a source no one has heard of (but addressing the question, right or wrong).
  6. Next comes an answer referencing any source about some other topic.
  7. Followed by a thoughtful, extended answer weighing various factors and suggesting further reflection.
  8. Next, a response pointing out that insufficient information is provided, and without context it's dangerous to discuss the topic. (Interesting how easily this gets trumped).
  9. Then come funny answers.
  10. But finally, all credit accumulated from the above attributes is futile if you are below answer #8.

Another interesting dynamic is that if you edit your response to improve it (possibly incorporating information from other answers), your answer will drop to the bottom of the group with your score (sometimes, sadly, below #8).

Any answer suggesting new, popular technology trumps an answer suggesting that an old way is a better way (think ORM/DAL).

And note that this process is becoming somewhat recursive/incestuous. Googling for an answer is incrreasingly bringing forth SO as a highly referenced source.


(1) In fact, Google is how I first discovered SO! - Eddie
The 'edit moves to bottom of the group' change happened recently; I'm not entirely convinced it was an improvement. - Jonathan Leffler
(2) I like a man who thinks of "recursive" and "incestuous" (and can spell the latter) as the same thing.... - Curt Sampson
Hence unlike Jeopardy, being first to the buzzer is more important than knowledge? - Accipitridae
love the ordered list - you could write an answer-generator based on that (but please don't call it "the Jon Skeeter") ;-) - Steven A. Lowe
6
[+6] [2008-11-12 18:43:23] Jim C

Why not have a accept answer by the person who asked the question and an accepted answer from the community? Either set a threshold based on the number of up votes, first answer to that threshold is accepted,a margin of up votes, it leads the second answer by x votes, or have a separate accepted button that a certain number of people have to click. You could restrict this by reputation. I am sure there are other ways to have the community select an accepted answer.


My sentiments exactly. Distinguish them visually - the highest vote getter vs. accepted by questioner. - Ken Gentle
(5) In most cases this is indicated by the number of up votes - Rune Grimstad
7
[+5] [2008-11-12 17:39:37] Hugo Bonacci

The best practice isn't always the answer to the question that was asked.

In your example Does Perl have PHP-like dynamic variables?, the question that was marked as accepted, regardless of if it should or should not be used, is the answer to the question he was asking. (I guess it is, I don't know PHP or Perl)

Of course, the fortunate thing about SO is that the dialog is available for others to debate if what is being done is the right way or not. I think someone reading that post will notice that the answer directly beneath has many upvotes and would consider it as well.


(2) there is a moral obligation to at least try to help people understand that what they want to do is a Bad Idea - Steven A. Lowe
I certainly agree, and SO has that built right into it already. - HBoss
Not exactly. If a bad idea is commonly accepted as a best practice then it is still up voted. - Chris Lively
8
[+3] [2008-11-12 17:41:12] Sam Meldrum

I agree.

It would be better if the answer the community feels is right (i.e. highest votes) appears above the accepted answer, with the accepted answer coming next.


(1) The OP is asking the question, not the community; therefore, the OP's opinion should take precedent (even when wrong). - Gnome
9
[+2] [2008-11-12 17:33:03] Arvind

In any system that has a bunch of humans, someone is going to look to game the system [1].

We could have central authorities, popularity, validity of edits or any number of metrics. No matter what, a really motivated person can find a way to game a system. In the end, an exception is not usually the rule. Some folks may end up accepting really wrong answers, but good answers do make it to the top in most cases.

[1] http://www.wikitruth.info/index.php?title=Gaming_the_system

It's not really about being able to game the system though. It's merely that right now, special consideration is given to an answer chosen by the least knowledgeable person. The person accepting a bad answer doesn't gain anything from doing so, he just doesn't know better - jalf
I agree with jalf. In many cases I've seen, it's not gaming, it's just an "A/B" problem: Newbie wants to do task A, and decides that B is the best method to do it, so he asks for help with B. But, being a newbie, he doesn't realize that there's a better solution for A than B. - Sherm Pendley
I should say that I also disagree with this answer, I just edited it. - Bill the Lizard
10
[+1] [2008-11-12 17:39:11] Won't

Users with a high enough rep should be able to change which answer gets accepted.

EDIT: preferably without changing the author


Not really, because then you'd have potential for edit waring like we have with open/close waring. - Kent Fredric
We can do that already- with enough rep you can just edit the accepted answer to be correct. But this has manifest problems of it's own. - Joel Coehoorn
@Joel: Hm, that's an interesting idea! - sep332
except that it changes the author, like i just did ;-) [please change it back later] - Steven A. Lowe
@Steven: It only appears to change the author on community wiki posts. I did the same thing to another answer. :) - Bill the Lizard
@Bill: That's OK. I'm interested in the discussion, not the fame. :-) - Sherm Pendley
@Sherm fame, responsibility, if you say something crap its unfair to let somebody else look like it was them :P - Kent Fredric
@Kent: Good point. - Sherm Pendley
11
[+1] [2008-11-12 17:48:25] flolo

The time limit for accpeting the answer would not change the people: they still use the first hint that seems to help, and hence they cant accept the answer, they would do nothing. So many answered question would stay unaccepted (except some high ranking person feels responsible to close open issues)


If that is the case, the person would have no real interest in accepting answers anyway if he leaves after 24h already. I expect many peoples don't behave like that. - Johannes Schaub - litb
(1) True, but at least the "accepted" flag wouldn't be there to mislead future newbies. - Sherm Pendley
12
[0] [2008-12-08 19:16:31] OIS

The problem with this site is the questions will remain, but once its past the front page they don't receive more answers or up votes. So the question will have the old questions but without new answers which are perhaps better. Or they will receive new better answers without any up votes.

Im thinking about questions for code examples with code example answers. Not the more subjective ones ("what framework is best for me?") or author specific ("where is the bug in my code here?").

And I hope older questions maybe tagged with "generalcodeexample" or similar will resurface with new answers, so new (hopefully better) answers can have a chance to gain some votes. Instead of old good enough answers remain on top.

  • "generalcodeexample" should be sorted by votes only IMO.
  • new answers long after question is asked should have a chance against old answers with accumulated up votes over time. (how I don't know, points are not as important as getter the best answer recognised and on top)

13
[0] [2009-01-24 23:22:51] Stephen Baugh

In my opinion ...

1, part of the problem is SO seems to expect you to accept a solution. Or at least suggests you accept one. Sometimes this leads to people accepting something that is 'almost' right. The same problem exists at most 'solution sites'.

2, the second problem is that programming is not exact and people have different styles. Something I think is perfect for my style or follows my methodologies may irate another person and vice versa. On many occasions I have seen code that executed perfectly but was basically useless in a team environment as no one could understand it without constant re analysis ... "who is right"?

3, Sometimes an answer is actually a solution to the question asked ... ie it works. For that person the problem is solved ... The answer might not however consider wider issues like security etc ... again a working answer providing a less than perfect solution. The person asking the question is unlikely to be aware of these wider issues.

Now I am a new user on SO, but have been observing for a while.

The thing I like about SO is that accepting a solution doesn't end the discussion. Through comments or new answers you (we) have the opportunity to correct or add to an accepted solution.

None of what I have said is about "popularity", is more about human nature and the ironically imprecise industry we work in.


14
[0] [2009-02-05 05:34:16] Eddie

Maybe we need there to be two answers, one selected by the person who asked the question, and a second selected by people with over a certain threshold of points, and only after the question had been open a few days. Only an experiment allowing this would show whether this improves the correctness of answers selected.


15
[0] [2009-02-05 06:04:33] David HAust

Ignore the 'accepted' answer and pay attention to the answer with the highest votes.

This is a much better indicator of answer quality.


16
[0] [2009-02-05 06:24:08] mPP

Voting and reputation are both bad measures.

  • How is it possible that someone with ~800 points doesnt know that they can refer to the documentation of String to find out how to find if a String contains a certain character ?

  • If most of your users dont know anything and they all vote for an answer does voting help ? Im sure there are many examples of bad answers being voted up by people with lots of points.

Its unfortunate that someone who didnt know something, then asks a question and gets a bad answer which leads them terribly astray. Its this sort of advice that leads the industry to be full of very incompetant developers who really should not be trusted.

Should it be possible for people to lose points if they give bad answers ?

I personally think it should otherwise the points currency has no value. In the real world people who are successful make lots of money (aka points) and people with dingbat ideas lose money and go broke. Should giving or voting for the wrong answer cost points - i think it should.


Punishing voting for the wrong answer would cause a lot of problems, but making the penalty for giving a wrong answer more severe might be good. - Brilliand
17
[0] [2009-02-05 06:43:07] davogones

Given the assumption that a user with high reputation is more likely to be correct than someone with low reputation, the following simple changes may help alleviate the problem:

  • Give high-reputation users the power to veto a marked answer and choose another one. OR, differentiate between the answer chosen by the asker, and the answer chosen by the masses.
  • Weight the vote of high-reputation users more highly

18
[0] [2009-02-06 03:31:07] mPP

Reputation means nothing in terms of a perons competance or knowledge. In the end if you post enough times youll get points. Reputation is a measure of activity not quality in any form.


19
[0] [2009-05-26 01:30:05] Jeffrey Kemp

You raise a valid point but having come from EE (many years ago, before it become the corporate monstrosity it is now) SO is SO much better. Sure, the wrong answer is accepted every now and then, but the question doesn't drop off the radar straight away, which I think is a huge strength of SO. New answers can be added much later and upvotes improve the chances that other readers will find the correct solution. As a relatively new SO user I enjoy going through old questions and upvoting what I think are good and correct answers.

I disagree with a 24-hour time limit only because I suspect that the number of times this happens is low in comparison with the number of times the correct answer is rewarded, and we want to encourage questioners to reward the correct answer. Hopefully after trying and testing it of course.

If an incorrect or suboptimal answer is rewarded, the downvote system should be sufficient to indicate that it is not necessarily the best answer. This will work as long as people are prepared to use the downvote system.


20
[0] [2009-07-19 08:23:53] kuroir

I think that the user that asked the question has the right to decide the answer... even if it's wrong. Besides, we have the point system to allow the community point out if there's a better answer.


21
[0] [2013-06-02 10:10:02] Lucas Malor

I agree with you. I think the best solution is to add the possibility to flag the accepted answer as "controversial".

and sort the not accepted answers by upvotes, as @jalf suggested


The non-accepted answers are already sorted by up-votes. - apaul34208
@apaul34208: I don't know why I thought they was not, thank you :) - Lucas Malor
22