By having for example money and human suffering as the metric.
Note this is a specific question.
Last month automaker Toyota announced a recall of 160,000 of its Prius hybrid vehicles following reports of vehicle warning lights illuminating for no reason, and cars' gasoline engines stalling unexpectedly. But unlike the large-scale auto recalls of years past, the root of the Prius issue wasn't a hardware problem -- it was a programming error in the smart car's embedded code. The Prius had a software bug.
We studied the Therac-25  software bug at University. I'm not sure it is history’s worst, but it really does go to show how much of a threat to our lives software bugs can be.
The Therac-25 was a radiation therapy device in which a software bug resulted in patients being exposed to lethal doses. The software bug was a race condition which would have been fine in the previous model (Therac-20) which had had its electromechanical locks replaced with software controls in the 25. http://en.wikipedia.org/wiki/Therac-25
I think one of the most expensive was the Ariane 5  bug.(US$370 million)
The Pentium bug  was rather expensive too.
A bug in the software of the Patriot Missile  killed 28 US Army soldiers at Dhahran. http://en.wikipedia.org/wiki/Ariane_5_Flight_501
The sendmail bug that allowed the Morris worm to propagate and introduced a generation of hackers to the wonderful world of buffer overflow (brought to you, more often than not, by the standard C library function
gets(), which is finally being removed from the 201X standard).
"Simply when you try to multiply 850 by 77.1 excel display the result to be 100000 !!!" http://groups.google.com/group/microsoft.public.excel/browse_thread/thread/2bcad1a1a4861879/2f8806d5400dfe22?hl=en#2f8806d5400dfe22
To make this the most expensive you would have to index the cost and convert it into today's prices.
The bug struck on the 22 July 1962 (the day before I was born!) and led to the loss of the Mariner 1  space probe that was due to fly to Venus.
If you believe that this meets your definition of a software bug, it must stand as one of the most serious ever. It also shows that there is nothing new about bugs in space/missile software. http://en.wikipedia.org/wiki/Mariner_1#An_Infamous_Bug
It is still debated whether it was actually caused by a bug or not, but the Flash Crash of the US stock market from May 2010 has to be the worst from a purely financial standpoint. The crash apparently wiped out over a trillion dollars worth of equity in a few minutes. While a large percentage of that was regained as the stock market quickly rebounded, there was certainly a lot of money lost (and made) that day.
A defect in the control software for the Soviet Urengoy - Surgut - Chelyabinsk natural gas pipeline resulted in " the most monumental non-nuclear explosion and fire ever seen from space ".
But, although this is an interesting story, it is not a bug per se as the CIA allegedly sabotaged the software which was subsequently stolen by the KGB for use in the project. Crazy stuff... http://en.wikipedia.org/wiki/Siberian_pipeline_sabotage
What really happened on Mars
Honorable mention to the Zune software bug that froze all the devices in Dec 31, 2008 before midnight. Zune Bug Explained  http://www.crunchgear.com/2008/12/31/zune-bug-explained-in-detail/
Its not really a bad software bug, but I guess the pilot was pretty surprised when it happened the first time:
F16 autopilot flipped plane upside down whenever it crossed the equator
I wonder how many trade secret housing and government computers were penetrated by the SSH unauthenticated session bug that allowed ssh access without a password. Still looking for the reference, but some people might know what I'm talking about.
The Program Trade defects associated with the 1987 Stock market crash  comes to mind. None of the firms using program trading thought through the implications of how things would change when the majority of trading was automated. http://en.wikipedia.org/wiki/Black_Monday_%281987%29#Causes
The British destroyer H.M.S. Sheffield was sunk in the Falkland Islands war. According to one report, the ship's radar warning systems were programmed to identify the Exocet missile as "friendly" because the British arsenal includes the Exocet's homing device and allowed the missile to reach its target, namely the Sheffield.
The Wired Magazine article  has some useful and interesting information on software bugs. http://www.wired.com/software/coolapps/news/2005/11/69355
The Patriot Bug  always comes up in questions like this. The thing to remember is that it wasn't actually a software bug - it was user error. The software required a reset every couple of days to keep timers in sync, and this was according to spec, in part to keep the cost of the components down, and in part because the components were designed a long time ago.
What they didn't take into account was that the Patriot crews didn't want to turn off their rigs for even the few minutes it would take to reboot, in case a missile happened to fly over while they were down. They didn't know, or had forgotten, that by keeping the rig active, they were making it less accurate over time, until after a couple of days without a reset it literally couldn't hit its target.
Maybe calling this "user error" is a little harsh. If the users' training had been adequate, this would not have been an issue. http://sydney.edu.au/engineering/it/~alum/patriot_bug.html
Coming soon: The Year 2038 problem , i.e. the fact that the POSIX time_t data type,
started in 1901-12-13 20:45:53 GMT will wrap around in 2038-01-19 03:14:07 GMT.
Will cause some computer software to fail at some point before, during, or after the year 2038 (from the wikipedia article) http://en.wikipedia.org/wiki/Year_2038_problem
Winnuke, Denial of service attack on Windows 9x was brutal.
I remember hearing about a missile that crashed on launch due to a typo in a Fortran program, wherein a period was typed instead of a comma, resulting in a program the was syntactically correct but semantically very wrong.
My vote is Y2K. With all the preparation. (Entered from my phone)
Visual Basic Script (VBS)! Or you can blame the ingenious feature of hidden extensions that is turned on by default (thank you Microsoft) which fooled people into thinking LOVE-LETTER-FOR-YOU.TXT.vbs was a text file.
That bug allowed a virus called 'ILoveYou' to spread more than rabbits in Australia causing an estimate damage of $5.5 billion.
You say it's not a bug!? Apple and Oracle are now watching you...