Another option is the free app Retinizer ^[1], which I used for a while with Mathematica 8. It cleaned up the text, but I found caused visual glitches with the documentation and added some instability throughout. That said, it made the text look beautiful which helped the eyes when reading a lot of code.

You can turn it on and off, and it works on other applications. Worth a try, with the pros and cons in mind.

Mathematica 10.0 and later now support retina resolutions on OS X.

Here's how to do it for more recent versions of Mathematica (specifically, 9.0.1):

1. Get Retinizer, mentioned in other posts
2. Show Package Contents of Mathematica, navigate to ../Mathematica/Contents/ and open Info.plist for editing.
3. Delete the lines for NSHighResolutionCapable:True.
4. Save the plist.
5. Open Retinizer and drop Mathematica onto it as instructed. It should succeed instead of throwing an error.

This answer is complementary to Ian's. Retinizer appears to work very well with Mathematica 9. It increases even the resolution of Graphics, but not Graphics3D.

To get a prettier view of a Graphics3D, one can use the following function:

retinize[g_, n_: 2] := 
   Image[Rasterize[g, "Image", ImageResolution -> n 72], Magnification -> 1/n]

Here's an example from the docs, processed like this:

The situation as far as I can guess (that's the operative word) is this: In order to supports Retina's high dpi, one has to compile the application using the Cocoa framework which has for long been the default application framework on Mac OS X anyway. Unfortunately, I don't think Mathematica does that. It uses the legacy Carbon framework. So in order to support Retina, they will probably have to rewrite a whole lot of front-end code.

By the way, this isn't just a display issue. The resolution of screen shots from Mathematica is also wrong. So if I paste a screen shot into Preview (a high-dpi application) and save it there, the result looks especially blurry when you embed it in a web page (or upload to StackExchange, for that matter). So I now try to avoid taking screen shots and instead use Export to PNG inside Mathematica. I also have a script that repairs the resolution for pasting into my Xemacs editor that I use to write web pages, but that's probably too specialized for this answer.

You can use one of the resolution setting utilities like switchResX to change the mode to 2880x1800. Then set the magnification in Mathematica to 200%. It works, but the menu bar is very small. Probably more trouble than it is worth.

Retinizer works, but beware that complex 2d graphics cause the rotating beachball to appear, and Mathematica might hang for quite some time.

Retinizer ^[1] is the only option to solve this, as in programming_historian answer. I just wanted to add that for version 9.0.1.0 of Mathematica the only way to "retinize" is to use Retinizer's "global mode": launch Retinizer and pres Alt-Cmd-G. Retinizing by dropping Mathematica icon on Retinizer doesn't work for 9.0.1.0.