This is one of the first things I do on a fresh install of Windows. Makes Sharepoint more enjoyable.
My Computer -> Tools -> Folder Options -> File Types tab:
Select the file type you want to modify (I picked Word). Click the Advanced button and get the Edit File Type dialog:
UNcheck "Browse in same window", click OK a couple times and you're done.