A while back I blogged about a problem I had installing Visual Studio 2005 on a fresh Vista install - it complained about requiring Windows XP SP2. I never found a satisfactory explanation, but copying the DVD contents to the hard drive worked around the problem. The stock suggestion was to check the Compatibility settings on the property page for setup.exe - but in my case no compatibility mode was set. Or was it?
Recently I tried installing Visual Studio 2008, and got the same error. Following a suggestion from the MSDN forums, I checked the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers. Lo and behold, it contained an entry for %tempdir%\setup.exe. But I'm not running setup.exe from the temp dir, I'm running it from the DVD, so that entry shouldn't affect anything, right? Wrong. When you run the installer from the DVD, it appears to copy itself to the the temp dir and run itself from there. At that point, the compatibility flag is in effect.
Where did the registry setting come from in the first place? I have no idea. But removing it allows the Visual Studio install to proceed.