My experience is that frustration usually leads to salvation, this took me two days and lead to a 90% solution and explanation.
Day 1
I just reinstalled all my software, thus the My Site link in Office save open dialogs disappeared. No problem I thought and went to MySite to click the "Set as Office default website" link. Well, it did not show, so I created a custom link with a content editor webpart to be able to call the script "setPersonalSite()".
This had no effect. By examining the script I found that an ActiveX control called PortalConnect.PersonalSite was involved. Because I'd tried just about everything to disable security (yeah, I just installed Windows XP SP2) on all levels I figured I could try calling the component directly from managed code through COM Interop.
No luck here either. A COMException of the unknown type blessed my console application which left me kind of blank. More research lead me to the assumed functionality of this code, which is in fact simply setting a registry entry (go figure). The key in question is
HKCU\Software\Microsoft\Office\11.0\Common\Open Find\Places\UserDefinedPlaces\PersonalSite
From there you'd think it would be smooth sailing, but no. The actual path to mysite is stored as a Pidl (Pointer to ID List) which is not even close to be accessible for strictly managed coders like myself. I spent some time staring on this, and this, and this, and this without beeing able to do
1. Manipulate the binary array in managed code from the RegistryKey.GetValue("Pidl") method
2. Calling unmanaged code to manipulate the path section of the damn Pidl
Day 2
Clarity appeared during the night. I managed to figure out that the ActiveX component actually didn't create the Pidl and SortAcending values on the PersonalSite key, but actually writes a Name and Path key, both strings. The first time an Office Open or Save dialog opens the path key is converted to a Pidl automatically. Thus changing the MySite link in Office can be done by deleting the Pidl and SortAscending values and inserting a Path string with another url. Click Open in Office and it's converted.
The second part of the problem was that I was unable to open MySite as a Web Folder (IE -> File -> Open -> tickbox). After examining log my Local Settings\Temp directory I found some guidance in the wecerr.txt logfile indicating something wrong in the response from IIS:
HTTP 0 The Web server, or a proxy server, returned a badly-formed HTTP response. Please notify the Webmaster for the server.
I had a feel this was related to security somehow to I retried something I was certain to have tried before: re-adding the portal url to Trusted Sites. I found that the url listed was on the http protocol, not https as we use. I changed this and reverted security settings for Trusted Sites to default level.
Then I went along and deleted the PersonalSite registry key which prompted MySite to display the "Set as Office default website" on page load.
Click Yes -> Fixed!
However, I tried the same approach on another machine and the procedure was successful all the way to actually opening the MySite from File Open. On this machine MySite was still not opening as a Web Folder with the same error logged in wecerr.txt file. Currently we see a couple of possible reasons; Microsoft AntiSpyware might have mixed it up or due to that extensive customization to the MySite in question.
Updates will come if we resolve this issue aswell.