ShawnVN's Blog

The saga continues.  Soon after I reported the terrible timing-sensitive bug in the VS automation layer, I got a friendly email from someone smarter than me, and with a microsoft.com email address.  That's the kind I like..!

I've forgotten how most of this COM/OLE goo works, but this kind soul reminded me that OLE message filters (ala IMessageFilter, but not the one you're thinking of!) are the key to avoiding call-rejected errors from COM plumbing.

Code follows.  Caveat emptor -- but it works for me!

using System;
using System.Runtime.InteropServices;

/*
This class implements an OLE message filter, appropriate for use with the VS automation clients.

Sample usage:

    [STAThread]
    private static void Main()
    {
        // We avoid instantiating the RCW as "new DTE()", because the both PIAs for v7.0 and v7.1 point 
        // to the same CLSID (VS 7.0's)... 
        Type latestDTE = Type.GetTypeFromProgID("VisualStudio.DTE"); //ver-indep progid
        EnvDTE.DTE dte = Activator.CreateInstance(latestDTE) as EnvDTE.DTE;

        // Register an OLE message filter to avoid "call rejected" exceptions
        MessageFilter.Register();

        // Drive the VS automation interface
        Installer1.AddToolBoxTab(dte); // or whatever

        // Unplug the message filter
        MessageFilter.Revoke();
    }
*/

class MessageFilter : IOleMessageFilter
{
    //
    // Public API

    public static void Register()
    {
        IOleMessageFilter newfilter = new MessageFilter(); 

        IOleMessageFilter oldfilter = null; 
        CoRegisterMessageFilter(newfilter, out oldfilter);
    }

    public static void Revoke()
    {
        IOleMessageFilter oldfilter = null; 
        CoRegisterMessageFilter(null, out oldfilter);
    }

    //
    // IOleMessageFilter impl
    
    int IOleMessageFilter.HandleInComingCall(int dwCallType, System.IntPtr hTaskCaller, int dwTickCount, System.IntPtr lpInterfaceInfo) 
    {
        System.Diagnostics.Debug.WriteLine("IOleMessageFilter::HandleInComingCall");

        return 0; //SERVERCALL_ISHANDLED
    }

    int IOleMessageFilter.RetryRejectedCall(System.IntPtr hTaskCallee, int dwTickCount, int dwRejectType)
    {
        System.Diagnostics.Debug.WriteLine("IOleMessageFilter::RetryRejectedCall");

        if (dwRejectType == 2 ) //SERVERCALL_RETRYLATER
        {
            System.Diagnostics.Debug.WriteLine("Retry call later");
            return 99; //retry immediately if return >=0 & <100
        }
        return -1; //cancel call
    }

    int IOleMessageFilter.MessagePending(System.IntPtr hTaskCallee, int dwTickCount, int dwPendingType)
    {
        System.Diagnostics.Debug.WriteLine("IOleMessageFilter::MessagePending");

        return 2; //PENDINGMSG_WAITDEFPROCESS 
    }

    //
    // Implementation

    [DllImport("Ole32.dll")]
    private static extern int CoRegisterMessageFilter(IOleMessageFilter newfilter, out IOleMessageFilter oldfilter);
}

[ComImport(), Guid("00000016-0000-0000-C000-000000000046"),    
InterfaceTypeAttribute(ComInterfaceType.InterfaceIsIUnknown)]
interface IOleMessageFilter // deliberately renamed to avoid confusion w/ System.Windows.Forms.IMessageFilter
{
    [PreserveSig]
    int HandleInComingCall( 
        int dwCallType, 
        IntPtr hTaskCaller, 
        int dwTickCount, 
        IntPtr lpInterfaceInfo);

    [PreserveSig]
    int RetryRejectedCall( 
        IntPtr hTaskCallee, 
        int dwTickCount,
        int dwRejectType);

    [PreserveSig]
    int MessagePending( 
        IntPtr hTaskCallee, 
        int dwTickCount,
        int dwPendingType);
}

Everybody has been following along with my little toolbox saga, right?

It just doesn't let up.  Today it threw me another curve -- a timing bug! -- apparently it's failing in our test lab, on slow machines (think: sub-900 MHz laptops, w/ slow busses).

Sometimes I get a "call was rejected by the callee" exception, sometimes just silent failure (no exceptions, but no new toolbox items either).  When the exceptions do occur, it's never in a predictable spot (just one of the several calls in to VS).

The bug seems to manifest on both VS 7.0 and 7.1, but I'm not 100% sure of that.  I managed to repro on my high-end dev box, by cranking my SETI@Home client's process priority up to "high".  ;-)

Sprinkling around a few Thread.Sleep calls seems to fix the problem nicely... well, not "nicely", but what else can I do?

        // Get DTE from version-indep progid (points to newest)
        Type latestDTE = Type.GetTypeFromProgID("VisualStudio.DTE");
        EnvDTE.DTE dte = Activator.CreateInstance(latestDTE) as
            EnvDTE.DTE;

        // Repeat attempt to drive VS, with ever-increasing handicap
        for (int delayFudge=100; delayFudge < 5000; delayFudge += 1000)
        {
            try
            {
                AddToolboxItems(dte,delayFudge);
                break; // exit loop
            }
            catch (Exception ex)
            {
                Debug.WriteLine(ex.Message + "\nTrying again...");
            }
        }

At what point do I just give up, and look for a new job?

A few weeks ago, I blogged about how to programmatically add .NET UserControls to a VS Toolbox tab.

Today my teammates and I got around to testing/updating the code for VS 2003 -- as it happens, the code I presented in the blog is correct, but in practice getting it to work proved to be another royal p.i.t.a....

This time, the problem we ran into lied in how we were automating VS:  by instantiating a new instance of EnvDTE.DTE, from outside the devenv.exe process. 

For VS 7.0, we went about this by cavalierly instantiating the managed wrapper for the VS automation model:  dte = new EnvDTE.DTE();  No problem, right?

But guess what -- the managed wrapper for VS (EnvDTE.dll) that's shipping with the v1.1 framework is binary identical to the one in v1.0 framework!  No kidding!  They point to the same underlying CLSID, which, as it happens, is the old (VS 7.0) CLSID. 

All of the VS automation interfaces are the same, so I guess MS decided they didn't need to ship a new PIA for VS 7.1.  But the CLSIDs have changed, in order to support side-by-side coexistance of VS 7.0 and 7.1...  and the PIA contains references to the CLSIDs!  Kaboom.  For our little use-case, anyway.

In the end, we determined the best/easiest thing to do was to instantiate the RCW by going through the version-independent ProgID (which seems to correctly point to the latest):

            // Get DTE from version-indep progid (points to newest)
            Type latestDTE = Type.GetTypeFromProgID("VisualStudio.DTE");
            EnvDTE.DTE dte = Activator.CreateInstance(latestDTE) as EnvDTE.DTE;

Hope that helps somebody...