So after wasting about an hour and a half debugging why in the world I kept getting an invalid cast exception, I discovered a quirk in the way ASP.NET loads assemblies. Apparently, the .NET framework loads everything in the \bin directory that even looks like a DLL. For example, if you have a file called DLL_ the framework will load it anyway. Why the hell it does this I don't know!!! It's not even a DLL for God's sake!
Now you might be wondering why in the world I have a file called DLL_ sitting in my \bin directory but that's easy to explain. Sometimes I want to use a different compilation of a DLL that might contain something like more debugging code, for example, in my ASP.NET application. So instead of moving or overwriting the old version or recompiling my application or even changing which version is redirected to what other version in the config file, I simply change the old DLL to DLL_ or something of the like and drop the new DLL in the \bin directory along side of it. Seems harmless and should work and in fact it does when .NET isn't being stupid and just loads DLLs.
I spent all this time trying to figure out this problem, changing the code I was debugging over and over again, and when I finally said what the hell let me see what the code base is for DLL which contains the class I'm trying to cast to, I saw that it was <Path>\SomeAssembly.DLL_. This is a bug as far as I'm concerned. And as far as I can tell, there is simply no application for this type of behavior so I'll just call it retarded.
By the way when I say ASP.NET, I'm working with SharePoint specifically.