BinaryFormatter can hang within modified stream?
This is definitely weird. I could have sworn the binary formatter in V1.0 would throw an error whenever the stream was invalid (can happen if the transport protocol burps or a file gets messed up on your disk), but with V1.1 installed, all the formatter does is hang.
Without going in to what I was testing, all I did was to go into the file and change a typename. I have two identical classes in my assembly, one named MyFish and the other MyDolphin. By changing MyFish in the binary file to MyDolphin the formatter goes to work and never comes back. I'm assuming this is due to some offsets in the file that are looking for specific field values (most likely a string value).
The fact that it never comes back really annoys me, because it creates a simple DOS attack against any applications that use the BinaryFormatter (by never I've only tested for about 5 minutes. I'm going to leave it running for a while longer though).
Here is the skinny as I see it:
1. Changing the length of the type name will result in an out of memory exception.
2. If you have a custom binder it will happen in the BindToType call.
3. Without a custom binder it will happen within the binary formatter itself.
4. I'm not sure if this is a string interning problem of some sort or what, but the system memory is completely consumed.
5. The attack stops when the application is terminated (can be hard since my entire machine was frozen most of the time) or an OutOfMemoryException is thrown which cancels the application out.