Comments

# TrackBack said on Friday, January 23, 2004 6:34 PM

# Scott Galloway said on Saturday, January 24, 2004 9:27 AM

That is just fantastic.This has long driven me nuts, it seemed the single most obvious place that would benefit from compression. Don't suppose you've figured out a way to support IIS compression - so the proxy sending 'compression supported' headers and transparently decompressing the stream as part of the generated proxy.

# Paul Glavich said on Saturday, January 24, 2004 7:39 PM

This is a different kettle of fish as you would not be dealing with SOAP headers, rather you would probably be implementing a new HttpHandler (at a guess). Once I get a bit more time, I may have a go at it.

# Marshall Brooke said on Monday, January 26, 2004 4:33 AM

This is a really nice piece of work. I've tried something similar, but failed to get it to work with WSE, I think because of the ordering of the SOAP extensions. Do you know how you can get this Compression extension to be executed as the first extension in the PipeLine?

# Paul Glavich said on Monday, January 26, 2004 6:12 AM

Thanx Marshall.

Not sure about the ordering you asked about as I haven't really tried or tested that aspect but if I get time I will have a look. Feel free to email me at glav@aspalliance.com if you want to work through it.

# Leonardo Torres Ochoa said on Saturday, February 07, 2004 5:08 PM

This is great DLL'... thank you friend..

# David Peckham said on Thursday, May 20, 2004 6:45 PM

Marshall, did you try setting the extension's Group and Priority in your Web.config?

# Darren Zwonitzer said on Friday, May 21, 2004 2:43 PM

Great piece of code, it helps a great deal! I do have one issue/question. Before applying compression, I was able to use the Context.Request.InputStream in the WebMethod on the server to retrieve the actual SOAP request. That stream now returns an empty string.

I was using this for debugging as well as for getting the length of the actual request. Is there anyway to get this information with this compression piece in place?

# Paul Glavich said on Monday, May 24, 2004 8:42 AM

Darren,
I haven't actually examined the InputStream within the context of the compression library before so not sure what might be happening. At a guess, the 'ChainStream' method that is overridden returns a newly allocated memory stream buffer, which initially will be nothing, so perhaps thats what you are receiving. The runtime uses this stream during the serialise/deserialise process. I haven't gone back into this code for some time though and am a bit rusty. I was actually thinking of hooking up some events for purposes such as logging/debugging, but I just haven't had the time.

# Ed said on Wednesday, June 09, 2004 11:55 AM

Hi,

The extension is great. Was very simple to use and makes a big difference to the transfer times but I seem to have discovered a bug. I get a soap exception on several characters such as "º", "©", "Ó", etc.

Strangely this will even happen when the Compression Levels is set to None! I tried playing with the requestEncoding, responseEncoding options in web.config but that didn't help.

Anybody have any idea. I'm trying to figure this out but I'm not familiar with these type of encoding? unicode? issues.

To test, the Web Method can be as simple as the following:

[WebMethod] [WSCompression(CompressionLevels.None)]
public string HelloWorld()
{ return "º";}

I call the method from an equally simple WinForms App. (I don't get the error in the auto generated browser app).

The exception reads:
"This is an unexpected token. The expected token is 'TAGEND'. Line 1, position 489"

Sorry for the length of the post. Hope somebody can figure this out.

Cheers,
Ed.

# Paul Glavich said on Thursday, June 10, 2004 8:23 AM

Hi Ed,

Not sure, perhaps the extension needs HtmlEncode the text before compressing, then do a HtmlDecode after decompression.

So try doing a Server.HtmlEncode and Server.HtmlDecode on the text to see if that helps. If that fixes it, then it would be pretty easy to implement. Dont know why it has problems when compression is none though as it bypasses the whole compression library then.

# Ed said on Friday, June 11, 2004 4:46 AM

Hi Paul,

Just figured out the problem. In Decompress()

change:
_bufferedStream.Write(System.Text.Encoding.UTF8.GetBytes(sb.ToString()),0,sb.Length);

to:
byte[] bytesSB = System.Text.Encoding.UTF8.GetBytes(sb.ToString());
_bufferedStream.Write(bytesSB,0,bytesSB.Length);

Looks like the encoding changes the length of the array when there are "special chars" thus calling the Length on sb returns the incorrect size.

Thanks again for this useful piece of code!

Cheers,
Ed.

# Xybex said on Thursday, June 17, 2004 6:25 PM

Guys anyone managed to compress a dataset?

# Paul Glavich said on Saturday, June 19, 2004 12:44 AM

If the Dataset is part of the web service payload, then it will be compressed. ie. If you are passing or returning a dataset, it will be serialised as part of the message, and will be compressed if you use this library.

# Roger Willcocks said on Monday, June 21, 2004 10:57 PM

[quote]
So the proxy sending 'compression supported' headers and transparently decompressing the stream as part of the generated proxy.
[/quote]

I've done this. I sell a version on my website, but the basic trick is to implement the HTTP Compression module by Ben Lowry on the server side.

And override the HttpSoapClient (not sure i've got the right name) class to handle the decompression.

Then change the proxy stub class to ingerit from your overridden class instead

# Pam said on Wednesday, July 07, 2004 5:13 PM

It is exactly what I am looking for but
I couldnot make it work with WSE 2.0.
I always get this message "EOF in header" from Decompress(). Any idea?

Thanks,
Pam

# Paul Glavich said on Thursday, July 08, 2004 6:55 AM

Pam,

Not sure, I have yet to try it with WSE2.0, however have a look at the comments from Ed. Perhaps try what he has done to see if it makes a difference. Would be interested to know how you go.

- Paul

# Pam said on Thursday, July 08, 2004 11:34 AM

I tried that but it didn't help. I guess it was something about WSE policy pipeline. Propably the compression and decompression has to be done after the soap request successfully passed WSE policy pipeline processing.

# Pam said on Wednesday, July 14, 2004 1:55 PM

I got web service compression working with WSE compression custom filter that only compresses soap envelope. So if there is anything out side soap envelope such as dime attachment, it will not be compressed.

Here is the link of morty's blog if anyone is interested in.
http://blog.morty.info/CommentView.aspx?guid=877aed1e-cc93-496e-ab5a-5818b25c316d

# Paul Glavich said on Friday, July 16, 2004 3:10 AM

Thanks for the info Pam. Much appreciated.

# Michael Proctor said on Friday, October 13, 2006 2:14 PM

OMG!!!!!! Thankyou for the Steroids for my WebService, it speed it up I would have to say over 5 times what it was. Why didn't Microsoft implement compression? wierd. Your effects are greatly appreciated Regards, Michael Proctor

# Brandon Lusk said on Thursday, March 01, 2007 2:40 PM

I just started using your compression library for a web service that returns datasets and it works great.  The only problem I'm having is that on a really large dataset I get an exception (Response is not well-formed XML->System.Xml.XmlException: Unexpected end of file has occurred.  The following elements are not closed: soap:Envelope. Line 121609, position 8160.)  Do you think there would be a limit on the size of the dataset?

Thanks,

Brandon

# Glav said on Saturday, March 03, 2007 6:46 AM

Brandon,

Absolutely there is a size limit. Check the <HttpRuntime maxRequestLength=".."> setting in your web.config. if not there, I think it defaults to 4096 which could be causing the issue if its a really big (and I mean big DataSet).

# Marvin said on Tuesday, March 27, 2007 11:28 AM

Hi Glav,

I'm just about to try your library. Will it work with PocketPC and Windows Mobile, .NET Compact Framework?

Thanks!!

# Kevin said on Friday, March 30, 2007 11:44 AM

The reason Brandon is getting that error is that there is a bug in this SOAP extension.    The issue is that UTF-8 encoded text when represented as an array of bytes (which is a necessary step in the compression transformation) can have anywhere from 1 to 4 bytes per character.  For space efficiency, the most common 128 characters are represented by one byte each.  This code assumes that ALL the characters are one byte each.  The code that copies the unzipped data stream into the SOAP message buffer assumes that the number of bytes it needed to copy is always equal to the number of characters in the unzipped string representation of the SOAP message.  That USUALLY is a very safe assumption, because the overwhelming majority of SOAP messages consist of nothing but the most common 128 characters.  However, if your SOAP payload has any “special” characters (that end up taking more space in the byte array then the code expects because they have multi-byte representations), then the end of the SOAP message gets truncated off, which causes the SOAP message to appear to be badly formatted XML (because the closing tag of the SOAP envelope gets left off). You should change the text encoding class to calculate the size of the byte array rather than rely upon the length of the string it pulls the raw bytes from.

# john said on Thursday, June 28, 2007 1:31 PM

how do you consume the compressed web service?

# Fermin said on Tuesday, October 02, 2007 7:19 AM

Helo

I try it, for my web services, they return DataSets

And dont work.

I made this modificaction in the Decompress method

private void Decompress()

{

if (_compressionLevel != CompressionLevels.None)

{

// Create a stream reader to give us a string representation of the network stream

StreamReader sr = new StreamReader(_networkStream,System.Text.Encoding.UTF8);

string netStr = sr.ReadToEnd();

sr.Close();

// Get the encoded bytes from the Base64 string.

byte[] tmpB = Convert.FromBase64String(netStr);

// Make a tmporary memory stream to work on

// (failure to do this i.e working directly on the network stream can have adverse effects)

MemoryStream tmpMem = new MemoryStream(tmpB);

// Use the Zip compression libraries to decompress the bytes

ZipInputStream zip = new ZipInputStream(tmpMem);

ZipEntry entry = zip.GetNextEntry();

// We need to read in the compress stream chunks at a time. The 'Read' method of the

// ZipInputStream can only read a max of 23191 bytes (I think thats its max)

int size=1;

byte[] buf = new byte[2048];

// System.Text.StringBuilder sb = new System.Text.StringBuilder();

while (size > 0)

{

size = zip.Read(buf,0,2048);

if (size > 0)

                       _bufferedStream.Write(buf, 0, size);

// sb.Append(System.Text.Encoding.UTF8.GetString(buf,0,size));

}

zip.Close();  // should implicitly close our memory stream.

//Write our tnetwork stream to the buffered stream so that it gets transmitted

// _bufferedStream.Write(System.Text.Encoding.UTF8.GetBytes(sb.ToString()),0,sb.Length);

}

else

{

// No Compression, plain text

StreamReader sr = new StreamReader(_networkStream);

string s = sr.ReadToEnd();

sr.Close();

_bufferedStream.Write(System.Text.Encoding.UTF8.GetBytes(s),0,s.Length);

}

_bufferedStream.Position = 0;  // If we dont do this, nothing is returned!

}

# Ramon Murillo said on Wednesday, January 02, 2008 5:00 PM

I'm uploading a large body to a web service. Would your library work as well?

# Glav said on Sunday, January 06, 2008 5:36 AM

Ramon,

It would work so far as compressing the content goes, but you really need to make sure the content can be decompressed. so the receiving service needs to implement this library or something similar in order to decompress it.

# Rob said on Monday, February 18, 2008 10:44 AM

Glav,

I need to consume a WS that uses WsSoapComression, but the client is in Delphi, how can I decompress the Stream??

It can be reached via the .dll??

Thnx

# brian said on Monday, May 05, 2008 3:25 PM

thanks, but I cannot find anywhere in your sample app that shows where to add the Compression stuff.

In my app i get this error:

{"There is a problem with the XML that was received from the network. See inner exception for more details."}

Inner Exception:

{"The data at the root level is invalid. Line 1, position 1."}

i'm very new to web services, help!

# Dan said on Wednesday, May 21, 2008 2:41 PM

I know this has been around for a long time but I figured I'd post a comment anyway as people (like me) are just finding this.

The encoding used in the soap extension along with the NZipLib is UTF8. This will not work with special chars outside of the standard ascii char set (above dec 127). To allow the soap extension to work with special chars simply change all code using Encoding.UTF8 to Encoding.Default. This will allow the extension to use the default encoding and code page throughout the process and keep the special chars intact and won't throw exception.

Hope it helps someone.

Dan

# brij said on Thursday, June 05, 2008 12:55 PM

Hi Dan,

I also tried your solution of converting encoding.utf8 to encoding.default, but it still returned me the same error

can someone please put up a through and complete package here.

Regards

Brij

# NL said on Wednesday, June 25, 2008 2:47 AM

Hi,

When I run the TestClient project and hit the "Call ReturnBigString" button that internally calls the ReturnBigString() method of your webservice. I get an exception "Invalid character in a Base-64 string." in Decompress() method line:141 byte[] tmpB = Convert.FromBase64String(netStr);

Please provide a solution.

Thanks,

NL

# Aurelien said on Thursday, June 26, 2008 5:58 AM

I have the same problem.

In fact, as soon as I am willing to use compression, the application crashes. If I set the compression level to none, then I do not have any problem. Setting it to anything else and I will receive the error {"The data at the root level is invalid. Line 1, position 1."}.

Have you got any idea where that can be comming from?

# Glav said on Thursday, June 26, 2008 7:44 AM

NL and Aurelien,

I haven't even looked at this code for years but your errors usually indicate one end has compression, and the other doesn't (and thus expects a valid SOAP XML but instead gets some compressed data), hence your error.

A few things could be happening but its really hard to tell. Have a look through the previous comments as there are some good suggestions around encoding in there.

# Qaiser said on Friday, March 20, 2009 7:39 AM

I am getting errror "Response is not well-formed XML" in case of high level. Please explain what can be wrong.

# eXavier said on Friday, June 12, 2009 4:16 AM

Hi,

The code has missing WSCompression attribute in the generated proxy (reference.cs file). Decorating ReturnBigString() method with [WSCompression(CompressionLevels.High)] gets the sample to work.

Leave a Comment

Title (required) 

Name (required) 

Your URL (optional)

Comments (required) 

Remember Me?