Web Service Compression

35 Comments

• That is just fantastic.This has long driven me nuts, it seemed the single most obvious place that would benefit from compression. Don't suppose you've figured out a way to support IIS compression - so the proxy sending 'compression supported' headers and transparently decompressing the stream as part of the generated proxy.
  

• This is a different kettle of fish as you would not be dealing with SOAP headers, rather you would probably be implementing a new HttpHandler (at a guess). Once I get a bit more time, I may have a go at it.
  

• This is a really nice piece of work. I've tried something similar, but failed to get it to work with WSE, I think because of the ordering of the SOAP extensions. Do you know how you can get this Compression extension to be executed as the first extension in the PipeLine?
  

• Thanx Marshall.
  
  
  
  Not sure about the ordering you asked about as I haven't really tried or tested that aspect but if I get time I will have a look. Feel free to email me at glav@aspalliance.com if you want to work through it.
  

• This is great DLL'... thank you friend..
  

• Marshall, did you try setting the extension's Group and Priority in your Web.config?
  

• Great piece of code, it helps a great deal! I do have one issue/question. Before applying compression, I was able to use the Context.Request.InputStream in the WebMethod on the server to retrieve the actual SOAP request. That stream now returns an empty string.
  
  
  
  I was using this for debugging as well as for getting the length of the actual request. Is there anyway to get this information with this compression piece in place?
  
  

• Hi,
  
  
  
  The extension is great. Was very simple to use and makes a big difference to the transfer times but I seem to have discovered a bug. I get a soap exception on several characters such as "º", "©", "Ó", etc.
  
  
  
  Strangely this will even happen when the Compression Levels is set to None! I tried playing with the requestEncoding, responseEncoding options in web.config but that didn't help.
  
  
  
  Anybody have any idea. I'm trying to figure this out but I'm not familiar with these type of encoding? unicode? issues.
  
  
  
  To test, the Web Method can be as simple as the following:
  
  
  
  [WebMethod] [WSCompression(CompressionLevels.None)]
  
  public string HelloWorld()
  
  { return "º";}
  
  
  
  I call the method from an equally simple WinForms App. (I don't get the error in the auto generated browser app).
  
  
  
  The exception reads:
  
  "This is an unexpected token. The expected token is 'TAGEND'. Line 1, position 489"
  
  
  
  Sorry for the length of the post. Hope somebody can figure this out.
  
  
  
  Cheers,
  
  Ed.
  

• Hi Ed,
  
  
  
  Not sure, perhaps the extension needs HtmlEncode the text before compressing, then do a HtmlDecode after decompression.
  
  
  
  So try doing a Server.HtmlEncode and Server.HtmlDecode on the text to see if that helps. If that fixes it, then it would be pretty easy to implement. Dont know why it has problems when compression is none though as it bypasses the whole compression library then.
  
  
  
  

• Hi Paul,
  
  
  
  Just figured out the problem. In Decompress()
  
  
  
  change:
  
  _bufferedStream.Write(System.Text.Encoding.UTF8.GetBytes(sb.ToString()),0,sb.Length);
  
  
  
  to:
  
  byte[] bytesSB = System.Text.Encoding.UTF8.GetBytes(sb.ToString());
  
  _bufferedStream.Write(bytesSB,0,bytesSB.Length);
  
  
  
  Looks like the encoding changes the length of the array when there are "special chars" thus calling the Length on sb returns the incorrect size.
  
  
  
  Thanks again for this useful piece of code!
  
  
  
  Cheers,
  
  Ed.
  
  

• Guys anyone managed to compress a dataset?
  

• If the Dataset is part of the web service payload, then it will be compressed. ie. If you are passing or returning a dataset, it will be serialised as part of the message, and will be compressed if you use this library.
  

• [quote]
  
  So the proxy sending 'compression supported' headers and transparently decompressing the stream as part of the generated proxy.
  
  [/quote]
  
  
  
  I've done this. I sell a version on my website, but the basic trick is to implement the HTTP Compression module by Ben Lowry on the server side.
  
  
  
  And override the HttpSoapClient (not sure i've got the right name) class to handle the decompression.
  
  
  
  Then change the proxy stub class to ingerit from your overridden class instead
  

• It is exactly what I am looking for but
  
  I couldnot make it work with WSE 2.0.
  
  I always get this message "EOF in header" from Decompress(). Any idea?
  
  
  
  Thanks,
  
  Pam
  
  
  
  
  
  

• Pam,
  
  
  
  Not sure, I have yet to try it with WSE2.0, however have a look at the comments from Ed. Perhaps try what he has done to see if it makes a difference. Would be interested to know how you go.
  
  
  
  - Paul
  

• I tried that but it didn't help. I guess it was something about WSE policy pipeline. Propably the compression and decompression has to be done after the soap request successfully passed WSE policy pipeline processing.
  

• Thanks for the info Pam. Much appreciated.
  

• I just started using your compression library for a web service that returns datasets and it works great. The only problem I'm having is that on a really large dataset I get an exception (Response is not well-formed XML->System.Xml.XmlException: Unexpected end of file has occurred. The following elements are not closed: soap:Envelope. Line 121609, position 8160.) Do you think there would be a limit on the size of the dataset?
  
  Thanks,
  Brandon
  

• Brandon,
  
  Absolutely there is a size limit. Check the setting in your web.config. if not there, I think it defaults to 4096 which could be causing the issue if its a really big (and I mean big DataSet).
  

• Hi Glav,
  
  I'm just about to try your library. Will it work with PocketPC and Windows Mobile, .NET Compact Framework?
  
  Thanks!!
  

• The reason Brandon is getting that error is that there is a bug in this SOAP extension. The issue is that UTF-8 encoded text when represented as an array of bytes (which is a necessary step in the compression transformation) can have anywhere from 1 to 4 bytes per character. For space efficiency, the most common 128 characters are represented by one byte each. This code assumes that ALL the characters are one byte each. The code that copies the unzipped data stream into the SOAP message buffer assumes that the number of bytes it needed to copy is always equal to the number of characters in the unzipped string representation of the SOAP message. That USUALLY is a very safe assumption, because the overwhelming majority of SOAP messages consist of nothing but the most common 128 characters. However, if your SOAP payload has any “special” characters (that end up taking more space in the byte array then the code expects because they have multi-byte representations), then the end of the SOAP message gets truncated off, which causes the SOAP message to appear to be badly formatted XML (because the closing tag of the SOAP envelope gets left off). You should change the text encoding class to calculate the size of the byte array rather than rely upon the length of the string it pulls the raw bytes from.
  

• how do you consume the compressed web service?
  

• Helo
  I try it, for my web services, they return DataSets
  And dont work.
  I made this modificaction in the Decompress method
  private void Decompress()
  {
  
  if (_compressionLevel != CompressionLevels.None)
  {
  // Create a stream reader to give us a string representation of the network stream
  StreamReader sr = new StreamReader(_networkStream,System.Text.Encoding.UTF8);
  string netStr = sr.ReadToEnd();
  sr.Close();
  
  // Get the encoded bytes from the Base64 string.
  byte[] tmpB = Convert.FromBase64String(netStr);
  
  // Make a tmporary memory stream to work on
  // (failure to do this i.e working directly on the network stream can have adverse effects)
  MemoryStream tmpMem = new MemoryStream(tmpB);
  
  // Use the Zip compression libraries to decompress the bytes
  ZipInputStream zip = new ZipInputStream(tmpMem);
  ZipEntry entry = zip.GetNextEntry();
  // We need to read in the compress stream chunks at a time. The 'Read' method of the
  // ZipInputStream can only read a max of 23191 bytes (I think thats its max)
  int size=1;
  byte[] buf = new byte[2048];
  // System.Text.StringBuilder sb = new System.Text.StringBuilder();
  
  while (size > 0)
  {
  size = zip.Read(buf,0,2048);
  if (size > 0)
  _bufferedStream.Write(buf, 0, size);
  
  // sb.Append(System.Text.Encoding.UTF8.GetString(buf,0,size));
  }
  zip.Close(); // should implicitly close our memory stream.
  
  //Write our tnetwork stream to the buffered stream so that it gets transmitted
  // _bufferedStream.Write(System.Text.Encoding.UTF8.GetBytes(sb.ToString()),0,sb.Length);
  }
  else
  {
  // No Compression, plain text
  StreamReader sr = new StreamReader(_networkStream);
  string s = sr.ReadToEnd();
  sr.Close();
  _bufferedStream.Write(System.Text.Encoding.UTF8.GetBytes(s),0,s.Length);
  }
  
  _bufferedStream.Position = 0; // If we dont do this, nothing is returned!
  
  }
  

• I'm uploading a large body to a web service. Would your library work as well?
  

• Ramon,
  
  It would work so far as compressing the content goes, but you really need to make sure the content can be decompressed. so the receiving service needs to implement this library or something similar in order to decompress it.
  

• Glav,
  
  I need to consume a WS that uses WsSoapComression, but the client is in Delphi, how can I decompress the Stream??
  
  It can be reached via the .dll??
  
  Thnx
  

• thanks, but I cannot find anywhere in your sample app that shows where to add the Compression stuff.
  
  In my app i get this error:
  
  {"There is a problem with the XML that was received from the network. See inner exception for more details."}
  
  Inner Exception:
  {"The data at the root level is invalid. Line 1, position 1."}
  
  i'm very new to web services, help!
  

• I know this has been around for a long time but I figured I'd post a comment anyway as people (like me) are just finding this.
  
  The encoding used in the soap extension along with the NZipLib is UTF8. This will not work with special chars outside of the standard ascii char set (above dec 127). To allow the soap extension to work with special chars simply change all code using Encoding.UTF8 to Encoding.Default. This will allow the extension to use the default encoding and code page throughout the process and keep the special chars intact and won't throw exception.
  
  Hope it helps someone.
  
  Dan
  

• 
  Hi Dan,
  I also tried your solution of converting encoding.utf8 to encoding.default, but it still returned me the same error
  can someone please put up a through and complete package here.
  
  Regards
  
  Brij
  

• Hi,
  
  When I run the TestClient project and hit the "Call ReturnBigString" button that internally calls the ReturnBigString() method of your webservice. I get an exception "Invalid character in a Base-64 string." in Decompress() method line:141 byte[] tmpB = Convert.FromBase64String(netStr);
  
  Please provide a solution.
  
  Thanks,
  NL
  

• I have the same problem.
  In fact, as soon as I am willing to use compression, the application crashes. If I set the compression level to none, then I do not have any problem. Setting it to anything else and I will receive the error {"The data at the root level is invalid. Line 1, position 1."}.
  Have you got any idea where that can be comming from?
  

• NL and Aurelien,
  
  I haven't even looked at this code for years but your errors usually indicate one end has compression, and the other doesn't (and thus expects a valid SOAP XML but instead gets some compressed data), hence your error.
  
  A few things could be happening but its really hard to tell. Have a look through the previous comments as there are some good suggestions around encoding in there.
  

• I am getting errror "Response is not well-formed XML" in case of high level. Please explain what can be wrong.
  

• Hi,
  The code has missing WSCompression attribute in the generated proxy (reference.cs file). Decorating ReturnBigString() method with [WSCompression(CompressionLevels.High)] gets the sample to work.
  

• Hi,
  
  I have got this working in Win Forms application but it doesnt seem to work for Windows Mobile application.
  
  Has anyone used this for a Windows Mobile application?
  
  Thanks,
  
  Wahid