Transferring large data when using Web Services

I've been working with complex reporting application and big part of the application relies on Web Services.

The client requests some operations by calling the Web Methods. The Web Service basically does everything on the sever where it is deployed and returns data to the client as byte[] array.

When we have some more complex methods, the maximum message size quota (which is 65536 by default) is exceeded.

When trying to get the byte[] back from the Web Service

byte[] rep = serv.getReport(json);

The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element.

As it says, the MaxReceivedMessageSize property value should be increased.

In Web.config under <system.serviceModel>, once you add web service reference, the bindings and client are generated.

It should look something like this:

<system.serviceModel>

 <bindings>

  <basicHttpBinding>

    <binding name="Service1Soap" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"

      allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard" maxBufferSize="65536" maxBufferPoolSize="524288" maxReceivedMessageSize="65536"

      messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered" useDefaultWebProxy="true">

    <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" maxBytesPerRead="4096" maxNameTableCharCount="16384"/>

    <security mode="None">

     <transport clientCredentialType="None" proxyCredentialType="None" realm=""/>

     <message clientCredentialType="UserName" algorithmSuite="Default"/>

    </security>

   </binding>

  </basicHttpBinding>

 </bindings>

 <client>

  <endpoint address="http://hajan/mywebservapp/MyService.asmx" binding="basicHttpBinding" bindingConfiguration="Service1Soap" contract="HSServ.Service1Soap" name="Service1Soap"/>

 </client>

</system.serviceModel>

So, pay attention on the <binding name="Service1Soap ...> element:

Whole line:

<binding name="Service1Soap" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard" maxBufferSize="65536"
maxBufferPoolSize="524288" maxReceivedMessageSize="65536" messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered" useDefaultWebProxy="true">

So, you should make change to the bolded attributes of the binding element.

Because I was not sure how big in size the byte[] array will be in the future, I have set it to the maximum of 2^31 - 1 or 2147483647 which is the biggest 32 bit number.

Next, we should pay attention on the transferMode, which by default is Buffered.

- Bufferred means that the request and response messages will be both buffered.

Other possible enumerations for transferMode are:

- Streamed which means both the request and response messages will be streamed, or

- StreamedRequest where the request will be streamed while response message buffered, or

- StreamedResponse where the request message will be buffered while response message streamed.

For those that don't know, Buffered means that the transfer will hold the entire message in memory buffer all until the transfer is completed.
Streamed means that only the message headers will be buffered, while the message body will be exposed as a stream.

In our case, after we've made change to the maxReceivedMessageSize from 65536 to 2147483647 - if we leave the transferMode="Buffered" we will get another error:

For TransferMode.Buffered, MaxReceivedMessageSize and MaxBufferSize must be the same value. Parameter name: bindingElement

True. It's required both these attributes to have same value, which is logically because the MaxReceivedMessageSize (at this moment) is larger than MaxBufferSize, and we are expecting the whole message to be buffered.

Here, we have two options:

a) You can change the MaxBufferSize to size of 2147483647

b) You can make the transferMode to: Streamed, StreamedRequest or StreamedResponse.

You can change the option you think is more suitable for your application, even streamed transfers can improve the scalability of a service especially when transferring large messages, by eliminating the need of large buffers.

In the MSDN library, I have found the following paragraph regarding decision of using buffered and streamed transfers:

"The decision to use either buffered or streamed transfers is a local decision of the endpoint for HTTP transports. For HTTP transports, the transfer mode does not propagate across a connection, or to proxy servers or other intermediaries. Setting the transfer mode is not reflected in the description of the service contract. After generating a proxy to a service, you can (it is allowed but not required) edit the configuration file for services intended to be used with streamed transfers to set the transfer mode. For TCP and named pipe transports, the transfer mode is propagated as a policy assertion."

Once you try to test it, the next error you might get is regarding the maxArrayLength property of readerQuotas element. It's all about the XML Reader of the XML data.

<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" maxBytesPerRead="4096" maxNameTableCharCount="16384"/>

The error is as follows:

The maximum array length quota (16384) has been exceeded while reading XML data. This quota may be increased by changing the MaxArrayLength property on the XmlDictionaryReaderQuotas object used when creating the XML reader. Line 1, position 205847.

So, we need to increase the maxArrayLength value because our byte[] array which is returned in our example exceeds the 16348 size. The max value you can set for maxArrayLength is again the max Int32 number which is equal to 2147483647. Once you are done with this setting, your web service will be able to transfer large data between your web service and the web service consumer - client application.

In the future, I will write more on the same topic using WCF, which indeed has a very bright future.
(I just got this suggestion from Vivek, who is one of the administrators in CodeASP.NET and recognized Microsoft MVP)

Hope this was useful.

Regards,
Hajan

This blog post is also posted on my CodeASP.NET community’s blog:
http://codeasp.net/blogs/hajan/microsoft-net/802/transferring-large-data-when-using-web-services2

Translation in Macedonian language:
http://mkdot.net/blogs/hajan/archive/2010/07/16/13868.aspx