Uploading large files to WCF web services

Recently, we need to allow users to upload large files to a web service. Fortunately, WCF does support this scenario (see MSDN: Large Data and Streaming). MSDN recommends:

The most common scenario in which such large data content transfers occur are transfers of binary data objects that:

• Cannot be easily broken up into a message sequence.
• Must be delivered in a timely manner.
• Are not available in their entirety when the transfer is initiated.

For data that does not have these constraints, it is typically better to send sequences of messages within the scope of a session than one large message.

Since our files do not have these constraints, we decide to break them up into sequences of segments, and pass the segments to the web service as byte arrays. There are several remaining issues: 

Firstly, by default, WCF uses Base64 encoding to encode byte arrays, resulting 4:3 ratio in encoding overhead. To avoid this overhead, WCF support both MTOM and Binary encoding. MTOM is a widely supported format that sends large binary data as MIME attachments after the soap message. Binary encoding is a WCF proprietary encoding format. It is extremely easy to use either formats. Just change the messageEncoding attribute in the binding element in the configuration from Text to Mtom or Binary.

Secondly, experiment shows that high level handshake through web service is fairly slow. Better performance can be achived by using a larger chunk size. By default, the maximum message size is 65536 and maximum array size is 8192. In order to transmit larger chunks, we need to increase the maximum message and array size. This can be done by increasing the maxBufferSize and maxReceivedMessageSize attributes of the binding element and the maxArrayLength attribute of the readerQuotas element. Note that increasing the maximum sizes also increases the risk of denial of service attack so select the size carefully.

Thirdly, although it is possible to configure the IIS to automatically compress HTTP responses, HTTP clients cannot be configured to automatically compress the HTTP requests. WCF has a sample custom compression encoder. It is fairly easy to use the compression encoder. Just reference the GZipEncoder.dll on both the server and the client side. The configuration settings can be extracted from the sample. The only thing that is not apparent from the sample is how to increase the message size. The maxReceivedMessageSize  attribute can be configured on the httpTransport element:

To change the maxArrayLength in readerQuotas, I actually have to make a modification to the sample code:

The finish proof-of-concept application can be downloaded here.