Contents tagged with .NET

  • Asynchronous Streaming in ASP.NET WebApi

     Hi everyone, if you use the cool MVC4 WebApi you might encounter yourself in a common situation where you need to return a rather large amount of data (most probably from a database) and you want to accomplish two things:

    1. Use streaming so the client fetch the data as needed, and that directly correlates to more fetching in the server side (from our database, for example) without consuming large amounts of memory.
    2. Leverage the new MVC4 WebApi and .NET 4.5 async/await asynchronous execution model to free ASP.NET Threadpool threads (if possible). 

    So, #1 and #2 are not directly related to each other and we could implement our code fulfilling one or the other, or both. The main point about #1 is that we want our method to immediately return to the caller a stream, and that client side stream be represented by a server side stream that gets written (and its related database fetch) only when needed. In this case we would need some form of "state machine" that keeps running in the server and "knows" what is the next thing to fetch into the output stream when the client ask for more content.

    This technique is generally called a "continuation" and is nothing new in .NET, in fact using an IEnumerable<> interface and the "yield return" keyword does exactly that, so our first impulse might be to write our WebApi method more or less like this:

     

            public IEnumerable<Metadata> Get([FromUriint accountId)
            {
                // Execute the command and get a reader
                using (var reader = GetMetadataListReader(accountId))
                {
                    // Read rows asynchronously, put data into buffer and write asynchronously
                    while (reader.Read())
                    {
                        yield return MapRecord(reader);
                    }
                }
            }
     

    While the above method works, unfortunately it doesn't accomplish our objective of returning immediately to the caller, and that's because the MVC WebApi infrastructure doesn't yet recognize our intentions and when it finds an IEnumerable return value, enumerates it before returning to the client its values. To prove my point, I can code a test method that calls this method, for example:

            [TestMethod]
            public void StreamedDownload()
            {
                var baseUrl = @"http://localhost:57771/api/metadata/1";
                var client = new HttpClient();
     
                var sw = Stopwatch.StartNew();
                var stream = client.GetStreamAsync(baseUrl).Result;
                sw.Stop();
                Debug.WriteLine("Elapsed time Call: {0}ms", sw.ElapsedMilliseconds);
             }

     

    So, I would expect the line "var stream = client.GetStreamAsync(baseUrl).Result" returns immediately without server-side fetching of all data in the database reader, and this didn't happened. To make the behavior more evident, you could insert a wait time (like Thread.Sleep(1000);) inside the "while" loop, and you will see that the client call (GetStreamAsync) is not going to return control after n seconds (being n == number of reader records being fetched).

    Ok, we know this doesn't work, and the question would be: is there a way to do it?

    Fortunately, YES!  and is not very difficult although a little more convoluted than our simple IEnumerable return value. Maybe in the future this scenario will be automatically detected and supported in MVC/WebApi.

    The solution to our needs is to use a very handy class named PushStreamContent and then our method signature needs to change to accommodate this, returning an HttpResponseMessage instead of our previously used IEnumerable<>. The final code will be something like this:

     

    public HttpResponseMessage Get([FromUriint accountId)
            {
                HttpResponseMessage response = Request.CreateResponse();
     
                // Create push content with a delegate that will get called when it is time to write out 
                // the response.
                response.Content = new PushStreamContent(
                    async (outputStream, httpContent, transportContext) =>
                    {
                        try
                        {
                            // Execute the command and get a reader
                            using (var reader = GetMetadataListReader(accountId))
                            {
     
                                // Read rows asynchronously, put data into buffer and write asynchronously
                                while (await reader.ReadAsync())
                                {
                                    var rec = MapRecord(reader);
     
                                    var str = await JsonConvert.SerializeObjectAsync(rec);
     
                                    var buffer = UTF8Encoding.UTF8.GetBytes(str);
     
                                    // Write out data to output stream
                                    await outputStream.WriteAsync(buffer, 0, buffer.Length);
                                }
                            }
                        }
                        catch(HttpException ex)
                        {
                            if (ex.ErrorCode == -2147023667) // The remote host closed the connection. 
                            {
                                return;
                            }
                        }
                        finally
                        {
                            // Close output stream as we are done
                            outputStream.Close();
                        }
                    });
     
                return response;
            }

     

    As an extra bonus, all involved classes used already support async/await asynchronous execution model, so taking advantage of that was very easy. Please note that the PushStreamContent class receives in its constructor a lambda (specifically an Action) and we decorated our anonymous method with the async keyword (not a very well known technique but quite handy) so we can await over the I/O intensive calls we execute like reading from the database reader, serializing our entity and finally writing to the output stream.

     

     Well, if we execute the test again we will immediately notice that the client line (var stream = client.GetStreamAsync(baseUrl).Result;) returns immediately and then the rest of the server code is executed only when the client reads through the obtained stream, therefore we get low memory usage and far greater scalability for our beloved application serving big chunks of data.

    Enjoy!

    Andrés.

     

      

     

     

     

     

    Read more...

  • MongoDB usage best practices

    The project I'm working on uses MongoDB for some stuff so I'm creating some documents to help developers speedup the learning curve and also avoid mistakes and help them write clean & reliable code.

    Read more...

  • Registering a file-less WCF Service Dynamically

     I know...  I know...   it has been a while since my last post, but I never forgot about it, just having a blast doing some fun things. Well, some of that stuff wasn't really THAT fun, but that's part of being a CTO and always there is room for improvement in some processes and structures in the company, and so...  after a lot of hard work, we are now in a much better position to actually start doing some fun stuff.

    And the fun part has begun, in thre form of a couple new and REALLY interesting and fun projects, where I participate not only as CTO but also I'm fulfilling a role of an Chief Architect as well, overseeing aspects of general architecture from development and infrastructure standpoints, and more important, doing a LOT of research and proof of concepts to hand over to our Architect Team.

    So, for this new project (let's say, Codename "E2") I'm in charge of researching some stuff, and here I will present the results of one of those topics: dynamic (runtime) registering of WCF services.

    Before jumping in today's topic, let me talk a little about the other topics I'm researching, just to paint a broader picture and set the stage for future posts. Our E2 project is a little ambitious in some aspects, and it's main motto is "Configurable Dynamic", so we are exploring ways to make this happen and the list of things to explore first are:

    • Dynamic registration of WCF services (today's topic)
    • Dynamic Data Access (ORM without Entities)
    • Dynamic Business Rules (or logic, if you want)
    • Dynamic Business Processes (workflows are a possibility here, but not the only one)
    • Performance of of the above (mainly IIS, ASP.NET, WCF) and how to optimize the platform.
    • Performance techniques to use: caching, profiling, monitoring, etc.
       

     Of course there are other topics (like Security, Scalability, Fault Tolerance, etc) but we will advance over those in time. For today's topic, let's explain a little what I'm talking about. For WWCF services there are two ways of let know the runtime environment that we have a Service class and we want to expose it to the world:

    1. The plain old .svc file approach: we need a file with .svc extension and this file will contain the Type information needed to activate the service.
    2. The new CBA (configuration based activation) approach: this is new in NET4, and therefore is possible to create a WCF service WITHOUT the svc file, using only a section in the web.config file (<serviceHostingEnvironment><serviceActivations>).

    While this second option is very interesting, we cannot do it at runtime and so the idea of this post was born. We can register HttpModules at runtime (the MVC3 project do that to register the HttpModule that handles Razor views) and so wwe can try to do something similar. That functionality (dynamically register an HttpModule) could be found in the "Microsoft.Web.Infrastructure" assembly using the "RegisterModule" method found in the "DynamicModuleUtility" class.

    We are going to take a similar approach to this helper method and use reflection to inject our service configuration somewhere so the runtime thinks it have a CBA Service (file-less) and can active it as any normal service. This is achieved in the "DynamicServiceHelper" class in the attached sample project, using the "RegisterService" method.

     This is the source code for the "RegisterService" class:  (some lines where removed for brevity)

     
    namespace System.ServiceModel
    {
        public static class DynamicServiceHelper
        {
            static object _syncRoot = new object();
            static FieldInfo hostingManagerField;
            static MethodInfo ensureInitialized;
            static FieldInfo serviceActivationsField;
     
            static DynamicServiceHelper()
            {
                ensureInitialized = typeof(ServiceHostingEnvironment).GetMethod("EnsureInitialized"BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.InvokeMethod);
                hostingManagerField = typeof(ServiceHostingEnvironment).GetField("hostingManager"BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.GetField);
            }
     
            public static void EnsureInitialized()
            {
                ensureInitialized.Invoke(nullnew object[] { });
            }
     
            public static void RegisterService(string addr, Type factory, Type service)
            {
     
                lock (_syncRoot)
                {
                    object hostingManager = hostingManagerField.GetValue(null);
                    
                    if (serviceActivationsField == null)
                        serviceActivationsField = hostingManager.GetType().GetField("serviceActivations"BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.GetField);
                                    
                    Hashtable serviceActivations = (Hashtable)serviceActivationsField.GetValue(hostingManager);
     
                    string value = string.Format("{0}|{1}|{2}", addr, factory.AssemblyQualifiedName, service.AssemblyQualifiedName);
     
                    if (!serviceActivations.ContainsKey(addr))
                        serviceActivations.Add(addr, value);
                }
            }
        }
    }

    This class have two methods: "RegisterService" and "EnsureInitialized". The first method is pretty self explanatory, it receives the service activation information (endpoint, service type and factory type) but the "EnsureInitialized" method deserves a little explanation. As you can see, we use the "ServiceHostingEnvironment" class, and this class is used internally to manage all WCF services running in our process. It uses an internal class named "HostingManager" that is in charge of doing the discovery of existing WCF services (by looking for ".svc" files in the application folder and reading the <serviceActivations> configuration section, so by using a little reflection we can see that this information is processed in the private method "LoadConfigParameters", and in that method we see that the list of services to activate is added to the private field named "serviceActivations" (a Hashtable).

    If you see the example code, you would also see there is another class named "Startup" that contains an assembly attribute named "PreApplicationStartMethod":

    [assembly: PreApplicationStartMethod(typeof(WcfService1.Startup), "Init")]
    

     This attribute says to the runtime environment (WAS) that should call the "Init" method in the "Startup" class, and this method is called to give you the chance to initialize your code so it's called almost before anything else so the WCF / WAS infrastructure is not yet initialized at this point. So, in this moment we cannot use anything on the "ServiceHostingEnvironment" class. Hopefully, this class also have an "EnsureInitialized" method (private, of course) so we can try to invoke it and hope it doesn't have any other dependency and will initialize what wew need to register our service. Luckily for me, it worked without any evident side-effects, so let's use it!

    Another method to register our services would be not calling the "EnsureInitialize" method and wait for something else to do it. In that case we can create (and dynamically inject) an HttpModule to register our service in it's "Init" method. That will work and I'm including that code in my sample as well (in fact it was my first approach) but it has an important drawback: it needs to something else to initialize the Http stack, and therefore our WCF Service (using TCP to make things more interesting) will not be activated if the hosting application doesn't receive an Http request first.

    Well, that's it, hope you enjoyed the reading and find the code useful (by the way, you can use it in any way you want, so I have to include here the usual "No warranties" disclaimer), and if you make some improvement or have any comments about my code I would like to know.

    You can download the source code from here: DynamicService.zip

    Best regards,

    Andrés G Vettori, CTO, VMBC

    Read more...

  • Projects upgraded from VS2008 to VS2010

    Yesterday I did my first test migrating one of our our rather complex VS2008 solutions to VS2010 (and TFS 2010). The solution and projects upgrade went generally well, I only had a few minor things to fix I'm detailing below:

    • All WCF client proxies were automatically regenerated. This is not neccesarily bad but we had touched our generated code (for example, to add a #pragma warning disable 1591 to avoid failing compilation because of the missing XML comments). To solve this I only needed to undo the changes produced in the generated files and everything worked as usual.
    • The new VS and/or compiler seems to be more strict in some things. Compilation failed because it found a couple malformed XML configuration files, a a couple conflicts about references (projects referencing directly and indirectly different versions of the same component). Fixing those errors was easy and I'm happy to find them automatically in VS2010.
    • I had to uninstall WIX 3 and install the (experimental) WIX 3.5 with Votive support for VS2010. Wix projects didn't compiled at first so I had to manually edit the .wixproj file and change the tool path. Also the new WIX 3.5 has a known bug where some of the project variables have the wrong value (for example, var.project.ProjectDir now includes a trainling "/bin" that is clearly wrong). A quick fix is to include a "/.." in the source path to reference the root project folder, but this is probably going to need rolling back this hack after the bug is fixed.
    • Some projects didn't upgraded automatically (WebTests, for example) and I had to migrate them manually. I'm investigating a issue with this projects because VS2010 seems to prompt me to migrate the same projects EVERY time I open the solution.
    • Tests projects (Unit tests, for example) are migrated to Framework 4.0 automatically. Those projects cannot run or compile using a previous version of the CLR and therefore ALL its references need to be compiled for Framework 4.0 as well.

    Well, that's all for now...   I'm going to test some build scripts now...  wish me luck!

     Andres G Vettori, VMBC, CTO

    Read more...

  • Retina.NET support for Microsoft SQL Server 2008

    The latest version of Retina.NETm the open source ORM published on CodePlex has now explicit support for MS SQL 2008. The new DataStore now supports the new data types (DATETIME2, TIME, DATE, DATETIMEOFFSET) and the new FILESTREAM access modifier for BLOBs columns.

    Check it out at http://retina.codeplex.com and download the latests binaries or sources.

    Andrés G Vettori, VMBC, CTO

    Read more...

  • Add Blackberry support to Forms Server 2007 or Sharepoint 2007

     A few months ago I was the architect for a project involving Microsoft Forms Server 2007 and we needed to support viewing Infopath forms on mobile devices, specifically Blackberry devices.

    The Microsoft Forms Server 2007 is a striped down Sharepoint 2007 server so all Infopath forms web rendering for mobile devices is done through the ASP.NET mobile controls support.

    Unfortunately, Blackberry devices aren't recognized so Infopath forms are render as standard web forms, wich aren't supported by Blackberry devices. Luckily for us, adding support for additional devices is not difficult and can be done by adding some information to a "compat.browser" file found in the "App_Browsers" folder inside the Sharepoint virtual directory. In this file you need to add the following:

    <!-- BlackBerry RIM Browser -->
        <browser id="RIM40" parentID="Default">
            <identification>
                <userAgent match="BlackBerry" />
            </identification>
            <capabilities>
          <capability name="isMobileDevice" value="true" />
          <capability name="javascript" value="true" />
          <capability name="ecmascriptversion" value="1.3" />
          <capability name="version" value="${version}" />
          <capability name="majorVersion" value="${major}" />
          <capability name="minorVersion" value="${minor}" />
          <capability name="supportsCss" value="true" />
          <capability name="frames" value="false" />
          <capability name="cookies" value="true" />
          <capability name="isColor" value="true" />
          <capability name="requiresLeadingPageBreak" value="true" />
          <capability name="requiresUniqueFilePathSuffix" value="true" />
          <capability name="supportsBold" value="true" />
          <capability name="supportsFontSize" value="true" />
          <capability name="supportsImageSubmit" value="true" />
          <capability name="supportsItalic" value="true" />
          <capability name="requires303Redirect" value="true" />
         </capabilities>
        </browser>

    With that your Blackberry browsers will be recognized as mobile devices and our Infopath Forms Server will be able to render the proper content.

    Hope you find this useful, seeU!

    Andrés G Vettori
    MCSE/MCSD/MCT
    Systems Architect

    Leader of the C# Community of the Microsoft Users Group Argentina

    Read more...

  • Retina.NET 2.0 Core & Designer Released

    Hey! just a quick announcement to say that I released a new version of Retina.NET Core and also the new DSL designer for VS2008.

    You can download the precompiled binaries or source code from the CodePlex Retina.NET page. As always, comments / suggestions / critics...  are all welcome.

    Thanks in advance,

    PS: the Retina Setup package still has some older documentation, I know...   sorry for that but I preferred to release the bits and fix the documentation later (I hear some helping offering there...?)

    Andrés G Vettori
    MCSE/MCSD/MCT
    Systems Architect

    Leader of the C# Community of the Microsoft Users Group Argentina

    Read more...

  • Retina.NET project active again

    The Retina.NET project was dormant for some time, but with some help I'm bringing it back to life.

    The last official release was made on October 2006 (v1.0.8.0) and at that time I was working on the new v2 release, but I never finished that task and so the project came to a stop.

    Nonetheless, I have managed to include a couple of interesting changes in those internal Retina.NET versions, and I have found that some other people also have made some useful changes in their own Retina.NET version and are offering to integrate some of those new features with the official project sources.

    So, if you used Retina.NET in some project and made a change or addition to the source code and you would like to see those changes included in an official project release, please stop by the CodePlex Retina.NET page or and let me know.

    Also, if you want to participate in the project in some way, you have an idea to implement some cool new feature on Retina.NET, or you have some time to improve existing documentation, samples or Unit Tests, please stop by the Retina.NET CodePlex site and let me know.

    I'm actively working on getting the new DSL designer that will replace the old EntityGen project, but there is some other work in progress like: LCG support, Generics support, LINQ support & VistaDB support.

    Thanks in advance,

    Andrés G Vettori
    MCSE/MCSD/MCT
    Systems Architect

    Leader of the C# Community of the Microsoft Users Group Argentina

     

    Read more...

  • Introduction to WCF Extensibility article published at LevelExtreme Magazine

    Hi everybody! I have been off radar quite a while, so I'm just writting this quick post to make everybody know that I have written and published (Thanks Martín Salías) a nice article (shameless plug, I know...) about WCF extenssibility mechanisms.

    The article can be viewed online at LevelExtreme Magazine WebSite home page or directly at the article.

    Enjoy!

    Andrés G Vettori
    MCSE/MCSD/MCT
    EDS Top Gun - .NET Capability Architect

    Leader of the C# Community of the Microsoft Users Group Argentina

    Read more...