Lists and their performance has been hot discussion topic for years. I have seen many examples and read a lot of stories about List class and I made conclusion that there is too much pointless opinions about it. In this posting I will show you how List<T> performs when adding elements to it and how it works internally. Be prepared for interesting discussion!
As a first thing I will introduce you how I measured List<T> performance and what results I got. After that we will see how List<T> is internally optimized. There are some changes that happen during optimizations and sometimes people consider these changes as scary ones. I will calm you all down, there is nothing so horrible as some authors say. In the end of the posting I will make some quick conclusions about my experiment and investigations.
My experiment
I wrote for loop that adds 10 million integers to List<int>. I made eight experiments with different initial List<T> capacity to see how initial capacity affects performance. In all of my tests I made sure I don’t make mistakes I described in my previous posting Common mistakes made when measuring the speed of code. Here is the example of my code I used to measure List<T> performance.
static void Main()
{
var watch = new Stopwatch();
var cycles = Math.Pow(10, 2);
var elements = Math.Pow(10, 7);
var listCapacity = 10000000; // (int)elements;
var times = 0D;
for (var cycle = 0; cycle < cycles; cycle++)
{
var inputList = new List<int>(listCapacity);
watch.Reset();
watch.Start();
for (var i = 0; i < elements; i++)
{
inputList.Add(i);
}
watch.Stop();
times += watch.ElapsedMilliseconds;
}
Console.WriteLine("Time: " + times / cycles);
Console.ReadLine();
}
For initial capacity 1 I got 0.78 seconds as average to add 10 million elements to List<T> and for initial capacity 10 million I got 0.33 seconds as average time. So there is two times difference although in time the difference is so small that we don’t recognize it. Here are the results.
As you can see it does not really matter how we choose the initial capacity of List<T>. There seems to the question if these is small initial capacity or there is enough capacity to fill list with elements without any need to resize internal buffers. By default, initial capacity is set to four elements.
How List<T> is optimized?
Source code of .NET Framework is available for everyone and my Visual Studio 2010 is configured to download .NET Framework symbols and source code when I debug my code. The Add() method of List<T> is as follows.
public void Add(T item)
{
if (_size == _items.Length) EnsureCapacity(_size + 1);
_items[_size++] = item;
_version++;
}
Before adding element to internal buffer there is check for capacity. If elements count is reaching the capacity of list then capacity will be increased automatically. Finding new capacity happens in internal method called EnsureCapacity().
private void EnsureCapacity(int min)
{
if (_items.Length < min)
{
int newCapacity;
if (_items.Length == 0)
newCapacity = _defaultCapacity;
else
{
newCapacity = _items.Length * 2;
}
if (newCapacity < min) newCapacity = min;
Capacity = newCapacity;
}
}
As we can see the new capacity for list is two times bigger than previous capacity. But as we saw from diagram above we don’t win much if we don’t set the capacity to final length of list at first place.
What happens when list capacity changes?
When capacity is increased then something happens in setter of Capacity. In my opinion these operations should be part of EnsureCapacity() and not part of some setter. Here is how internal buffer is resized to match the new capacity.
public int Capacity
{
get
{
return _items.Length;
}
set
{
if (value != _items.Length)
{
if (value > 0)
{
T[] newItems = new T[value];
if (_size > 0)
{
Array.Copy(_items, 0, newItems, 0, _size);
}
_items = newItems;
}
else
{
_items = _emptyArray;
}
}
}
}
We can see that this is the place when blasphemy called array copying happens. Now we know how list capacity is optimized and how internal buffer is increased. Let’s see now how the number of array copies changes with elements count in list.
Comparing this diagram with previous one we have new question – how array copying affects overall performance so lightly? We can practically see correlation only in last part of diagrams. What is going on?
How is implemented resizing of internal array?
Okay, let’s open Reflector and hunt Array.Copy() as far as we can. Pretty soon our journey ends with call declared like this.
[MethodImpl(MethodImplOptions.InternalCall)]
[ReliabilityContract(Consistency.MayCorruptInstance, Cer.MayFail)]
internal static extern void Copy(Array sourceArray, int sourceIndex,
Array destinationArray,
int destinationIndex,
int length, bool reliable);
This is the final point we can reach. On picture we can show it this way.
The big question is what is happening in this external method. I discovered the answer when I measured performance of different array copying methods. All safe managed code methods gave me pretty similar results. Extremely heavy performance hit came when I moved to unsafe code and pointers.
When I compare performance of Array.Copy() method with my own array copy method then their performance is almost equal. As there is no other way I know that should end up with externals and give extremely fast results I am very sure that internally .NET Framework copies arrays using some technique that I tried out with pointers.
Conclusions
Based on previous discussion I have three conclusions.
- Initial capacity of list affects it’s performance. When we avoided internal buffer resizings we got almost twice better results than with buffer corrections. Although we got performance better it is not so much better that we should work hard to find out final capacities of our lists at any price.
- List<T> is optimized very well. When list capacity is reached then capacity of list is doubled and that leaves room enough to add new elements without internal buffer resizing. For 10 million elements with initial capacity 1 we got only 24 array copy operations.
- Array copy is not a real problem. Array copy in .NET Framework is implemented (expectedly) in unsafe code using pointers. You can try out how fast Array.Copy() works when compared to custom code that copies elements from one array to another. Array copy is extremely fast and it is not bottleneck in List<T> work.
During times I have seen problematic ways how developers measure the speed and performance of their code. There are some easy tricks that help you make your measurements way better and accurate comparing to measurements made not so well. In this posting I will give you some hints how to get more accurate results when measuring the speed of your code.
1. Don’t use Debug configuration
This is one common problems that developers do. They forget to switch configuration to Release and therefore they are measuring the assemblies that are full of debug symbols and debugger interactions.
Code compiled without debug symbols and other debugger stuff works sometimes hundreds of times faster than Debug version of same code. On live environments you must use code compiled with Release configuration. There is no point to measure Debug code because it brings too much noise and you have no idea how debugger and other tools affect the measuring process.
2. Use Stopwatch instead of DateTime.Now
Measuring the speed of code is often done using DateTime.Now values before and after measuring. This is wrong because resulting time contains also time delays when current thread was suspended by operating system by example.
If you want better results then use Stopwatch class. This class is perfect fit for scenario like this. Here is the example.
static void MeasureCalculation()
{
var watch = new Stopwatch();
watch.Start();
// do your calculations here
watch.Stop();
Console.WriteLine("Time: " + watch.Elapsed);
}
If you want to find out more about internals of Stopwatch class then please read it’s documentation from MSDN Library.
3. Measure only the code you need to measure
To get better results you have to measure only the code you need to measure and nothing else. This is not very uncommon way how some developers write their measuring code.
static void MeasureCalculation()
{
var watch = new Stopwatch();
watch.Start();
var inputList = GetInputData();
RandomizeInputs(inputList);
// do your calculations here
inputList.Clear();
watch.Stop();
Console.WriteLine("Time: " + watch.Elapsed);
}
This is wrong because with calculations we want to measure we also measure how much time it takes to load data, prepare it before calculations and clear the list with input data after measuring is done. These three activities should not be the part of measuring. Let’s write this code the way it should be.
static void MeasureCalculation()
{
var inputList = GetInputData();
RandomizeInputs(inputList);
var watch = new Stopwatch();
watch.Start();
// do your calculations here
watch.Stop();
inputList.Clear();
Console.WriteLine("Time: " + watch.Elapsed);
}
This code measures only the calculation and nothing else.
4. Turn off diagnostic messages while measuring
Before measuring we need to make sure that our code contains only the functionalities that we need to measure. Still developers forget different diagnostic messages there and we cannot be sure in measuring results anymore. Take a look at the following code.
static void MeasureCalculation()
{
var inputList = GetInputData();
RandomizeInputs(inputList);
var watch = new Stopwatch();
watch.Start();
// Start calculation
Debug.WriteLine("Initializing data for algorithm");
// Some more line
Logger.WriteLine("X(Y, Z)=" + z);
watch.Stop();
inputList.Clear();
Console.WriteLine("Time: " + watch.Elapsed);
}
Well… who the hell is Logger and where it writes it’s output? Is it screen, Windows event log, ULS log, some file, database or what? It doesn’t really matter where output goes. The problem is that the output is there and we don’t need it to be there. So, let’s remove all those diagnostic and tracing messages from code we want to measure. And we are back where we were before.
static void MeasureCalculation()
{
var inputList = GetInputData();
RandomizeInputs(inputList);
var watch = new Stopwatch();
watch.Start();
// Start calculation
watch.Stop();
inputList.Clear();
Console.WriteLine("Time: " + watch.Elapsed);
}
Nice and clean code and measuring is focused only to code that does something we need to measure. Besides diagnostic messages you should remove also all other code that is not necessary in the context of code to be measured.
5. Measure more than once and calculate averages
I think you have seen something like this in office or in internet:
Q: “How long it took to run?”
A: “On my machine it takes about 1.5 seconds to run”
Q: “1.5 seconds… Is it average or just one run?”
A: “Just hit F5 and read the result, that’s it”
Well, my question is – what other processes were running on same machine that possibly may affect the results? Okay, there are a lot of processes. Instead of using one measuring round we need to make more runs and take their average. Here is the example.
static void MeasureCalculation()
{
var inputList = GetInputData();
RandomizeInputs(inputList);
var watch = new Stopwatch();
var cycles = Math.Pow(10, 6);
var times = 0D;
for (var i = 0; i < cycles; i++)
{
watch.Reset();
watch.Start();
// Call calculation
watch.Stop();
times += watch.ElapsedMilliseconds;
}
inputList.Clear();
Console.WriteLine("Time: " + (times / cycles));
}
This code measures quick calculation and we have no problem to make million measuring cycles. When cycles are done we calculate average time that one calculation took. If you want to analyze or process measuring results you can save them after watch is stopped. This way we don’t affect the code to be measured.
Conclusion
Measuring the speed of code is not complex task. There are common mistakes that I listed here and if you avoid these mistakes you get way better and accurate measuring results. It is also possible to use testing frameworks but I focused here on quick and simple measuring method so you get your code measured right there where it is. You don’t have to install any additional tools or spend your time setting up more complex measuring environment. As you saw – measuring the speed of code is not rocket science.
Some ASP.NET web applications use web services to get some data that they display to users. Some pages may lay hardly on web services and these pages need some optimization to work better. In this posting I will show you how to use web services behind your ASP.NET page asynchronously and perform a lot of queries to web services. Sample code included!
Asynchronous page processing
As an introduction let’s see how ASP.NET supports asynchronous calls behind pages. Although we can use our own threading mechanism and avoid what is offered by ASP.NET it is more safe and time saving to use ASP.NET own infrastructure.
The image on right that I stole from MSDN article Asynchronous Pages in ASP.NET 2.0 shows you how synchronous and asynchronous pages are processed by ASP.NET. I recommend to read this MSDN article because you can find also other methods for asynchronous calls. I will introduce you how to use PageAsyncTask based calls.
If you look at diagram on right you can see that asynchronous processing of registered tasks happens between PreRender and PreRenderComplete events in separate threads.
If we have data bound controls on our page we have to move data binding to PreRenderComplete event because this is the closest phase of page processing where asynchronous calls are finished. For PreRenderComplete event all asynchronous tasks are done.
Getting started
Now let’s go through simple page that performs asynchronous call to web service. We are using simple web service with one method that is able to wait for given amount of seconds before sending out response. Right now we don’t need waiting functionality. I will show you later how to make more than one call and then we will need delay parameter. Our web service method is here.
[WebMethod]
public string HelloWorld(int delay)
{
if(delay > 0)
Thread.Sleep(delay);
return "Hello World, delay: " + delay;
}
Our hello world method returns just hello world sting and time delay given to method.
NB! Before going to code you have to make one modification to your page. Open it in mark-up code view and add Async="true" to Page directive. Without this parameter you get the exception!
Now let’s add some code behind our asynchronous page. Some explanations too. We will define page level variable called _service that keeps the instance of web service client class. When page initializes we will create new PageAsyncyTask and provide it with references to BeginRequest and EndRequest operations. First one of them initializes new asynchronous call to web service method and the second one is called when asynchronous call is finished.
private readonly DelayedHelloSoapClient _service =
new DelayedHelloSoapClient();
private Random _random = new Random();
protected override void OnInit(EventArgs e)
{
base.OnInit(e);
var task = new PageAsyncTask(BeginRequest, EndRequest,
null, null);
RegisterAsyncTask(task);
}
IAsyncResult BeginRequest(Object sender, EventArgs e,
AsyncCallback cb, object state)
{
var param = _random.Next(1, 5) * 1000;
return _service.BeginHelloWorld(param, cb, null);
}
void EndRequest(IAsyncResult asyncResult)
{
var answer = _service.EndHelloWorld(asyncResult);
Debug.WriteLine(answer);
}
Delay parameter for hello world method is generated by using page wide randomizer. Again, we don’t need this randomizer to be at page scope right now but we will need it in next section. If you need to call only one web service during page processing then you can use this code and make refactorings you need.
Calling multiple web service during page processing
Now let’s take more complex scenario and let’s suppose we need more than one web service call to be made. Lazy as I am I am using same web service for all calls. If you have different services then each one of them requires separate task with separate BeginRequest and EndRequest methods. Now let’s use this code behind our page.
private readonly Random _random = new Random();
protected override void OnInit(EventArgs e)
{
base.OnInit(e);
for (var i = 0; i < 100; i++)
{
var task = new PageAsyncTask(BeginRequest, EndRequest,
null, null, true);
RegisterAsyncTask(task);
}
}
IAsyncResult BeginRequest(Object sender, EventArgs e,
AsyncCallback cb, object state)
{
var service = new DelayedHelloSoapClient();
var delay = _random.Next(1, 5)*1000;
var hash = service.GetHashCode();
Debug.WriteLine("Started " + hash + ", delay: " + delay);
return service.BeginHelloWorld(delay, cb, service);
}
void EndRequest(IAsyncResult asyncResult)
{
var service = (DelayedHelloSoapClient)asyncResult.AsyncState;
var hash = service.GetHashCode();
var answer = service.EndHelloWorld(asyncResult);
Debug.WriteLine("Finished " + hash + ", " + answer);
}
In OnInit we registered 100 calls to web services. Yes, 100 calls… I know it is not normal but we need a lot of instances to be test the results better. BeginRequest method is now tricky. We create new instance of service each time and provide it to BeginHelloWorld() as state object. When EndResult is fired we read the service for given call out from state parameter and call EndHelloWorld() method to get data back.
Run application and see how fast it runs. On my pretty old and heavily loaded laptop this code takes about 30 seconds to run. When I remove delay then all these 100 calls are made during 1.1 seconds.
Storing and displaying results
Now let’s see how we can store and display results returned by web services. Here is the full source of my page class. Notice that I addded Stopwatch to measure the time that web services take to run. I also added DataTable called _answers to page scope where web service instances write answers they got.
public partial class _Default : Page
private readonly DataTable _answers = new DataTable();
private readonly Stopwatch _watch = new Stopwatch();
private readonly Random _random = new Random();
protected override void OnInit(EventArgs e)
for (var i = 0; i < 100; i++)
// last parameter (true) means that we want
// parallel execution of task
var task = new PageAsyncTask(BeginRequest, EndRequest,
null, null, true);
protected override void OnLoad(EventArgs e)
_answers.Columns.Add("InstanceId", typeof (int));
_answers.Columns.Add("Answer", typeof (string));
IAsyncResult BeginRequest(Object sender, EventArgs e,
AsyncCallback cb, object state)
var service = new DelayedHelloSoapClient();
var delay = _random.Next(1, 5) * 1000;
var hash = service.GetHashCode();
Debug.WriteLine("Started " + hash + ", delay: " + delay);
return service.BeginHelloWorld(delay, cb, service);
void EndRequest(IAsyncResult asyncResult)
var service =
(DelayedHelloSoapClient)asyncResult.AsyncState;
var hash = service.GetHashCode();
var answer = service.EndHelloWorld(asyncResult);
Debug.WriteLine("Finished " + hash + ", " + answer);
var row = _answers.NewRow();
row["InstanceId"] = hash;
protected override void OnPreRenderComplete(EventArgs e)
base.OnPreRenderComplete(e);
Debug.WriteLine("Time: " + _watch.Elapsed);
answersRepeater.DataSource = _answers;
answersRepeater.DataBind();
Take a careful look at EndRequest method. Because we are using threads we have to lock _answers table so it is not used by multiple threads at same time. I locked _answers for as short time as possible. Still I got 1 second additional time due to this lock. Also notice that we bind data to Repeater in OnPreRenderComplete method. If you do it in PreRender method then threads are not run yet and _answers table is not filled with results.
Conclusion
Using threads behind ASP.NET pages is not very complex task if you use built-in mechanism that takes care of running threaded operations. We used PageAsyncTask to register new asynchronous tasks and we gave service instance as state object to asynchronous calls so we were able to get correct service channel later when we finished asynchronous call and read data that web service method returned. We also saw that using ASP.NET built-in support for asynchronous pages it was very easy to store the results and display them in data bound controls. All we had to do was to move our data related functionality to PreRenderComplete event of page.
I am auditing one big legacy application. I use Visual Studio 2010 modeling projects to visualize the design of this legacy application. When I wanted to get my UML diagrams to image files so I can insert them to documents I found no option for it. As it turned out we have to write extension for modeling projects and deploy it as Visual Studio 2010 extension. Here is the installer and source package of my UML.SaveAsImage extension.
I still cannot believe that we need to write extensions to get our UML diagrams to image files but okay, it’s done now and let’s say – problem is solved. At least this one. In my work I used code example by Cameron Skinner. He has very good blog posting titled as Save a diagram to Image File. Here are steps how I wrote this extension.
1. Create new Layer Designer Command Extension project
As a first thing you should create layer designer command extension project. If you don’t have support for modeling projects you should take more powerful edition of Visual Studio 2010.
Name it your project as you like and click OK.
2. Add functionality to Command class
You can find Command class from your new project. This class is created by default. Here is the modified code from Cameron Skinner’s blog I referred before.
using System.ComponentModel.Composition;
using System.Drawing.Imaging;
using System.Linq;
using System.Windows.Forms;
using Microsoft.VisualStudio.ArchitectureTools.Extensibility.Presentation;
using Microsoft.VisualStudio.ArchitectureTools.Extensibility.Uml;
using Microsoft.VisualStudio.Modeling.Diagrams;
using Microsoft.VisualStudio.Modeling.ExtensionEnablement;
namespace UML.SaveAsImage
{
[Export(typeof(ICommandExtension))]
[ClassDesignerExtension]
[UseCaseDesignerExtension]
[SequenceDesignerExtension]
[ComponentDesignerExtension]
[ActivityDesignerExtension]
public class SaveAsImage2Extension : ICommandExtension
{
[Import(typeof(IDiagramContext))]
public IDiagramContext DiagramContext { get; set; }
public string Text
{
get
{
return "Save as image";
}
}
public void Execute(IMenuCommand command)
{
var dslDiagram = DiagramContext.CurrentDiagram.GetObject<Diagram>();
if (dslDiagram != null)
{
var dialog = new SaveFileDialog
{
AddExtension = true,
DefaultExt = "image.bmp",
Filter = "Bitmap ( *.bmp )|*.bmp|" +
"JPEG File ( *.jpg )|*.jpg|" +
"Enhanced Metafile (*.emf )|*.emf|" +
"Portable Network Graphic ( *.png )|*.png",
FilterIndex = 1,
Title = "Save Diagram to Image"
};
if (dialog.ShowDialog() == DialogResult.OK &&
!string.IsNullOrEmpty(dialog.FileName))
{
var bitmap = dslDiagram.CreateBitmap(dslDiagram.NestedChildShapes,
Diagram.CreateBitmapPreference.FavorClarityOverSmallSize);
bitmap.Save(dialog.FileName, GetImageType(dialog.FilterIndex));
}
}
}
public void QueryStatus(IMenuCommand command)
{
if (DiagramContext.CurrentDiagram != null &&
DiagramContext.CurrentDiagram.ChildShapes.Count() > 0)
{
command.Enabled = true;
}
else
{
command.Enabled = false;
}
}
private static ImageFormat GetImageType(int filterIndex)
{
var result = ImageFormat.Bmp;
switch (filterIndex)
{
case 2:
result = ImageFormat.Jpeg;
break;
case 3:
result = ImageFormat.Emf;
break;
case 4:
result = ImageFormat.Png;
break;
}
return result;
}
}
}
Execute() method of this class is activated when user right-clicks on design area of some UML diagram and selects “Save image as …” from context menu.
3. Modify VSIX package definition
Before we can publish or deploy our extension we have to make sure it installs correctly. Double click on source.extension.vsixmanifest file to open it in VSIX definition editor.
NB! You must fill Author field to get extension installed correctly! Otherwise your extension may be installed incorrectly.
When author name is inserted save manifest settings and compile your project. You can find VSIX installer from Debug or Release folder. Now you can install your extension.
Conclusion
Visual Studio 2010 modeling projects have many things I miss but is is good to know I can write my own extensions to modeling projects. The only bad point I found was problem with VSIX installer. It was created without any notices when Author field was empty and it was also installed but incorrectly. After creating package where Author field was filled everything started work.
In my last posting I introduced my idea about common membership provider for ASP.NET that is able to support multiple authentication providers. Before writing membership provider we need support for some authentication providers to get an better idea how to use them together. In this posting I will introduce you how to use OAuth protocol with Twitter.
Before we start …
… let me make some important notes and give some advises so you know better what is going on:
- OAuth is secure open-source authentication API that is used by Twitter and many other providers. You can find out more details about OAuth from their homepage at http://oauth.net/. There is also official specification for OAuth by Internet Engineering Task Force (IETF): The OAuth 1.0 Protocol.
- To make our sad lives easier there is library for .NET that makes it for us easier to support OAuth and OpenID protocols in our systems. This cool library is called DotNetOpenAuth. This library is also used by stackoverflow and MySpace. We will use the same library, so go and download it before starting with code.
- You need to register Twitter application. Go to page Twitter’s application registration page and register new application. You need consumer key and consumer secret parameters of your application later.
- Copy “Sign in with Twitter” to some of your web application’s folders where other images are. You need this button on logon page later.
- This is pretty raw stuff here and how to finish it so it works fine for you is up to you this time.
- I am sorry for format of code but I found no better way to get it more readable here. When my new blog is finished somewhere in future then there will be way better layout that is designed with .NET code in mind. This far, please survive my current blog! :)
I suppose you can Twitter application registered by yourself and also it should not be very hard to download DotNetOpenAuth and add it to your ASP.NET MVC application.
Token manager
Before we can connect to Twitter we need token manager that handles security tokens. Right now we will use InMemoryTokenManager that comes with Twitter application block for Enterprise Library (you can find it from samples folder of your DotNetOpenAuth download). The class is here, you can take it using copy and paste. I removed code documentation from this class because I plan to replace it later by my own token manager.
internal class InMemoryTokenManager
: IConsumerTokenManager, IOpenIdOAuthTokenManager
{
private Dictionary<string, string> tokensAndSecrets =
new Dictionary<string, string>();
public InMemoryTokenManager(string consumerKey, string consumerSecret)
{
if (String.IsNullOrEmpty(consumerKey)) {
throw new ArgumentNullException("consumerKey");
}
this.ConsumerKey = consumerKey;
this.ConsumerSecret = consumerSecret;
}
public string ConsumerKey { get; private set; }
public string ConsumerSecret { get; private set; }
public string GetTokenSecret(string token) {
return this.tokensAndSecrets[token];
}
public void StoreNewRequestToken(UnauthorizedTokenRequest request,
ITokenSecretContainingMessage response)
{
this.tokensAndSecrets[response.Token] = response.TokenSecret;
}
public void ExpireRequestTokenAndStoreNewAccessToken(string consumerKey,
string requestToken, string accessToken, string accessTokenSecret)
{
this.tokensAndSecrets.Remove(requestToken);
this.tokensAndSecrets[accessToken] = accessTokenSecret;
}
public TokenType GetTokenType(string token)
{
throw new NotImplementedException();
}
public void StoreOpenIdAuthorizedRequestToken(string consumerKey,
AuthorizationApprovedResponse authorization)
{
this.tokensAndSecrets[authorization.RequestToken] = String.Empty;
}
}
This class may seem a little bit complex at first place but when you have Twitter connected and you play with it a little bit then you get the idea. There is really nothing complex and implementing you own token manager should be easy.
Twitter client
Now let’s write Twitter client. This is class that does all the dirty work for us. It is better for us to have this class instead of pushing all this functionality to our logon page controller methods. For one authentication provider it doesn’t seem like very bad sin but if we have three or five different authentication providers supported then our logon page controller will look like war field. So, let’s fight against chaos right from start.
public class TwitterClient
{
private string UserName { get; set; }
private static readonly ServiceProviderDescription ServiceDescription =
new ServiceProviderDescription
{
RequestTokenEndpoint = new MessageReceivingEndpoint(
"http://twitter.com/oauth/request_token",
HttpDeliveryMethods.GetRequest |
HttpDeliveryMethods.AuthorizationHeaderRequest),
UserAuthorizationEndpoint = new MessageReceivingEndpoint(
"http://twitter.com/oauth/authorize",
HttpDeliveryMethods.GetRequest |
HttpDeliveryMethods.AuthorizationHeaderRequest),
AccessTokenEndpoint = new MessageReceivingEndpoint(
"http://twitter.com/oauth/access_token",
HttpDeliveryMethods.GetRequest |
HttpDeliveryMethods.AuthorizationHeaderRequest),
TamperProtectionElements = new ITamperProtectionChannelBindingElement[]
{ new HmacSha1SigningBindingElement() },
};
IConsumerTokenManager _tokenManager;
public TwitterClient(IConsumerTokenManager tokenManager)
{
_tokenManager = tokenManager;
}
public void StartAuthentication()
{
var request = HttpContext.Current.Request;
using (var twitter = new WebConsumer(ServiceDescription, _tokenManager))
{
var callBackUrl = new Uri(request.Url.Scheme + "://" +
request.Url.Authority + "/Account/TwitterCallback");
twitter.Channel.Send(
twitter.PrepareRequestUserAuthorization(callBackUrl, null, null)
);
}
}
public bool FinishAuthentication()
{
using (var twitter = new WebConsumer(ServiceDescription, _tokenManager))
{
var accessTokenResponse = twitter.ProcessUserAuthorization();
if (accessTokenResponse != null)
{
UserName = accessTokenResponse.ExtraData["screen_name"];
return true;
}
}
return false;
}
}
Our TwitterClient has two methods. StartAuthentication() is called when user clicks on Twitter button. FinishAuthentication() is called in Twitter authentication callback page to finalize authentication procedure. As a next thing I will show you how to mix all together.
Logon page
Okay, I expect you have default ASP.NET MVC application with AccountController and no markable modifications there. Let’s go now through changes using minimal steps. As a first let’s open LogOn.aspx view and add the following markup there.
<a href="<%= Url.Content("~/Account/StartTwitterAuth/") %>">
<img src="<%= Url.Content("~/images/sign-in-with-twitter-darker.png") %>" />
</a>
Now let’s open AccountController and add the following lines of code to the beginning of controller class.
static private readonly InMemoryTokenManager TokenManager =
new InMemoryTokenManager("<consumerKey>", "<consumerSecret>");
Replace <consumerKey> and <consumerSecret> with values you got from your Twitter application page. I know it is not nice solution but it works for me now and of course I will let you know how to solve it better as soon as I work out better solution.
Now add two new controller methods to AccountController.
public ActionResult StartTwitterAuth()
{
var client = new TwitterClient(TokenManager);
client.StartAuthentication();
return null;
}
public ActionResult TwitterCallback()
{
var client = new TwitterClient(TokenManager);
if (client.FinishAuthentication())
{
return new RedirectResult("/");
}
// show error
return View("LogOn");
}
Now we have Twitter support added to our ASP.NET MVC application.
Testing Twitter authentication
After compiling your application put breakpoint to TwitterCallback method on the line where redirect is made. Now run your application and log in using Twitter. When application starts go to log on page and click on Twitter button. After successfully logging in the breakpoint will be hit. Check out the value in UserName property of variable called client. You sould see there you Twitter username.
Conclusion
Getting Twitter authentication work with ASP.NET MVC using DotNetOpenAuth was pretty simple thing to do. There was also no complex issues from Twitter side and everything started to work smoothly. Besides Twitter there are many other sites that provide authentication options to external applications using OAuth protocol. With little modifications to code provided here you can use some other provider instead of Twitter or add additional OAuth providers to your ASP.NET MVC application.
I am trying to implement form based authentication (FBA) membership provider that is able to support multiple authentication mechanisms. Take it as an self-training experiment. In this posting I will introduce bases of my experiment and introduce my current plans and ideas.
Why membership provider for multiple authentication mechanisms?
There are many people in the world who are users of some social network or some open services provider. Today Facebook and Twitter are very popular social networks. Also Google and Microsoft Live services are very popular although they are not social networks. But all mentioned systems provide their own authentication channels also to other applications. The question is: why should I have another username and password if I have already one at some of systems I mentioned before?
Here, in Estonia, we are building local Microsoft community portal with local Microsoft. We analyzed the situation and decided to use Live ID as authentication mechanism in our portal. We were a little it worried too because we were afraid that there are load of users who are not happy with Live ID as authentication mechanism. And guess what – as time has shown us there is no reason to worry! Our ~2000 users (Estonia is small country) are all happy with Live ID and the ones who doesn’t have Live ID yet can go and join Live ID – it’s free.
If you have no idea where I live then take a look at map on right. The small and strong screen splash there is Estonia. If you don’t see it well then feel free to click on this image to see it at original size. :)
Profile store
To give you better idea what I am thinking about take a look at the following diagram.
Every user has one account. Each account has account specific information and custom profile fields. Profile fields can be customized because different sites need different profiles. My profile store will be common enough to live over almost all common situations. Or at least I hope so.
Authentication tokens is the place where I store tokens for users returned by different authentication mechanisms. This way each user is able to use authentication mechanism he or she likes. Also users will be able to use Twitter and Live ID both if they like. Same goes for all the other authentication providers I am able to support.
Implementation details
I have some implementation details on my mind. I am sure that the end result is maybe a little bit different but in big picture things hopefully does not change a lot.
- There will be one common membership provider that is able to authenticate users to ASP.NET site by querying the authentication tokens table.
- It is possible that each authentication provider gets it’s own membership provider that is extended from common one. I am not very sure yet if this is the great idea.
- Usernames will be in form: ProviderName:ProfileName so I am able to detect which authentication mechanism I have to look to find out correct profile.
- There will be additional field for profile or membership provider that returns me ProfileName without authentication mechanism information.
- Data will be held in SQL Server at first place. Later I plan to support also the other databases or data stores.
- There will be common login page that is automatically able to offer authentication mechanisms it knows about.
I think these points are good enough for start.
What’s next?
Well…. as a next thing I try to implement support for different authentication mechanisms and when all common providers are covered then it is time to write generic membership and role providers for ASP.NET. My first four candidates are Twitter, Facebook, Google and Live ID and after these I plan to write providers. Stay tuned, it will be interesting experiment!
I am looking at features of different versions of Visual Studio and I less and less understand how and why is the quality of software handled as fun of few seasoned professionals. Although Visual Studio is best development platform I have ever seen I really don’t understand why Microsoft hates people who want to provide quality to their customers.
Usually n00bs and start-ups start with minimum versions to get their work done. Many of them start with some version of Visual Studio Express or cheapest one of the full versions. These products have this far caused me hell load of problems because specially n00bs have no way how to control or measure the quality of their work. n00bs make also a lot of stupid mistakes and they need some way to rollback those mistakes. And don’t forget how many other mistakes n00bs can make. Now let’s put it all together and see how hard is for n00bs to get started their professional life on Visual Studio.
- No advanced debugging – there is IntelliTrace but not for guys who need it at first years almost every day. They have to waste their and their seniors time to get mysteries solved. I need IntelliTrace rarely because usually stack trace of Exception is enough for me. But I have seen n00bs who try to debug their applications using only message part of exceptions. IntelliTrace is only available with Visual Studio Ultimate.
- No versioning support – SVN, Git and other version controlling systems are must-be in today’s software development. I cannot see this in Express editions of Visual Studio. Guys come from universities and they have no idea how to save their work and how to rollback the changes they don’t really need. Well… if we take Eclipse then it is only matter of minutes to get connected to some version controlling system.
- No support for database projects – you must be very professional guy if you want to be able to develop your databases as integral part of your project. At least so good that seniors agree to buy you at least Premium license of Visual Studio 2010. On all other versions of Visual Studio 2010 you have to manage your database versions manually. Okay, I have seen what it means specially for younger d00ds who come from schools.
- No testing support – another must-be thing is integral support for some testing framework. This far I have seen no testing support in Visual Studio Express editions and I still wonder why n00bs on MS platform are forced to produce crap – forced by money and sick marketing politics. If I take Eclipse… well, you can start with tests after couple of minutes. And using some hacks I can make nUnit work with Express Editions too, but is it normal?
- No code metrics – code metrics is also there to help only guys like me who are coding dinosaurs and know without Visual Studio what is the current state of code. N00bs are usually not able to write code like pros and for them measuring code quality daily is mandatory. Code metrics are available starting from Premium Edition of Visual Studio. Well… those who are starting must be wiser than pros to get their job done. Seems like not very normal situation to me. Okay, I solved the problem with NDepend and I am happy with it.
- No support for profiling – well… same stuff as previous. It is something that also beginners need to do and still they have to find other ways how to do it. Like using some external frameworks, struggling with them and wasting time instead of getting wiser and smarter on mainstream tools.
I don’t think that all these features must me 100% available for “smaller” versions but at least some level these features must be there. It is awful waste of time right now to get n00bs work at Visual Studio if you don’t want to buy a lot of money and still you need external tools to accomplish some team tasks (I have very good CI story to tell that ends with simple question – why TFS when I have working solution for free that takes less time to set up and manage then TFS?).
I think that instead of building another tools like WebMatrix Microsoft should put their focus heavily on quality topics for couple of years and work out all editions of Visual Studio (and other development tools) with software quality as most important goal in their mind.
Lately I finished reading the famous patterns book by Martin Fowler -
Patterns of Enterprise Application Architecture. This book introduces and describes design patterns you will usually see when building enterprise applications. There are also examples for patterns and although they are simple and primitive they give you the idea how to implement or use specific patterns. If you are working on enterprise applications every day you will find this book very useful and I’m sure you will use it as manual. Yes, you can read this book as any other book and you can also use it as manual. In the end of this review you can find table of patterns with links to short introduction for each pattern.
Are these patterns platform specific somehow? No, they are not. It doesn’t matter what platform or language you are using. Of course, you must know your platform well because some of problems solved in this book maybe also already solved on platform you are working on. Don’t try to push all available patterns to your application – use patterns of this book when you face problems that your tools doesn’t solve or that your tools solve poorly. Of course, you can consult with this book if you are not sure how to implement one or another pattern in your application.
My suggestion is to read this book with two other books (of course, there are more recommended books but let’s make a quick start, okay?):
As the result you have better idea of patterns, domain driven design and how to use them in your .NET projects. Enterprise application patterns and domain driven design are both somehow complex topics and Applying Domain-Driven Design and Patterns gives you some ideas about how to apply this knowledge in practice.
Table of contents
Introduction.
Architecture.
Enterprise Applications.
Kinds of Enterprise Application.
Thinking About Performance.
Patterns.
The Structure of the Patterns.
Limitations of These Patterns.
I. THE NARRATIVES.
1. Layering.
2. Organizing Domain Logic.
3. Mapping to Relational Databases.
4. Web Presentation.
5. Concurrency (by Martin Fowler and David Rice).
6. Session State.
7. Distribution Strategies.
8. Putting it all Together.
II. THE PATTERNS.
9. Domain Logic Patterns.
10. Data Source Architectural Patterns.
11. Object-Relational Behavioral Patterns.
12. Object-Relational Structural Patterns.
13. Object-Relational Metadata Mapping Patterns.
14. Web Presentation Patterns.
15. Distribution Patterns.
16. Offline Concurrency Patterns.
17. Session State Patterns.
18. Base Patterns.
References
Index.
Table of Patterns
Martin Fowler introduces all these patterns also on his homepage. There is special patterns catalogue for that. Patterns here are linked to Martin Fowler’s catalogue.
Today I worked out simple application side object locking solution for one server scenario. My motivation came from ASP.NET architecture forum thread How to solve concurrent site issue where one user asks for locking solution that works without changing database. Here is my simple and primitive solution that should help this guy out. I added also sample solution to this posting.
My solution is simple:
- Create class for locked items.
- Create manager class that holds locks and manages them.
- Clear locks when session ends.
- Create page to release all locks.
If you need something more serious then you should keep locks information in database or better than that – use some lock server. Also you may consider developing WCF service.
LockItem class
As a first thing let’s create class that keeps lock information. I call this class as LockItem. This class doesn’t hold references to locked objects – only type and ID as there are many business layers that doesn’t use globally unique identifiers for objects.
public class LockItem
{
public Type ObjectType { get; set; }
public int ObjectId { get; set; }
public string SessionId { get; set; }
public DateTime LockedAt { get; set; }
}
LockItem class also contains property for session because otherwise it is not possible to know what objects session should release. LockedAt property is used to release expired locks.
LockManager
LockManager is the most complex part of this example. It handles all the locking actions and keeps internal locks registry that contains information about locks. Here is the source for LockManager.
public static class LockManager
{
private static List<LockItem> _lockedItems;
static LockManager()
{
_lockedItems = new List<LockItem>();
}
public static bool IsLocked(BusinessBase obj)
{
var locksQuery = from l in _lockedItems
where l.ObjectType == obj.GetType()
&& l.ObjectId == obj.Id
select l;
return (locksQuery.Count() > 0);
}
public static bool Lock(BusinessBase obj, string sessionId)
{
if (IsLocked(obj))
return false;
lock (_lockedItems)
{
var lockItem = new LockItem
{
ObjectType = obj.GetType(),
ObjectId = obj.Id,
LockedAt = DateTime.Now,
SessionId = sessionId
};
_lockedItems.Add(lockItem);
return true;
}
}
public static void ReleaseLock(BusinessBase obj, string sessionId)
{
lock (_lockedItems)
{
var locksQuery = from l in _lockedItems
where obj.GetType() == obj.GetType()
&& obj.Id == l.ObjectId
&& sessionId == l.SessionId
select l;
ReleaseLocks(_lockedItems);
}
}
public static void ReleaseSessionLocks(string sessionId)
{
lock (_lockedItems)
{
var locksQuery = from l in _lockedItems
where sessionId == l.SessionId
select l;
ReleaseLocks(locksQuery);
}
}
public static void ReleaseExpiredLocks()
{
lock (_lockedItems)
{
var locksQuery = from l in _lockedItems
where (DateTime.Now - l.LockedAt)
.TotalMinutes > 20
select l;
ReleaseLocks(locksQuery);
}
}
public static void ReleaseAllLocks()
{
lock (_lockedItems)
{
_lockedItems.Clear();
}
}
private static void ReleaseLocks(IEnumerable<LockItem> lockItems)
{
if (lockItems.Count() == 0)
return;
foreach (var lockItem in lockItems.ToList())
_lockedItems.Remove(lockItem);
}
}
You can write special page to administrator interface that has button for calling ReleaseAllLocks(). It is good idea because at least during development I am sure you are creating many situations where locks are not released due to exceptions.
Release session locks when session ends
When session ends you have to release all locks that this session has. Otherwise those objects will be locked until you restart your web application or call ReleaseLocks method. Although ASP.NET Session_End event is called god knows when and you should not use this method due to this uncertainty I base my example exactly on that event. Copy this method to your Global.asax.ascx file.
protected void Session_End()
{
LockManager.ReleaseSessionLocks(Session.SessionID);
}
Now we have almost everything in place to start using locks.
Testing locks
I created simple view to test locks. You need two browsers to test if locks work okay. One browser locks the object for 20 seconds and the other gives you messages that object is locked. You have to make this second request with another browser during this 20 seconds when object is locked. Here is the ASP.NET MVC controller action for that purpose.
public ActionResult Index()
{
Response.Clear();
Response.ContentType = "text/plain";
var product = new Product { Id = 1, Name = "Heineken",
Price = 1.2M };
if(!LockManager.Lock(product, Session.SessionID))
{
Response.Write("Object has already lock on it!\r\n");
Response.End();
return null;
}
Response.Write("Object successfully locked\r\n");
Response.Flush();
Thread.Sleep(20000);
Response.Write("Releasing lock\r\n");
LockManager.ReleaseLock(product, Session.SessionID);
Response.Write("Lock released\r\n");
Response.End();
return null;
}
If you are using old ASP.NET application then paste this code to Page_Load event of your application.
From here …
… you can go on and make your own modifications and customizations to this code. Before going to live with your own locking strategy test it hundred times at least. To make administration easier you can also create page that shows active locks and lets them release one lock at time or by session ID.
Sample solution
Today I tried out ASP.NET MVC HTML5 Helpers Toolkit by Dean Hume. Although browsers today support HTML5 partially it is time to start playing with it. As soon as HTML5 gets common standard in web you should be able to react fast. In this posting I will introduce you HTML5 Helpers Toolkit and some HTML5 field types.
HTML5 Helpers Toolkit is easy to install. Just download the package, unpack it and reference the DLL you found from package. That’s all. You don’t have to do anything more. If your browser doesn’t support HTML5 then HTML5 fields are rendered as usual textboxes and you can extend them using jQuery by example.
E-mail field
HTML5 defines many types for input boxes. One of the types is e-mail. Here is the demo of e-mail field with placeholder value. Yes, in HTML5 textboxes support placeholders and we don’t have to mess with JavaScript anymore. This is how e-mail box is defined.
<form>
<%= Html.Html5TextBox("userEmal", InputTypes.InputType.Email,
"plah@plah.com") %>
</form>
And here is the result on Firefox 4 Beta.
Nice, isn’t it? I discovered one cool thing when I tested HTML5 Helpers Toolkit with Opera. I tried to submit form where e-mail address is not valid. And this is what happened:
Form was not submitted and all I got was this error message in red box. Cool!
Slider
As a next thing let’s see how to get numeric slider on page without any JavaScript. Numbers in slider box definition have the following meaning: minimum is 1 and maximum is 50, step is 2 and default value is 25.
<form>
<%= Html.Html5Range(1, 50, 2, 25, null) %>
</form>
And here is the result on Chrome.
If we want we can use usual HTML to decorate slider with numbers mentioned above. But I like the result – simple, calm and clean.
Date picker
Now let’s do something that only Opera seems to support right now – let’s create HTML5 date picker. Definition is here.
<form>
<%= Html.Html5TextBox("deliveryDate", InputTypes.InputType.Date) %>
</form>
And here is the result on Opera.
Seems good to me. And again – no additional JavaScript is needed. Until other browsers also start supporting this kind on input box you can use jQueryUI Datepicker component.
Conclusion
Although HTML5 is not here yet it is coming very soon and seems like ASP.NET MVC will be soon ready for it. Although HTML5 Helpers Toolkit doesn’t provide all HTML5 controls yet it is still great start and I think this project is worth keeping eye on it. It is very easy to install and very easy to use and I suggest you to try it out as soon as possible.
More Posts
Next page »