Zack Owens : ClubStarterKit

ClubStarterKit – Caching for performance

zowens — Fri, 11 Dec 2009 15:43:56 GMT

First of all, if you haven’t heard, I recently released ClubStarterKit v3 Preview. If you haven’t had a chance to look at it, I highly encourage you to take a look at the whole new codebase.

My whole goal when building any application whether it’s an application for a client or the open source ClubStarterKit project is to make the app as fast as possible. There are quite a few layers to the caching mechanisms in CSK that I think could be applied to many other projects. In fact, some of the caching ideas came from an internal web framework we use at eagleenvision.net.

There are two “caches” that CSK uses. One is the HTTP Cache that IIS and other web servers use in the background. The other is the client’s web cache. I will give a brief overview of each in this post.

Web Cache

The web cache is the simplest to understand of the two types of caches. When a request comes down from the web application, a response key is set on the server side that informs the browser to not look up the same file until a certain date. The trick we can do is to set the date as far as possible so the cache NEVER expires. (In CSK I think the cache is something like 5 years…). But what happens when you change your website? That’s where the application ID comes into play.

At every application startup, an identifier is pushed in storage (we use HTTP Application) of a string-based token. In CSK the token is the DateTime of the insertion into the cache so that there are no other collisions. This token is appended onto every CSS file request, image request, and Javascript file request so that the client doesn’t have to wait every time a page is loaded for the same CSS, image and Javascript files to load into the browser when running a particular “application instance”. If, for some reason, the application ID is not passed to the request, then the response isn’t cached.

When a part of your application changes, you have the ability to reset the application ID yourself. In CSK you just navigate to /sitecontent/reset and the application ID should be reset.

The obvious advantage to this strategy is the reduction in unnecessary bandwidth. And users don’t have to wait for something to download that they already have on their computer. So there are some real benefits to using client-side caching of static files.

HTTP Cache

Just like the web cache, the HTTP cache reduces unnecessary bandwidth. The center of the HTTP cache, however, is around the database. It can sometimes be costly to hit a database for the same query. To counteract this we use HTTP caches. These things store data onto the application server and store them for a certain amount of time until it has expired or is expired. A value can be expired by the application server, when the specified TimeSpan is reached or when the item is removed from the cache by the web application. In the CSK, an item is expired from the cache when something is added, such as a new article or a forum post. Once the item is pulled from the cache, the next request forces the cache to go to the database for the query result and store it in the cache.

In the CSK we are also refreshing caches for every application id. This just ensures that there isn’t a leak in the caching mechanism and the data can be easily refreshed by a website owner.

The rationale for this feature is that hitting the database is a lot more costly when you’re dealing with load. Memory cache is really cheap comparatively. So it just makes sense to “hold” the query results until they are expired by either the application’s usage or the refresh of the application ID. In the worst case, ASP.NET removes the item from the cache because of lack of storage. In that case the query is regenerated anyways.

In the CSK there are a few abstractions that I will detail in later posts that are particularly useful when dealing with data. These are the CollectionDataCache, used for storing a collection from the DB, PagedDataCache, used for storing a paged list from the DB, and the SingleItemDataCache, used for storing single items from the DB. All these abstractions are sortable, constrainable, and easy to use. They all use the HTTP cache as the backing store. The abstractions also take care of the data access using the UnitOfWork and Repository patterns we employ in CSK.

There is also an HttpSession cache, which is particularly useful when storing user data. It operates off the same cache interface as the HttpCacheBase.

The Changes Ahead

Like I said, I took a lot of code from the internal framework I built a few years ago. The code I put into the CSK was really useful in that framework. But in the application of CSK, there is a sense of “code smell”. I would really like to do a few things differently, namely abstracting caches even further for CollectionDataCache, PagedDataCache and SingleItemDataCache. So maybe you might want to store a paged list into the session state. Currently you can’t do that without writing your own cache, which isn’t bad… it’s just not particularly fun to do. I would like to add, what I am calling, “cache strategies” to the infrastructure. So expect changes down the line for further abstraction.

So you’ve seen that caches are particularly useful when dealing with data, whether it’s file data or database data. As always, questions, comments, and other feedback are GREATLY appreciated. Just send me an email, comment on this blog, or post on the ClubStarterKit forums.

ClubStartKit is Reborn… With a new release

zowens — Wed, 09 Dec 2009 06:07:00 GMT

The title says it all. That’s right, I’m back on ClubStarterKit. I’ve tried to assure the CSK community in past blog posts that it isn’t dead. In the last year, I’ve done a lot of thinking and outside programming. I’m FINALLY in a position where I have really progressed in terms of project management skill and web design. Many of the opinions I held when I started the project have simply changed. I’ll detail some of those in a minute.

Here’s a link to the new release.

Open source software is really tricky, especially for someone who has to go to school and keep up with the day job. It’s simply TOO MUCH for someone else to demand so much on a person. However, demanding open source work onto yourself is highly rewarding. That’s why I’m back.

Why the time off?

Well, I simply had to build my business. Ultimately, that takes a lot more priority over any other open source work. As harsh as that is, I simply had too much going on. Also, I was in a bit of a rut with CSK.

The project started from a need by the community to extend the club starter kit originally started by Microsoft that shipped in the box as a starter kit with Visual Studio 2005. I found plenty of ways to extend the kit for myself and decided to share my extended version. It was a tool for beginners. My role was to share my knowledge of ASP.NET with everyone else.

But there was a point where I was just copy and pasting and hacking the crap out of something that wasn’t mine and was written by a bunch of other people. I ripped a lot of things out, especially the data access. What I put in it’s place was a disaster as well, though. I simply didn’t have the forethought to imagine a site that was fully extensible. I simply put out something that was hacked and hacked and hacked. I didn’t put something out that EVERYONE could use, I put something out that EVERYONE had to figure out every little piece and extend. That’s simply irresponsible on my part.

So I decided I needed to completely rewrite the thing. There was a point were I was sick and tired of writing VB. To me, it looks WAY too verbose. I couldn’t stand looking at it… and C# was my primary language at a point anyways, and still is.

The big news

Now I’m ready to come back. I finally have something that I put my stamp on. It’s MINE. I wrote it. There’s not a feeling in this world more satisfying for me than to say a piece of code is completely my creation. No hacks, no copy paste (ok, there’s a little… but at least I know what it does!) and certainly something I think everyone can use, whether it’s bits and pieces or the whole thing.

What’s new!

Well… for starters, it’s written entirely in C#! While I know some VB people are shivering in their boots, I plan on upgrading the web project to VB if there is an apparent need from the community. I’m not here to waste my time converting something to VB if there’s not a need. C# is such an expressive language and it’s something every VB dev should at least TRY to look at. More on that down the line.

Another big new thing is that it runs ASP.NET MVC, not classic web forms. I know that there is some serious turmoil out there over this subject, but it really is a much better way to create web applications in ASP.NET in my opinion. There’s not very much of a difference, as it turns out. It’s a very comfortable shift… more than I thought it would be. It makes applications much easier to understand, write, and test. I can entertain comments, questions, and concerns by email over this. I’m willing to fight for this :) But really, MVC rocks. It’s simply amazing.

I know it hasn’t been released yet, but this release will be targeting Visual Studio 2010 and the .NET Framework 4.0. There’s a lot of really cool new ASP.NET features that I think CSK would really benefit from.

Now to the big thing that might just turn some people off… we’re primarily going to use NHibernate for data access. I plan on adding a few more providers (Subsonic would be the first to the list). I really started off with NHibernate as a learning experience. It’s a very complex, but WELL written data access solution. I’ve VERY pleased with the results. There’s a learning curve with NHibernate, to an extent. But NHibernate isn’t as hard for simple operations, like what CSK provides. I think I can go back to the fundamental part of CSK and provide some really good samples for people to learn from. NHibernate isn’t really all that hard as long as you have something to look at. Trust me, reading the documentation isn’t as glamorous as it sounds :). Hopefully I can abstract that away from the average CSK user.

A huge reason for adding NHibernate as the primary data access point is because we can use it to generate our DB for us, often called “Domain Driven Design” (although we aren’t completely “DDD”). I can’t tell you how much trouble I’ve had with the SQL stuff in the past. If you start with your data model first, a tool like NHibernate or Subsonic will just generate all the DB code for you! For me, that’s HUGE! I hate having people say the new version broke their DB. That' hurts. Now we can just have the damn thing upgraded!

Speaking of the DB… we will be starting from scratch on that too… sorry to say. I know plenty of people will be really disappointed by this. Just how it is. Hopefully from now on, we will have a better upgrade path. I hate breaking people’s compatibility simply because we release a new version. I’ll be assessing a possible upgrade path for those running v3 beta 1 and v2. But we shall see. This would be a GREAT contribution if someone in the community could code this out.

The fact that we didn’t have a lot of tests in the last few versions proved we really didn’t have a great product. We just didn’t. That simple. I knew that this time, I needed to test EVERYTHING. While I don’t claim to be a TDD expert, I really tried to test first. I’ll detail contributions in a bit, but I can tell you, everything needs to have tests. While we’re a little light on tests right now, hopefully that will change as we get closer to v3.

What’s taken out

There was a serious outcry after v2 for a league management component. What happened in v3 Beta 1 was… interesting. I haven’t really seen anyone use it, and the feedback was mixed. So for this iteration, it’s not going to be a core part of the product. If there is a serious need, I think we (as in the community) should plan and implement an extension to CSK for league management stuff.

League management was a really good idea, but I think the ideas some people had were just out of my scope. It’s something someone needs to write a spec for. I can’t just create what I think will fly. Just not something I’m willing to waste more of my time on.

Also on my list of things taken out, I’ve finally decided to separate the logic of the application into separate projects. While the Visual Studio Express people might be upset, it really comes down to manageability of the project. If you create a large web application, it really defeats the purpose of expansibility. There are 6 main projects: Core, Infrastructure, Web, Data.NHibernate, Domain, and Tests. The core and infrastructure projects are meant to be distributed with ANY web project. It’s not CSK specific. So you can really use those as toolkits. It has some really good logic in it. I’d love to provide something that isn’t just for our little open source project. I will, however, be shipping a template with compiled versions of Domain, infrastructure, core, and Data.NHibernate. Hopefully that will help those who use express. We’ll see. I’d love to hear your feedback on this.

Contributions

In the past, I’ve said everyone can just contribute. That’s going to change. We’re starting with a whole new codebase that I wrote. Because of this, I think it’s time we started accepting patches rather than add contributors. We’ve had only a tiny bit of contribution in the past anyways. So here’s my decision.

I’m going to take every contributor off the list. It’s a tough choice, but it has to be done. I will start to accept patches. These patches HAVE to contain tests. Once you’ve contributed in that way, I will then consider making you a contributor to the source repository.

The release

So I’ve told you wants new, and what not, but I didn’t say anything about a release. Today I am releasing the new codebase. I’m going to name this ClubStarterKit v3 preview. While I would love to call this v4, I think that not having a final v3 would be kind of strange. So this codebase will eventually become version 3. My plans from this point is to get feedback on what I have an release Version 3 when it’s ready. No promises, no false hopes. It’ll be done when I say it’s done. In the mean time, you can download the preview here.

A call to action

I’ve done my thing. I’ve built what I thought you’d want. Now it’s YOUR turn to help me out. Tell me what you like, tell me what you don’t like, tell me what should be in there, tell me what the future of CSK looks like. I need your feedback. It’s the essential ingredient in any open source project. We need to pick up anew and make this a sustainable and polished project that we can call a community-driven project!

As always, thank you for your support and contribution. We all can give each other a helping hand in learning to develop software.

ClubStarterKit - where do we go?

zowens — Wed, 30 Jul 2008 01:40:28 GMT

Wow, what a year. Not exactly what I'd expected. Before I dive into where we are now, let me preface this post by saying I have NOT forgotten ClubStarterKit! Although I was not the original author, I feel like it's my baby. I can't just forget about it. Not going to happen.

You might be quick to point out that I have done virtually nothing since the v3 beta 1 release. This, my friends, was a big mistake. Not v3 beta 1, which I will discuss later in this post, but the fact that I was not clear in what I hope to get out of the project. I do take the blame on the lack of development. I'm not going to sit here and tell you I have neglected to look at the code in a while. This is partially due to my work life (I have to make some money.. know that I have 2 jobs) and my personal endeavors (as I am a high school student with more to do than dedicate all my time to ClubStarterKit). No one is perfect. But I do sincerely apologize to the ClubStarterKit community for my lack of communication.

My hopes from the project

When I started the project, I knew that there was a void. ClubStarterKit v1 was an obvious success because of the idea of sample-driven learning. It did a lot of things that people needed. Now, I really wanted to bring CSK to the level of open source. With this, I opened up the door for contributors. This is an area that I found fairly problematic. We did get some amazing contributions, such as the CMS and the RSVP system. Without these core contributions, I would have spent more time building the features rather than tracking down bugs or expanding other CSK features. Trust me, building out a feature for CSK is not easy.

Essentially I had hoped for the ClubStarterKit to be a community effort where there would be multiple core developers. In my idea of "core developer", I seem to be the only one. To me, CSK seems more like a sample that goes through active development.

It is my hope to not be a nag about this, but come on CSK community. You guys do some amazing stuff. Then you demand something out of CSK without contributing? I just don't see how this whole thing can work out like this. I'm not a free contractor. I don't even ask for donations!

I have been asked by one member of the community to forget my role in CSK and pass it along to another developer. Guess what, no one stepped up. Interesting. Guess it looks like I'm still in, win or lose.

What's the deal with the inactivity?

Like I said, there are many factors to this. Mostly, I am starting to regret hard-coding SubSonic into CSK. I now know, after completely hard-coding this stuff in, that I should have allowed external DALs that people wanted. Not everybody like SubSonic and it's query tool. Some people like NHibernate, LINQtoSQL, XML, whatever. I seriously screwed up. My fault, guys. I've learned, let's move on.

A big reason for inactivity was time. I just couldn't devote a lot of time on CSK like I used to. Luckily, this is about to change for me. Just know that I'm going to have more time to work on CSK. Period.

What's next?

I hate to say it, but v3 beta 1 was kind of bad. I really wanted something fresh. That just wasn't what we got with v3 beta 1. The ideas were there, just the delivery was poor. So this is what I plan on doing. What I want to do, and I would LOVE to hear feedback from you on this, is to completely scrap the current codebase and move over to the ASP.NET MVC framework. I am seriously in love with ASP.NET MVC. Honestly, it will move development, deployment, and testability along. I am in the preliminary planning stages. Also, for development, I would like to use C# and convert the C# code over to VB. The fact that the original kit was released in both languages made it all the more appealing. With a refreshed codebase, we can make this process much easier on the person converting (probably me :) ).

Also, we will have an extensive service layer that can be extended to use a custom DAL or any form of data storage.

Lastly, the features listed in my last post will go in.

So, what do you think? I would LOVE to hear your feedback AS LONG AS IT IS PRODUCTIVE AND WILL BENEFIT THE PROJECT.

THANKS FOR YOUR CONTINUED UNDERSTADING CLUBSTARTERKIT COMMUNITY!

Source Code conversion as part of a Build Process

zowens — Sat, 12 Apr 2008 04:10:20 GMT

If you're like me, you want a particular piece of code in your particular language. For example, a lot of people want the Club Starter Kit 2.0, 3.0 in C# since it has been written in VB. Me being the lazy open source developer that doesn't get paid :), I keep delaying the CSK conversion to C# because of my personal stuff. Then I was thinking, what if it were a part of the build process? Then I wouldn't have to go through the arbitrary process of converting a whole application to another language for a release that I will most likely be changing.

Before I go ahead and implement this myself, I was wondering if anyone out in the community has done this sort of thing before? I'm sure it wouldn't be that hard, what with the CruiseControl.NET API being what it is. I have already scoped out some ideas of how to implement. But I was wondering if anyone has ever built this plugin.

PLEASE leave a comment on the blog or email me (zowens2009 at gmail dot com)

THANKS COMMUNITY! I will make you proud if this isn't out there ;)

Future of ClubStarterKit project

zowens — Tue, 22 Jan 2008 19:20:00 GMT

It's been almost a year since I started the ClubStarterKit project as an open source project up on Codeplex. Since then, we've released Version 2, which added functionality to the original starter kit, and the first Beta of Version 3, which tried to change the data architecture with SubSonic.

The problem is that I've sort of abandoned the project for a while to work on my personal projects. But in that time, I've been really looking at better ways to put together the ClubStarterKit. This post is sort of a Version 3 feature list/roadmap that hopefully we can stick with!

The Data Layer

We tried to do something cool by redoing the data by using SubSonic. In my eyes, this was only a partial success. The reason I chose SubSonic in the first-place was because the original app really was just a presentation of data, there wasn't really much that went on and from a programmer's perspective, there was a lot of logic just encapsulated in the SqlDataControl. Having inline SQL in a SQLDataControl is really unmanageable and can be a pain to update. With SubSonic, it's easy to update a data sequence to contain an additional field.

What I want to do next is have a Data Provider. I realized that not everyone wants to use SubSonic. Some people want to use ADO.NET and I'm sure that some people want to use LINQ now that it is available. What I want to do is give people the option in the provider. The default will still be SubSonic, since I strongly reccomend it, but we will also natively support ADO.NET and LINQ and possibly anything else you guys might want us to support :)

Wiki

I've wanted to build a wiki for quite some time. Our CMS does do some pretty cool stuff, which we will be building upon with the final release, but I also want to build a wiki for those clubs that need such functionality. I'll be evaluating some of the wiki stuff I want to put into the ClubStarterKit's wiki engine, but I definitely want it's functionality.

Photo Album

I really want to revamp the Photo Album functionality. I really want to make it almost Flickr-like with commenting, tagging, etc. But I also want to add the functionality of a sweet photo-album with effects. I would love to add something like the Lightview since it's really amazing to look at :)

What about all the questions I have in the forums?!

I know that I have fallen behind on my forum answering. For that, I TRUELY apologize. I do have a life, you know :) But there really isn't any excuse for not answering people's questions. In the next couple of weeks, I will be cleaning out the issue tracker, taking care of some things, and get those forum questions answered! There are WAY too important for me to just blow them off.

New Website?

I am considering building a website for the project. Codeplex is GREAT, but there are some things I would really love to just make myself into a website. It would most-likely run off a modified version of the ClubStarterKit (which I WILL post for you guys to download) and will show off what a modified version could look like. More on that later.

If there are any question about the roadmap, the project, or feature requests, PLEASE LEAVE A COMMENT! If you have an urgent issue or you just want to talk with me one-on-one, email me (zowens@eagleenvision.net). I really want to start getting this rolling again as this starter kit should be on the cutting edge of ASP.NET design!

Thanks to the community and the contributors of the project on your feedback and we hope to start getting your ideas and suggestions into the product!

Quite The Surprise for ClubStarterKit!

zowens — Thu, 13 Sep 2007 19:05:00 GMT

I was just catching up on some RSS feeds and I saw something really cool... the ClubStarterKit open source project was put up on the main ASP.NET site! (http://www.asp.net/downloads/starter-kits/codeplex-club/)

Although I didn't really ask for this to go on there, I think it is appropriate. THANK YOU SO MUCH WHOEVER PUT IT UP! Hopefully we can continue to provide a great community for the ClubStarterKit... and this should help :)