Rob Gillen's WebLog

Codestock Session

[This is cross posted from here]

I’d like to thank all of you who attended my session today at CodeStock. I had a great time talking with you all and sharing my experiences with SharePoint and TFS with you all.

Downloads from today’s session:

• Powerpoint Deck
• My customized version of STSDev
• The Samples I built in prep for today
• The samples we built during the session today

Also, I promised a collection of links for the tools I had installed.

• STSDev on CodePlex (this does *not* contain most of the fixes demonstrated today)
• SharePoint Installer
• Sand Castle
• Sand Castle Help File Builder
• Zoomit – Utility for zooming in and out during a presentation
• Smart Tools Enhanced Site Actions

Customizations to STSDev 1.3

[This is cross-posted from here]

As part of my session on Deployment and build using TFS and SharePoint for CodeStock 09 I took the source code from the STSDev project on CodePlex (http://stsdev.codeplex.com) and made a number of modifications. Some of these I would classify as clearly bugs, but most of them are simply adjustments to the core to fit my needs/desires. I’m documenting them here and providing a zip of the source for the benefit of those attending my session. These changes and source code are completely unsupported and you use them at your own risk. That being said, I hope that they are helpful and speed you in your integration between SharePoint and TFS. NOTE: unless specified, all of these changes are to the “Core” project.

1. First minor change is that I moved the solution file up one level to the parent folder. This is truly nothing but a nit-pick but seems to make source control trees happier and therefore something that I almost always do.
2. Upgraded the projects/solutions to Visual Studio 2008
3. Added an app.config file with an assembly binding redirect pushing old references to Microsoft.Build.Framework to utilize version 3.5.0.0. This is one of the changes needed to get support for .NET 3.5 working properly.
4. Changed the target framework property for the stsdev.csproj to .NET Framework 3.5. This change, in concert with the previous, allows the 3.5 selection in the UI to work properly.
5. Major Change: Support for alternate bin paths. The 1.3 version as published on CodePlex always uses the compiler output in projectdir\bin\debug when assembling the *.wsp file. This happens regardless of what build configuration you have selected (yes, even release). This doesn’t work for TFS builds since the output is, by default, in a different location on the build server. To support this use case, the following changes were made:
   • Program.cs: Changes were made prior to calling SolutionBuilder.RefreshDeploymentFiles to support the passing in of an additional parameter indicating the alternate bin path.
   • Changed the Create method of Builders\DeploymentFiles\CabDdfBuilder.cs to accept the alternateBinPath as an additional parameter.
   • Changed the Create method of Builders\DeploymentFiles\CabDdfBuilder.cs such that, if an alternate bin path is provided, it will use that value when adding references to assemblies rather than the otherwise-hard-coded /bin/debug/
   • Updated the first overload of the RefreshDeploymentFiles method in SolutionBuilder.cs to pass the alternate bin path to the second overload of RefreshDeploymentFiles.
   • Updated the second overload of the RefreshDeploymentFiles method in SolutionBuilder.cs to build the TargetPath from the alternateBinPath if provided and otherwise to use the default.
   • Updated the second overload of the RefreshDeploymentFiles method in SolutionBuilder.cs to pass the alternateBinPath parameter to CabDdfBuilder.Create().
   • Updated the CompleteSolution method in SolutionBuilder.cs to pass an empty value for alternateBinPath to RefreshDeploymentFiles since, during the initial project creation, it is ok to utilize the default /bin/debug path.
6. Major Change: When creating a project, always create a parent solution directory in the same way that Visual Studio defaults to. This is helpful when working in a source controlled environment with multiple projects in the same solution. To support this feature, the following changes were made:
   • Changed the <REFRESH /> property in Resources\Common\Microsoft.SharePoint.targets.xml to utilize the $(ProjectDir) variable rather than $(SolutionDir).
   • Changed the Create method of SolutionFiles\SolutionFileBuilder.cs to accept the solution directory as an additional parameter.
   • Changed the Create method of SolutionFiles\SolutionFileBuilder.cs to change the directory for where the solution file is created
   • Changed the Create method of SolutionFiles\SolutionFileBuilder.cs to reference the *.csproj file in subdirectory rather than in the same directory as the *.sln file.
   • Added  a property called ProjectDirectory to SolutionBuilder.cs to hold the full path to the project directory (now distinguished from SolutionDirectory).
   • Updated the RunCreateSolutionWizard method in SolutionBuilder.cs to set the Project Directory property.
   • Updated the InitializeSolution method of SolutionBuilder.cs to create the new (child) project directory and update the location appropriately
   • Updated the InitializeSolution method of SolutionBuilder.cs to set the SolutionBuilder.TargetPath based on the ProjectDirectory property rather than the SolutionDirectory value.
   • Updated the CreateDeploymentFiles method of SolutionBuilder.cs to use the proper path for the files (based on ProjectDirectory).
   • Updated the RefreshDeploymentFiles() (first overload) method of SolutionBuilder.cs to optionally set the ProjectDirectory property if it isn’t already set.
   • Updated the second overload of the RefreshDeploymentFiles method of SolutionBuilder.cs to set the current directory to the value of ProjectDirectory if it is set, and otherwise to use the SolutionDirectory value.
   • Updated the second overload of the RefreshDeploymentFiles method of SolutionBuilder.cs to utilize the ProjectDirectory value rather than the SolutionDirectory path.
   • Updated the LoadSolutionConfigFile method in SolutionBuilder.cs to use the ProjectDirectory property and remove the pathing ambiguity that was leading to incorrect path lookups.
   • Updated the CompleteSolution method in SolutionBuilder.cs to pass the SolutionDirectory property to SolutionFileBuilder.Create() since it can no longer assume that the solution file should be in the same directory as the project file.
7. Major Change: Added support for the Release|AnyCPU project configuration to support integration with TFS Build. In support of this feature, the following changes were made:
   • Added <REFRESH-TEAMBUILD> property in Resources\Common\Microsoft.SharePoint.targets.xml file. The value of this property is similar to that of <REFRESH /> but has an additional parameter referencing the alternate bin path that TFS creates by default.
   • Added a Release target in Resources\Common\Microsoft.SharePoint.targets.xml. This is targeted specifically at the TFS build and is therefore similar to the ReleaseBuild target but calls the $(REFRESH-TEAMBUILD) exe rather than $(REFRESH)
   • Added a “Release” value to the Configurations string array in SolutionBuilder.cs. This causes the release config to be added to the generated solution and project files.
8. Major Change: We utilize SharePoint Installer to create a nicely-packaged version of the resulting solution making it nearly brain-dead easy to install a new solution. The only real thing unique or project-specific in a SharePoint Installer package is the setup.exe.config file that passes a number of parameters to the executable allowing it to adapt for your particular solution package. This feature change causes STSDev to generate the initial draft of this file and to add it to the solution items. To this end, the following changes were made:
   • Created a new file, SolutionFiles\SetupConfigFileBuilder.cs to represent the template and logic for the SetupConfigFileBuilder class.
   • Updated the CompleteSolution method in SolutionBuilder.cs to call SetupConfigFileBuilder.Create and create the new config file.
   • Updated SolutionFiles\SolutionFileBuilder.cs to include a pointer to setup.exe.config and thereby include it in the solution.
9. Major Change: We utilize Sand Castle and Sand Castle Help File Builder to produce developer-targeted API documents for our projects. Sand Castle Help File Builder needs a configuration file (xml) with a number of property values to configure it to run properly. This feature allows the basic file to be generated by STSDev and for it to be added to the solution items collection. To this end, the following changes were made:
   • Created a new file, SolutionFiles\SandcastleHelpFileBuilder.cs to represent the template and logic for the SandcastleHelpFileBuilder class.
   • Updated the CompleteSolution method in SolutionBuilder.cs to call SandcastleHelpFileBuilder.Create and create the configuration file.
   • Updated SolutionFiles\SolutionFileBuilder.cs to include a pointer to the generated *.shfb file and thereby include it in the solution.
10. Major Change: By default, STSDev would not only generate manifest.xml and the ddf file, but it would add these files to the project. While in many cases this is innocous, it causes heartburn in a source-controlled environment. Firstly, purely generated files have no business being in your source tree. Secondly, since the user doesn’t edit them, once they are checked in, they are most often left as such, resulting in errors on compile because the files are marked as read only, preventing stsdev.exe from updating them properly. To this end, the CreateDeploymentFiles method of SolutionBuilder.cs has been updated to remove the calls to ProjectBuilder.AddSourceFile() following ManifestBuilder.Create() and CabDdfBuilder.Create().
11. Added an image, PlanetIcon.gif to the resources section and replaced all hard-coded references to africanpith.jpg with references to PlanetIcon.gif. This is nothing more than a branding change for the output projects and has no other affect on the program’s functionality. Affected files include: stsdev.csproj, SolutionProviders\SimpleFeatureSolutionProvider.cs (AddSolutionItems method), Builders\SourceFiles\FeatureBuilder.cs (Create method)
12. Adjusted the InitilaizeSolutionProviders method of  UserInterface\SelectSolutionType.cs to select by default the Web Part Solution (C# Assembly) project type on load rather than the empty solution. This change is nothing other than a convenience feature during testing and use (that’s almost always the project type I use).
13. Adjusted the RefreshDeploymentFiles method of SolutionBuilder.cs to fix some seemingly obvious bugs in Console output using incorrect values.

Following the session on Friday, I’ll update this post with the actual source code and any other changes made during the presentation.

Speaking at CodeStock

[This is cross-posted from here]

I’m privileged to have been given the opportunity to speak at CodeStock (details below) this coming Friday. I’ll be speaking on the topic of Deploying and Packaging SharePoint solutions using TFS. The abstract for my session is:

Have you been using the VS Extensions for SharePoint to create SharePoint packages and found yourself wondering how best to integrate with your source control platform and build system? Consistent packaging of SharePoint solutions can be a challenge and is not for the faint of heart. Come to this session and learn how our team utilizes TFS, Team Build, SandCastle, SharePoint Installer, and STSDev in concert to produce consistent installation packages for our SharePoint/MOSS environment.

CodeStock is about Community. For Developers, by Developers (with love for SysAdmins and DBAs too!). Last year an idea started at CodeStock to mix Open Spaces within a traditional conference. This year we're going to crank things up to 11 and rip off the knob - and you're being drafted to help!

• Keynote by Microsoft RIA Architect Evangelist Josh Holmes
• From Developer to Business Owner roundtable with guest Nick Bradbury creator of HomeSite, TopStyle, and FeedDemon
• 50+ break out sessions + Open Spaces (self-organizing sessions)
• Grand Prize: VSTS 2008 Team Suite with MSDN Premium
• Virtual sessions with Jeffery Richter and John Robbins

Space is limited so register today at CodeStock.org

Common API Set for Cloud Storage?

I’m working on a project which has as one of its goals the “publishing” of some very large datasets (order of 1PB) to the “cloud” for consumption by the general populous for use in scientific research. Rather than designing/inventing our own API, our decision has been to provide an interface consistent with the APIs produced by some of the leading cloud storage providers. Our goal would be that if someone is already used to/has tooling to working with cloud data sources such as Amazon’s S3 service or Microsoft’s Azure Blob storage, those same tools/experiences should directly apply by simply changing the http endpoint.

Unfortunately, neither Amazon or Microsoft seem poised to provide an open source implementation of their interfaces (from a service host perspective) so we are faced with the challenge of reverse-engineering their server-side interfaces. Neither is overly complex, so this certainly isn’t rocket science, but as I’ve been digging/poking at them the last few days I’ve found myself wishing there was more commonality between them. There are certainly many similarities and the “lift” moving between them is not heavy, but that’s actually the rub… they are so similar in many ways that I find myself wondering why they aren’t identical… are the deltas actually adding any value, or are they simply the repercussions of different groups of people solving the same problem while trying to convince the world they are not copying one another. I should be clear that I’ve been focused on the REST APIs and haven’t spent any time with the SOAP interfaces so I can’t speak to the commonality or lack thereof at that level.

From a client perspective, they are similar enough that there are many tools (both OSS and commercial) that can front-end either platform, so the differences aren’t *that* significant. John Spurlock has done some good work in providing some libraries for C# that provide wrappers for the REST APIs and a client application (http://www.codeplex.com/spaceblock and its foundation http://www.codeplex.com/resourceful). and Alin Irimie has assembled a simple chart illustrating some of the similarities (http://www.azurejournal.com/2008/11/amazon-s3-vs-azure-data-storage/).

Clouds and Traditional Hosting Companies

NOTE: this blog has moved to http://rob.gillenfamily.net Please update your links and feed readers appropriately.

I sat on a conference call yesterday wherein we discussed our thoughts on where the cloud was going and how it was going to impact the “traditional” hoster. Our company has been working with hosters for many  years (I started in 2000 and they had been doing it before that) and have seen cycles of service come and go. It used to be that just providing personal web + email was all you had to do to be successful. Then, the “Application Service Provider” (ASP) model started and morphed into providing business-related services and eventually to where companies outsourced entire portions of their infrastructure (think Hosted Exchange, Hosted CRM, Hosted OCS). Throughout much of this, however, locality seemed important – not in reality, but in perception. The vast majority of people would choose to host their services with a company that was close to them, or they had learned by reference from a friend or colleague.

As we walked through the current market directions, and in later conversations on the same topic, I’ve become more and more convinced that unless significant changes are made, the traditional “hoster” is destined for failure. Web hosting, data hosting, email, etc. are all being commoditized by the big boys (read Microsoft, Amazon, Google) making it almost impossible for the small hoster to compete. I expect to see a rise in the number of white label services, and also in the number of mixed-metaphor hosters – What I mean by this last statement is hosters who will present a unified stack of services to their customers and while much of those services may simply be white-labeled reselling of someone else’s services, some key aspects of the “stack” will be provided directly by the hoster. The key is specialization, or niche services. Maybe a hoster offers a specific financial or ERP system hosted and then marries that with services from other vendors.

Another aspect of the call that was interesting to me was the postulation that these hosters were looking for ways to “get into the cloud” – I chuckled a little because they, of all people, understand the benefits of the cloud as they have been cornerstones of such for years. They’ve built their businesses on providing “cloud” hosted services. They understand where the margins are/are not – and know better than most that it’s all about scale. This is precisely why I think that the existing model is in trouble – a moderate size hoster with 1000 – 5000 servers simply cannot compete with the big three who are buying datacenter blocks in shipping container-units – each of which hold between 2000 and 4000 servers.

The game is changing… those who innovate and adapt will be here tomorrow to talk about it. Those who don’t, won’t.

2009 BJU Programming Contest

NOTE: This blog has moved to: http://rob.gillenfamily.net. Please update your links and feed readers accordingly.

I had the privilege of being one of the alumni-judges at the annual Bob Jones University Computer Science departments programming contest. This was the first time I’ve participated in this type of contest and I found it very interesting. The CS department had a fairly slick harness for executing the contest and supporting the judging in multiple languages and multiple platforms. As with anything of this nature, there were a few bumps in the road, but nothing of any consequence.

The contest turned out wonderfully… we  had around 35 contestants (I lost count because we overflowed the one room and had to use a different room). There were 10 problems of various difficulties to be solved in a 3-hour time window. The contestants could solve the problems in any order, and could choose both their platform (Windows/Linux) and their language (C#, Java, Python, C++, Ruby). To my surprise, many of the contestants switched between languages rather than using just one as I would have expected. Every contestant solved at least one problem properly and all of the problems were solved by at least one person. The distribution of problems solved was pretty balanced as well.

As a judge, my job was to monitor the queue for submitted answers, run the submissions through the test harness and reply on the results back to the contestant. I was a bit amazed (though I shouldn’t have been) at the wide variety of coding styles and levels of verbosity to solve the same problems. The contestants could also submit questions to the judges, and the favorite for the day was “can I leave and not come back?”.

I’d like to congratulate the winners and all of the contestants for a fine job and look forward to participating in next year’s event.

Misplaced Modifier?

I was on a few websites this morning (go figure) and noticed a handful of visual missteps (mostly minor). One of them was on the Mix 09 conference page and made me chuckle a little.

On the page talking about being able to watch Mix online (http://visitmix.com/News/Watch-MIX-Online-Ask-the-Gu) they, I think, are trying to encourage you to leave a comment.

Or, they really don’t like me, and are simply telling me to leave.

Project Templates for SharePoint Development

I've had the privilege of working with an organization for almost two years now doing alot of SharePoint development. There's a team of approximately 75 developers that cover the gammut of skill sets and experiences, most of whom are working, to some degree, with C# and SharePoint development. One of the things that has come up repeatedly, is "how do I get started"? or "what project template should I be using"? These are good questions without a completely clear answer. This post (and hopefully some following) are intended to discuss what we are using, how it evolves, what options we discarded, and why. I spent last Friday teaching a class attempting to bring our team up to speed as to how to structure their SharePoint solutions/projects in VS and get them integrated with Team Build and packaged for deployment in our organization. As the day wore on, I realized how "janky" the "elegant" solution I had been using felt to someone new to the problem set. The supposed elegance was simply relative to the pain I had been experiencing doing it the "old way"... there has to be a better answer for the causal SharePoint developer.

I should stop here to add a caveat that prior to this assignment I spend a year or so (on and off) working on a custom VSIP toolkit for Visual Studio that included custom project and item templates as well as menu items, dockable toolwindows, custom build tools, etc. so I'm a bit biased towards using the built-in extensibility hooks for Visual Studio (especially since its gotten so much easier with the 2008 release).

A year and a half or so ago we started by looking at the WSS Extensions for Visual Studio.. these were interesting and "felt" like the right answer because "they were from Microsoft... certainly they must be the best approach". While this sounded good (and in theory should have worked out) we ended up with a handful of issues... the first being their very slow support for VS 2008. Secondly, we often found a webpart project that suddenly stopped working (i.e. F5 deploying) and couldn't figure out how to get it working again... there seemed to be a bit too much "magic" going on behind the scenes. Also, It was completely unclear how one would take the resultant project and integrate it into a build system (i.e. Team Build). I'm sure it is possible, but it didn't seem to add much value beyond the initial development.

We then looked at STSDev (codeplex.com/stsdev). This is an interesting project that is *almost* my silver bullet. I like the way the template projects are setup, the layout, and the work that the post-build events work (automatically maintaining the ddf and manifest files as well as building the wsp). I have a handful of gripes with how they layout the project structure (no root folder for the solution) and the variables they use (or don't) for certain things. My biggest gripe is that I'm left wondering why they didn't leverage the existing, built-in templating features for Visual Studio (i.e. why can't I go File --> Add New --> web Part). Why should I have to train developers on yet another paradigm for creating their projects? Is their launcher really any better? I think not.  That being said, to this day, this is still the mechanism we are using for the majority of our work, but my dissatisfaction with the tool is the driver for this post and quest for a better end-to-end solution.

Because many of our developers were building webparts based on the SmartPart, we found ourselves looking at the SmartTemplates project. There was some very interesting things learned from the way in which Jan implemented this, but still some difficulties presented themselves when we tried to look at this tool relative to the larger problem of our entire SharePoint development environment (webparts, "standard" features such as menu items or application pages, smart parts, and Team Build).

While at PDC, I found myself talking with the Blueprint guys (http://codeplex.com/blueprints). Near as I can tell this is the successor to the GAX tools and looks to be very interesting. I spent part of yesterday afternoon studying the approach and found some very nice features (i.e. ability to update/maintain the platform via an RSS subscription). Unfortunately, the platform is still in beta and, at least in my testing, doesn't feel ready for primetime... maybe in a few months...

So, today I'm going to start out very basic and investigate simply building a custom project template that uses the built-in T4 templating features of VS. The objective is to have a project template that a developer can click on that will create a web application project, configured for development of ascx controls as smart-parts, along with a folder structure similar to what STSDev gives you supporting the build and auto generation of wsp files, as well as preparing the project for Team Build. 

.NET is a Smorgasboard?

Like many other .NET devs I often find myself expecting to be current in all of the existing and up-coming tools/technologies in the Microsoft/.NET platform. Frankly, I don't know how that is possible, especially with the pace at which MSFT (not to mention the surrounding ecosystem) is releasing tools and platforms. Over the past few years, my approach has been to know "enough" about the various tools/technologies so that I can be conversant, and also know when a particular toolset applies to my current project, thereby warranting a "deeper" dive into that area. Such has been the case for me with WPF and WCF (much of my work over the past while has been in the SharePoint/web space meaning WPF - until SilverLight - didn't have much of a play and we hadn't yet seen a need to switch from standard ASMX for our services). They fell into the bucket of tools I had seen while walking along the smorgasboard, but I simply hadn't decided I needed to consume them yet.

Scott Hansleman describes the .NET Framework and the MSFT tool suite as making it easy to "fall into success" (I'm sure I'm not quoting him correctly, but the idea is the same). Essentially, the tool set, while robust and quite capable, is approachable and relatively easy to simply build something. Especially when you compare it to other languages such as C++ -- in C#/.NET it is relatively easy to build "okay" code, and not that hard to build good code and almost (yes, there are plenty of exceptions) hard to write *bad* code. It is much easier (at least in my opinion) to write bad C/C++ code and much harder to write good C/C++ .code. I agree with him 100% - once you have a core competency on the platform, picking up the basics of the "new" stuff becomes almost trivial

I was recently working on a project (someone else did most of the coding - I did some of the design and proof-of-concept work) and I was able to see this in action. We were building a security-focused app, being deployed to a mixed environment of XP and Vista machines, and we had a 6-7 week window to build it, test it, and have it deployed. We ended up building a Windows Service that hosted a WCF service, a desktop application using WPF, a webpart for SharePoint and an IIS-hosted WCF service. We made heavy use of the cryptography libraries which, oddly (to me) were one of the areas that the other developer had prior experience with, however neither he nor I had done any real work with WCF and WPF. The technologies offered us quite a bit as far as functionality and form, even for two guys who weren't "experts" in them - that's where the "magic" lies - I'm reasonably comfortable with the MSFT dev stack, and I'm handed two completely new-to-me technologies, and with a relatively small amount of effort, I'm able to use them in my application and reap the benefits they bring. Now, certainly there's quite a bit more functionality that WPF/WCF bring to the table than what we used or "grok'd" during this project, but it did what we needed to and quickly - making me want to dig further into those technologies and to use them for other projects.

Does the “Cloud” help the Public Sector?

[Warning… rambling mind walk following]

I’m here at PDC 2008 and having a rather good time attending sessions, talking with experts, speaking with vendors, etc. It has been a pretty good conference and I’ve done a better job than previously at managing my ability to pay attention (i.e. I don’t necessarily *have* to attend a session during every available slot, and, amazingly, there’s a good chance that taking a break b/t will help me pay attention better at the next one…

Anyway, I’ve been thinking about the Windows Azure announcements and having discussions with a couple of my colleagues and also the CIO where I’ve been working for the past couple of years and I’m left wondering what value the “cloud” and the cloud computing platform provides me with… Don’t get me wrong… I completely get it for the consumer market… there is huge value there, and some of my work with some non-profits and consumer-facing organizations might benefit significantly from these services. I can even see benefits in some internal-to-the-same-business corporate scenarios, however I think the uptake is going to be slow simply due to an unwillingness (warranted or not) for companies to “trust” Microsoft (or Google or Amazon for that matter) with their core corporate data.

The problem for me, is that for the past few years I’ve been working at an institution in the US public sector… we’ve been “pushing the envelope” a good bit and are more willing than most to adopt new technologies… but I don’t think that I as an app architect for this organization, nor I as a citizen of this country, necessarily want us posting federal information to “a cloud” controlled by someone other than the same federal entity… and I don’t think that this is necessarily unique to the US federal space… I would have an equally difficult time seeing any other government being willing to take this leap of faith.

However, the principals of the cloud are, in fact, very interesting… should those of us in the public sector be banished to a life of permanent disconnectedness? I think not… What I’m leaving the conference wondering… is how does a federal entity (for any country) provide a “safe cloud” to its users? One in which much of the flexibility/extensibility that is provided in the “public” cloud is still available, however wherein the data and services are completely controlled by the specific government… This, of course, brings up a number of issues… if the given agency completely hosts/controls the “cloud” services… do they cease to be “cloud” services and simply manifest themselves as renamed corporate services? This is certainly a possibility but then I think those of us within the various agencies/sectors need to think of ways to broaden the scope of smaller clouds that we will be building… can the Gov’t of India (US, Canada, or insert country name here) build a single set of “cloud” services that is available only for use by it’s authorized agencies and still see manifested the benefits of scale, abstraction, etc. that are being touted for the “public” cloud? Does a given government build a single cloud? or one for each agency? or some mix in the middle? Will MSFT (or some other vendor) provide tools and platforms that not only allow you to connect to “their” cloud, but also allow you to build your own cloud?