Tales from the Evil Empire

About Orchard Governance and Microsoft

Back in September, we did something with Orchard that is kind of a big deal: we transferred control over the Orchard project to the community.

Most Open Source projects that were initiated by corporations such as Microsoft are nowadays still governed by that corporation. They may have an open license, they may take patches and contributions, they may have given the copyright to some non-profit foundation, but for all practical purposes, it’s still that corporation that controls the project and makes the big decisions.

That wasn’t what we wanted for Orchard. We wanted to trust the community completely to do what’s best for the project. This is why we organized elections for our new Steering Committee and had five members elected. Anyone who had ever posted anything on our forums that wasn’t spam could be a candidate, and had five votes to cast.

We got 9 candidates, and after a week of vote casting, we had our committee elected (see http://orchard.codeplex.com/discussions/271355 for the details of the vote). Our five members are elected for a year and they each get a vote in any major decision. Two out of those five are Microsoft employees, which means the company doesn’t even have a majority (even though I have a veto right that I never used, as the elected “benevolent dictator” of the group). We have public online meetings every week.

The committee’s role is to give the project’s strategic direction, and to decide what feature areas are going to be in the next releases. It is not to make technical decisions or build the product (more on that in a moment). The weekly meetings are used to give status on current development, do bug triage and make decisions as needed.

This was not just the right thing to do. It did have a very visible impact on the involvement of non-Microsoft developers. Shortly after the election, we started calling for teams to form around our 1.4 release’s features. Feature teams are being given a feature scope by the committee, and they are then free to make their own decisions on technical design. They regularly interact with the committee, which gets to decide when the features make it into the main branch.

This was quite successful as we are preparing to launch what will be the first release of Orchard where entire features have been written by the community. In fact, 3 out of 4 of the new feature areas of 1.4 have been written mostly by non-Microsoft developers.

To my knowledge, this makes Orchard the most open of all Microsoft-initiated Open Source projects. It now belongs entirely to the community, and absolutely everything is not only done publicly but also in a way where community participation is actively sought after and encouraged.

My hope is that as the project becomes more and more successful, our model gets copied by other corporate-initiated projects, at Microsoft and elsewhere. I would love to hear your thoughts, dear readers, on this.

What’s the deal with Orchard and NuGet? Does Orchard suck?

Damn, Rob saw right through the PR lies of Phil Haack about the new NuGet gallery and figured it all out. Kidding, kidding. For your enjoyment and context, here is Rob’s post:
http://wekeroad.com/2011/12/06/nuget-and-orchard/

Phil made a thoughtful answer to the post where he gives the rationale behind the decision:
http://wekeroad.com/2011/12/06/nuget-and-orchard/#comment-380571113

What I want to address here is what that means (or not) about Orchard.

“Why wasn’t it fast before?”

Because by default, back when the NuGet gallery was first built, Orchard was grossly under optimized. We made a lot of progress, but it still is under optimized out of the box --in a number of ways-- which is why we’re having this discussion for the 1.4 release:
http://orchard.codeplex.com/discussions/274921

One thing you cannot accuse the Orchard team of is premature optimization… But seriously, there are options available today: installing the cache module, or the combinator module. Or custom, in-site optimizations.

So could the NuGet team have optimized its gallery instead of mobilizing several engineers over a few months to rewrite it? Sure they could have, but it so happened that they didn’t, for reasons Phil explained better than I could.

As a matter of facts, our own gallery site, which just passed the million-download mark, still uses that same codebase that NuGet was using before. The new NuGet gallery doesn’t display total download stats (because the home page is now static html) but I believe it’s around 3 million so we’re looking at a similar scale here. Just visit http://gallery.orchardproject.net/ and you’ll see that it’s performing quite well. So what’s the deal here?

Well, we did spend the time to apply a few optimizations:

• We profiled the code and fixed the biggest bottlenecks. Those fixes, of course, now benefit all Orchard users.
• We profiled the database queries (using Ayende’s excellent NHProf) and optimized the biggest offenders.
• We activated output caching.

All optimizations that NuGet never saw applied. And well, see for yourself, it’s not too bad. We’re not quite done, there is a whole lot more we can and will do, with what limited resources we have (hint: a lot less than NuGet).

We’re not the only ones: more and more big sites are going online with Orchard, with good performance.

But Rob also says something interesting and expresses a perception that I’ve seen many times:

“I’ve deployed Big CMS’s before as a solution for clients and every single time we decided to move away. They’re great for getting off the ground – but after a while there’s just too much friction.”

That is true of some CMS, but not all. In the case of Drupal or Orchard (not what Rob used, I checked), my own experience and that of many others is that because the content model is factored at precisely the right level of abstraction, it’s actually sometimes harder to get off the ground, but the friction diminishes over time. Where a more traditional application requires you to pile functionality on a relatively monolithic data model, modern CMS allow you to add functionality in a completely decoupled and unobtrusive way that puts the composition exactly where it needs to happen, somewhere between individual properties and full types.

Source-controlled database backups

I’m not very satisfied with traditional database backup solutions. It seems like almost no progress was made since SQL Server 6.5 (at least for the built-in tools). They are still outputting monolithic binary backup files that only do the job if the only thing you expect from backups is the ability to recover from catastrophic failures. I expect more, but before I explain, we need a disclaimer:

I am no expert of database backup and may very well miss some crucial points or some aspects of the current state of the art. The solution exposed in this post is the result of my own experiments and has no pretense at exhaustiveness or even reliability. It’s something I use on my own simple projects, and no more. Database backups are serious business and your data is valuable. Use this at your own risk.

What I wanted from my database backup solution can be summarized as follows:

• Works on Azure databases (can you believe that as I’m writing this there is no valid built-in backup solution on Azure?)
• Backups are not stored in the same place as the database
• Has more than one state to rollback to
• Can be automated
• Easy to restore
• Can make diffs between backup states
• Can merge backup states

The last two, diffs and backups, can seem a little peculiar and require some explanation. I work on several Orchard projects, where a team of developers collaborate. The team is virtual, with people in the US, and in various places in Europe. Because Orchard is a CMS, the database contains more than just data: it also stores the configuration of the site, something that developers modify as part of their routine job. Since it is impractical to work off a central database because of latency, and also because one developer can break everyone’s instance with a bad change, we need multiple copies of the database, and a way to reconcile them.

What works in a distributed fashion, resolves conflicts, and stores history? Why, a Distributed Version Control System of course! This is what gave me the idea of storing my database backups in a private repository on BitBucket, using Mercurial as the DVCS.

In order to be able to do diffs and merges, we need a text-based persistence format. I chose to use the bcp command-line utility to emit that format because my past experience of it has been that it’s fast and reliable. The code could in principle use the bulk copy SQL commands, but having never worked with them I went for bcp. If someone wants to modify the code and make a pull request using bulk copy that’s fine by me. Bcp produces flat text files for each table that are very easy to diff and merge.

My solution, HgDBackup, comes with two command-lines that automate bcp, one for backup and one for restore. HgDBackup.exe enumerates the tables in the database and spits out one text file per table. HgDRestore.exe enumerates the tables, tries to find a text file with the same name, and if found, drops all rows in the table before importing the contents of the file into the table.

The schema of the database is not backed up, so you need to do that as a separate operation every time it changes, or as a separate automated task.

In the distributed development environment I described above, the process to take a new version of the database will be as follows:

• execute HgDBackup
• commit local changes
• pull and update from the repository
• execute HgDRestore
• run tests
• commit if a merge was necessary
• push to repository

Both commands have built-in help that you can bring up with hgdbackup /? or hgdrestore /?.

The backup command responds to the following flags:

• /Server or /S: the address of the server to backup (e.g. tcp:myazuredb.database.windows.net or .\SQLExpress)
• /Database or /D: the name of the database to backup (e.g. Orchard)
• /User or /U: user name to use on the database
• /Password or /P: password to use on the database
• /Output or /O: output directory where the tables will be dumped (e.g. C:\backup)
• /Template or /T: connection string template

The restore command responds to the same flags, except that Output is replaced with Input.

The default connection string pattern is SQLExpress with integrated security (so no password has to be on the command-line).

For example, here is what I use to backup one of my Azure databases:

HgDBackup.exe
  /S:tcp:something.database.windows.net /D:Orchard
  /U:mysubscription@weirdstuff /P:**********
  /O:C:\Projects\nwazet.Backup\Tables
  /T:Server={0},1433;Database={1};User={2};Password={3};
     Trusted_Connection=False;Encrypt=True;

And here is what I use to import that backup into my local SQLExpress instance:

HgDRestore.exe
  /S:.\SQLExpress /D:Orchard
  /I:C:\Projects\nwazet.backup\Tables

That nwazet.backup directory is a clone of a private repository on BitBucket. It’s of course important that the repository is private because databases contain secrets.

The code is available on BitBucket:
https://bitbucket.org/bleroy/hgdbackup

You may also download the already compiled commands:
http://weblogs.asp.net/blogs/bleroy/HgDBackup.zip

And remember, use at your own risk…

From ScrewTurn Wiki to Markdown

I'm in the process of moving the Orchard documentation site from ScrewTurn Wiki to a Mercurial + Markdown system, where revisions are managed through source control tools instead of a fully online wiki. We see quite a few advantages in doing that, but that's a story for another post.

Today, I just want to post on that quick and dirty tool that I've built to translate the ScrewTurn Wiki markup into Markdown. It's not a masterpiece in any way, it's not exhaustive, but I thought it may be useful to others, so here it is…

https://bitbucket.org/bleroy/screwturn2markdown/src

State of .NET Image Resizing: how does imageresizer do?

I've written several times before about image resizing in .NET and how the various built-in solutions (GDI, WPF and WIC) compare in terms of quality, speed and size. I'll put the links to my previous articles at the end of this post for reference.

Several readers have since pointed me to the imageresizer library, which is pure .NET and thus has no problems running in medium trust. Medium trust is an issue that has plagued existing options, preventing many people from using the best available approach. I was doubtful though that a purely managed library could come anywhere near the native Windows libraries in terms of performance. The best way to find out, of course, is to run a benchmark. Fortunately, I still had the code for my previous benchmarks so I just had to add imageresizer code to it.

I excluded IO from this test by measuring the processing time for each library working on binary streams. This also levels the playing field as some of them start processing while they are reading a file, making it harder to separate file reading overhead from image processing. By working with streams, the comparison is more meaningful. The timing code is wrapped around only the meaningful code.

The first problem that I hit with imageresizer is that it has a strong dependency on System.Web. This is really unfortunate and very probably unjustified. The library should be split between the pure image processing bits, which should have no System.Web dependency, and the request handling and caching bits, which can depend on System.Web all they want.

Since I'm mostly interested in web applications, I changed my command-line code into an ASP.NET Pages application, which probably saved me some configuration headache. The code is linked to at the end of this post. One thing I have to say is that the API feels good, natural and reasonably concise, a nice change when compared with the noisy Interop-like code we had to write before. It's nice to have something that is native .NET rather than a thin layer on top of Windows APIs that stink of C.

var settings = new ResizeSettings {
    MaxWidth = thumbnailSize,
    MaxHeight = thumbnailSize,
    Format = "jpg"
};
settings.Add("quality", quality.ToString());
ImageBuilder.Current.Build(inStream, outStream, settings);
resized = outStream.ToArray();

I really liked that I don't have to do size calculations myself, and I can just give it a square or rectangle inside of which it has to fit. You have many options to customize that. It's unfortunate that the quality setting has to be set this way instead of being promoted to a 1st class property like other settings but I can live with it. Notice how the library can work with streams, which is very important for web apps. One thing I really didn't like though was that some APIs, such as Build, take untyped object parameters and then decide internally at runtime what to do with them. Why the author didn't use strongly-typed overloads here is a mystery to me.

So how does imageresizer do then? Well, mostly as expected. In terms of size, I found it to be a little greedier than its peers, but really nothing dramatic:

The quality of the images is where imageresizer shines. See for yourself (quality 85% for those images):

imageresizer	WIC

The images look sharper (in particular the hummingbird one), and a little bit more saturated. Because of that, some moiré effects seem more visible (the shot before last makes that the most obvious. Still, the resizing is overall the best I've seen so far.

Now of course, while managed code is usually quite fast, image processing is heavy in the sort of operation where native code has a definitive advantage. That is reflected in the results I got in terms of processing speed:

Imageresizer is clearly the slowest. That was expected, but now we know by how much. It is 31% slower than GDI (I excluded the 50 quality point here), and almost 4 times slower than WIC.

Of course, unless your application is doing lots and lots of resizing operations, speed may not be that important, in particular if you are doing appropriate disk caching. Imageresizer shines in that department by making disk caching super-easy. The library, after all, is very web centric and tries hard to do everything right for the web scenarios. We should not underestimate how hard it is to get web image resizing right.

That leaves us with the unfortunate conclusion that we still don't have an image resizing silver bullet. I'll try to summarize all that we know in a table that will hopefully help you make a decision in the context of your specific application:

Benchmark code:
http://weblogs.asp.net/blogs/bleroy/ImageResizeBenchmark.zip

Results:
http://weblogs.asp.net/blogs/bleroy/ImageResizerBenchResults.zip

Previous articles:

• The fastest way to resize images from ASP.NET. And it’s (more) supported-ish
• Server-side resizing with WPF: now with JPG
• Resizing images from the server using WPF/WIC instead of GDI+
• What InterpolationMode and CompositingQuality to use when generating thumbnails via System.Drawing
• Generating image thumbnails in ASP.NET

The imageresizer library:
http://imageresizing.net/

So what's new in Orchard 1.3?

Orchard 1.3 was released yesterday night with some really neat features that I will outline in this post. I will come back in depth on some of those with full-length posts. Let's start with the simple but super-useful ones…

1. Draft Preview

Once you've created and saved a content item, there will now be a "Preview" link next to its summary:

Previously, it was impossible to view unpublished contents. This was one of the most requested missing features, and there it is.

2. Delete Content Types and Parts

Speaking about features high on the wish list, you can now delete the custom content types and parts that you built.

3. Markdown

Not everyone likes editing contents in Wysywyg editors, for various reasons. There has always been the possibility in Orchard of switching the flavor of the Body part, but the only available alternative to the default "html" so far has been "text", which was only giving you a plain text box. Now there is a new option, "markdown".

Markdown is a neat text syntax that has the advantage of being simple, rich and human-readable.

I especially like the code sample convention of starting a line with four spaces. Very handy for writing documentation. Oh, about that, did I mention that we are planning on switching the Orchard documentation from the current wiki to Markdown?

4. Title Part

Until now, the title of Orchard content items has been kept in the Route part. That was an inconvenience when you wanted to give a title to a content type that did not have a vocation to have its own URL, for example widgets.

Title is still there in the Route part, but what we did is introduce a new interface, ITitleAspect, that the Route part implements.

public interface ITitleAspect : IContent {
    string Title { get; }
}

In all the places where we were using the old title, we are now using the interface instead. If you need to add a title, but not a URL to your own content types, you can either make one of your parts implements ITitleAspect, or you can add the new Title part, that already implements the interface in a ready to use form.

5. Rules

Rules are a truly wonderful feature that enables you to add event-driven behavior to your site from the admin UI. Let's create for example a new "Comment Notification" rule and add a "content created" event that we'll configure to be triggered on the Comment content type:

Once this is done, we can add an action to be triggered when that event fires. We'll choose "Send e-mail", which will raise the following form, where we can specify who will receive the e-mail, and what will be the subject and body:

The subject and body of the e-mail can use tokens, which are another new feature that I'll get to in a minute.

Here is how the content creation event was wired in a handler:

OnCreated<ContentPart>(
  (context, part) =>
    rulesManager.TriggerEvent("Content", "Created",
    () => new Dictionary<string, object> { { "Content", context.ContentItem } }));

When a piece of content is created, an event is triggered on the rules manager, giving it the content item being created as the context. The event itself is implemented in a class that implements IEventProvider. I'll write more on creating your own events in a future post.

Actions are classes that implement IActionProvider. Both event and action providers can expose their own UI forms for their configuration, which brings us to the next feature…

6. Forms

So far, to expose admin UI, a module writer had to create controllers or drivers, view models and views. Starting with this release, they can use a much easier forms API. This is what rule events and actions are using to expose their UI. The forms API makes it possible to describe a form in a nice, compact and extensible form that is using, you guessed it, dynamic shapes.

Shape.Form(
    Id: "ActionEmail",
    _Type: Shape.SelectList(
        Id: "Recipient", Name: "Recipient",
        Title: T("Send to"),
        Description: T("Select who should be the recipient of this e-mail."))
        .Add(new SelectListItem { Value = "owner", Text = T("Owner").Text })
        .Add(new SelectListItem { Value = "author", Text = T("Author").Text })
        .Add(new SelectListItem { Value = "admin", Text = T("Site Admin").Text }),
    _Subject: Shape.Textbox(
        Id: "Subject", Name: "Subject",
        Title: T("Subject"),
        Description: T("The subject of the e-mail."),
        Classes: new[] { "large", "text", "tokenized" }),
    _Message: Shape.Textarea(
        Id: "Body", Name: "Body",
        Title: T("Body"),
        Description: T("The body of the e-mail."),
        Classes: new[] { "tokenized" }
        )
    );

What you see here is the form that is in the e-mail action picture above. Of course, each of the shapes in there will be rendered by a template that can be overridden (as usual). The advantage of such a description is that it can be enriched dynamically. A comments form for example could have a captcha inserted dynamically by another module, just by injecting a new shape in the form. I encourage you to dive into existing implementations to understand how you can use this new API, until we get the chance to write some good documentation.

7. Tokens

Tokens are system-wide variables whose values can vary depending on the context. They can be used in a variety of places, one of which is rules, as you can see from the example above (the subject of the e-mail that is sent when new comments are added uses a token to describe the name of the commenter). I've blogged about tokens before, so I encourage you to read these posts to know more:

http://weblogs.asp.net/bleroy/archive/2011/07/22/future-orchard-part-1-introducing-tokens.aspx
http://weblogs.asp.net/bleroy/archive/2011/07/27/future-orchard-part-2-more-tokens.aspx

8. New Content Manager Methods

Building contents by composing small parts is an extremely flexible approach, but it makes it more difficult to build efficient database queries. There is always caching, but we also needed to optimize things a bit. We did that by adding some new methods to the content manager, such as GetMany, which is taking multiple content ids in one query. We also added the Aggregate attribute that can be used to specify records that should be eagerly loaded. For example, the Tags property of the TagsPartRecord and the TagRecord property of ContentTagRecord are marked with the attribute, because when the tags are included with a content item, you don't want the act of actually getting the tags associated with the content item to be lazily executed (that is a SELECT N+1 condition). If you want the tag part, there is a very high chance that you'll want to actually access the tags themselves: the part in itself has nothing else of interest. With the attribute, the tags will be fetched at the same time and from the same query as the item itself.

9. The rest

There is a lot more in this release, such as RSS for lists (it's just there, not much to say about it), some localization improvements, the task lease API that enables better synchronization across web farms, and many, many bug fixes and performance improvements (in particular, if you use multi-tenancy, you should feel the difference). If you want to see everything that was fixed, take a look at the list on CodePlex:

http://orchard.codeplex.com/workitem/list/advanced?keyword=&status=Closed&type=All&priority=All&release=Orchard%201.3&assignedTo=All&component=All&sortField=Votes&sortDirection=Descending&page=0

What's next?

The current plan for the near future is to quickly release a 1.4 version with the focus on the new Projector feature that will enable you to query your contents in any way you want directly from the admin UI. This, I can assure you, is going to be a killer feature.

PIX-6T4, the DIY console, has an Orchard site

If you're a regular reader of this blog, you may have noticed the electronics posts lately. With my friend Fabien, we've been building a little handheld console from readily available parts. It's also programmable in C#, using a library that we built. It's great fun really. Our plan is to build a kit and a book out of it, to help people learn the basics of digital electronics and C# programming.

We've built a site for the project over a few nights last week, using Orchard and hosting on Azure. I thought I'd throw in a little return on experience and talk about the modules we used.

The site can be found here: http://www.pix6t4.com

The first thing I did to build the site was to build the theme. I started from the built in Theme Machine, and just added the markup and styles for the header. I then changed the fonts from the stylesheet and layout files, added a few custom styles such as the atomic bullet lists or the round corner menus, some transparency for the body and that was pretty much it. The Theme Machine is extremely flexible and let me get exactly what I wanted without changing much markup at all. Most of the changes are CSS.

I then packaged and deployed my Azure image onto our Azure instance. We are using multi-tenancy in order to host multiple small sites on a single Azure instance, so I just created a new tenant for our new domain name. After the second or so it took me, I was able to go through setup for the tenant, and pointed the site to the same SQL Azure connection string I'm using for the main tenant, but using a difference table prefix.

Here are the modules we're currently using (I installed those on my dev machine and added them to the Azure solution so that they get uploaded with the image, as you can't install modules directly on Azure):

• Google Analytics: very simple to use, just register the new site on Analytics and copy the provided script into the module's admin page. You are done, Google starts accumulating stats on your site from this moment.
• Content Types: as you add modules to your Orchard site, you will often want to extend existing types with new parts. You need the content types feature for that.
• Import Export: the Azure story for backups is, well, lacking at this point in time. We'll use export to work around this and download the contents of the site on a regular basis. It's not the same as proper automated backups but it's way better than nothing.
• Vandelay Industries: well, I built this one. I has several useful features, such as favicon settings, Feedburner, tag clouds, etc.
• YouTube Field: We added a Youtube field to the page content type, to make it super-easy to add video to our pages. I had to make a couple of modifications to the module (nice that all modules come with source code): I overrode the template so that it uses the iframe YouTube markup rather than object and embed. As a side note, the default markup has a Lady Gaga video hard-coded into the archaic <object> tag. Nobody noticed I suppose because no recent browser uses that nowadays. I also modified the driver so that it only returns a shape if a video has been configured and null otherwise. I will contribute back those changes of course.
• Contact Form: works as advertised. You can create a contact form page, and every time someone fills the form, we'll get an e-mail notification. The module also has a report page where you can see all the contact requests. I did have to fix a bug in the driver to prevent double entries in the database. I will also contribute that change back.

And well, that is it really. It's a simple site, and it was a simple job to set it up, as it should be. :)

Building a simple Fritzing component

This is me shaving a yak. Shaving the yak, if you don't know, is what you do when a seemingly simple task necessitates many recursive and unforeseen sub-tasks in order to be carried out.

The story goes like this… Let's say that I want to paint a picture, but I'm in a shack in the middle of the desert and I don't have a canvas, brushes or paint. I do have a piece of strong cloth, a few pieces of wood, and the rocks around the shack can provide some pigments that I can mix with some oil. For the nails, I can extract some from the shack's structure itself. For the brush, I'll need some quality hair. Well, long story short, before I know it, here I am, in the middle of the desert, shaving a yak.

Today's metaphorical yak is the representation in Fritzing of a $0.95 part, a knob potentiometer. Fritzing is a wonderful Open Source tool for designing electronic circuits. Its only shortcoming is that its library of components is not yet complete enough that it can be used to design all circuits. In my case, it's lacking the SD card reader that I'm using, analog sticks and… this small $0.95 potentiometer. Well, in fact, for the potentiometer, I could have easily used one of the stock components from Fritzing that is close enough, but for my first component design, I wanted to start with a fairly simple part so I went ahead with it anyways, with the hope of having a better fit in the end.

A Fritzing component is typically made of five XML files that can be optionally packaged together using a zip format. Four of these files are SVG: the icon, the breadboard view, the schema view and the PCB view. The fifth file is a special XML format with extension .fzp.

Each of the SVG files should be crafted with precision, following the component's datasheet, that always contains the precise dimensions of the part. I showed some of the techniques for this in my previous post. I found that manually authored SVG is a lot easier to debug for Fritzing, and debugging you will do…

The SVG elements constituting the graphical representation of the component in each of the views should be enclosed in layers (g elements) that bear a specific id. For the breadboard view, that is "breadboard", for the schema view, "schematic", and for the PCB, you need at least three layers: "silkscreen", "copper0" and "copper1". The silkscreen describes the white indications on the PCB, whereas copper0 and copper1 are the top and bottom copper layers.

In addition to the graphical representation of the component, the file must also contain objects that represent the connectors for the part. In the case of the potentiometer, there are three connectors named leg1, leg2 and wiper.

<rect id="connector0pin" x="184.5" y="440" fill="none"
      width="76" height="76"/>
<rect id="connector1pin" x="438.5" y="440" fill="none"
      width="76" height="76"/>
<rect id="connector2pin" x="692.5" y="440" fill="none"
      width="76" height="76"/>
<rect id="connector0terminal" x="184.5" y="440" fill="none"
      width="76" height="76"/>
<rect id="connector1terminal" x="438.5" y="440" fill="none"
      width="76" height="76"/>
<rect id="connector2terminal" x="692.5" y="440" fill="none"
      width="76" height="76"/>

In the breadboard view, those connectors are each represented by a pin and a terminal (see code above). The way I understand those is that the pin is the representation of the physical pin, whereas the terminal is the part of it that plugs into the breadboard, where the actual contact is made. The terminal, therefore, has to be precisely positioned and dimensioned to align with breadboard holes. [UPDATE: Jonathan Cohen from the Fritzing team is telling me that the terminal doesn't have to be of a specific size but that the center of the element is used as the contact point. The terminal element is not even mandatory; if it's not there, the center of the pin element is used. Still, the distance between the pins/terminals needs to be precisely 0.1 inch in breadboard view (it doesn't matter in schema view, and PCB view should be precisely matching reality)] I found Fritzing to be quite touchy with this, with earlier designs I've made failing to click easily into the breadboard if at all, or partial fits of only some of the connectors. With the hand-written SVG for the potentiometer though, I found the fit to be easy and error-free. The terminals must be, in my experience, named "connectorNterminal" where N is the index of the terminal, starting at 0. The pins are named similarly "connectorNpin". Terminals should be 0.76mm squares (0.03 inch in the weird and irrational unit system some parts of the world insist on using) and 2.54mm apart (or 0.1 inch). The full set of conventions that Fritzing expects can be found here: http://fritzing.org/fritzings-graphic-standards/.

For the schema view, I re-used the existing potentiometer template, something that one should not shy away from. I also has similarly id'ed pins and connectors as in breadboard view:

 <rect width="1.894" x="17.797" y="1.824" height="9.783"
       id="connector2pin" />
 <rect width="0.867" x="18.276" y="1.14" height="0.684"
       id="connector2terminal" />
 <rect width="1.893" x="17.801" y="52.541" height="9.783"
       id="connector0pin" />
 <rect width="0.867" x="18.28" y="62.253" height="0.687"
       id="connector0terminal" />
 <rect width="1.894" x="29.861" y="42.203" height="9.783"
       id="connector1pin" />
 <rect width="0.866" x="30.409" y="51.986" height="0.684"
       id="connector1terminal" />

Remaining is the PCB view, the most important if you are doing all this with the purpose of having a real PCB printed, like I do. No mistake is allowed here, the dimensions have to come exactly right as they will have to fit with reality, in other words the actual physical component. As I described before, the PCB view has at least three layers, the silkscreen (that I like to be the outline of the component to facilitate assembly), and the two copper sides of the PCB. One difference with the other views is that connectors have to be described as patches of copper on one or both of the copper layers (usually both, unless you're doing a surface-mounted component). The two copper layers are described from the same point of view, and are not mirror images of one another, so if they are identical one can be a simple copy of the other:

<g id="copper0">
    <rect
        style="fill:none;stroke:#FFBF00;stroke-width:50.8"
        x="152.5" y="408" height="140" width="140"/>
    <circle id="connector0pin"
        style="fill:none;stroke:#FFBF00;stroke-width:50.8"
        cx="222.5" cy="478" r="69.85"/>
    <circle id="connector1pin"
        style="fill:none;stroke:#FFBF00;stroke-width:50.8"
        cx="476.5" cy="478" r="69.85"/>
    <circle id="connector2pin"
        style="fill:none;stroke:#FFBF00;stroke-width:50.8"
        cx="730.5" cy="478" r="69.85"/>
</g>
<g id="copper1">
    <rect
        style="fill:none;stroke:#FFBF00;stroke-width:50.8"
        x="152.5" y="408" height="140" width="140"/>
    <circle id="connector0pin"
        style="fill:none;stroke:#FFBF00;stroke-width:50.8"
        cx="222.5" cy="478" r="69.85"/>
    <circle id="connector1pin"
        style="fill:none;stroke:#FFBF00;stroke-width:50.8"
        cx="476.5" cy="478" r="69.85"/>
    <circle id="connector2pin"
        style="fill:none;stroke:#FFBF00;stroke-width:50.8"
        cx="730.5" cy="478" r="69.85"/>
</g>

Each pin in PCB view is usually a thick circle (with a thick square around it for pin 0), and there is no terminal in this view.

Now here's a pitfall I fell into: one might be tempted to include the connector description only once, and to surround it with nested g tags for copper0 and copper1. I had seen this in existing components and thought it was a neat way of not repeating myself. Well, I had to pay for that economy with a good hour of debugging why my component was not loading in Fritzing, giving me a cryptic "unable to create renderer for svg" message. The weird thing was that the PCB view was rendering fine in the component palette, but not on the design surface. Once I unwound the copper layers and just made them plain and boring copies of each other, the component started to work.

Once you have your four SVG files, all that remains is to build the .fzp file that will tie it all together. Fritzing contains a decent tool for this, where you pick the four files and then start identifying additional properties and connectors. I used this as a starting point and then edited the file it produced in a text editor, once again for better control.

One thing you need to know when you do this is where Fritzing is putting your files. You will need to get acquainted with the \Users\YourUserName\AppData\Roaming\Fritzing\parts directory (may be hidden on your system, requiring some tweaking of your OS settings to appear). The .fzp is in the user subdirectory, and the SVG files have been copied into the respective subdirectories of svg\user. Fritzing also renames the file to include some goo for unicity. I prefer to rename those files to something entirely human-readable and then modify the corresponding references inside the .fzp file.

The .fzp has fields for author, title is the name of the part that will appear in the component palette, and label is what Fritzing will use to generate a unique name for the component instance on the design surface (R23 for example for the 23rd resistor on the surface). You can optionally include a url tag that will appear as a hyperlink above the component's description. Great to point to the datasheet or to a place where the component can be purchased.

<module fritzingVersion="0.6.2b.07.11.5217" moduleId="TSR-3386UT">
    <author>Bertrand Le Roy</author>
    <title>TSR-3386UT Square Trimming Potentiometer</title>
    <label>R</label>
    <date>2011-08-07</date>
    <url>http://www.sparkfun.com/products/9806</url>

Then we have tags, to facilitate search.

<tags>
    <tag>potentiometer</tag>
    <tag>poti</tag>
    <tag>adjustable</tag>
    <tag>resistor</tag>
    <tag>sparkfun</tag>
</tags>

The properties section should compile relevant data about the component, usually a copy of what can be found in the datasheet. Important pieces of information are the family and the Size, because that will neatly classify your component together with others of the same kind. For example, for my little knob, I can drag the stock potentiometer onto the design surface and then pick "Rotary – 9.53mm" in the size drop-down to switch from the default potentiometer to the one I designed.

<properties>
    <property name="Manufacturer">Suntan</property>
    <property name="Size">Rotary - 9.53mm</property>
    <property name="Maximum Resistance">10k&#937;</property>
    <property name="family">Potentiometer</property>
    <property name="Track">linear</property>
    <property name="Type">Rotary Shaft Potentiometer</property>
    <property name="part number">TSR-3386</property>
</properties>

The description can be simple, or quite complex, with support for encoded HTML.

<description>Trimpot 10K with Knob</description>

Then comes the meat of the file, with first pointers to the four views and declaration of the relevant layers.

<views>
    <iconView>
        <layers image="icon/TSR3386UT_icon.svg">
            <layer layerId="icon"/>
        </layers>
    </iconView>
    <breadboardView>
        <layers image="breadboard/TSR3386UT_breadboard.svg">
            <layer layerId="breadboard"/>
        </layers>
    </breadboardView>
    <schematicView>
        <layers image="schematic/TSR3386UT_schematic.svg">
            <layer layerId="schematic"/>
        </layers>
    </schematicView>
    <pcbView>
        <layers image="pcb/TSR3386UT_pcb.svg">
            <layer layerId="copper0"/>
            <layer layerId="silkscreen"/>
            <layer layerId="copper1"/>
        </layers>
    </pcbView>
</views>

Then we have the description of the connectors. Each connector is described by an id, a type (male or female), a friendly name, and a description. Under this come pointers to the elements inside each view that represent the connector. The layer attribute is what you expect, the name of the layer containing the connector (don't try to be too smart, stick to the standard names). The svgId attribute is the id of the pin object, and terminalId is the id of the terminal object. In PCB view, there is no terminal, but there is a p tag for each copper layer.

<connector id="connector0" type="male" name="Leg1">
    <description>Leg1</description>
    <views>
        <breadboardView>
            <p layer="breadboard" svgId="connector0pin"
               terminalId="connector0terminal"/>
        </breadboardView>
        <schematicView>
            <p layer="schematic" svgId="connector0pin"
               terminalId="connector0terminal"/>
        </schematicView>
        <pcbView>
            <p layer="copper1" svgId="connector0pin"/>
            <p layer="copper0" svgId="connector0pin"/>
        </pcbView>
    </views>
</connector>

And… this is it, we're done. All we have to do now to share our creation is to package it (select an instance of the component on the design surface and select Part/Export in the menu) and upload it to the contribution bug: http://code.google.com/p/fritzing/issues/detail?id=875.

You'll be able to download my own potentiometer from there, or you can use this direct link.

Have fun shaving your yak, and don't hesitate to tell me in the comments what I did wrong or misunderstood. I'm a n00b.

Reference documentation on Fritzing components can be found here:
http://fritzing.org/support-us/developer/fritzing-part-format/

Authoring SVG with a text editor

SVG definitely is an increasingly interesting skill, especially as it's making its way into HTML 5 as an officially allowed grammar inside of HTML documents. Most SVG is authored through some kind of tool, and it's absolutely the way to go for artistic drawings. I used Inkscape (open source) and Expression Design in the past for that (I can't afford Illustrator), and I've been happy with the results (to your right and left).

When you are doing this kind of artwork, the quality of the generated SVG doesn't matter much, and well, it's a good thing as what those tools spit out is rather horrendous. Think HTML from Word.

There is a category of drawing though that is better authored by hand, because it needs to be human-readable and tweakable, and because it requires a level of precision in coordinates that would be tedious to achieve in a tool. A lot of technical drawing enters this category.

One of the things I do outside of work is electronics. I use Fritzing to design printed circuit boards (PCBs), and it's an excellent open source tool, but its library of components is not as complete as what can be found in commercial offerings, so I have to author some of the components I need myself. Today I'm working on a very simple one, a potentiometer I got from Sparkfun (illustration to your right).

Fritzing has three views: breadboard, were components are drawn in a semi-realistic fashion, schema and PCB. In this post, I'm going to explain how I built the breadboard view of the potentiometer.

At the beginning of the document is some rather uninteresting doctype declaration:

<?xml version="1.0" standalone="no"?>

<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">

Then we have the svg tag where, apart from some namespace declarations, the size of the document is declared:

<svg width="9.53mm"
    height="9.53mm"
    viewBox="0 0 953 953"
    id="TSR3386U-T_Breadboard"
    version="1.1"
    xmlns="http://www.w3.org/2000/svg"
    xmlns:xlink="http://www.w3.org/1999/xlink">

We define the size in millimeters here to make the design from the component's datasheet as easy to translate as possible. The viewport attribute is here to map a system of coordinates on these dimensions. The values we set here maps 9.53mm to 953 coordinate steps. The result of this is that a coordinate of 100 will be a millimeter from the origin (all the other coordinates in the document will be without units and will use those proportions). This also means that you can resize the whole document from this single place.

A very important construct to understand in SVG is groups, represented by <g/> tags. This allows us to make logical grouping of forms and give structure to the document. For Fritzing components, I put a group with id "breadboard" around all shapes to identify the breadboard view of the component. Inside of that group, I have subgroups for the outline of the component, and another for the shapes forming the knob.

The outline is made of a single filled path tag:

<path
   style="fill:#0000A0;stroke:none"
   d="m 0,0 l 953,0 0,953 -75,0 0,-38 -803,0 0,38 -75,0 z" />

The style is self-explanatory (and familiar if you know CSS), but the unexpressive d attribute demands more attention. It describes the path using a small DSL. Each command in the path DSL consists in a letter and a set of coordinates. The convention is that a lowercase letter is used for relative coordinates, and uppercase is for absolute coordinates. We use relative coordinates here. We start by moving (m command) the pen to the top-left corner (0,0), then we draw lines (l command) 9.53mm to the right, 9.53mm down, 0.75mm left, 0.38mm up, 8.03mm left, 0.38mm down, and 0.75mm left. Then z (like zero) closes the path.

The design of the knob needs three concentric circles, an arrow on top and small grooves around the knob. When adding the shapes for each of those elements, it's important to order them from the bottom up as SVG uses that to determine how overlaps are handled. The circles are simple shapes, described by the coordinates of the center (cx, cy) and the radius (r):

<circle
   style="fill:none;stroke:black;stroke-width:20"
   cx="476.5" cy="419" r="419" />

The grooves around the knob are more interesting. They are thin lines going from the center of the knob, all around it. We could draw each line by computing its extremities' coordinates, but that would be no fun at all. Fortunately, SVG has a special tag, <use/>, that can make clones of other elements, simple or complex, and apply geometrical transforms to them. So all we have to do is draw one line, and then we can rotate copies of it for all the others.

<line id="grooveception0"
    style="fill:none;stroke:black;stroke-width:5"
    x1="476.5" y1="419" x2="57.5" y2="419" />

Here's a use tag that takes the line above and rotates it 4 degrees around the center of the knob:

<use xlink:href="#grooveception0" transform="rotate(4, 476.5, 419)" />

The transform here is a simple rotation specifying the angle in degrees, and the x and y coordinates of the center of the rotation. Now we could make 88 more copies of this to complete the circle but that would be boring. Instead, let's divide and conquer, and make increasingly big recursive copies of the grooves.

<g id="grooveception5">
 <g id="grooveception4">
  <g id="grooveception3">
   <g id="grooveception2">
    <g id="grooveception1">
     <line id="grooveception0"
       style="fill:none;stroke:black;stroke-width:5"
       x1="476.5" y1="419" x2="57.5" y2="419" />
     <use xlink:href="#grooveception0" transform="rotate(4, 476.5, 419)" />
    </g>
    <use xlink:href="#grooveception1" transform="rotate(8, 476.5, 419)" />
   </g>
   <use xlink:href="#grooveception2" transform="rotate(16, 476.5, 419)" />
  </g>
  <use xlink:href="#grooveception3" transform="rotate(32, 476.5, 419)" />
 </g>
 <use xlink:href="#grooveception4" transform="rotate(64, 476.5, 419)" />
</g>
<use xlink:href="#grooveception5" transform="rotate(128, 476.5, 419)" />
<use xlink:href="#grooveception5" transform="rotate(256, 476.5, 419)" />

The neat thing about this method is that the number of copies necessary only grows with the logarithm of the number of grooves. In our case, we have 7 use tags instead of 89.

And this is pretty much it. I'm attaching the finished SVG file to this post so you can check it out in its entirety, and here is the result:

Download the SVG file:
http://weblogs.asp.net/blogs/bleroy/Samples/TSR3386U-T_Breadboard.zip

So you don't want to use placement.info?

In Orchard, the UI gets composed from many independent parts. We wrote a lot of code to handle that fact without friction. It is easy to add a new part or remove an existing one without breaking anything. One ingredient in this is the placement.info file.

The role of placement is to dispatch the shapes that represent content types to local zones and to specify in what order they should appear. it separates the templates and their orchestration, implementing some healthy separation of concerns.

It is quite powerful and flexible, but it represents a sort of inversion of control that can be quite puzzling to designers: we are used, in other systems, to the layout pulling the different "includes" that constitute it. With placement, the "includes" are getting pushed into zones according to placement. This is similar to master pages in spirit, a concept we walked away from for similar reasons.

There is always a little pause in the learning of Orchard when people have to discover placement. In this post, I'm going to show how to do layout more explicitly, without placement. Whether it is a good idea or not, the possibility is there and will be easier and more familiar to some.

On my personal blog, the summary of a post is rendered by the regular ~/Core/Contents/Views/Content.cshtml template:

<article class="content-item @contentTypeClassName">
    <header>
        @Display(Model.Header)
        @if (Model.Meta != null) {
        <div class="metadata">
            @Display(Model.Meta)
        </div>
        }
    </header>
    @Display(Model.Content)
    @if(Model.Footer != null) {
    <footer>
        @Display(Model.Footer)
    </footer>
    }
</article>

This is not doing much except defining Header, Meta, Content and Footer local zones where placement can inject the shapes for the various parts forming the content item. We want to do without placement so these zones are not going to help us.

Here is the placement for summaries:

<Placement>
    <Match DisplayType="Summary">
        <Place Parts_RoutableTitle_Summary="Header:5"/>
        <Place Parts_Common_Body_Summary="Content:5"/>
        <Place Parts_Tags_ShowTags="Footer:0" />
        <Place Parts_Common_Metadata_Summary="Footer:1"/>
        <Place Parts_Comments_Count="Footer:2" />
    </Match>
</Placement>

We can see here where each shape is supposed to go: title in the header; text in content; tags, date and comment count in the footer. We'll now reproduce the exact same markup, but without placement.

The first thing to do is to override the Content.cshtml template in our theme with a more specialized alternate, Content-BlogPost.Summary.cshtml. This alternate will be used when rendering a blog post in summary form.

As always when writing templates, Shape Tracing is our best friend.

On the left side of the shape tracing tool, we can see the parts that entered the composition of the content summary rendering, confirming what placement showed.

On the right side, the Model tab enables us to drill into the shape used to represent Content, which is what the "Model" will represent from our template.

For example, we can see that the title can be rendered with @Model.Title. We can drill further into the content item to find the body:

Similarly we can find the tags, creation date and comments. Some of those are less trivial to render than the title, and we'll have to copy and adapt from the existing templates for each of the shapes.

The following template will render the markup that we want:

@using Orchard.ContentManagement
@{
    if (Model.Title != null) {
        Layout.Title = Model.Title;
    }
    var bodyHtml = Model.ContentItem.BodyPart.Text;
    var more = bodyHtml.IndexOf("<!--more-->");
    if (more != -1) {
        bodyHtml = bodyHtml.Substring(0, more);
    }
    else {
        var firstP = bodyHtml.IndexOf("<p>");
        var firstSlashP = bodyHtml.IndexOf("</p>");
        if (firstP >=0 && firstSlashP > firstP) {
            bodyHtml = bodyHtml.Substring(firstP,
                         firstSlashP + 4 - firstP);
        }
    }
    var body = new HtmlString(bodyHtml); 
}
<article class="content-item blog-post">
    <header><h1>
        <a href="@Model.Path">@Model.Title</a>
    </h1></header>
    <p>@body</p>
    <p>@Html.ItemDisplayLink(T("Read more...").ToString(),
        (ContentItem)Model.ContentItem)</p>
    <footer>
@{
    var tagsHtml = new List<IHtmlString>();
    foreach(var t in Model.ContentItem.TagsPart.CurrentTags) {
        if (tagsHtml.Any()) {
            tagsHtml.Add(new HtmlString(", "));
        }
        tagsHtml.Add(Html.ActionLink(
            (string)t.TagName,
            "Search",
            "Home",
            new {
                area = "Orchard.Tags",
                tagName = (string)t.TagName
            },
            new { }));
    }
}

@if (tagsHtml.Any()) {
    <p class="tags">
        <span>@T("Tags:")</span>
        @foreach(var htmlString in tagsHtml) { @htmlString }
    </p>
}
    <div class="published">
        @Model.ContentItem.CommonPart.CreatedUtc.ToString(
            "MMM dd yyyy hh:mm tt")
    </div>
    <span class="commentcount">
        @T.Plural("1 Comment", "{0} Comments",
            (int)Model.ContentItem.CommentsPart.Comments.Count)
    </span>
    </footer>
</article>

The problem with this approach is that although we did without placement and still got the same rendering, there is a lot of repetition here, notably from part templates. This is inconvenient because if we start applying that sort of method everywhere, we are going to repeat ourselves a lot, which will eventually become a maintenance nightmare.

What we really want to do is to render the same shapes as before, and to let their templates do their job. These shapes do still exist but are not obvious to find.

As part of its job preparing the shape tree, the controller action that was responsible for handling the current request will have made a number of calls into ContentManager.BuildDisplay. Some of those calls were for the summaries of blog posts. BuildDispay calls into the drivers for each of the parts forming the content item. This is how the shapes for each part get created. It will then use placement to dispatch those shapes into zones. Those zones have not yet been rendered, and in our case never will, but the shapes are there, ready to be used. Shape tracing is not showing them, but they are under one Model.NameOfTheZone or another. Of course, we don't know what zone they are in, or at what index, so to find them we have to scan the model for zones and the zones for shapes.

I wrote a little helper to make that easier and added it to my theme's project file:

using System;
using System.Collections.Generic;
using System.Linq;
using ClaySharp;
using Orchard.DisplayManagement;

namespace Util {
    public static class ShapeHelper {
        public static dynamic Find(IShape model, string name) {
            var zones = new Dictionary<string, object>();
            ((IClayBehaviorProvider)model).Behavior
                .GetMembers(_nullFunc, model, zones);
            foreach (var key in zones.Keys
                .Where(key => !key.StartsWith("_"))) {

                var zone = zones[key] as IShape;
                if (zone == null ||
                    zone.Metadata.Type != "ContentZone") continue;
                foreach (IShape shape in ((dynamic)zone).Items) {
                    if (shape.Metadata.Type == name) return shape;
                }
            }
            return null;
        }

        private static readonly Func<object> _nullFunc = () => null;
    }
}

This is using some Clay wizardry to enumerate zones and then the shapes within them to find one with the specified name. Now with this helper, we can replace most of the code in the template and get something fairly clean and easy to understand:

@using Util
<article class="content-item blog-post">
    <header><h1><a href="@Model.Path">@Model.Title</a></h1></header>
    @Display(ShapeHelper.Find(Model, "Parts_Common_Body_Summary"))
    <footer>
    @Display(ShapeHelper.Find(Model, "Parts_Tags_ShowTags"))
    @Display(ShapeHelper.Find(Model, "Parts_Common_Metadata_Summary"))
    @Display(ShapeHelper.Find(Model, "Parts_Comments_Count"))
    </footer>
</article>

And this is it, we now have a very explicit layout for our blog post summaries, we are getting the same markup without placement, and the code is still nicely factored and maintainable. Of course, if you go that route, you will have to modify your templates every time you add a new part or field instead of relying on placement, but then again that's what you wanted…