Future Orchard Part 1: Introducing Tokens

After a long phase of cleanup on the new Orchard 2.0, we are now busy designing new features. We are focusing on a few foundational pieces, and on enabling e-commerce on top of the platform. In this post, I'm going to expose the basics of the preliminary design for one new foundational piece: Tokens.

You could technically confuse Tokens with a fancy String.Format, or with a very lightweight templating solution, but you'd be slightly wrong in both cases. Its usage is what sets it apart from both. You will use tokens whenever you need to build a string by inserting named environmental variables into a string that has placeholders. That is it. No code, no loops, no ifs, just formatted substitution, but really the keyword here is environmental, and by the end of this post it should be clearer what we mean by that.

Let's start with trying to implement Mad Libs. If you have the following Mad Lib:

_____________! he said ________ as he
  exclamation            adverb

jumped into his convertible ______ and
                              noun

drove off with his ___________ wife.
                     adjective

Somebody could pick at random: "Praise the Flying Spaghetti Monster", "vigorously", "left nipple" and "smelly", and we would get the following weird sentence: "Praise the Flying Spaghetti Monster! he said vigorously as he jumped into his convertible left nipple and drove off with his smelly wife."

What's fun and creative about it is that the "format string" and the tokens that go into the placeholders come from different persons, and none of them know in advance what the other will write.

Similarly, Tokens in Orchard 2.0 can come from many sources, and the list is fully extensible. The way you use the API is by first injecting an ITokenizer dependency. Once you've done that, you can call Replace to start formatting strings:

_tokens.Replace(
  @"{MadLib.Exclamation}! he said {MadLib.Adverb} as he
    jumped into his convertible {MadLib.Noun} and
    drove off with his {MadLib.Adjective} wife.",
    null);

Without doing anything else, this will only yield a rather meaningless "! he said  as he jumped into his convertible  and drove off with his  wife." Being used to String.Format as you are, you may be a little frustrated by this and be tempted to directly inject the values for the tokens as an additional parameter to Replace. After all, that null does look a little suspicious. Well, if that is what you want, you should probably continue to use String.Format. But this is not what we're trying to do here. What we want is for whatever other player is available to provide us with values to inject. This is where the second part of the puzzle appears: token providers.

public class MadLibTokenProvider : ITokenProvider {
  public void Describe(DescribeContext context) {
    context.For("MadLib", T("Mad Lib"), T("Random words for Mad Libs."))
      .Token("Exclamation", T("Exclamation"), T("A random exclamation."))
      .Token("Adverb", T("Adverb"), T("A random adverb."))
      .Token("Noun", T("Noun"), T("A random noun."))
      .Token("Adjective", T("Adjective"), T("A random adjective."));
  }

  public void Evaluate(EvaluateContext context) {
    context.For<IMadLib>("MadLib", () => new MadLib())
      .Token("Exclamation", madLib => madLib.GetExclamation())
      .Token("Adverb", madLib => madLib.GetAdverb())
      .Token("Noun", madLib => madLib.GetNoun())
      .Token("Adjective", madLib => madLib.GetAdjective());
  }
}

public class MadLib : IMadLib {
  public string GetExclamation() {
    return PickOneOf(
        "Praise the Flying Spaghetti Monster",
        "Holy rusted metal, Batman",
        "!@#$%");
  }

  public string GetAdverb() {
    return PickOneOf(
        "vigorously",
        "cheerfully",
        "alarmingly");
  }

  public string GetNoun() {
    return PickOneOf(
        "left nipple",
        "rocket",
        "jacket");
  }

  public string GetAdjective() {
    return PickOneOf(
        "smelly",
        "pink",
        "absurd");
  }

  private string PickOneOf(params string[] strings) {
    return strings[(new Random().Next(strings.Length))];
  }
}

public interface IMadLib {
  string GetExclamation();
  string GetAdverb();
  string GetNoun();
  string GetAdjective();
}

The Describe part is pure metadata that will be used to provide users hints on what tokens can be used. The second part, Evaluate, is doing the actual evaluation of the tokens. It's accepting tokens of type "MadLib" and of names "Exclamation", "Adverb", "Noun" or "Adjective". The specific implementation should be fairly simple to understand as it's just picking at random in lists of strings.

Because we now have a token providers that knows how to handle all four of our Mad Lib tokens, and because our provider is picking them at random, we have recreated the fun of Mad Libs.

Decoupling the token values from the formatting site is more than a cosmetic improvement. Now our tokens are available everywhere. I can reuse the MadLibTokenProvider for other Mad Libs of course:

_tokens.Replace(
  @"My {MadLib.Noun} is {MadLib.Adverb} {MadLib.Adjective}.",
  null);

Or I can reuse them to add some fun to an e-mail notification, although the notification module knows nothing of Mad Libs (note that the string wouldn't be hard-coded but rather come from configuration):

_tokens.Replace(
  @"{MadLib.Exclamation}, {Site.Admin.FullName},

A new {MadLib.Adjective} comment has been published on
{Site.Name} by {Comment.Author}:
{Comment.Text}

<a href=""{Comment.AdminUrl}"">Click here to moderate</a>.",
  new {Comment: newComment});

In addition to this mix of tokens from various providers, something interesting and new is happening here. We have a number of tokens that are just environmental, such as Mad Libs, the site name, and the full name of the site administrator, but the comment object is provided as local context at the formatting site.

You may have noticed how in our Mad Lib token provider, we provided an expression that creates a new MadLib object when evaluating token values:

context.For<IMadLib>("MadLib", () => new MadLib())

This defines the default, or global, or environmental MadLib to use, but if you needed an IMadLib implementation that was choosing in a different list of words, for example words that actually make some kind of sense in this context, all you would have to do is override the default by just providing it in your context object:

_tokens.Replace("...my Mad Lib...",
  new {MadLib: mySpecialMadLibVocabulary});

This new special object would become the argument to the Lambda expressions for the token evaluation, and it will just work, without a change in the token provider.

In summary, token providers are exposing a vocabulary of tokens that can be used everywhere with the guarantee that any given token is always going to work, even though the values themselves can be global or contextual. The feature effectively gives users a predictable and stable vocabulary that they can use in dynamically populated strings.

Next time, I'll show some more subtle features of the new Tokens API, such as how to build deep graphs of tokens.