Justin Rogers : Plug-In Framework

Implementing AI wars where code is the primary asset. (focus on Terrarium)

Justin Rogers — Tue, 28 Sep 2004 11:44:00 GMT

The one thing I've taken away from the .NET Terrarium in terms of AI development is that players value their code over everything else. Protecting code from being viewed by other users is extremely difficult and hard to implement. After all, the focus of a distributed AI game is to pass the intelligence all over the place, to multiple machines, so it can operate across a large number of machines and potentially become the dominant system. If the code leaves your machine, it becomes possible that other users are going to steal your code, plain and simple. You can try trusting some other environments, perhaps a central server that hosts your code, but even then your code exists somewhere that you aren't in charge of basic protection.

Code as an Asset:
The development of new AI in the Terrarium was driven by the decompilation and examination of existing code. To show you how pervasive code dissassembly was, we introduced an Animal Farm, or location where users could share code. Even with this central repository of code, we constantly saw users working in ILDasm and ILAsm to produce verbatim copies of existing AI. The existing code as considered an asset by not only the developer, but also by the rest of the players. Controlling that asset meant possibly controlling the Terrarium's game world. Code is such an asset that users will go to any length to get their hands on it.

What is the best way to protect your assets? Well, don't create a situation where your assets fall into the hands of other users. Why do you think ASP .NET is so popular and Windows Forms is still riding coat tails? People can protect their code assets when it comes to server side coding, but find it much harder to protect code on the client. This is a common problem. So if you want code protection, you need to host your assembly either at a central server level or from your own machine. If you follow these guidelines then users will be more likely to participate and generate professional level AI while still protecting their investment in programming the code.

Execution Time as an Asset:
We've identified three places to run the AI. You can operate at the local client level, at the server level, or at a distributed client level. Identifying execution time as an asset, load balancing across the greater distributed network of all clients is going to yield the best results in terms of execution time. The resources are apparently limitless as long as you can maintain a large user base. In reality, users aren't always connected. The only 100% up-time resource is going to be your server. You can run AI on the server, but then you run the risk of not having enough execution time to schedule the AI as the game becomes more and more popular. You can offload this risk by allowing users to host their own personal servers and implementing relatively simple queueing routines.

The most limited location to run the AI is on the local client. Each user that produces AI hosts a networkable interface through which their AI can operate. This is the most resource consumptive option because it entails replicating a large amount of game state down to the local client and the propagation of AI commands back up to wherever the game is being hosted. Your AI is bound by the amount of processor time you want to give it. Further, you can only host a single instance of your AI at a time (potentially more in a more robust hosting environment).

Up-Time as an Asset:
Looking at our three models again, the natural next step is examining up-time. If you are programming an AI it is truly an investment of time and effort, but should it also be an investment of your own computer time? In the distributed model it up-time becomes a non-issue as long as the server is available to server AI assemblies down to distributed clients that do all of the processing. The server model is fairly identical in that your AI is playable as long as the server is up and people are selecting it as an opponent. The problem with local client interfaces is that your AI is only around as long as you are. You have to maintain an interface for people to connect through.

Can we solve the up-time problem for local only hosting? Yes, if we trust someone else to host our code for us. This is a sticky point, but we can share a hosting location with a number of friends, family, colleagues where we each host an AI off of the same box. This box might be a web server or similar online resource, and once the AI is loaded, it now acts as an AI interface in addition to it's other roles. A third option is the pay for play option where you AI is hosted by the game originator for a small maintenance fee proportional to keeping their server up and running with your AI on it.

Penalties:
This is really important and something a lot of people don't think about. How many different ways can code hose a machine? Plenty right? You can get stuck in an infinite loop and never return, you can run a machine out of memory, you can screw up a static initializer, you write finalizers in your code to hang the GC. Who should be penalized for bad code? If you allow AI to run on the server or distributed clients then everyone is going to pay the price for bad code. Forcing code to run at the local client level means only the original AI developer is going to pay the penalty for bad code.

Can you get around all of those issues in server or distributed hosting? Sure, but I offer the following advice. Fixing the hosting problems comes in the form of adding constraints. These constraints are based on usage scenarios and known patterns of attack. They leverage existing code protection features as much as possible. In the end, you have a system capable of doing nearly all of the things the original system could do, minus a select few things that are considered unsafe. From the other end of the spectrum, you can explicitly enable features one by one. This direction is safer, but much more limited. It is akin to a custom scripting language because you have full control over what the underlying code is capable of. .NET doesn't currently offer good facilities to make this a possibility. You can't explicitly enable access to only certain classes, and disable all other access. You can't hook into the memory management functionality and suppress creation of objects once the component has reached it's limits. They may be available as Whidbey roles around in the form of hosting protections, but that will be a code-path much more difficult to code than the average .NET programmer will be able to handle.

At the end of the day the penalties should probably be placed right back on the AI developer. In a real game, AI developers are just as responsible for bugs as the rest of the development team. In fact, if there is a bug in the AI, it often takes down the entire game. In a developer game the model is switched because the AI now becomes an unknown and the AI developer may not bear the brunt of poorly written code. In fact, there are issues with developer code in the Terrarium that only manifest after the code has traversed several machines and is widely available when an odd code-path is run through.

I'm leaning more and more towards a local client hosting model for developer based AI games for a number of reasons. I think I've mentioned the potential network models that allow AI and Networked players to operate almost transparently by taking advantage of a network protocol. Now stack the asset protection on top of that along with the ability to remove advanced code hosting features and you have yourself a definite winner.

Multi-cast delegates are potential trojan horses for protected eventing...

Justin Rogers — Thu, 23 Sep 2004 06:42:00 GMT

I posted on some security options for eventing when you are using custom storage. While I stopped short of full examining the potential of the various systems, I also stopped short on pointing out some additional security concerns. Here is the previous posting: Some security considerations for systems with events.

The offending pointer was in regards to protecting the event model through type checking the Target of the delegates before adding them. Well, the code really isn't as accurate as it could be. What you have to realize is that all delegates defined in C# are multi-cast delegates. What in the hell is a multi-cast delegate? Well, it is a linked list of delegates with each containing a pointer ot the previous one. A reverse linked list would be a good way to describe it.

Why do they reverse it? Well, think about that for a minute. The most common way of combining delegates is to add a single new delegate to a long list of existing delegates. By taking the new delegate and setting its previous pointer to the existing delegate list, you've effectively reduced appending a delegate to a single operation.

Then how in the hell do you execute a reversed list in the proper order? Well, they bash the stack in order to make it work by recursively calling the invocation method back up to the beginning of the list. Once all of the methods have executed the stack unwinds down to the last entry and they are done. They also have to order the list correctly for GetInvocationList. For this they run through and get an accurate count, then fill the array from last to first.

So what in the hell did I do wrong? Well, by checking the Target I am checking the type of the last added delegate, but I'm not checking the types of all of the delegates. With that said, a user can hide multiple delegates that are invalid on the front end of a valid delegate. This would look something like:

EventHandler handler = new EventHandler(this.Foo);
handler = (EventHandler) Delegate.Combine(handler, new EventHandler(allow.Foo));
cea.Woot += handler;

The code is incomplete above, but the idea is to create an event handler pointing to your current instance, say an instance of BadClass, but then combine the delegate with another delegate pointing to an instance of GoodClass. Hard to believe that something so simple completely destroys our initial security model. You can get around this problem by fully examining the invocation list of a particular delegate and then adding only those delegates that are valid to your existing storage. Knowhing more about the storage, we can also use our own linked list to combine delegates together and then we can remove calls to GetInvocationList and perhaps eek some additional performance out of systems that rely heavily on eventing.

using System;

public class Runner {
    private static void Main(string[] args) {
        ControlledEventAttachment cea = new ControlledEventAttachment();

        Console.WriteLine("Firing");
        cea.Fire();

        Console.WriteLine();
        Console.WriteLine("Attaching");
        Allowable allow = new Allowable();
        allow.Attach(cea);

        Console.WriteLine("Firing");
        cea.Fire();

        Console.WriteLine();
        Console.WriteLine("Attaching");
        Hidden hide = new Hidden();
        hide.Attach(cea);

        Console.WriteLine("Firing");
        cea.Fire();
    }
}

public class ControlledEventAttachment {
    private EventHandler woot;
    public event EventHandler Woot {
        add {
            if ( value.Target is Allowable ) {
                woot = (EventHandler) Delegate.Combine(woot, value);
            }
        }
        remove {
            woot = (EventHandler) Delegate.Remove(woot, value);
        }
    }

    public void Fire() {
        if ( woot != null ) {
            woot(this, null);
        }
    }
}

public class Allowable {
    public void Foo(object sender, EventArgs e) {
        Console.WriteLine("Allowable");
    }

    public void Attach(ControlledEventAttachment cea) {
        cea.Woot += new EventHandler(this.Foo);
    }
}

public class Hidden {
    public Allowable allow = new Allowable();

    public void Attach(ControlledEventAttachment cea) {
        EventHandler handler = new EventHandler(this.Foo);
        handler = (EventHandler) Delegate.Combine(handler, new EventHandler(allow.Foo));
        cea.Woot += handler;
        cea.Woot += new EventHandler(this.Foo2);
    }

    public void Foo(object sender, EventArgs e) {
        Console.WriteLine("Hidden");
    }

    public void Foo2(object sender, EventArgs e) {
        Console.WriteLine("Not So Hidden");
    }
}

Some security considerations for systems with events.

Justin Rogers — Thu, 23 Sep 2004 00:54:00 GMT

For just a moment, relax your guard and don't think about the common usages of eventing that occur every day. The quick answer to solving any security concerns is to do a code review, run your application in a debugger to find offending code, and claim that since you own all of the source you don't need extra measures to protect yourself. I'm fine with that. This model is for plug-in architectures that allow eventing as a core messaging feature. We'll examine the concepts of messaging and break-downs in the system from a couple of different angles.

No Twosies
This is a prominent issue. User code identifies a location where they need to add an event handler for something and this location is possibly re-entrant. Normally, you might call this a mistake, but I think it is a mistake that can be easily rectified by some custom event code. Not only that, but user code fixes for these issues require that the offending code be replaced, a state variable protect whether or not the event has been added, or finding the proper location to remove the event. All of these can be difficult compared to the custom event code.

add { customEvent = (EventHandler) Delegate.Combine(Delegate.RemoveAll(customEvent, value), value); }

We end up using removal code to ensure that the delegate doesn't already exist before adding the new delegate. We can also use locks to ensure multi-threaded protection during these operations.

First Come, First Serve
Another prominent issue. If you always know your event is only going to be consumed by one object at a time, then you can easily protect yourself from adding multiple handlers. This is generally found in places where it only makes sense for one piece of code to handle an event. Have you ever seen what the effects of 10 paint event handlers? Again, you might call this poor code, but each time you want to paint a new glyph, you might think it smart to add a new event handler while the option is checked in place of yet another branching block. Let's ensure we keep it to one at a time.

add { if ( customEvent == null ) { customEvent = value; } }
remove { if ( customEvent == value ) { customEvent = null; } }

What is the Password?
The majority of the events you see are communal beasts of multi-casting, but you may not always want someone to be able to add themselves as an event source. Here is a game related scenario that might be of interest... As AI players walk around the world, they look for event sources. These sources are objects in the world that might perform some actions and they give notification before they do. A naively programmed AI tries adding event sources that don't have any impact on them, they simply aren't in the effect zone, and so that AI builds up more information about the world and uses more resources than the other AI. On the same note, pushing the requirements of adding events to the eventing zones themselves, means that AI not even interested will be notified and thus again resources are wasted. The solution is to enforce some sort of password.

private event FireEventHandler BelchFlames {
    add {
        Player p = value.Target as Player;
        if ( p == null || p is Mech ) { return; } // Only add players and non mechs that are affected
        belchFlames = (FireEventHandler) Delegate.Combine(Delegate.RemoveAll(belchFlames, value), value);
    }
    remove {...}
}

Get Out of Here!
Another odd moment in the life of an event is when you realize that when it fires and when the delegates are called are two different times. The rationale here is another game concept. While the game engine needs events from each game object to schedule things to happen, AI components need to run in a specialized sand-box, timed, and most likely run at a different time from when the original event first fired. One of the larger mistakes in AI programming is nesting scripts too deeply and events are a form of nesting. An example would be a group leader subscribed to picking up base communications that in turn relays the information to the group participants. That is three levels already and you might not be prepared for all that mayhem when the event is fired.

add {
if ( value.Target is GameEngine ) { gameEngineSink = value; }
else { fullSink = (EventHandler) Delegate.Combine(fullSink, value); }
}

Notice how we collect events from different sources (by examining the Target) into different collections. The game engine, when it wants to be notified of a particular event, will be placed in a specialized holder, while the rest of the objects get collected into a secondary holder. We'll pass the secondary holder off to the game engine at some point and allow it to call each of the objects from within the timing sand-box.

I'm Tired of Waiting!
While the above solves locality issues, it doesn't necessarily solve concurrency issues. While running AI code in the sand-box is definitely a great thing to do, we may also want a bunch of code to run in response to an event while we finish some other processing. You can toss individual delegates off onto separate threads. These are single-cast, non-combinable delegates that you get from calling GetInvocationList. You have to be far more careful with respect to thread-safety and the operations in the delegates really need to be designed so they can run asynchronously while the rest of the application continues on.

private void OnMessageReceived(MessageInfo mi) {
    if ( messageReceived != null ) {
        Delegate[] callbacks = messageReceived.GetInvocationList();
        for(int i = 0; i < callbacks.Length; i++) {
            ThreadPool.QueueUserWorkItem(new WaitCallback(RunMessage), new Package(callbacks[i], mi));
        }
    }
}
private void RunMessage(object state) {
    Package p = state as Package;
    if ( p != null ) { p.Callback.DynamicInvoke( new object[] { p.MessageInfo } ); }
}

Now, as messages are received, we'll quickly dispatch them off to other threads to be handled. If the other threads are running neural networks, doing character recognition, or running variations of a specific algorithm, they can update their internal state and return the threads to the pool after they are complete. This never stops the rest of the game from doing its thing and back-ups in the thread pool can result in some auxillary code to throw out messages until things catch up.

Quit Throwing Stuff at Me!
The code running in an event handler can crash and burn just like any other code. If this happens during a normal event invocation, you may not process other, operable events. One option is to break out the invocation lists and call each delegate safely, while removing the offending delegates that are causing problems. I think this is an excellent pattern, especially in plug-in architectures where you expect plug-ins to surface errors through exceptions since they have no other manner of indicating an exceptional situation.

A secondary issue would be disposed objects. A Form for instance can be in an unusable state, but still be the target of a delegate. Since most event oriented code is terrible about removing or unhooking events (where do you put the code, did you put it in the right place, does it cover all code-paths that lead to an unusable object, is the remote object still alive even) it might be smart to protect against such a common mistake.

private void OnFilterImage(Bitmap b) {
    if ( messageReceived != null ) {
        Delegate[] callbacks = messageReceived.GetInvocationList();
        for(int i = 0; i < callbacks.Length; i++) {
            try {
                callbacks[i].DynamicInvoke(new object[] { this, b });
            } catch {
                messageReceived = (BitmapHandler) Delegate.Remove(messageReceived, callbacks[i]);
            }
        }
    }
}

Now, if you have an error in a single filter, the rest won't affect you. You can even notify the user in the catch block, and ask if they would like to proceed with the rest of the events, especially if this is a run of sequential filters that all need to complete to achieve a desired effect. However, if you've been processing for 10 minutes, and this is the last filter, you may just want to get it over with and finish the process.

All of these considerations are for systems where events are considered important. You can fix each of the issues in turn by removing events and using another pattern entirely. In fact in most cases I don't use events myself. The direction I'm taking here is that of a highly dynamic plug-in oriented architecture, where eventing solves many information and messaging problems without a lot of code.

Terrarium: Did we do the right thing with the creature event model?

Justin Rogers — Sun, 05 Sep 2004 00:51:00 GMT

History
Why did we use the eventing model? Well, because it made sense that creatures only needed to spend processing time on things they were curious about. It seemed like it might make things easier because a basic creature could hook only one event while a more complex creature could hook many events. It was also a great demonstration of how you would build pluggable apps within the .NET Framework. However, at some point we broke the model a bit and added some virtual methods for serialization and deserialization of creature data.

At no point did we consider revisiting the model and changing it out for another model that might be more appropriate. The backwards compatibility mindset was always in place and we didn't want people to have to recode their creatures entirely as versions changed. It also became rather difficult to change the pattern away from the 10 or so events and large eventing infrastructure that was in place. With that in mind we kept the pattern.

Pattern
Our pattern was to simulate the concept of discrete inputs. Certain things may or may not happen in a given round, and other things may not happen for the entire life-time of the creature. No reason to check for these conditions every tick of the game loop. We created several separate events that comprised certain worldy actions such as attacking, defending, eating, etc... We also needed the creatures to be notified even if nothing of importance happened. This ensures the creature gets some time to process irregardless of the state of the world. These are the Load and Idle events. The Load happens before all other events are called, while the Idle happens after all other events are called. Load allows you to set things up and prepare for action, Idle lets you operate after world notifications have been propagated. Easy enough right?

Complexity
We definitely made some mistakes. First, there are a lot of events. Many creature authors are curious about which events they need and the order the events happen in. We have some documentation, but that doesn't help when you are first picking up the Terrarium for the first time. You want things to be as plain and simple as possible. Another level of complexity is added in that many developers wind up doing work in each event that may or may not overwrite work done in other events. You could blame this on the person writing the code, I rather blame it on the model and blame myself for allowing it.

Additionally there is the problem of storing information about discrete events having happened. The author could store the information, but we already store it within the base class. This isn't immediately obvious. I mean why would we store the information for you and give you an event that gives you the same information? It makes more sense for you to be notified that it is your turn to process and then allow you to investigate the state of the world.

What about time-slicing? Since we time-slice you, each event we call is taking a bit more of your time. Additionally it is harder to load balance your time across ten events than it would be across a single event. You have to store the amount of time spent in each of your events in order to get an accurate measure of how much time you used and how much you have left to process. For path-finding algorithms you may want to make the most effective use of your time by knowing about how much time you have left, even if the measure is somehow relative.

No Guarantees
We don't guarantee that your events will fire. In fact, we can skip you if you are taking too long, and the only way you know is by examining a special property that tells you for how long you were skipped. Any in process events continue processing, so while you aren't being notified, your move may complete or you may get teleported. If you are relying on the eventing model to keep your private state in check you are in for a rude awakening.

Single Cast Events
Having single-cast events isn't very useful in our model as you can tell from some of the downsides. In fact, in trying to demonstrate a plug-in model, we desperately failed. Normally, plug-ins will hook up to some global events that exist on the game engine and not exhibit a bunch of local events that are only consumed by the plug-in.

Fixing the Model
The model is quite easy to fix. Simply don't use events. There is no reason to use events in our model, rather with a base-class already in place we simply supply a single callback method, virtual of course, for the creature to implement all logic. This drastically reduces the complexity of the code required to implement the creature into a single entry point program. While you might think that authors couldn't code creatures that were nearly as functional as before, they actually have more time to operate now and have a less complex model to program against. As I'm making these changes to the source it simplifies other portions of the engine and helps refine the layout of the interfaces between the creature base, the engine, and the UI.

Working With What You Have
You have some options already as a creature developer. By writing a simple base class that manages all of the events for you and stores local information in a very minimalistic format, you can write your derived creatures to use the new local information from within a single method that is fired by the Idle event. At one point we had a work-item open to ship this class within OrganismBase itself. The idea would have been to have the Animal class for anyone interested in programming the old model, but also a new SingleCallAnimal supporting the new, simpler syntax.

.NET Immutability Tip #3: Protect your properties AND your methods.

Justin Rogers — Mon, 03 May 2004 23:15:00 GMT

A common immutability practice is to simply protect the property setter with an immutability flag. Take a simple class that has a single integer field for to back our property and a single boolean flag to mark it immutable.

using System;

public class SimplyImmutable {
    private int field1 = 0;
    private bool immutable = false;

    public SimplyImmutable() {
    }

    public int Field1 {
        get { return field1; }
        set {
            if ( !immutable ) {
                field1 = value;
            }
        }
    }
}

As soon as the immutable flag is set, you can no longer set the field value. This is nice. Time to add a few methods. First we need a MakeImmutable method. This will set our flag. Normally, our application will set this before giving any other code access to the class. Second, we'll add an IncrementField method, but we'll make a mistake, so see if you catch it.

public void MakeImmutable() {
immutable = true;
}

public void IncrementField() {
field1++;
}

See the mistake yet? We accessed field1 directly, and didn't use the immutability flag. This problem goes away if you use the property instead of the private field or if you use the immutability flag. In this small example the problem is easy to spot. However, in a larger class with several properties, many backing fields, and maybe 10 or more methods, you can fail to find issues early in the game. The gem here is to use the property inside of our methods so that we don't have to write a bunch of extra code in each of them to handle immutability.

public void IncrementField() {
Field1++;
}

What happens if you are setting the Field1 property multiple times? You start to incur the wrath of the flag check on each set. If this happens, then go ahead and move your immutability check into your method and then convert over to using the private field again. Note if you do this process by first using the property and then only later converting over to use the private field and the immutability flag, you can do it on an as needed basis and ensure that you are protecting yourself.

public void IncrementField() {
if ( !immutable ) { // Work with field multiple times, if not then just use the property instead }
}

This tip has many natural extensions into other gotchas that I'll cover later. Think for a bit about making our fields protected so they can be accessed from derived classes, and how that might introduce resource access issues. I'll be examining this abstraction in the next tip.

[Terrarium] How do we pass creature assemblies around?

Justin Rogers — Fri, 23 Apr 2004 23:56:00 GMT

This is the primary feature of the .NET Terrarium. The ability to safely and easily pass around creature assemblies and run them on end user machines. There are several considerations for this process including security, transport, loading, and storage. I've already talked about the Terrarium Model and security over in the Plug-In Framework (Foreward): Defining the end goal, so I'll be talking about the later three now.

Transport
Transport can be very easy (WebRequest), or fairly difficult (Socket). Today, you can easily use web services or web requests to grab information off of a central server, and the Terrarium does this all the time. We have several services that go out and grab extinct creature assemblies off of the web server. This is probably the easiest way to get a creature assembly.

The P2P methods are a bit more difficult. We actually created a miniature web server HttpWebListener (thanks to Lance Olson) and use that for P2P interaction. So are we really using Socket? Yeah, but not the way you would think. Each client still issues basic WebRequests for data from each peer, but we host those little miniature HTTP servers in order to intercept the commands. This is actually pretty cool and I'm sure everyone will enjoy having the source code at their fingertips for this little gem pretty soon.

Since HTTP is a stateless protocol I can tell you now that we have to implement a message based system whenever we communicate with other peers. All of the messages originate on one side of the pipe (the client) and will change depending on the response from the listener. This is a fairly easy process, made quite complicated by requring multiple connections, one for each message, but overall it isn't that bad. Sample messages would be, do you have assembly A? Here is assembly A, did you get it okay? Here is a new creature, are you receiving it okay?

Storage
Now that we have the assemblies getting tossed around we have to store them somewhere. We can't store them in the same place as our executable. We can't store them in Program Files and still act under “Run as Normal User”. We can't really store them anywhere, except for the users local settings. Ah, we found a spot. Since we have to store assemblies off in some remote directory, how do we go about loading creature assemblies when they are requested?

That is where our assembly caching logic comes in. The basic process starts inside of the game logic. You see the game logic knows about the assembly caching logic, and so rather than calling an Assembly.Load on a creature's full name, we simply call into your assembly cache and pull out an assembly. This works great. However, there are still times when an Assembly.Load might get called outside of our code (deserialization) and we need to be aware of that so we can return the assembly when probing fails. For that we hook AssemblyResolve, verify that the assembly being requested isn't one of our own, and finally look things up in the assembly cache.

There is one hang-up here. We never make use of private paths, so the run-time doesn't control the probing, we do. We get a lot more storage potential out of this, including monitoring how many assemblies are currently loaded, how many are in the assembly cache, and finally the ability to run clean-up logic if we want to. We also don't automatically download the assemblies if they aren't already in the assembly cache. We could make a web request and search for the assembly on the server, however, we rely on other game logic to ensure the assembly is there.

To note some additional security we've layered on our cache. Each time the Terrarium starts, the cache directory is dynamically changed to some random directory name. No brute force, I know where your assemblies are hidden, attacks. This prevents users from storing things in assemblies and then taking advantage of them later through email, URLs on their website, or any number of other hacker routes.

Loading
This is the most complex aspect of assembly use that we have. Not only do we have many layers of code access security protection, but we grep the assembly IL for bad constructs, load the assembly in multiple ways to prevent context caching of assemblies, and a few other goodies. I've detailed the code access security items and IL grepping within Plug-In goals document, so you can see the real deal there. I'll talk a bit about loading assemblies for the first time and the process they go through.

Loading an assembly for the first time is a pretty big deal. We obviously run the IL grep over the assmebly before we do anything, but once that process happens, we have a number of other checks that have to be run. Namely, we have to get a bunch of assembly attributes off of the type, make sure the image isn't bad, etc... Simply enumerating the IL won't give us this full verification that everything is kosher, and we don't want to lock the file on disk. So we use the byte[] overloads of the Load method. The method supports loading of just the assembly, or what most people don't know, debug symbols as well. Took some time to get all of this code written correctly to allow all of that precious debug information to make it to it's final location in the assembly cache.

Once the assembly is verified, we check it against the Species service to make sure it is unique, add it to the list of creatures, and store it's assembly remotely. If this is a new creature for sure, we save the assembly into the assembly cache (copying symbols if necessary), and load the assembly again. This time for real, since this will be the assembly codebase that we use for creating creatures.

The entire process is fairly quick. Later as the application shuts down and restarts, we'll be responsible for loading creature assemblies over and over again. They'll always come out of the assembly cache from there on out. Some hang-ups do occur. If the original assembly (before copy) was located in the probing path, a call to Assembly.Load by the serialization code might load the wrong assembly. We prevented this by some ingenious serialization, but I'll talk about that later. You also have to make sure all of your code loads from the assembly cache as well, and doesn't call Load, else that probing issue will catch you sooner or later.

[Plug-in Framework] Adendum to the goals document

Justin Rogers — Tue, 20 Apr 2004 01:12:00 GMT

I've added the Terrarium model and World Builder model definitions. The Terrarium model is based on the hardships in implementing the code hosting technology for the .NET Terrarium and the World Builder model is focused on slightly improving on the Terrarium model by extending the IL Verifier and talking about the use of factory types to manage memory allocations for types.

Plug-In Framework (Foreward): Defining the end goal

.NET Immutability Tip #2: Be careful of unprotected types in the executable.

Justin Rogers — Sun, 11 Apr 2004 05:48:00 GMT

I've seen this happen quite often. When developers create their immutable types, they assume the only view the user will ever have is of the type through some interface. Normally the interfaces are defined before-hand and included in some library that the user is supposed to link to. This can easily be demonstrated by mocking up an IWorldView interface and then using this in our user code.

// Interfaces.dll
public interface IWorldView {
    int WorldSizeX { get; }
    int WorldSizeY { get; }
}

// UserPlugin.dll
public class UserClassConsumingIWorldView {
    public void EngineCallMe(IWorldView iwv) {
        // do some work
        int area = iwv.WorldSizeX * iwv.WorldSizeY;
    }
}

From the code above, you would assume that the user sees the world through IWorldView goggles. That might not be the case. When they develop their plug-in they have access to your executable as well. While some .NET languages won't let you link against executables, C# will. This can lead to some access elevation in that the user might be able to case IWorldView to a type with more access.

using System;

// Game.exe
public class WorldView : IWorldView {
    private int _worldSizeX;
    private int _worldSizeY;

    public int WorldSizeX { get { return this._worldSizeX; } }
    public int WorldSizeY { get { return this._worldSizeY; } }

    public void ResetWorldSize(int x, int y) {
        this._worldSizeX = x;
        this._worldSizeY = y;
    }

    private static void Main() { }
}

Woops, we defined our WorldView class as public making it available to the outside world. Now the UserPlugin can cast to this guy and make use of ResetWorldSize().

using System;

public class UserClassConsumingIWorldView {
    public void EngineCallMe(IWorldView iwv) {
        // do some work
        int area = iwv.WorldSizeX * iwv.WorldSizeY;
        ((WorldView) iwv).ResetWorldSize(0, 0);
    }
}

You won't make this mistake twice, believe me. Making the WorldView class internal will fix any issues with this possible casting and put you on your way. In this case we assumed immutability of the world view because we thought we had control over the user's access when we really didn't through a common programming mistake. Users in the VB world might never recognize this error, since they wouldn't be able to include Game.exe as a reference to begin with.

Joel's Lightweight Code Gen spells SUWEET for small scripting languages in games.

Justin Rogers — Sat, 10 Apr 2004 18:46:00 GMT

Reading Joel's blog and having lunch with him are two different things. You never really see all of the possibilities of a technology until you see the twinkle in someone's eye and realize that the technology might be slightly more powerful than you originally realized. Today I want to cover function based scripting languages and how these can easily be implemented using the LCG inside of any existing MUD or MMOG system.

The syntax for small scripting languages is not similar to what you see with C# and VB .NET. Small scripting languages are generally functional and not necessarily centered around the creation of objects. In the game world, a developer might create new functions in order to perform specific actions. Each of these functions is then attached to a game world object and fired as part of an invocation list as needed. Code is not used to define everything in the game world. Invocation lists might be stored in a data file with a list of object id's, and their attached function id's. Object's might not even be hard objects, they might just be scalar data or structs. The data something contains, might also not readily identify it's object type, but instead the functions it handles. Let's take a sample case.

> You see a mossy rock in front of you.

Mossy rock? Why is it mossy? Why is it called a rock? You might think someone created a mossy rock object (MossyRockObject : MUDObject) somewhere and compiled it into the game. This is probably not the case. More likely is that they attached the GetName function and that function is diving into the database or some other data file to get the name of the object. Is mossy even part of the game? Or is that something else? Maybe the GetName method is calling a GetAge function or the game engine is calling a GetAge function. In this case the possibilities of getting a name could chain together a large number of similar functions that all return pieces of the objects name.

// Note all functions are assumed to turn return an Object data type and the
// engine is capable of handling the rest. I'm making this very JScript like.
function GetName(ObjectID) {
var name = DatabaseName(ObjectID);
return name;
}

Now, I'd actually have to attach that function after I made it. Each function could be in it's own file. I could store them in XML. I could pre-parse them into some byte-code type structure that I could quickly convert to IL later. Lots of options. So we add a command to our MUD or MMOG that allows us to attach a function to an object. Remember that mossy rock? Well, in our case we simply told the game system to create a new MUDStaticObject. This object is derived from MUDObject and has properties that make it act as a static object within the game world (aka a rock isn't a mobile). Once we create the object, we have an ID. The UpgradeObject {id} {function name} is now capable of attaching a new function. The system is loose. The new function is now something that can be called by name, or it can be added to any of the objects invocation lists. An invocation list is important, because we want the game engine to be able to generically support something like name's, but not necessarily only through the GetName function (note we could have allowed many instances of the GetName function, one per object, but I'm choosing invocation lists instead ;-).

An invocation list for our object looks something like, invocationListNames = { functionIdGetName }. Note there is only a single ID when grabbing a name. We might not even call this a list but instead an invocation bind point so that we have the ability to limit the number of functions for this particular feature to just one. If we wanted to upgrade our name from say “rock” in the database to “mossy rock”, we just swap out the methods.

function GetExtendedName(ObjectID) {
    object = GetObjectByID(ObjectID);
    name = object.GetName(ObjectID);
    age = object.GetAge(ObjectID);
    prefix = “shiny “;

    switch(age) {
        .... // Change prefix
    }
    return prefix + name;
}

That was easy. Dynamic methods make the above possible. My MUD can take textual input through a world builder, create this new method, and then at run-time, when I compile the method I can convert my script to the appropriate IL to make things happen. In the case above, I'd still have to use UpgradeObject to attach GetExtendedName and then add it to the invocation bind point for grabbing somethings description. That isn't a big move. Then we use some global functions GetObjectByID to surface a structure capable letting us call methods that we think might exist on our object. We could have just called GetName, but what if that function didn't exist on the object? With the object. syntax we are adding some checks in there to make sure that method has been attached. If not you can exit out of the script or do whatever else is necessary.

I'm getting pumped guys. This is the kind of interface I've been waiting on for a long time. It is the type of interface that will allow Terrarium to become more functional and provide an easier to replicate sand-box. It will certainly create an easier to create plug-in framework environment. Whidbey is Whidbey though, and that means some time before you get to produce production level apps on this stuff. I'm going to try and go back and fulfill a promise I made months ago to actually complete my plug-in framework documentation that would allow you to create strong plug-in environments today.

References:
[1] A fun lunch with a couple of Rotor fans
[2] hello, world... LCG (Lightweight Code Gen) style!
[3] Late-bound invocation notes - CallVirt, Delegates, DynamicMethod, InvokeMember.

Note that I didn't actually cover 3. However, this article provides an EXTREMELY fast solution to some common scripting problems. I'll try and blog about why later when I get more time.

.NET Immutability Tip #1: Nothing is immutable.

Justin Rogers — Fri, 09 Apr 2004 05:24:00 GMT

I figured I'd start with the obvious. You can never control a machine 100%, so there is always the opportunity that whatever systems of protection you have in place, they can be overcome. This same principle applies to security and cheating systems as well.

Things start to become mutable in .NET under several circumstances. There are always the platform methods of interacting with memory, for instance grabbing the physical address of a variable and setting the data on it. Then there are the debugger tools. These make the process of interacting with memory a bit easier, and if you are being run under a debugger, you can throw immutability out the window. Finally, .NET offers private reflection. I'd say the majority of code that gets run, is run under security that would allow private reflection. Maybe I should lock my dev machine down a bit more, but I've even grown to rely on various private reflection techniques to gain performance improvements in my app (see ImageFast).

Even saying all that, under normal circumstances the paradigm does calm down a bit. The system does allow us to protect our data and gain immutability of sorts through visibility modifiers, denial of advanced code execution and reflection rights, usage of the readonly flag, and special design considerations. The remainder of the tips in this series will focus on ways to ensure immutability given normal circumstances rather than assuming extraordinary circumstances. I'll try to base the tips on usable scenarios such as the plug-ins, general software design, and gaming logic in addition to provide thorough code samples describing each technique. (note: if I get enough response to republish the tip series in VB, I will. I plan on aggregating all of these tips together into a single article location anyway, and providing code samples in multiple languages will be easy)