CodeSmith Generator 6.0–Status and IntelliSense

Work on Generator 6.0 is progressing pretty nicely. We are code complete on our new parser, new compiler, new caching mechanism, out of process template execution, and WPF based property grid. There is a lot to talk about there, but I have also started working on implementing IntelliSense and I wanted to talk about that today.

Better working IntelliSense has been our biggest request probably from day one. People tend to take IntelliSense for granted because Microsoft has done a great job with it in Visual Studio and it almost becomes transparent. We just use it and expect it to work without thinking about what goes into it. I was just as guilty of this as everyone else until I started working on implementing it in CodeSmith Studio. Think about it for a second, you have a collection of text files and you need to make sense of them… what goes into that? Here is a simplified list of the steps and parts required for implementing IntelliSense.

• Tokenize – the first step is to take that text file and create a series of tokens from it.
  • This sounds easy, but in and of itself it can be pretty hard because of ambiguities in languages where certain things have different meanings depending on their context.
  • In our case, things are even more complicated by the fact that we are mixing random languages together.
  • Also, keep in mind that this needs to be happening in real time as the user types without slowing the editor down.
• Parse – next we read the tokens in, make sense of them, validate them and construct an abstract syntax tree along the way so that we can have some sort of structure that can be more easily understood, traversed and transformed.
  • This is expensive to build and needs to happen in a background thread and constantly be updated as the user is typing.
• Gather Type Information – we need a database of reflected and parsed type information from any referenced assemblies and source files so that we can use that information in resolving and knowing what members are available for any given type.
  • It is expensive to reflect over every single type in every single referenced assembly. So you need to cache this information in a structure that makes it really fast to load and search.
• Gather Context – when an IntelliSense request is made, we need to figure out where we are and provide as much information about our surroundings as possible.
  • Due to the fact that active typing is going on, we have to use a combination of token scanning and looking at our most recent AST to provide this information.
  • Also due to active typing, we have to work with an AST that has information that is slightly out of date and we will need to translate that out of date information to the current source.
• Resolve – now we need to take all of the information that has been gathered and use it to figure out what we are looking at and what options are available to show in the IntelliSense list.
  • There are a lot of rules that come into play here like member scope and precedence for extension methods to name a couple.

Another thing I would like to address is that many people in the past have said to us… "just use Microsoft's IntelliSense". Unfortunately, it's not that easy. Microsoft does not expose their IntelliSense services and, even if they did, we most likely would not be able to use them because they would be very much geared around their use cases which would make them unusable for ours.

As you can see, there is a LOT of work that goes into providing that handy little drop down menu that magically pops up and makes it so that you don't have to memorize an entire framework library. We are completely re-writing every single one of those pieces in CodeSmith Generator 6.0 and our end goal is to make it so good that you completely take us for granted because it just magically works.