June 2003 - Posts
James Avery blogs about an article written by a person who calls himself 'Angry Coder' (I thought people called me that ;)) and that article ventilates the opinion that code generators are not useful, to say it in a politically correct way. I agree with James Avery that the article misses the advantages of code generation and how it can benefit you in your daily life. I'd say: the article is totally wrong. Here's why.
In 1993, a professor of the CS department I was a student of held a lecture about code generators. In short it came down to the fact that typing code statements is time consuming, boring and error prone, and has nothing to do with programming, which is defining algorithms, data-structures and functionality specifications, in a language independent way. If you can transform the actual logic, the functionality you want to implement, from a higher abstract level to code which can be compiled by a compiler into executable format without having to type it in, it would be a real time saver plus would give you (almost) error-free software.
Code generators do that. They let you define functionality on a higher abstract level and do the typing for you. LLBLGen v1.x (the open source, free version) f.e. generates code to do the CRUD operations on an SqlServer catalog (C#/VB.NET/t-sql) and some more. You then do not have to type in that code, you can be sure it works and you can go on with what's really important: writing Business Logic (BL).
Even in that area, the days of typing code are numbered, at least for the majority of applications. Look at BizTalk server. It lets you design your BL with standard code blocks. In other words: it generates the glue between default blocks that can be reused. If you want an extra piece of functionality that is not there, you can add it or buy it. Adding it can force you to type some code, but not much. "People who do not understand that code generation is the future will be the first who will not be able to keep up with the rest who do", I said in the comments on James' blog. Print it out on a piece of paper and glue it on your monitor. Code generation is the future, simply because it saves time, is less error prone and it saves the programmer from boring coding sessions, sessions which will result in more bugs than interesting coding sessions.
I really feel sorry for those people who do not understand what power code generation really has. They are doomed to do boring work which can be done by computers in a fraction of the time without any error. Or does the author of that visionary article think he can program a DAL + BL facade tier-set for an application which has 100-s of tables, without errors? It did cost me 2 months to do it for my upcoming LLBLGen Pro application, to develop the framework and the code which is used in the templates. The generator can do it now in 2 seconds. Without 1 error. I definitely know what I'd do if I had to choose between the two :).
Code generators are tools, like a compiler, editor or debugger. Good tools help the user, so a good generator will help the user of that generator. It should be: which generator will I use?, and not: shall I use a generator or not?, like it is already natural to choose a higher order language or a sophisticated editor over their lamer equivalents. If you, as a developer, be aware of the fact that developing software has nothing to do with typing code but has everything to do with choosing which algorithms, which patterns, which OO structure, which database model, which sets of functionality to implement and so forth, you also be aware of the fact that the fastest and most clever way to realize all that is to specify what you want to develop in a tool which will then the best way how to implement what you want to develop. Code generators help you to reach that ultimate goal. We're not there yet, but I'm sure one day we'll. Until that time, code generators are the closed things we got to get near that ultimate goal. Don't stay behind.
I've blogged before about wishes I had for the IDE I use on a daily basis: Visual Studio.NET (currently v2003). Since that blog I've made a new list of new wishes not mentioned in the previous wish-list. I've also included wishes for .NET and the .NET API. The list is very long so I decided to post it as a story. You can find the story by clicking here. Below is the list of items I discuss.
IDE
- Generation of a makefile (C#, VB.NET)
- The ability to define post-build steps which call plug-ins.
- Cache the Help Index list between coding sessions.
- Visual Studio.NET should contain a lot more wizards and helpers for real applications.
- Expose a better object model for macros.
- A callers / callees graph would be nice.
- Appdomain instance reference viewer.
C# Editor
- Give Tooltip intellisense a better brain.
- Coding helpers should be smarter.
.NET Controls
- Replace the winforms datagrid.
- Replace the winforms treeview.
- ListView: sorting means sort it, not fake it.
.NET API
- Make all classes unsealed.
- Make more classes serializable.
- More classes should have abstract methods and virtual methods.
- Implement more base classes.
- Pay more attention when designing namespaces.
- Add an API to work with XML the OO way.
- If serialization to disk succeeds, deserialization should succeed too.
- Make event handlers / delegate definitions maskable with [Nonserialized].
.NET CLR
- An easy way to get a reference of an object in the current appdomain.
- Where are the monikers?
- Make assembly code unloadable from an appdomain.
Click
here for the complete article.
Apparantly I changed this blog posting instead of posting a new one, therefor this posting seems changed, it's not. :)
Below is a screenshot of the application, LLBLGen Pro, I've been working on for the past 6 months. LLBLGen Pro is the big brother of the DAL generator LLBLGen and generates a complete O/R mapper.
Click to enlarge
At the left you see the entities found automatically by the tool and all the relationships which are detected and added automatically (in other words: you add all these entities and their relationship definitions with 2(!) mouseclicks). In the middle you see the current state of the entity definition editor (not finished yet). Any change made is reflected directly in the project explorer at the left, everything related is updated, including the relations. At the right you see the catalog explorer, which lets you browse through all the schemas in the catalog of the project, including all tables, fields, stored procedures, resultsets and views.
LLBLGen Pro is released later this summer.
Yesterday I was playing around with VS.NET and discovered that I didn't use any macro's at all. So I opened up the macro IDE and looked at the examples. It's pretty easy stuff. So I wrote my first macro. It is a set of macro's actually which create #region ... #endregion sections. Two pre-defined ones for members and properties and a generic one which pops up an input box for the region name. Bind them to a hotkey and you're set. I'm pretty sure the code can be more efficient, it's my first attempt to write a macro that manipulates an active document in the IDE, so if I'm doing things in an inefficient way, that's the reason ;)
The initial macro I wanted to create was a create property macro which also created the related member variable in the region I've defined for that. Unfortunately, the IDE's code elements collection doesn't contain region sections, so this would be a hard thing to do efficiently.
The code is pasted below. Copy it into a new macro project and bind the macro's to hotkeys. Have fun :). Oh, you can of course modify the macro to work with VB.NET, but be aware of the fact that VB.NET doesn't support regions in a lot of places in your code.
Usage: select some lines in your code, call one of the macro's, and it's a region, baby :) You can also apply the macro's on empty selections.
''' Makes the selected lines the class property declarations region
Sub MakeMemberRegion()
MakeRegion("Class Member Declarations")
End Sub
''' Makes the selection the class property declaration region.
Sub MakePropertyRegion()
MakeRegion("Class Property Declarations")
End Sub
''' Makes a generic C# region of the selected lines and asks for the name of the region.
Sub MakeRegion()
Dim regionName As String = InputBox("Region name: ")
MakeRegion(regionName)
End Sub
Sub MakeRegion(ByVal regionName As String)
Dim selection As TextSelection = DTE.ActiveDocument.Selection()
Dim start As EditPoint = selection.TopPoint.CreateEditPoint()
Dim endpt As EditPoint = selection.BottomPoint.CreateEditPoint()
DTE.UndoContext.Open("Make Region")
Try
' insert the start of the region
start.Insert("#region " & regionName)
start.SmartFormat(start)
Dim cursorPoint As EditPoint = start.CreateEditPoint()
start.Insert(Environment.NewLine)
If selection.IsEmpty Then
start.Insert("#endregion")
start.SmartFormat(start)
start.Insert(Environment.NewLine & Environment.NewLine)
Dim ts As TextSelection = DTE.ActiveWindow.Selection
ts.MoveToPoint(cursorPoint)
Else
' insert the end of the region
If Not (endpt.AtStartOfLine) Then
endpt.Insert(Environment.NewLine)
End If
endpt.Insert("#endregion")
endpt.SmartFormat(endpt)
endpt.Insert(Environment.NewLine & Environment.NewLine)
End If
Finally
DTE.UndoContext.Close()
End Try
End Sub
With all the new emerging heros with their books about modern age technology about how to develop good, working software the right way, you'd almost forgot that in the last 15 or so years, a lot of people have already done that on a lot of occasions. How come?
They read this book:
It's from 1988. It's written by Edward Yourdon. It won't contain a single line on patterns, object oriented system design or the internet. It will however provide you with the key elements of what's really important: the base knowledge behind the material the new heros on the block are talking about.
You see, knowing what a domain model is, what a table module is, is very nice. But if you don't know how to do a thourough analysis of the problem you have to solve with your software in the first place, your knowledge about how to put the results of the analysis into practise is moot.
It's a litte expensive, but if you really get it what's inside, you will thouroughly understand what software engineering is all about and it will then not matter anymore which method you'll use to transfer your analysis to implemented functionality: you then know what you're talking about.
I have a copy now of Harry's new book, 'The Order Of The Phoenix'. It's even thicker than its predecessor :). Thankfully my wife volunteered to read it first, so I can continue reading Hillary's Bo^H^H^H^H^H^H^H^H John T. Lescroart's latest ;)
Mads Haugbø Nissen blogs:
I've tried to make a similar framework [as Paul Gielens has] but I'm leaning more and more towards discarding OR-mapping as an alternative. Clemens Vasters gave me the final push in a breaktime discussion a month ago during the Scalable Apps tours visit in norway. Successful OR mapping might be flawed by definition because you'll never fully be able to ensure that you are operating on the same object.
He further has some other points against O/R Mapping.
When you look at O/R mapping the way Mads describes it, you will conclude than that he's right: you can't assure two or more threads are working on the same entity unless they operate in the same appDomain and even then. But that's not the point of O/R mapping. O/R mapping is a way to work with data on a more abstract level. It doesn't suffer more from the 'concurrency problem' than any other database code outside the RDBMS itself: every line of code outside the RDBMS which is not shared in a single thread is potentially a danger to the integrity of the data in the RDBMS, since two or more threads can execute DML statements using that line of code at the same time, resulting in potentially concurrency problems.
It therefor doesn't matter how you interact with a persistent storage such as an RDBMS: be it through low level stored procedure calls, through objects from an O/R mapper layer, by calling code in a DAL, each time you use that kind of code, you run the potential risk of working on data that is currently also targeted by another thread. Then there is the problem of (a) reading data from the database (b) modifying it (c) and saving it again. When the application starts action (b), the data read from the database can already be altered by another thread. This can be solved by locks in the persistent storage but that doesn't scale well. If you look at it closely, it is again a concurrency problem: a user who wants to execute the sequence (a), (b) and (c), can void the work of another user who is already in action (b). Is this the result of O/R mapping? No! It is related to code that is run outside the persistent storage. So any low level DAL code will suffer from this too. O/R mapping is therefor not a mechanism that suffers from this disadvantage alone, every C# code accessing a database does.
If you as a developer are aware of that fact, that no matter what you do, the code you'll write in C# or VB.NET is potentially not working with the correct data because it modifies data outside the persistent storage, you can do something about it, by implementing concurrency control, preferably by using functionality locking, described in a previous blog. From that moment on, you can use an O/R mapper layer without any problems, simply because the actions performed on the persistent storage are not hurting other users.
So basically it comes down to: realize that the data that's stored in-memory (be it a datatable, a custom class, a string of text) is a copy of data that is stored in the persistent storage and that other threads in the application (possibly on other machines as well, f.e. in a webfarm) will not see your version of the data. If that's not what you want, be sure that other threads do see your version of the data, by a) updating the persistent storage and b) keeping your in-memory copy in-sync with the persistent storage.
O/R mapping is a very useful abstract way of working with entities stored in a database in the form of values stored as rows inside tables and views, which does not introduce new problems compared to other .NET code, but does make working with data more transparent for the developer. Can the developer forget about the database? No. As the developer also can't forget about files in a filesystem, protocols when working with networks etc. An entity is a concept that both is defined in code and in the database model, and thus in code you'll have relations as you have them in your database model. The database is there as the persistent storage, the concept 'entity' is part of the database too. You can't ignore that simply because you are in the C# editor and not typing SQL.
The only small problem I have with writing an O/R mapping layer is that the relational databases which are used to persist the entity data into are not capable to store inheritance trees, simply because the relational model doesn't support inheritance of subtypes: you can't store a supertype-subtype relation in DDL, you have to use attribute values for that, which is not workable (change a value in an attribute and the entity holding that value is suddenly of another type, that's not what I call 'robust'). However this is not a problem per se: if you look at the tables found in a schema in a catalog / tablespace, you can use these entities as your entities in your code. Your OO abstraction will then be a little less abstract than with a database that's pure OO, but the abstract way of working with data which is brought to you by O/R mapping is still possible, very useful and at a highly abstract level. With OO databases, O/R mapping can only get better, but O/R mapping is already very useful today.
Jimmy Nilsson asks on his weblog
if I've dropped stored procedures for everything, according to my dynamic query blog. Well, no, I haven't. The reason for this is simple: you should use the technology that fits the job best. If a couple of actions are simple in SQL and hard in C# or other 3GL (which is highly likely, since SQL is set-based, C# is imperative), why not write that code in SQL?
However, the occasions in which you should decide to use SQL instead of an imperative language are rare, since the vast majority of database activity is related to the 4 CRUD actions: Create, Retrieve, Update and Delete. In other words: the database activity is the end of a chain of logical components (CUD) or the start of a chain of logical components (R) and not a chain of logical components itself. This implies that it is not that important in what form you realize these CRUD actions: all in SQL procedures or in an imperative language with dynamically formulated queries. In fact, in some areas (UD), an imperative language with dynamically formulated queries is more efficient, because it can tailor the SQL statement to the task it has to perform, while the SQL procedures are inflexible due to their compiled nature and because they form an API that is called, which interface is not changing a lot.
At the moment I only use stored procedures for some retrieval actions actually, since these can be complex and can be hard to formulate using a dynamic query engine. I'm not referring to SELECT statements with joins but SELECTS with aggregate functions, resultset columns which get their data from a function call so on and so on. It's hard to define a set of predicate classes which are generic towards most databases plus have the same expressional power as SQL statements as SELECT function_f(Foo.Column3) as Column1, Foo.Column2 FROM Foo WHERE Foo.Column4 = function_b(Foo.Column5). It's then better to choose for those particular cases the SQL route and to write those functions / procedures in SQL stored procedures so you can have the best of both worlds.
Isn't that going to become a mess, both embedded, dynamic SQL and stored procedures? No I think it isn't. The same as having 20 libraries in your solution isn't going to become a mess because you've split the code up into 20 libraries and not 1. Having part of the code of the application inside the RDBMS isn't going to create a mess either, there is just a part of the code stored in the RDBMS, like there is some part of the application's set of datastructures stored in the RDBMS instead of in memory / the CLR, using constructions not known by the imperative language.
Views
However I think I'll move away from stored procedures in the future completely though, since more and more databases are supporting views. Views are awesome. Most people don't use them, but they have so much power in them. Once you've used them occasionally, you'll understand the true power that is embedded in the concept of views. Views are simply virtual tables which are defined using a SELECT statement with whatever resultset you want, even new columns based on calculated results. Views can make complex SELECT statements simple and easy to understand and are very beneficial in the world of dynamic queries, because you can hide complex SQL statements in the target views you select rows and columns from using those dynamic formulated queries. Views are also very clean: they are usable as regular tables, thus an extension on the set of tables, which opens up new ways of defining database logic: you can build new logic on top of a virtual table which is formed by database logic in a clean, simple way, which was impossible with a set of retrieval stored procedures, since you can't join with or select columns from stored procedures.
So to sum it up: stored procedures can be a good choice for complex SQL-related actions that are easier formulated in SQL than in an imperative language. Stored procedures are also a good choice when you have a complex retrieval query because it is hard to define a generic set of predicate classes for dynamic queries which have the same power as their SQL counterparts. Views are however an even better choice, since you can then have a well defined set of tables (virtual and real) and select rows and columns from these tables using dynamically defined queries, so both worlds benefit from this. I highly recommend views to anyone who is now struggling with complex retrieval queries constructed in stored procedures.
Scoble (among others) writes about a situation which is a given in any marriage where one person is gifted with 'geek-genes' and the other lacks every single one of them. Here in the Netherlands we even had a term for those wives which feel neglected because their husband is spending more time behind the keyboard than with her: "Computer weduwe" or in English: "Computer Widow".
Now, I'm married too and I don't have these situations here. I think one of the reasons for that is that my wife loves her computer: she plays The Sims whenever she finds the time, writes her novels on it (she's a writer) and uses it helping with projects in my company (functional designs / research / front-end development). Another reason is that she knows I'm a true geek. I'm 32 now and am programming since 1985, you can't simply say to a person who wants to do one thing in life: software development, to back off and spend free time away from the machine that makes that one thing possible. In other words: she respects that I'm a geek and I am spending a lot of my time typing code, shaking my head while looking at a monitor and other activities developers perform on a daily basis.
However as Paul writes, it's not a one way street, it's a two way street. You can't expect from a person you want to spent your life with to simply agree with how you think he/she has to spend his/her life. I live near the beach (in Scheveningen) and when it is nice weather, it's great to spend some time near the sea, sit in the sun, drink a couple of beers. There are times I'm in the middle of a routine I have to finish (I have my office in my house) and my wife asks me to stop coding immediately and to get to the beach. Well, it's a two way street, so in most of these situations I indeed choose the beach over continuing programming. But fortunately for me, the weather sucks most of the time so I'm not confronted with a lot of these situations. :)
I also know the situation where your girlfriend / wife simply can't understand what's so darn interesting about spending a lot of time typing cryptic texts, staring at a monitor and being most of the time not that happy because what you try to accomplish is not what you be able to achieve. The whole concept of writing software by seemingly endless periods of misery with a happy ending is surreal to them. In the period 1988-1993 the girlfriend I had at that time was a person who simply didn't understand that concept. I was still studying at the university and programming graphical routines on my Amiga 500. She didn't understand that I found it hard to spend 2 weeks in a tent on a camping site without my computer. Needless to say: it didn't work out very well and frankly, now I'm glad that period ended.
So, the moral of the story? Have respect for each other's interests, be aware of the things which are very important to the other person. If you read that sentence two or more times, you will understand that indeed it is a two way street, because from the Point Of View of the 'geek' in the relationship, the other person is the non-geek who also has things which are very important for that person.
... for my blog, that is. I tried to mimic the style we use for our company website. It has some bugs actually. Working on it :)
If you still see the old skin, hit cntrl-refresh, or if you do not see a different skin at all, you probably are running a browser which doesn't understand javascript or stylesheets. (Not that IE does understand CSS completely but let's not get started on that one).
Update: it seems to work now. :) I've to test it on some other browsers though.
More Posts
Next page »