Yet Another Post on Dynamic Sql vs. Stored Procs

I continue to be amazed at the typical discussion of "dynamic sql" vs. stored procs.  The arguments almost always assume that "dynamic sql" equates to hard-coded sql in your application.  This is simply not the case for anyone I know that has ever argued for "dynamic sql".  So first, let's all agree that hard-coded sql in your application is a bad practice.  Now we can move on and really talk about "dynamic sql" vs. stored procs.  Before defining "dynamic sql", lets define stored procs.  That may sound silly and unnecessary -- but seriously -- what are stored procs?  Stored procs are blocks of hard-coded sql in your database!  That's right -- they are hard-coded sql -- just moved to a different physical tier.  Like any other piece of code, these blocks of sql must be written (or generated) and maintained.  The frequently recented argument that stored procs free developers is absurd since it just moves the burden somewhere else!  Its also stated that stored procs allow you to change things without a recompile -- but that's true to a small extent only.  Any serious changes in your data-model are also still going to require changes in your app to know how to work with those changes.  In other words, stored procs are an API, and if you change the API then you're going to have to change your application too!  That said, its certainly true that small optimizations can be made in your stored procs without a recompile, but how often does this really occur?  Most optimizations involve things like proper index tuning, the setup of separate file groups, and other database "tricks" that are not related to sql.  In fact, if the sql in your stored procs is so poor that it can benefit from optimization then you really need to question if you should be writing any sql in the first place!

Which brings me nicely back to "dynamic" sql -- so lets talk about what "dynamic sql" really is.  "Dynamic sql" is sql that is generated at run-time, typically by an O/R mapper, and it is also "parameterized".  Note that it is not hard-coded sql, not in your application or in your database -- so there is no sql to write or maintain anywhere!  You aren't just moving the burden from your developers to your DBAs -- you are removing this burden entirely.  Next, note the run-time piece -- this is critical for some serious optimizations that you simply cannot get with stored procs.  That's right, do your stored procs allow you to retrieve data for any criteria and/or sort on any set of fields?  Do your stored procs allow you to only retrieve a specific "page" of data, as opposed to all of the data?  Do your stored procs allow you update only the fields that have changed?  Do your stored procs make it easy to use optimistic concurrency?  Do your stored procs allow you to sell your product with support for any database?  These are real sql-level optimizations -- and they are made by good O/R mappers, but typically lacking from stored procs!  Everyone loves to focus on "performance", but that is typically done with lower-level things like index tuning and file groups, not by tinkering with sql.  And in that sense there is absolutely no reason why "dynamic sql" cannot perform just as well, if not better in some cases, than stored procs.  Yes, the length of the sql statements are larger for "dynamic sql" than they are with stored procs, but that realistically has little to do with performance when you look at the bigger issues.  And yes, stored procs make it "easier" to secure your database, but its actually quite easy to use application roles with "dynamic sql".  And one last thing to note is the "parameterized" part of most "dynamic sql" -- this means that you are just as safe from sql injection attacks as you are with stored procs.

Now this does not mean that I believe stored procs are evil and that everyone should use "dynamic sql" -- so please don't assume that.  What I am saying is lets please make sure we are all talking about the "right" things.  No one has been saying to use hard-coded sql in your applications -- unless you follow all the examples in MSDN, books, and articles that is.  :)  If you want to save development time and maintenance, then you should either be using an O/R mapper or some serious code-gen (like CodeSmith) -- end of story.  And if you really want to optimize your database, then you should get your DBA involved in the design of the database and stop giving him silly stored procs you wrote.  Is there really any value to have developers create poor database designs and then expect your DBAs to fix it all by "optimizing" your stored procs?  The problems usually are far more serious, and too late, if the database was designed poorly in the first place -- and the best optimizations are usually done elsewhere if its a well-designed database.  Again, that doesn't mean that stored procs are bad -- but please don't assume that the job of your highly paid DBA is merely to clean up your crappy design and stored procs.  Whether you use an O/R mapper with "dynamic sql" or code-gen with stored procs -- both of these practices attempt to free your entire team from the routine cut-n-paste so that they can focus on real design!  And maybe that's the real issue -- for sadly, most developers (and maybe many DBAs), especially in the Microsoft camp, are simply not able to do anything beyond simple cut-n-paste!  And if that's the case (which I fear it is), then O/R mappers and/or code-gen is simply going to leave them with blank stares -- since they will no longer have a clue what to do.

Published Sunday, September 05, 2004 11:03 AM by PaulWilson

Comments

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Good writeup !

and I think that perhaps the *REAL* issue is the bad database design problem!!!

I am working on some stuff right now that I am 99% sure the performance problems come from the first db design having been done by a non-dba who was more used to working in a big firm where the sql was handed to him...

two key tables have no relation and others that are related have none... and so indexes have to be built and extra code to prevent relational errors....

I am trying to fix it but have to step with caustion as 4 programs access the sql db and have some odd code embeded in each app.

one benifit to s-procs at times is this:

if none of the apps had hard-coded sql then fixing the db would be a lot simpler.

O/R maping and code gen are things I want to use more of but this is a system written in part in MS J++ part in Sun Java part in C# and now some is moving into Web Services.... so each change has to be tested for each apps embeded logic... a pain!

Sunday, September 05, 2004 12:20 PM by Denny Figuerres

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Hi Paul,

I'm one of the developers of DocuLive, an electronic patient
record system used in Norway and in Denmark.

DocuLive's kernel is constituted of close to 120 tables and
600 stored procedures. The originality of our approach is that
the kernel, implemented in T-SQL, *IS* what you could consider as
an O/R mapper. So, if I come to your questions, I could anwser "yes" to almost all of them, excepted "do your stored procs allow you to sell your product with
support for any database?", but that's only because we've
never had the occasion (and were never asked) to migrate it
to another SQL platform.

More information on DocuLive is available here:
http://www.xhovemont.be/PrvImages/DocuLiveECOOP2004.pdf

That's a short DocuLive presentation done at ECOOP 2004.

DocuLive in Norway (in Norwegian):
http://tinyurl.com/6kuqz

Sunday, September 05, 2004 1:26 PM by David Brabant

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Thanks David. There's no doubt there are ways to use stored procedures in such ways -- but I don't think its typical, nor is it what most of the people debating "dynamic sql" vs. stored procs have in mind. Again, I'm not so much arguing against stored procs as I am for trying to better focus the discussions on the real issues.

Sunday, September 05, 2004 1:31 PM by Paul Wilson

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

To me, a true developer/application architect should have the ability to design a decent database from the ground up. I pride myself in being able to do this, but always make it a point to get the design reviewed by a *qualified* dba (let's face it, some aren't...).

So to Denny's comment, if an application designer is designing a database with no keys and relationships, they probably shouldn't be building software (IMHO) anyway. You MUST be multifaceted to succeed in app dev these days.

Sunday, September 05, 2004 2:50 PM by Jay Nathan

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Paul,

I won't allow the interaction between my client application and database to be controlled by an O/R mapper. I have never found a piece of code that can look at my database and provide me some type of interface into that database that provides acceptable performance. I know how to setup a database with primary keys, foreign keys, and indexes. My code provides a high performance interface.

Whenever I have seen an O/R mapper, it has always had some serious performance problems compared to what I can manually write. I have found that the client will pay for this type of performance.

As for the argument about dynamic sql vs. sprocs, well i don't see enough programmers that know how to properly call sprocs, so I try and stay out of them because they need code that they can support and understand. They can understand dynamic sql. As for the suppossed security problems, well if you just take the input and try and put that into the database, well yeah, there are problems with that and you can get what you deserve. I have some parsing routines that resolve the sql injection security problems.

For me, a properly designed database combined with proper sql statements provides the best performance and are secure.

Wally

Sunday, September 05, 2004 3:16 PM by Wally

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

That's totally fair Wally -- I like a good O/R mapper, but its definitely not for everyone. I do hope you do some code-gen though, which you can do with CodeSmith using your own templates and style.

That said, while I don't have a problem with anyone doing there own thing (if done well), I do think that good O/R mappers do not suffer significant performance loss -- in some cases they even do better.

Sunday, September 05, 2004 8:27 PM by Paul Wilson

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

BTW, I am not trying to trash the Wilson O/R mapper. :-) I have much more confidence in you than in most developers (read 99.5%) and in your capabilities to create a feature rich app.

I've been dealing with this project that has an O/R mapper and the O/R mapper is driving me crazy. Basically, it is an O/R mapper that they rolled on their own. Yuck. As you can guess, I have a bad taste in my mouth.

I have tried to stay away from the code generators for two reasons.
1. Performance reasons as mentioned above.
2. Problems in making the finetuning type of changes that a code generator does not like me to do.

Wally

Monday, September 06, 2004 9:42 AM by Wally

# Back to Dynamic SQL vs. Stored Procs

I read an interesting post from Paul Wilson regarding the never-ending debate of whether to use stored procs or dynamic SQL for your data layer. As I mentioned before, I've always been a stored proc kind of guy. Why? Really just because I love them. I love to get a change of scenery everynow and then and work with some T-SQL. But after reading Paul's post I got thinking. I've never really even given the dynamic SQL route a try. We'll I've tried it, I love the idea of O/R mappers, but really just as that...a try - knowing the whole time I was just seeing what it was about then going back to my typical stored proc route.

Monday, September 06, 2004 2:00 PM by TrackBack

# Back to Dynamic SQL vs. Stored Procs

I read an interesting post from Paul Wilson regarding the never-ending debate of whether to use stored procs or dynamic SQL for your data layer. As I mentioned before, I've always been a stored proc kind of guy. Why? Really just because I love them. I love to get a change of scenery everynow and then and work with some T-SQL. But after reading Paul's post I got thinking. I've never really even given the dynamic SQL route a try. We'll I've tried it, I love the idea of O/R mappers, but really just

Monday, September 06, 2004 2:01 PM by TrackBack

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Nice post and a good discussion/description of true dynamic sql. Thanks.

Monday, September 06, 2004 2:12 PM by Bil Simser

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

I also won't argue either way yet (my jury is still out), But don't dismiss that "move the burden" argument as absurd - moving burdens from one group / person to another is critical to team software development.

The fact that the responsibility stays in house but moves to a different group can be an asset, not a detriment (or absurdity).

Monday, September 06, 2004 6:10 PM by Philip Rieck

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Great post. It really got me thinking. I've always been an sp-guy but maybe it's time to at least try another approach.
Starting a new version of my framework soon and I'll implement ORMapper into it.
BTW, I'm using CodeSmith for code and SPs, so that's never been a real problem. ;)

Tuesday, September 07, 2004 3:28 AM by Patrik Dahlén

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

A good friend of mine had a hard time understanding the concept of dynamic sql in O/R mapping. It took me the whole evening explaining and finally convinced him to read some articles regarding this. :-)

This is a great post Paul. I think this will clarify some issues regarding "dynamic sql" which some people thinks its hard coded somewhere in the app.

Wednesday, September 08, 2004 12:39 AM by Chris Ongsuco

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Wally,

I ask you to check out code gen again. You don't have to be stuck with your issues #1 and #2 with code gen. You can always modify what you generate afterwards if you need fine tuning. And especially with something like CodeSmith, the performance of the code-gen'ed code is only as bad as the template you write.

Certainly it'd be nice if you can code-gen everything in the build process with no touch-ups but chances are you have to, even if you do something like create a derived class to "fine-tune".

Just my 2 cents

Wednesday, September 08, 2004 2:26 PM by Jiho Han

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

In my apps I never user stored procs. I even avoid using views. For these reasons:
Stored proce make me dependent on the type of database (Oracle, MySQL, SQL Server)
Part of the source code is stored in stored procs and another part in C# or VB. How do you make agreements across a development team which code to put where?
Stored proces can only be organized on their filename. How do you control a few hundred store procedures? How does every developer know what each procedure does?


Friday, September 10, 2004 2:00 PM by Joris

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Hi!

The I am currently using Oracle 9i, and I just switched from SQL Server... just two weeks ago.

now the problem is in Oracle 9i or I duno if it is true for any Oracle World, that Stored Proc are pain in the @$$ they are totally disgusting way of getting value via Stored Proc. Like you have to write the Cursor for every return parameter? you can't get "single value" return via Stored Proc, it is allowed only via Function.

I was very keen at first that it can be the same thingy, as I am from SQL Server, but this hit me too hard! Oracle Stored Proc are not so "developer friendly"

Anything anyone can say about it?

The alternate approach? I am using "Views" and then shorten my big giants SQL and then gettin the data via Views and then applying some logic in my .NET Code (in my Business Layer).

Any very nice alternative of Oracle Stored Proc?

Take Care,

Mudassir

Monday, September 13, 2004 2:09 PM by Mohammed Mudassir

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Yes, Oracle stored procs are "harder" no doubt. :) They're aren't any tricks that I know of to make it "easier".

Monday, September 13, 2004 3:57 PM by Paul Wilson

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

my feelings exactly ...

for many members of the camp ... "Copy and paste is NOT code reuse"

Tuesday, September 14, 2004 1:46 PM by Greg Young

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Paul, one of the problems with the "dynamic sql" debates might be it is always discussed in abstract, at least everything I've read. Maybe it would help if you (and/or Frans) could blog a series of posts that give concrete examples showing a stored proc and how dynamic sql would replace it including the dynamic sql that would be generated and the .NET code that would be used to call it. I'm suggesting a series of blog posts showing many different scenarios. Maybe that would help people better understand the benefits of using dynamic sql? (or give them more ammo to attack the downsides? :)

Friday, September 17, 2004 2:29 AM by Mike Schinkel

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Mike,

From what I understood, the concept of OR mapping is not to ACTUALLY replace stored procedures. It simply tells the developers out there that "hey, you can also use DYNAMIC SQL when developing your CRUD and its also fast, easy to maintain, etc!" They explained the benefits when using it and of course there also trade-offs. But some people tend to misunderstand the concept of "dynamic sql" because they think creating these queries is by string concatenation. And this post, I believe, is to clarify the misunderstood concept.

Now the question of dynamic sql as replacement for stored proc? For me its not. We build app depending on the needs of our clients and their "situations". So its always a matter of choice.

Friday, September 17, 2004 6:08 AM by Chris Ongsuco

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

The concept of dynamic SQL existed way before the O/R mappers became popular. That's why most people relate to dynamic SQL as being queries created by string concatenation and hard coded in the application. And from that perspective it's quite understandable that people don't like dynamic SQL.

Just because O/R mappers can do dynamic SQL (and in a very controlled way) doesn't mean that people will start think about dynamic SQL as SQL generated by O/R mappers.

So you have:
- Original dynamic SQL (that most people relate too)
- Original stored procedures
- Dynamic SQL generated by O/R mappers

And depending where you are coming from you will relate to dynamic SQL differently and that's why the discussion of dynamic SQL vs stored procedures never ends since people are not discussing the same thing. The discussions are even more common now when O/R mappers are out there using "the dangerous" dynamic SQL.

When it comes to dynamic SQL generated by O/R mappers it's quite different than the hard coding approach since the O/R mapper generates the dynamic SQL in a controlled manner. Based on certain rules it will generate the code in a certain way.

So even if the end result is dynamic SQL I wouldn't say it's dynamic SQL as most people think of dynamic SQL. It's dynamic SQL generated by an O/R mapper. So you should call it something else than dynamic SQL when entering the classic stored procedure discussion. Call it O/R mapper generated SQL.

Saturday, September 18, 2004 7:42 AM by Magnus

# Examples of O/R Mapping vs Stored Procedures

Saturday, September 18, 2004 12:08 PM by TrackBack

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Magnus,

What is so "DANGEROUS" about generating SQL at runtime? Care to explain why?

Sunday, September 19, 2004 11:48 PM by Chris Ongsuco

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

Chris Ongsuco: Once again, you did the same thing (discussed in abstract!) I'm thinking we need SOURCE CODE EXAMPLES!! :)

Monday, September 20, 2004 4:35 AM by Mike Schinkel

# S dot One heeft het over .NET » Dynamische SQL versus Stored procedures

S dot One heeft het over .NET » Dynamische SQL versus Stored procedures

Wednesday, September 29, 2004 4:45 AM by TrackBack

# Examples of O/R Mapping vs Stored Procedures

Thursday, October 14, 2004 3:55 PM by TrackBack

# re: Yet Another Post on Dynamic Sql vs. Stored Procs

I absolutely refuse to believe that SQL generated by OR mappers or whatever else would perform well by default.  Dynamic SQL seems as if it is doomed to poor performance, unless the people using the OR mapper or otherwise changing the dynamic SQL are severely limited in what they are allowed to do.  Otherwise, what your 'user' ends up doing could eventually lead to long-running queries.  

This scenario seems worse to me than hard coded SQL in code or in a stored procedure.  Your tuning efforts involve figuring out the exact things your user did that ended up generating long running sql, and then you have to ensure your "tune-up" won't impact any other series of actions the user may take.

So, perhaps dynamically generating SQL may be ok under severely limited circumstance.  Otherwise, bad idea.

Monday, January 29, 2007 6:28 PM by MLKnight

Leave a Comment

(required) 
(required) 
(optional)
(required)