Paul Gielens:ThoughtsService

"What I extracted from Jeremy’s article by reading between the line is that big (half the project) upfront research including business case, requirements understanding etc eventually lead to these so called "Frameworks". Sounds familiar"
So, the theory computer science is partly based on is somewhat wrong?

Couldn't it be upfront research you ran into isn't done properly?

I mean: how else do you determine WHAT you're going to develop, HOW you're going to do that and above all: WHY?

By ad-hoc 'let's see where we end with this' approach? Or with research ?

One of the key mistakes embedded in agile development is that it assumes refactoring is free. Let me tell you: it's not, far from it. Especially in OO systems where refactoring of designs can lead to severe delays of the project as a whole, it's key to know as much information about the functionality you have to realize with the software you're supposed to be writing, UP FRONT, so you have the best chance to have the 'best' design in the first attempt.

Essential is proper functional research and with the results of that proper technical research of the subset of the results of the functional research that's chosen to implement in the system.

All this talk about actual code writing is missing the point: you can't write a single line of code unless you have a proper understanding what the code has to realize, which functionality it has to provide (or better: be the realization of). Now, agile nor not, to get that proper understanding, you have to do research. Thourough research. So you get a strong connection between functionality description and actual realization in code of that functionality AND vice versa. This is the key element of maintainable, extensible code: how else are you going to extend the code in the future? Based on WHAT makes you modify abc.cs and not def.cs? Based on what makes you adjust class A and not class B and C?

A classic example is when you design a set of classes S and after a year a new set of functionality (which would have been appeared on your functional research list if you would have done your homework) F has to be implemented and requires you to either refactor S for 80% so F can be build with a core where S can build upon and also the new class set T, or build a new class set T to implement F.

Every time I present this example to some Agile-sympatisant, s/he comes up with "But we DO proper research!" aha. but when? and how? I think it would be best if the agile community focusses more on that aspect instead of the project phases when everone is typing code already. Because what on earth are these typing fingers trying to realize? Apparently what the mind cooks up at that moment...

Hi Frans,

Thanks for you (speedy as always) reaction, my comments:

> One of the key mistakes embedded in agile development is that it assumes refactoring is free.

Agree. Sometimes it pays of writing tests and writing the code to support that tests and eventually end up refactoring that code to incorporate new features compared to looping in circles with a mediocore business team.

> Thourough research. So you get a strong connection between functionality description and actual realization in code of that functionality AND vice versa. This is the key element of maintainable, extensible code.

Agree! One of the reasons why I like the idea of DDD because it focuses on this exact part.

What I miss in all your reactions in various blogs and newsgroups regarding this topic is the fact that you seem to forget agile is here for a reason. Apparently people feel comfortable in practicing agile philosophies. I think your opinion is formed on 1) the fact that you’re building products in contrast to projects and 2) your one of those (with all respect) guys who grew up with SDM, waterfall, analysis and design, case tools etc. and 3) (and I’m not to certain of this one) with SD being largely a one man’s shop you’re both in control of process and realization.

I’m not saying agility is a silver bullet, nor a plaster on blunt practices to make them more enjoyable. Requirements and a business case carved in stone for a release sounds like a dream come true but that’s not the reality in many projects (isn’t that why it’s called project... that’s a whole other discussion).

"What I miss in all your reactions in various blogs and newsgroups regarding this topic is the fact that you seem to forget agile is here for a reason."
I know. I use a semi-agile approach for several years now, though I've been bitten by the fact that refactoring isn't cheap nor free, and that my only conclusion is that thinking/researching up front always pays off in the end, which greatly (IMHO) reduces the reason to use an agile approach to begin with.

"Apparently people feel comfortable in practicing agile philosophies. "
Of course, because it appeals to the average developer: no boring long phase up front but a lot of code hammering right away!... Sadly, most developers out there think that developing software is about writing code.

"I think your opinion is formed on 1) the fact that you’re building products in contrast to projects "
I've done a lot of large projects before doing that, Paul (for at least 6 years) :). And I don't see a product as different as a project: a product also has to solve a problem, similar to a project. A lot of developers forget the core reason why there is software: because it solves a problem.

"and 2) your one of those (with all respect) guys who grew up with SDM, waterfall, analysis and design, case tools etc. "
I'm not familiar with terms like 'SDM', smells like those old Volmac/Capgemini methods. What I indeed support is what Yourdon CS. have been preaching for decades. And you can't deny it makes sense, it's perhaps 'old tech' but still valid today (IMHO).

"and 3) (and I’m not to certain of this one) with SD being largely a one man’s shop you’re both in control of process and realization. "
I don't think that's relevant: I too have to deal with customer demand, do research and write code to solve those problems (and I'm not alone but that's another thing ;)). What I've learned in the 6 years before I started to develop a product was that if there's no research, forget it. Research is key to a succesful project. You might be lucky and start with your own ideas and it might be that the customer is happy with what you produced, but how many times have you tested if the result you produced is actually solving the problem it has to solve?

To set the record straight, SDM has nothing to do with Volmac/Capgemini at least not the way I intended it. See this resource for valuable information on SDM http://www.liberty.edu/informationservices/development/index.cfm?PID=6175.

The posts on this weblog are provided "AS IS" with no warranties, and confer no rights.The opinions expressed herein are my own personal opinions and do not represent my employer's view in anyway.

>Research is key to a succesful project.

Research is definitly key to a succesful product, much less so to a succesful project. I see LLBLGen as a product, something I custom make for a single client as a project.

You probably all know the picture of the rocking chair hanging on a tree. Where sales, tests, developers, everyone has a different view on the project. You can research all you want, think of what it's going to be, write it, and in the end you notice it's not what the business wanted then, and even less what the business wants right now. Because (A) we didn't understand the problem fully and (B) because the problem has changed or shifted.

I've been to a presentation by Mike Cohn, one of the founders of the Agile Alliance. He said we don't need a missile that's fired and with GPS hits an exact, predefined spot. We need a guided missle. We need a missile that can be steered, so that when the customers moves away from one point, the missile can change direction.

Now of course we need to know where the customer is at when we start to build. But we cannot do all the research up front!

I asked Mike Cohn about DDD, and he has read Evans' book as well. He thinks DDD is great to do research, but when you start to develop, don't get hung up on the UML you've drawn. Let the tests guide you.

Yes, refactoring is not free. Nothing is. But I do believe (and see) that when TDD is done right, you get a better design with less classes. Your DDD "assumptions" can lead your thoughts when you are thinking of and creating your tests.

(TDD incl Refactoring) < (BDUF == Too much up front research)

> I asked Mike Cohn about DDD, and he has read Evans' book as well. He thinks DDD is great to do research, but when you start to develop, don't get hung up on the UML you've drawn. Let the tests guide you.

Practicing TDD is no excuse to let the code drift away from your problem domain. TDD and DDD go hand in hand.

> Your DDD "assumptions" can lead your thoughts when you are thinking of and creating your tests.

Based on personal experience?

You don't think DDD is BDUF? I'm not saying it's as bad as... well, as Frans does BDUF ;)

But TDD is about design, so is DDD. They don't "just" go hand in hand.

What makes you think DDD is BDUF?

"I know. I use a semi-agile approach for several years now, though I've been bitten by the fact that refactoring isn't cheap nor free, and that my only conclusion is that thinking/researching up front always pays off in the end, which greatly (IMHO) reduces the reason to use an agile approach to begin with."

I think the problem you've been dealing with Frans is that you are using a "semi-agile" approach. Agile doesn't work nearly as well unless you totally commit to it. This is something a lot of developers are finding out as they try Agile techniques for the first time (this is also part of the reason XP teams have a "coach" - to help guide them and make sure they are utilizing all facets of the methodology).

"One of the key mistakes embedded in agile development is that it assumes refactoring is free. Let me tell you: it's not, far from it. Especially in OO systems where refactoring of designs can lead to severe delays of the project as a whole, it's key to know as much information about the functionality you have to realize with the software you're supposed to be writing, UP FRONT, so you have the best chance to have the 'best' design in the first attempt. "

Agile does not assume refactoring is free. What Agile does assume is refactoring is an integral part of the design process. It's how the design evolves over time. What Agile assumes is that no matter how much research you do up front, you're design will change by the end of the process. So rather than fight that, Agile embraces it and uses it to its advantage. This is how we end up with the "best design" at the end - because we realize the "best design" at the beginning is an impossibility.

"it's key to know as much information about the functionality you have to realize with the software you're supposed to be writing, UP FRONT"

I think, Frans, you might not have investigated Agile techniques as deeply as you could. Agile does this investigation up front. At least Extreme Programming does - with story cards and the Planning Game. With Agile, we attempt to discover as much information about the functionality before a line of code is written. Items are then prioritized, and the iterations begin with TDD as the guide for development. Design is morphed over time as the requirements change and we refactor to meet those changes.

"I mean: how else do you determine WHAT you're going to develop, HOW you're going to do that and above all: WHY?"

If you don't know what you're going to develop before you start, Agile won't help you :)


Agile isn't an excuse to simply start coding without thinking first. If people are applying Agile techniques in such a manner, then they are not doing Agile development. Agile is about leveraging the inherit flexibility of software to create better, more reliable code and thus higher quality finished products - it allows us to improve our design over time as requirements change.

Hi Chris,

First of all thank you for your comment!

What you wrote is in essence what I tried to explain with the AbNQY syndrome. Do you have any horror/success stories to share regarding teams committed such a hybrid process?

Chris: so in short, when you're using Agile development, you're too doing a lot of research up front, but also a lot of refactoring during the development phase, am I correct in that conclusion?

What I find particular troubling is that you say: "it allows us to improve our design over time as requirements change.", without addressing the severe impact this can have on the development phase.

A simple requirement change can make months of work become useless. It's IMHO easy to say "we'll improve our design when requirements change", but it's IMHO really hard to avoid severe complications during development because of changing specs.

The more you can foresee up front changes, the less you run the risk of having to throw away a lot of work because of a requirement change.

What I'd like to know is:
1) What amount of time do Agile development rules prescribe, so the research you talk about is done properly ?
2) What does Agile development have in store to avoid the severe risk of having to throw away a lot of work because of a design change?

All the stories I've been reading lately about Agile development don't address these 2 key elements. In fact, they skip 1) altogether as every Agile story / article posted on for example weblogs.asp.net is about writing code.

I'm a data-access guy, and know that doing 'agile' database development is not that smart. At least if you want to be able to make changes to your several terabytes of data in 500+ tables in the coming 3, 4 years without creating a mess. So research is KEY to be succesful. Though what's the amount of research that should be done? THe same as what was done before 'agile' became a buzzword? Not likely as with the refactoring time build in, projects using agile development will take longer, so it's likely to be less.

IF it's less, how much less and how are you going to make up for the lack of time you spend on research up front, as less research up front is a higher risk during the project that work performed will become void or requires a tremendous amount of refactoring. And as you said, refactoring isn't cheap.

>>You don't think DDD is BDUF?
>What makes you think DDD is BDUF?

I'm not, I'm asking you! :)

I haven't got experience, but I guess DDD _IS_ design up front. Now perhaps/probably that's not bad, if you can still let your TDD do the design. Again, I haven't got experience there, perhaps you do. I am curious though.

See http://weblogs.asp.net/pgielens/archive/2005/12/02/432094.aspx

DDD is more a theoretical/thinking thing.

Dennis,

If you haven't ready Eric Evans book. I would suggest reading it. I haven't been with a team yet, that has invested alot of time in DDD, but in reading and talking to people, I don't get any sense that it is counter-Agile.

One of the premises is to keep the business people and the development people speaking the same language. Eric speaks of Martin F's ideas and is generally Agile friendly in his writing. I don't see what would be difficult to integrate his ideas into an Agile delivery effort. In fact I think the things that he talks about are the kind of things that happen in successful Agile projects. Issues such as learning the business domain and maintaining close contact between business guy and techie guy. Getting them to each see the same picture and see the same 'Vision'. This doesn't all have to happen before a line of code is written and I don't think DDD suggests it should, it only suggest these things be valued. Of course there is a lot more to the book, but the ubiquitous language was the practice that most struck me in reading it.

"Now of course we need to know where the customer is at when we start to build."

With a guided missile, we don't even need to know the *exact* location of the customer, let alone the exact wind conditions etc. We just need to fire roughly in the right direction and can make adjustments while it's flying.

And still it's much more likely to hit the target than firing a missile with more precise targeting upfront, but a worse guiding system.

"A classic example is when you design a set of classes S and after a year a new set of functionality (which would have been appeared on your functional research list if you would have done your homework) F has to be implemented and requires you to either refactor S for 80% so F can be build with a core where S can build upon and also the new class set T, or build a new class set T to implement F."

Are you sure that F requesites would be exactly the same if gathered them a year ago? If the answer is "yes", then you are 100% right, you shouldn't be doing Agile in a well know and stable requisites environment. Lucky you...

Frans (and others),

Sorry, first, for my neglect of this thread. I meant to come straight back here and keep track of it, but things got away from me.

I'll address your comments directly Frans:

---------------------------------------------
Chris: so in short, when you're using Agile development, you're too doing a lot of research up front, but also a lot of refactoring during the development phase, am I correct in that conclusion?
---------------------------------------------


Yes, mostly. We do some research up front because implementing things like a framework require some thought, and we have to think about things like: Do we want a client-server architecture, or web services? Should we use a thin client, or a smart client? What's the best solution to this particular problem?

We ask those questions, do research, and try and get answers to them early on, and those guide some very big parts of the project.

But what I want to get clear is that at a lower levle in Agile development, refactoring is part of the *design* process. It's part of *how* we design. TDD is the design method we use. So the whole concept of design-first, code-second, test-last is a paradigm that must be cast aside before you can really understand Agile.

Agile is about using TDD to actually *do* design as-you-go. Design isn't something that gets done once, then forgotten. Design is actually done every day, every coding session. That is why we use TDD - because writing the tests first forces us to think about the actual design of the specific problem we're trying to solve. We write a test, it fails because we haven't written implementation yet. Then we write the class that solves that problem, but only enough to make the test pass. We don't do more in anticipation of things that may or may not occur.

That's going to sound really simple (and maybe even stupid), but that's the basis for the process. As you write more tests and code to make those tests pass, patterns emerge in the design. That's when we refactor, abstract, and simplify the design.

Now, if a pattern emerging can cause us to refactor part of our design, what's to stop a requirement from doing the same thing? Nothing. So we use that to our advantage. Our process is already flexible because of the way we use TDD to design and implement, so now we can handle requirement changes.

It's a process that happens daily. Design actually gets considered at every test, every minute of every day. We're always designing.


---------------------------------------------
What I find particular troubling is that you say: "it allows us to improve our design over time as requirements change.", without addressing the severe impact this can have on the development phase.

A simple requirement change can make months of work become useless.
---------------------------------------------

That happens when two things happen: (A) The requirements were never clear to begin with (so we messed up somewhere when we got them from the customer) and (B) we've built a lot of code in anticipation of things that may not ever happen.

Well, we try and solve (A) by how we go about getting requirements. We involve the customer in a way that other methods do not. They are part of our team, actually. (More on this in Beck's book).

As for (B), Agile methods preach "do the simplest thing that could possibly work". And by doing so, it prevents you from writing tons of extraneous code that might never get used, and would become 'useless' if a requirement change hit it.

I know it sounds weird, but it works. I would definately encourage you to pick up Kent Beck's book on Extreme Programming. I'm here to tell you, there's more about Agile that I can ever possibly put in this one feedback box. You owe it to yourself to get the book :)

---------------------------------------------
It's IMHO easy to say "we'll improve our design when requirements change", but it's IMHO really hard to avoid severe complications during development because of changing specs.
---------------------------------------------

Agile doesn't improve design only when requirements change. Agile does design *every day*. We actively think about the design every time we write a test, because we don't attempt to solve all problems before we actually encounter them. Every programmer has a responsibility to improve the design if they see a chance to do it. So, by the nature of the Agile process, design becomes everyone's responsibility, and it becomes integral to the everyday process of writing tests and code.


---------------------------------------------
The more you can foresee up front changes, the less you run the risk of having to throw away a lot of work because of a requirement change.
---------------------------------------------


Agile people will say, "The more Agile your process is, the less impact any single requirement change can possibly have on your design and code".


---------------------------------------------
What I'd like to know is:
1) What amount of time do Agile development rules prescribe, so the research you talk about is done properly ?
---------------------------------------------

Whatever is necessary. Again, I'd say pick up Kent Beck's book on XP (Extreme Programming Applied). Read about the Planning Game, Story Cards, and the Iteration process. It is a TOTALLY different way to gather requirements from the customer than you are used to, but believe me - once you do it and see it at work, you will be sold on it as the way to do up-front research.

---------------------------------------------
2) What does Agile development have in store to avoid the severe risk of having to throw away a lot of work because of a design change?
---------------------------------------------

It's the process of how we design that saves us from having to do massive design changes. TDD is our design process. As Scott Bellware said on his blog, "Testing is a side effect." The goal of TDD is really design.




---------------------------------------------
All the stories I've been reading lately about Agile development don't address these 2 key elements. In fact, they skip 1) altogether as every Agile story / article posted on for example weblogs.asp.net is about writing code.
---------------------------------------------

Yeah, Frans, I'd say when you need to do is get to the root of the guys who invented Agile. In this case, the eXtreme Programming I'm talking about, the guy to learn from is Kent Beck. His XP Applied is really good, and goes through all of the things that we do to make Agile work as a development and design methodology. Sometimes, the core stuff gets lost in the blogosphere because people want to focus on one or two things specifically.


---------------------------------------------
Though what's the amount of research that should be done? THe same as what was done before 'agile' became a buzzword? Not likely as with the refactoring time build in, projects using agile development will take longer, so it's likely to be less.
---------------------------------------------

It's completely dependent on the particular task Frans. In Agile, we don't attempt to solve problems that we're not presented with. It's all about providing business value to the customer, which means working, reliable, bug-free code. We ask the customer what they want. They provide us with the requirements via story cards. Everything they can possibly think of, with the knowledge that we *know* they are going to think of additional requirements, or changes to these requirements, as time progresses. In fact, in XP we don't even call the story cards requirements - they are a "promise" to have a conversation later about the feature. So they're really a loose requirement that will be firmed up when we actually start to work on it.

Agile is about solving problems, not anticipating problems that *might* occur.

I really can't add much more - you need to get Beck's book to see the whole picture. Approaching Agile with only blog knowledge - I can totally see why you have so many valid questions about the process. There's a lot of holes. Get Beck's book, it will answer them. When you see the process in total, it will be like pulling back from the surface of the Earth and seeing it from space :) You'll get the whole view, and then it will make sense.

----------------------------------------------
Hi Chris,

First of all thank you for your comment!

What you wrote is in essence what I tried to explain with the AbNQY syndrome. Do you have any horror/success stories to share regarding teams committed such a hybrid process?

----------------------------------------------

I'll have some stuff to share in the next few months and coming year as my team progresses through it. We're new to Agile, but we're seeing the benefits of doing things this way already. I've been doing things in a mostly Agile fashion on my own for about the last 5 years, and when I read Kent Beck's book it was finally putting a name to a face - I was finally able to put words and terms to processes and ideas that I had been naturally practicing for the past few years. What I had not been doing, Kent's book showed me, and we've worked to incorporate those things.

Keep track of my blog, slow is it is on Agile right now. It will pick up after the New Year, and probably get a lot of Agile stuff around May/June as we get into the thick of our iterations.

<a href="http://cigarsworld.net/Fonseca-cigars.php">Fonseca cigars</a> is one of the most exquisite cigar brands in Cuba. Reviews describe this brand as

offering to the consumers all the necessary ingredients for making the experience of enjoying this type of cigar, as unique and special as possible.

What is your telephone number?

<a href=  ></a>

<a href= elite-sex-naruto-porno.hakaka.com >how do they score the new sat</a> <a href= free-lesbains-sex-games.hakaka.com >whitepages</a> <a href= free-video-a4u.hakaka.com >samaritan counselling services utica ny</a> <a href= giada-de-laurentiis-br.hakaka.com >verizonwireless</a> <a href= http://eufrat-cock.hakaka.com >science projects for fourth graders</a> <a href= free-animal-sex-passwords.hakaka.com >westfield home</a> <a href= fetish-photos.hakaka.com >chicago bead sources</a> <a href= child-bikini-model-cartoon.hakaka.com >1984</a> <a href= free-nightshot-porn.hakaka.com >fireplaces</a> <a href= freeware-amateurporno.hakaka.com >netherlands beaches</a>