So, it has been exactly a week since I have been to dotNETwork 5th gathering. This is yet one of my too late events coverage (maybe I should write a long boring post about why that happens lately in as much annoying details as possible!). Let’s hope we have something worth the wait this time …
Intro (No tech – you can skip)
This time I’m talking about the second session in the last gathering. The 1st session was already covered earlier. This session is special to me for two reasons: It was delivered by my of my dearest friends in the field, Mohamed Samy. I know Mohamed four years ago, since He was moving from VB 6 to C#, until he became one of the super guys in the field of enterprise computing and architecture in the .NET in Egypt, being a technical architect in ITWorx, one of the top posters in MSDN forums, and finally getting his Solution Architecture MVP lately (which he tells us a story about). The second reason is that I had the pleasure to see the latest part of his preparations for the session, and it was very interesting to see “the making” as well as “the show”.
Anyway, getting back to the session… The microphone stopped working at the very beginning. Mohamed moved to the middle of the rows and looked like a student leader talking to his fellows. this warmed things up a lot as well. He started warming them up by going really quick on his start as a developer, and choosing to continue in the technical career path (technical lead, architecture) while most prefer to take the manager path (project management, etc). He told us about his MVP how he was so happy to earn it in “Solution Architecture” then having the “Solution Architecture MVP” canceled, and having to choose a product specific MVP award instead. He went for “VSTS MVP” as relatively a bit close to his work, having “Solution Architecture” as a “competency” to the “VSTS MVP” award.
As we got lost in our way to the CIC (Canadian Collage, where the gathering took over), he mentioned that software is like finding the CIC, you ask people you find in your way about the road to go, each gives you a different answer. You need to get to that road and not ask again.
Going more into the session
Buzzwords
Mohamed went through how “XML” suddenly became a buzzword. People tell you you’ll get rid of SQL and Oracle and replace them with XML files. It’ll solve all developer problems, and all this kind of talks. He bought a 1500+ pages book about being “proficient in XML” and he still didn’t understand what XML is really about. This is a buzz, just like the university sign, it tells you go certain way and everybody just follows. Later Mohamed recognized the simple fact that XML is just a text format based on tags that can be consumed within any platform. This is good because it solves communication problems between platforms (say COM from Microsoft and CORBA from Java) that were problematic because of using specific binary formats not text formats that before XML it was meaningful to prepare an entire PhD on communication between such platforms!
From this example, we see that every buzzword is said to solve all our problems. Few years later, the buzzword will just get deleted in favor of newer one. People need to understand the “concepts” behind the buzzwords not just how to use the tools and technologies related to them. He made fun of how people earlier earned so much money by writing about “Java & XML” when “Java” and “XML” were both buzzwords!
Patterns, Anti Patterns and Architecture
Getting more into the topic Mohamed asked the audience whether they know what patterns are. Well, few did! So, he explained “patterns” as solutions to common problems. He said that even architecture has much to do with this (more on that coming). Same as in building, there`re patterns in designing a living room or even a kitchen (like American furniture designs and such).
What happens is that someone faces a problem and solves it. Someone comes with better solution etc. and many people like it and repeat the same thing when solving the same problem. Someone draws UML diagrams and writes explanations for what the problem is and how the solution works, and this last work is what we call a pattern.
So, if patterns are the common practices we do to solve a repeated problem, then, following naming analogy, Anti Patterns will simply simply what what we should NOT do when soling our problems!
How about architecture? Lets use analogy with architecting buildings again. If a building buildings “designer” (similar to software designers) worries about the best use of spaces inside the buildings themselves, then the “architect” worries about the space “between” each building and the other, the green areas that should exist, where to have a garage, what ration should be between the building size and the width of the street it is build in, etc…An architect worries more about the interaction between systems and each other. Many applications have to talk to each other. Looking at a sample finance application, you may have an HR application that applies bonuses for example, for that, the HR application will have to talk to say a payroll financial application to apply the bonus payment.
But, not all applications run on the same platform, even more, each of them is likely designed in different way than the other. But we still need applications to talk to each others. That’s why we look at standard protocols, like HTTP, HTTPS and such. Those enable us to perform “integration” between those systems.
SOA, Who ??
SOA (
Service Oriented Architecture) is an architectural style. What dos this mean? Well, back to buildings analogy. Mohamed showed pictures of example of Islam and Romanian architecture. All the audience could recognize each. The common parts in Islamic or Romanian architecture are what define an architecture style. The common parts between different architectures are what define the architecture style.
When you recall a 3-Tier architecture, you know what you are talking about, data access, business logic and UI. When talking about “Service” Oriented Architecture, you expect to see services.
What is the goal of SOA? Every architecture style has a goal. In buildings, why are the ceilings of the old houses very high? It was a pattern that solves a common problem. They no good fans or ACs back in that time, so, they made ceiling very high so that warm air goes up and less warm air remains in the normal height – to understand what the goal of SOA is, let’s get more on what SOA is.
SOA is an architectural style that emphasizes standards based integration between systems. This means enabling the components of different systems to talk to each other using standards. That’s one reason why WCF is good BTW, because it enables you to make components of the same system talk to each other in binary way, while still allowing components to talk to other components outside the same system using say XML. Clearly, this is not the easiest way to build applications nor one that guarantees success of the system. Not all applications should be built leveraging SOA, actually, trying to do SOA in all applications is a common mistake, Mohamed says. A Point of Sale (POS) application in a really small shop for example, has nothing t do with SOA.
Mohamed gave an example of Egypt back when there was no interaction between countries, all the goods are either made locally or very hard to get. Now, with all worldwide agreements, borders on getting and sending stuff are removed. SOA features standards: HTTP, SOAP, etc… It’s all about friction free interaction between different systems (again, standards based integration). It’s just a way of designing your application, a way of thinking.
Of course there’s lots of buzz/hype around SOA. You hear all that stuff about “Loose Coupling” and such. Some people might say applications should not not even know each other and have locators to tell them how to find each other. These are points of different opinions. Who said you need this overhead (or you don’t)? You still in all times have the overhead of translating from your internal system types and data to the standards format and the reverse overhead on the other system, so, if you are designing say a real time application with thousands of transactions in seconds, SOA will not work. If someone tells you 2e’ll go for SOA to improve performance, just tell him “you are so wrong”!
So, If I understood SOA is designing the system having services communicating with services of other systems using standard protocols like HTTP, and formats like XML, etc, etc… What is a “service” at all then? A service is an interface to your business process. When you are renewing your car driving license, you fill a form that is agreed upon between you and the responsible department, typically (does not quite happen in Egypt) you should be able to give it to only one terminal, without having to know say how many employees it goes to inside. This makes me personally think of a service as a method that will be called remotely with standards protocol and also parameters in standard format that a method caller should not worry about what the implementation of the method actually looks like.
The 4 tenets of SOA
By this we mean those tenets:
- "Boundaries are explicit"
- "Services are autonomous"
- "Services share contract and policy not class"
- "Service compatibility is about policy"
Boundaries are explicit
Mohamed showed a picture of a country border. There’s a fence, a STOP sign, a guy holding a microphone. If you are going full speed in the highway from Egypt to Libya, you have to stop at the border between the two countries (slows your total average speed). You also have to show say your passport and maybe visa (Authentication and Authorization). Services also have logical boundaries, could require authentication and authorization.
Also, when you re doing a money transfer from a country to another. you need to convert you local units in your country to the other country’s local units. For services, this is changing from the internal data types and schemas in your system to those of the system that has the service you want to connect to. Crossing the boundaries and doing the transfer takes time, that’s why we said that SOA is not the way to solve application slowness.
So, what are the boundaries of each service? We define that when we are architecting our services. We first agreed that those are logical boundaries. An example is, who should be able (and hence responsible) for changing a customer address? A sales application or a marketing application ? This is one kind of the questions we ask our selves when putting the boundaries.
Services are autonomous
If there’s a problem in having fish in Libya, should Egypt be affected by that? Of course no. Same with services. Each service should specialize in one operation. Services should be as least dependent on each other as possible.If service A depends on service B internally and for whatever reason service B is down, Service A should (ideally) still work with 100% its performance, or (more realistic) should provide downgraded performance. It should not be completely stopped.
Back to driving license renewal analogy. If the renewal requires getting your traffic offences, and those are got from say the ministry of justice that is down, it should tell you to come another day, or better off offers to give the renewed license to your house that other day. Note that “another day” is a degraded service.
The point is to make sure to not crash if the dependency is not found or down. Otherwise, a failure in one service, will fail all dependent services, that will cause failure to other services depending on them and so on, that you can easily find your entire company business down because of this.
Services share contract and policy not class && Service compatibility is about policy
The two are related. Back to countries analogy, to get the visa to travel from Egypt to Canada, you need to get police clearance(showing you are not a criminal). This is both a contract and policy. The contract is the police clearance certificate itself. The policy is that if the one applying is a criminal, s/he will not be allowed to get the visa.
For services, the contracts are the schema of data transferred between those services. The contract could be different from platform to another even when using standards. For example, if your service returns a customer record by sending a DataSet containing that record after applying XML serialization, the Java fellows trying to call your service will have hard time parsing the XML that gets generated from this serialization. Instead of that, you should send a typed Customer object XML document. A document that has the information needed clearly without any platform specific additional tags.
Policy is the protocol of the contract. For example, if the service requires authentication/authorization, works on HTTPS not HTTP or so, has certain timeout for results, etc… al this would be parts of the service policy. There’s an overlap between the policy and contract though (is the schema being XML or binary or what a policy or a contract?). In general, the policy is the definition of what’s allowed and required in the conversation between the service and its client (caller).
Thinking of contracts and policies, Mohamed showed that you should not require more data than you need. If your services just changes the address of the customer, it should require a “Customer Address” document not an entire “Customer” object document. This is called “Just Enough” Validation (MSDN). This is good also because the data is meant to come from a likely remote location, requiring less data will make things go faster.
One reminder on polices as well. This is like whether to use SSL or not, what protocol to go to, what to enforce, etc.. Mohamed mentioned the WS-* specifications that can give you in-the-box encryption, trust, transaction and other services, but there was no room to talk about WS-* here as we needed to move to the title of the session, SOA Anti Patterns ….
The Anti-Patterns
The session was originally called “Patterns and Anti Patterns of SOA”, due to time limitation, and prior permission from the dotNETwork guys, the session was split to two parts. One for the patterns, and one the anti patterns, which Mohamed has chosen to start with Anti Patterns as to clear up the thinking before telling us what we should be doing. He’s more interested in what SOA is NOT than what it IS. Let’s see what he talked us about…
Remember, this is what you should NOT do.
Anti Pattern: “CRUDy Interface”
OK, so we want to start SOA, and want to expose our system components as services. Let’s start by the easiest parts. Our system will always have to deal with Create (Insert), Read (Select), Update, and Delete operations (CRUD). Lets create services that encapsulate those – WRONG.
Changing a record can have so many side effects, let’s take an example of changing a customer address. This sounds very direct (update customer set address = …), and that’s it. Not really! If you change a customer address, many things could change as well. Tax rate for example. In Nasr City district in Cairo there’s some area called “Free Zone” or “El mantea’a El Horra”. If your business is in it, you’re exempted from taxes for certain years. If your business is anywhere else in Nasr City or Egypt in general you do not get that. So, changing an address from or to this area even still within Nasr city can have a big effect. Also, you may want to send an email to the branch that the customer will be transferred to when changing address, so, this is taking an action not just changing the address.
It happens in reality. If you change your address in the ID card, it does not affect your driving license, passport info, or bank account. Some guys came with an argument that it should, as al the systems should have the same database. Mohamed said that in reality we could have very good system for banking running ith IBM stuff, a good system for car licenses running on SQL server, while the ID system might be Oracle based. Because the systems are already packaged into different databases and database schemas, they just cannot share the same DB.
Anti Pattern: “Enumeration”
This means something like CustomerService.MoveNext();. A virtual curser (meaning a pointer, not SQL curser). This allows the client to treat the data and iterate (loop) on them as if they were already loaded to the client, while they’re truly on the server. For the server to be able to enable this. It’ll have to hold the entire list for each client, and that would be a huge performance headache on the server, and the service should not have to worry about caching this and storing the last position and such. Remember (from the tenets) that services are autonomous. A service should care about one operation.
To explain this, Mohamed mentioned a system he joined in the past after it was created. There was too much work done on the DB server for updating recording dependant on each other and/or updating some complex calculations fields, etc.. This was done via SQL dynamic cursors. The cursors have been running all day locking the data in the DB from being accessed by any application.
The point is, the consumer (AKA client) of your service should not be the one who controls your server resources. You have no guarantee who’ll call the services. Even more, you have no guarantee who wrote the code that will call your service (even if you know which system it is via authentication), and you cannot guarantee whether they’re good developers or not.
Anti Pattern: “Chatty Interface”
This is when you require say 5 different service calls to do the same thing. Something like Customer.MarkForDelete(); Customer.Delete();. Two calls to perform deletion. This way you required the consumer to go all the way long from the client to your server and backwards two times instead of one. Well, not just that, how about if “MarkForDelete();” was called first and “Delete();” was never called??
This is actually something you may want to do in some certain cases, but most of the time you don’t. The problems with this are the additional time things will take to perform just one operation, and that it can easily get your data into an inconsistent state. If some of the steps were called and the next steps were for whatever reason not called, your system now has a weird state!
Anti Pattern: “Loosy Goosy”
Why don’t you just make all your services a single services that looks like XmlDocument PerformOperation(XmlDocument input) {…} ? BAD BAD BAD.
The point here is the difference between implicit and explicit behavior. If you do the example above, how would the clients know the way to call a certain operation than another? How will they know what data you need to be filled in the input XML or what to expect in the returned XML? An explicit behavior DOES matter. When creating your services, give them good names that shows what operation each service does perform. When defining your inputs, remember the “Just Enough” Validation rule. Make XML schema for the fields you need only better than using a bigger XML schema that already exists. A services that updates a customer address only required a CustomerAddress document (With address and ID inside) not an entire Customer object document. You should also be explicit about what the service returns, whether nothing, a document with certain info, etc..
Why is implicit behavior bad anyway? Mohamed gave a very funny example.He went to work on an old COBOL based government system for salaries (salary laws were defined in 1936 and had endless changes afterwards all handled by this system),. The system was a single long routine with no modularity at all, and when trying to make it more modular, Mohamed got stopped at an entry point to the application that takes 3 flags.ZEFT (Asphalt), MIDO (fun name), SOSO (fun name). He couldn’t know what those 3 flags refer to until he asked the people using the system. They told him the woman who created this system has chosen MIDO as per the name of her son, SOSO as per the name of her daughter, and ZEFT (meaning asphalt) as per the name of ….. whatever :D.
People kept a notebook including what exactly the operations that are performed when you set ZEFT, MIDO and SOSO to certain different numbers. You can imagine what maintenance nightmare Mohamed found himself in, and this is exactly the nightmare you give your clients when you go for an implicit behavior. Mohamed says it’s like a house of dominos. It doesn’t survive long.
Last thing Mohamed mentioned was about the “Just Enough” Validation. What would you if your system earlier required a customer document with single field for address and now happened to have different fields (Line1, Line2, City,…). You then need to version your services. He showed how this can be done with XML serialization of objects. The main idea is you use your platform native types as long as you are inside the same system’s boundaries, and serialize/de-serialize that to/from different XML documents when returning/accepting XML data in your service. You can apply “version” and “optional” flags to allow versioning of the documents.
Just after the session, I asked Mohamed to show his Links slid, and guess what? He had my blog (with the old GeeksWithBlogs URL he likes more than this one) in his SOA recommended readings!!! I can’t pretend like I didn’t know it, having asked him for the references myself made it very embarrassing to me. But anyway, thanks a lot, Mohamed for this invaluable trust.
Conclusion
As I mentioned before, these are only the Anti-Patterns. The patterns are to come in the next gathering in July (God Willing).
Now to my comment: Mohamed did well for a guy explaining this to people who did not know what design patterns are. I’ve seen few advanced developers sitting in between and they were looking satisfied too. The microphone problem was a good thing for him. Everybody was happy, but for me, I was still missing some. Maybe that’s why he kept kidding with me about along the session “you feeling OK now, Meligy? :D :D” :D :D :D.
I didn’t like that he assumed SOA is all about SOAP and WS-* while other paradigms exist like REST and such, but I cannot blame him at all, as mentioning such would probably cause a huge disturbance to people and could simply get them completely lost. I also was sometimes losing the software example that maps to the building/countries/real-life examples he gave. That was in few cases though. Maybe I just wanted to hear so much technical stuff though! One more thing is that he had a very interesting flow of the talk that he could highlight and clarify a point either before or after its original place in the presentation. This didn’t cause people confusion, but it was really tiring when came to transfer my session notes into this blog post, that I feel it does not tell the content of the session in a proper way.
Generally it was a very good session. If you could please people with different knowledge levels to good percentages in such a complex topic, then you have done something. Good job, Mohamed. You DID meet my expectations. I’ll be there for the Patterns session (God Willing).
BTW, Mohamed is now working on getting his VSTS related portal, theVSTSguy. I’ll update this blog as soon as he gets it up and running.
Technorati Tags: dotNETwork,
.NETwork,
Local Events,
Egypt,
SOA,
SOA Anti-Patterns,
SOA AntiPatterns,
SOA Anti Patterns,
Mohamed Samy,
Architecture,
theVSTSguy