WinFS delayed again: corporate politics or incompetency?

Friday, Microsoft brought the news that WinFS, the highly anticipated new filesystem annex object store, is delayed again. It is now said they hope to release a test version in late 2006 and it will not be present in Longhorn server as well.

If WinFS isn't ready for the Longhorn server release in 2007, something serious is holding it back. WinFS is already in development for a long time, and 2007 is more than 2 years away. That's an awful lot of time for a filesystem. When I read articles like the one on news.com, I instantly get al kinds of questions in my head. One I got from reading this particular article was : "Is this the same company that wrote SqlServer 7 almost from scratch in 4 years that is now incapable of producing a filesystem within twice that much time?". Reality gives the answer: yes, apparently it is the same company.

My somewhat paranoid mind however refuses to accept the spin Windows Server Chief Bob Muglia gives to the fact that WinFS will not show up for a long period of time. I mean, somewhere on this planet a guy named Hans Reiser wrote with just a couple of people the Reiser4 filesystem, which sports a large amount of complex functionality and is one of the fastest and most secure filesystems known today. (Read more about ReiserFS by clicking here). How could mr. Reiser succeed while Microsoft with all their power failed? A couple of reasons pop up: 1) Reiser4's functionality is small compared to what WinFS will do. 2) Microsoft hired the wrong people. 3) Microsoft has given WinFS a low priority.

According to what mr. Muglia said, it's clearly option 1. Is this really true? For what we've seen from WinFS so far, I can only conclude: no, it's not. Muglia said:

"This isn't a relational database, this is a brand-new data model, and it satisfies a whole class of applications that frankly have been unsatisfied from a data model perspective since the beginning of history. We've been working on things like this for a long time."

A brand new data-model... Now, I'm not a manager, so I don't know the jargon managers tend to use, but in the world where I live in, the professional software engineering world, a 'brand new data-model' doesn't exist. A couple of decades now a lot of very bright people are doing research on different data-models, database formats and how to handle data. But frankly, the last decade, it's more about the handling of the data than the datastore itself, basicly because everything to be said about storing data has already been said.

So it's all about the handling of the data, i.e.: creating information from structured or unstructured data. Let me dig up an old slashdot posting I made some time ago:click here. The only way WinFS can be something 'brand new' if it's something like that: files are views on data objects in a datastore. Is this something that's new or hard to do? Well... no. You see, Windows works, as every other good OS does, with a virtual filesystem. This means that what the OS API sees as the filesystem is actually a virtualized view on all filesystems mounted. This means that if you want to plug-in a datastore as a filesystem, all you have to do is write a filesystem driver for the virtual filesystem and applications can use your datastore as a filesystem. With an object store at the filesystem core, the virtual filesystem should be extended to offer access to that store as well: not only via the concept of a file, but also via the concept of an 'object'. But the bottom line is: a 'file' is just a view provided by a driver, how it's stored internally is not important, that's the job of the driver and datastore (be it NTFS, FAT32 or SqlServer).

The real problem is in the applications: if they save a file which is actually a collection of objects, and the file arrives as one big object at the gates of the filesystem driver, it can't chop it up in a clever way so it can re-use the individual objects. But this is not the problem of the WinFS team, but the problem of the people writing applications. In other words: deliver the filesystem first, then help application developers migrate their applications to the new filesystem format. This can never be the cause of the delay, as the more you delay the filesystem, the more time it will take to make people convert their application structures to the new filesystem API.

Let me get back to my original question: How could mr. Reiser succeed while Microsoft with all their power failed? With all due respect, but I simply don't buy reason 1) that it is so incredibly complex that it is so incredibly hard to do that it takes such a long time (we're talking 8-10 years) to create this filesystem called WinFS. I also don't think Microsoft hired the wrong people. So we arrive at reason 3: Microsoft has given WinFS a low priority. I actually think, but I'm speculating here, this can be close to the truth. The reason why I think this, is fairly simple: Microsoft is a demand-driven company. It has to make money, so it delivers what people want. A simple, yet, effective strategy, used by a lot of companies around the world.

Unless you've lived under a rock in the last couple of months, it's pretty obvious what the next big thing will be on the desktop: search. Everyone who has seen the Steve Jobs video where he demos the desktop search power of MacOS X 'Tiger', knows it: this will be huge, if not the next big thing which makes an OS a real OS instead of a toy. "At least on the desktop", you say? No, it goes far beyond the desktop. People in corporate networks see the part of the network they're allowed to see as part of their desktop. So a search for some data based on some criteria might look like it's a desktop search but might turn out into a network wide search. The company which can bring that power to the desktop is king.

Microsoft has a problem though: time, or better: Time to Market. It works against them: Google, Yahoo and others have already announced their desktop search engines which will deliver to the Windows desktop what mr. Jobs showed for MacOS X. But not in 2006 or 2007, no, it will be next year, or even: today. So what do you do when you're Microsoft? You continue with this huge project that will deliver a big leap forward in how we work with data, but it will take a couple of years to complete, or you look out (pun intended) your window and buy a small company with a clever guy who can deliver what your customers want on time and what will be delivered is good enough for these customers? The theoretical academic will say: "Go for the huge project!", but a person who knows how business works will say: "Go for the clever guy in the small company!". So Microsoft bought Lookout, and desktop search will come to you in Longhorn client, and because WinFS is delayed, probably also in a server version in Longhorn server (but that's speculation from my part).

In software engineering a famous saying goes: "Nothing is more permanent than a temporary solution.". The temporary solution here is the desktop search functionality that will be included in Longhorn. It's temporary because WinFS will show up eventually in the future, but... will it? Would you invest a lot of money in developing WinFS if you had a temporary solution that is good enough for your customers? Would you invest a lot of money in developing WinFS so that it can replace MS Access and SqlServer desktop edition? Just to replace a search functionality (remember, the user doesn't care how it works) that is good enough, i.e.: it doesn't make a difference in usability?

You do the math.

Published Sunday, December 12, 2004 1:47 PM by FransBouma

Comments

# re: WinFS delayed again: corporate politics or incompetency?

Sunday, December 12, 2004 8:26 AM by Paul Wilson
You're probably right, but I've always had a slightly different impression of what MS wants to do with WinFS -- of course it might be the wrong impression. I thought they were not just wanting to move the "file" to the relational datastore, but also have a significant amount of the "file" schemas to be an integral part of WinFS. One of the examples I've seen, although again maybe I misunderstood, was that whether you use Outlook or Outlook Express or some non-MS client, all of these apps would work with the same schemas for contacts, calendar, email, etc. Similary for all the imaging applications, and I assume most other such common apps. These standard schemas would then make search, the very thing you state is what's important right now, much more workable!

OK, so that's what I've thought WinFS was supposed to be. That doesn't seem trivial to me, although I think the difficulties would be much more on the schema design side than the technical side, as you have rightfully noted. What troubles me, if my understanding is anywhere close to correct, is that I don't want MS doing anything like this at all! They may know what's needed for Outlook and Outlook Express, but they have no idea what my custom apps need, so are they going to make this open enough to be usable for everyone? And do I really want my information stored in such a standard schema anyhow, since it seems that will just make it easier for other rogue apps to get to all my valuable data?

So I think that either WinFS should be doable and MS is just screwing up again, or they are going for something much more -- something that may make things far easier -- for both legit uses and other rogue uses too!

# re: WinFS delayed again: corporate politics or incompetency?

Sunday, December 12, 2004 4:21 PM by Robert Scoble
Frans: yeah, come back on Monday. And watch the videos on Channel 9. You'll see how a team shipped a really cool new product in only seven months.

# re: WinFS delayed again: corporate politics or incompetency?

Sunday, December 12, 2004 7:37 PM by Shannon J Hager
Desktop Search would be easier with WinFS but today's desktop search products (like my fave, the incredible and free Copernic Desktop Search) use an index instead. WinFS was my favorite "pillar" of Longhorn but I think MS got a lot of wild ideas and tried to incorporate them all and ended up with something they just couldn't handle. It is very unfortunate.

I hope the lack of ObjectSpaces means more people will try out LLBLGen Pro.

# It's happened before...

Monday, December 13, 2004 1:39 AM by Peter da Silva
Anyone remeber the talk about this new object-oriented desktop for Windows NT that was going to make IBM's object-oriented desktop for OS/2 look sick? Was that going to come out in Cairo? I forget, I'm pretty codename-deaf to begin with, all I remember is that this was the coming thing, and whatever happened to Windows 95/98 it was just a stopgap.

Remember that what we got in NT4 was the Windows 95 desktop. It even had 16-bit code back in its hindbrain, like some digital r-complex left over from the precambrian. And that's what we still have on the desktop... the nifty new object-oriented desktop got shuffled off to the same place Xenix and Microsoft Bob ended up.

# re: WinFS delayed again: corporate politics or incompetency?

Monday, December 13, 2004 4:04 AM by Frans Bouma
Dare: You say: "At its core, WinFS was about storing strongly typed objects in the file system instead of opaque blobs of bits. The purpose of doing this was to make accessing and manipulating the content and metadata of these files simpler and more consistent." (in one of the links you provided). I don't think this is true. The reason for this is that what you describe is exactly why object databases failed: data is just data, it depends on the CONTEXT what the meaning of it is. So storing it in a strongly typed way is ok, as long as the strong type is not too stong, i.e. it has to describe the data to retrieve the data, not the meaning of the data.

Let's say you're right. Why would WinFS be necessary? Would all of a sudden developers drop support for oracle, sql server and friends and store their data in WinFS? I hope not. WinFS is 'a' (not the) way to expose data of an app to a generic interface, f.e. the shell search functionality, so the user can find data back (or better: make information out of data, by specifying a query and data matching the query becomes information in the context of the query. This is fundamental. ) and open the app to grab the data in context. Or do you think that when outlook 2009 stores the contact information in WinFS, you can use it from a 3rd party app? You still need office/outlook.

Desktop search engines do the same: they index data, have a query interface and the user can find data back, open the app to use the data in context. (data found in a PDF can only be utilized if the pdf is read/opened)

So I absolutely disagree with your on the fact taht WinFS is not about search. Its sole purpose is to provide search to the desktop, by providing a platform for application developers to make searching any type of data very easy.

Or are you thinking it is for application developers so they don't have to deal with databases anymore, just store objects in the 'store' ? That would be a big problem, because it would cut into their sqlserver sales on one side and it would provide a problem to the developer on the other side: when to move from filesystem object store to a database because the store gets too big? And as if there are no solutions to using data as objects today ;)

# re: WinFS delayed again: corporate politics or incompetency?

Monday, December 13, 2004 5:04 AM by Ralf Westphal
I mostly agree, it´s a political decision. But partly I also think, it has to do with the vision behind WinFS: Microsoft has/had too big a vision of what it wanted to accomplish. It wants to make everybody happy, to finally provide a silver bullet for all(?)/most data access problems, to close the impedance mismatch chasm between OO and relation world. And I guess, this task is (currently) too large even for Microsoft - or maybe especially for Microsoft since they need to provide backward compatibility with all sorts of stuff.

So my conslusion is: Live in the now! The larger the hype around Longhorn technologies the more I get the feeling I should look at what I´ve got today.
It´s nice to get a glimpse of what the crystal ball shows once in a while. But doing business today, most companies cannot (and should not) focus so much on what possibly might come sometime in the future. Avalon, Indigo, WinFS: Really nice - but not manifest for many, many months. And that means: not relevant for most (not all) developers.

So even though I´m a Microsoft Regional Director for Germany, I shut out most of the noise coming from the hype around Avalon, Indigo and WinFS. The only real and tangible thing, right now, is Whidbey Beta 1. .NET 2.0 and improvements to VB.NET/C# and SQL Server I can already use today and which will be generally available in June 2005 are (somewhat) relevant to my customers and me.

Above that... I look beyond Microsoft. There´s more to care about than Avalon, Indigo and WinFS. What about O/R Mapping for .NET today? What about easier GUI programming today? What about tuple spaces/shared virtual memory for (cross-platform) communication? It´s all here today and needs to be mastered.

# re: WinFS delayed again: corporate politics or incompetency?

Wednesday, December 15, 2004 4:00 PM by Diego
Winfs is much more than search, but with this delay....i start to think that winfs has the "second system" syndrom. What winfs wants to provide to apps can be done now - it's just much harder to do. IMHO, WinFS has become too complex. In that timeframe they could rewrite NT core, instead they're just creating a tool to solve a problem that maybe it's not so important.

# re: WinFS delayed again: corporate politics or incompetency?

Thursday, December 16, 2004 5:29 AM by Alex James
Frans,

I think there are a couple of issues at work here:
1) There is a lot of perfectionism creaping into the WinFS team, it is all or nothing. Why slip WinFS, surely a stripped down feature set would still be useful?
2) The business side of MS has probably quite rightly got the feeling the WinFS technical team is unlikely to ship anytime soon (see 1) and has started to focus on immediate threats like Google.

My reasoning goes like this: I suspect the MS WinFS team is working on a development model that is the exact opposite of XP development, they are trying to resolve every problem they can think of before they need to.

For example do you really need to support Multi Container cascading security updates? Surely this with this idea you introduce a whole heap of GUI paradigm problems, this is probably where they are having trouble, how do you present this stuff to the User?

The idea that things can appear in many places and more dangerously can be modified or (affected by other modifications) that are not immediately obvious is where a heap of problems come from.

I am convinced that they are making it more complicated than it needs to be because they have too many academic brains involved and not enough pragmatists. A pragmatist would question the need for some of this stuff, but it seems this isn't happening, and as a result the whole project slips, instead of release2, because it is very hard and they aren't prepared to compromise at all.

Of course this is all just a guess ;)

# re: WinFS delayed again: corporate politics or incompetency?

Friday, December 17, 2004 3:03 AM by Jeremy
I don't think WinFS is "about search" more than it simply "enables search". It's MUCH easier to search a relational database than to search a file system. Why? Keyword: Relational. If the data is related in someway, ORGANIZATION (which trumps search in all forms) is what defines the system.

I have a theory and I probably should be slapped for this but I believe WinFS as it is is perfectly fine. I believe that they are simply having a very difficult time introducing it into the Windows code base and have it actually function properly. Much of Longhorn now still has many XP roots and I think Longhorn isn't that "new and improved" to easily just drop in WinFS and have it automagically work.

WinFS as a concept is probably already done but the incredible time consuming process is to make Windows itself work with it. Sure there WERE file system drivers you could just drop in but this is something completely different. Maybe they could have started with a file system driver and built on the existing technology but part of the equation could be that they decided to start from scratch and are realizing that Longhorn isn't quite the total sum of all parts it claimed to be when it was first conceptualized.

Microsoft sacrifices a LOT for backwards compatibility, lets not forget this. Sometimes I question that they sacrifice too much, but they've done a good job about keeping a happy medium. Backwards compatibility may be coming into play here though there's really nothing they could be worrying about with WinFS except maybe MSDE (which they proposed WinFS would do away with completely).

I'm probably completely wrong since I haven't kept up with any of this since I heard about the initial cuts. Also there was a ton of Longhorn talk in blogs around the WinHEC build release but things have severely died down since then so I don't really keep up with it any more. It doesn't mean I can't speculate and form conspiracy theories since I don't particularly work on that team.