Jon Udell gets assimilated! ;)

I have been a fan of Jon for 18 years. This is great news. I think he will have a very important role in the ongoing Microsoft change.

With Rory and Jon on board, Channel 9/10 looks like the dream team.

Now, let's see how I do it this week...

Transactional File System in Windows Vista (Part 2)

In my last post I tried to describe what I think is the single most important issue with the implementation of TxF (and TxR).

Now I want to explain how I think the pre-existing file name-based APIs could have been changed (or could still be changed in future versions) to allow for opting-in Transactions for virtualy any code that works with files, including things like System.Data.DataSet.WriteXml(String).

I take this just as an exercise for my mind, as I am almost sure that someone thought about this solution, but then discarded it for a reason I cannot discern.

First of all, let’s say that all code running inside or on top of Windows, when it needs to access the file system, ends up invoking on of a relatively small set of Win32 APIs. Of those, some are file-name based (like CreateFile, DeleteFile, SetFileAttributes, etc.) and some are file-handle based (like GetFileSize, ReadFile, WriteFile, etc.).

Second, you always have to call first a name-based API to get a handle you can use with a handle-based function.

Third, no new Transacted versions of handle-based APIs were created. Instead, handle-based APIs get transactional behavior only if the handle passed is already associated with a transaction.

(Still, it bugs me why some name-based APIs were not replicated but instead behave like in the beta 2 model, becoming transactional depending on the ambient transaction).

So, you only signal that want to participate in an existing transaction at the time you call name-based APIs. In the current model, the way to do it is calling the transactional version of the function (like CreateFileTransacted, DeleteFileTransacted, SetFileAttributesTransacted, etc).

But what all those functions have in common in the first place, is that they receive a file name as a parameter!

(Edit: What follows is my proposition, not how TxF/TxR works in Vista. Somehow, after some editing I got the text wrong. )

What if we change rules a little: If the file name is prefixed with a moniker like, for instance “txf:” or “txfile:”, then it becomes transactional. You can understand it as designating a new namespace for TxF, one that points to the same file system store, but behaves differently.

There are already many rules about file names. However, for me it makes sense to add just one more in this case. Of course the exact form of the prefix is not important (NTFS seems to prefer other kind of prefixes).

By this plan, calling CreateFile(“txf:foo.txt”…) would be equivalent to calling CreateFileTransacted(“foo.txt”…). Something similar could be done with TxR.

I think this change would integrate perfectly with the model that shipped in Vista, meaning that you could mix and match calls to the Transacted functions with calls to the “normal” functions with the moniker prepended to the file name.

Also, existing client code and code “hidden” across the programming stack would not need to opt-out. It would be automatically be not transactional because its hard-coded file names (or registry key names) would not contain the transactional moniker.

But the real benefit would be that this could enable us to do things like myDataSet.WriteXml(“txf:foo.xml”) or [insert any other function that takes a file name as a parameter](“txf:bar.ext”), without waiting for a revision of them. Virtually all the programming stack could enjoy the benefits of TxF, without requiring modifications.

I can only think of a few functions that try to parse the file name and that could fail on the presence of the prefix, but I think those would be rare exceptions.

What do you think? If you like it, you can vote for this as I entered a closely related suggestion to the .NET Framework team in Microsoft Connect.

Transactional File System in Windows Vista

In November, 1998, Microsoft Transaction Server was about a year old and SQL Server 7 was just arriving. I had at hand the task of coding a small CRM-like application in Visual Basic 5. Among other features, it had to upload unstructured documents and keep them linked to rows in a database.

I had one important decision to make: Should those documents be stored in the database itself or in the server file system?

SQL Server 6.5 had a lot of limitations with its lack of row locking and some performance issues with BLOB columns.

On the other side, the file system lacked transactional capabilities, and I lacked the ability to create a Compensating Resource Manager.

A transactional file system would have been super useful.

In November 2006, eight years later, Windows Vista is available. Transactions were introduced as a new feature of NTFS, named TxF. The Windows Registry is also getting support for transactions in Vista, under the name of TxR.

Before TxF, for instance, if you wanted to get ACID-like behavior from multiple file system operations, you could, but you had to fiddle a lot with temporary files, renaming, etc. With TxF you just issue something like a "begin transaction", then do your stuff in NTFS, and last, you commit or roll back the whole thing.

This way, TxF pushes best practices under the rug, and pushes the developer one level of abstraction up regarding files.

Surendra Verma, Developer Manager in the CFS group, explained how TxF/TxR works in Channel 9 some months ago. But it is interesting to note that after the video was recorded, there were major design changes to TxF/TxR.

As Jim Johnson explains in this first, second and third posts, from Beta 2 to RC1, TxF API changed from "implicit transaction enlistment" model that worked with the existing Win32 file APIs to a more explicit model for which new "Transacted" versions of some APIs were added.

In the first version, you just did something like:

EnterTransactionScope();
// do whatever file work with your favorite file APIs
ExitTransactionScope();

Everything you did in the middle got automatically enlisted in a thread specific ambient transaction.

In the new model, you have to do something like this (some function names were invented):

hTransaction = GetTransactionHandle();
hFile = CreateFileTransacted(... hTransaction ...);
// do whatever, but now using new *Transacted APIs
CloseHandle(hTransaction );

The complete listing of APIs that were affected by TxF is here.:

If you take a look at it, all the APIs for which a new Transacted version were created are file name-based. Besides, some existing APIs were updated and are now transaction aware, meaning that they acquire transactional behavior in the presence of a file handle that is associated to a transaction (for file handle-based APIs) or in the presence of a thread level ambient transaction (for yet another group of file name-based APIs).

The reason Microsoft change models, as explained by Surendra in the discussion of the video in Channel 9, is that the more simple original version, had a major drawback:

Between any pair of EnterTransactionScope()/ExitTransactionScope(), every single file or registry operation made by any code, even code lost in the middle of the programming stack was automatically and forcefully enlisted in the ambient transaction, acquiring a behavior that was often not intended at the time such code was created.

You could not opt-out.

So, if implicit transactions means that current code will break or misbehave, it is good that they abandoned this path.

On the other side, the main tradeoff of the new version, in my opinion, is that it is "too explicit":

Only those new APIs and those that have been changed will get transactional behavior. So, the hundreds, if not thousands, of higher level APIs that somehow affect the file system, won't get the possibility of having transactional behavior until the whole stack gets updated.

Now, you cannot opt-in.

For a .NET developer like me, this means that I have to use a lot of interop, or wait until new versions of System.IO.FileStream, methods like System.File.Delete, and even that the brand new APIs in System.IO.Packaging get revised to include the option of using transactions.

I have been thinking of a deceptively simple change they could do to the existing file name-based APIs, that could solve this issue. I must be missing something, or they would have implemented it on Vista.

I tried to discuss my idea con Surendra, but he is probably having vacation after shipping Vista :)

I will try to explain the idea in my next post...

Arbitrary precision types for .NET (and some musings)

I came up today to this article that describes two new arbitrary precision types for .NET. One is an Integer, and the other a Decimal (I think it is not correct to call it a Double as in the title).

Although I don't often use arbitrary precision math, this is a welcome addition. It is also interesting to think how nicely such a feature can integrate in the existing framework, given generics and the unified type system.

Speaking of which... I found recently that the Parse and TryParse methods that numeric value types share do not belong to a common generic interface, reducing its usefulness in some scenarios.

I hope we will someday see something like IParseable<T> together with IComparable, IEquatable, IConvertible and IFormattable.

And what about the MinValue and MaxValue constants? Shouldn't they belong to a common type?

There is also this internal class called Number that seems to hold the implementation of methods of every numeric type. I wonder if they could somehow introduce a Number class as a common ancestor without breaking value types rules.

Ashamed

I got the call, I went there, but then I disappointed myself.

No matter if I was trying to create a recursive function that wouldn't suck as much as the usual f(n)=if(n<=2,1,f(n-1)+f(n-2)). I should have just written this in a couple of seconds as I did now! Or maybe the more correct f(n)=if(n<2,n,f(n-1)+f(n-2)) (that is still only correct for natural numbers, but doesn't check it of course).

But I was too much out of shape.

How can this have happened to me? It should have had some impact that as a kid I was fascinated by the Fibonacci Series, Tartaglia's (Pascal's) Triangle and the Golden Number. I even discovered (or reinvented) myself the Binet formula when I was a teen. What for then? Maybe I should have applied for my dream job 15 years earlier.

I have never felt so uneducated before. I should probably go back to the University and get a Master in Computer Science, if my brain can still take the challenge. I cannot forgive myself for getting stuck with something so simple as a recursive Fibonacci.

Windows Live Writer

Hmmm... So far it feels nice :) Good job!

Rory's Best Post Ever

Dude! Congratulations! I hope you are living the best time of your life (so far) too.

Spaces Comming to Live Right Now

I remember I was one of the first to get my own MSN Space that night, and as a result, I got trapped in a bug and could not use the the site for some time (I don't use it a lot anyway) .

Now I think tonight I catched them in the act again. They are transitioning MSN spaces to their new Live home, and they still have some work to do.

Take a look at how a famous space (Ray Ozzie's) looks like right now:



Update: Around 40 minutes later, everything is comming together nicely. No more missing images or CSS. It is amazing how the new look captures the escense of the MSN space theme you choosed, but still looks ver LIVE like.

Rory Blyth: Blogging is Stupid

Just read this gem:

One of the examples of a risk that Alfred brings up in his post is my tendency to write about my depressive nature and the drugs I take to try and undo the crap job nature did on wiring my neurons. I’ve been told similar things by some of my friends in the “blogosphere.” They tell me I’m nuts for posting about my mental problems. I agree. But that’s part of being nuts, so what’s the confusion about?

Is it stupid or not? Well, I guess it is stupid, but it is more stupid when you cannot write like him :)

Fyodor Dostoevsky

I am very grateful to Juan and to Stef for introducing me to the writings of this "crazy" russian genious.

The more I read, the less I blog.

ADO.NET Entity Framework on Channel 9

Channel 9 published a very interesting video about Microsoft's O/R Mapping tool that will be included in a future version of ADO.NET. It is actually a remake of a video that was published a couple of months, also in Channel 9, only to be shutted down soon after.

As almost every .NET developer does, I use ADO.NET every day, and have been a fan from the first hour.

But as an architect -in a limited sense I must add- I have to deal a lot with both theoretical and empirical aspects of multi-tier development, so I have been looking for the silver bullet for the Data Access Layer for years.

Pablo Castro, Technnical Lead of ADO.NET (and a native from Buenos Aires!) was very nice answering the comments I posted to Channel 9. I will reproduce my own writting here, but you can read his very interesting answer directly on the site.

As I mentioned before, one of my great concerns is how team development will look like with the Entity Framework. I took some time to detail my thoughts:

First, many real life projects are partitioned in modules, so their data layers are partitioned likewise.

Often, there are sets of tables that are used exclusively in each module, and a set of tables that are common to all. Yet, there are some tables that are reused in more than one application (typical examples are security, navigation, etc).

Besides, building a useful data layer is not done in one step nor does it take a single day. It is more often an evolutionary and error-prone process in which a programmer “imports” objects from the database each time he/she realizes they are mentioned in the specification.

During this process, errors that affect maintainability (duplications, improper use of naming standards, etc.) are very usual.

So, here is a short list of features that I would like to see in the Entity Framework (some are actually hard requirements). Of course I ignore if any of these are already included:

1. Partitioning of the conceptual model in multiple files and assemblies.
2. Referencing and extending (entity inheritance) between entities defined in separate files and assemblies.
3. Creating reusable “libraries” containing entities and mappings that can be reused by different modules or different applications.
4. "Incremental" reverse engineering of databases (I think this one is already in the graphical design tool).
5. Support for basic refactorings (unification, replacement, renaming, etc).
6. Very readable and maintainable XML (it should be easy to merge two files with a source code comparison tool).
7. Efficient and easy serialization of entities and entity sets outside the database.
8. Separation of the conceptual model from the persistence logic (take a look at what Steve Lasker does with typed datasets).
9. A migration tool for typed datasets XSDs.
10. A degree of resiliency to some schema changes.

If you can answer any of my doubts, I will be grateful. My boss is pressing me to evaluate O/RM products, and I am telling him to way for your framework everyday :).

All in all, I am very enthusiastic about this new piece of technology. I think it could make my life and the life of many other developers a lot easier and also more interesting!