IsNullOrEmpty for IEnumerable

This is not extremely relevant, but String.IsNullOrEmpty() has become a very popular time saver, and I think the same concept should be applicable to arrays and collections.

I found on Microsoft Connect that someone already added a suggestion to add IsNullOrEmpty to arrays on 2005.

This is not a big discovery, but I have been playing with extension methods in Orcas and they are so nice!

What if I define this?

public static class IEnumerableExtensions
{
    public static bool IsNullOrEmpty(this   System.Collections.IEnumerable source)
    {
        if (source == null)
            return true;
        else
            return !source.GetEnumerator().MoveNext();
    }
}

Once you import the appropriate namespaces, all these things are possible:

string a = null;
Console.WriteLine(a.IsNullOrEmpty());

var b = new Dictionary();
Console.WriteLine(b.IsNullOrEmpty());

MemberInfo[] d = MethodInfo.GetCurrentMethod().DeclaringType.GetMembers();
Console.WriteLine(d.IsNullOrEmpty());

var g = from f in d
where f.MemberType == MemberTypes.NestedType
select f;
Console.WriteLine(g.IsNullOrEmpty());

I would like to see something like this included in System.Linq.Enumerable static class. Then it would be available to everyone, by default.

Update: I added a more complete entry as a suggestion on Microsoft Connect.

Update 2: At the Connect site, Mads teaches me why he thinks IsNullOrEmpty as an extension method is really a very bad idea. Basically, using the variable.Method invocation syntax on a method that is meant to work when the variable is null, it is very inconsistent with the instance method invocation semantics one usually gives to this syntax in languages like C# and VB.

I still think that the method, probably defined as a static method, would be nice to have on Enumerable (because it is already a well-known place to find methods that apply to the IEnumerable interface). Also, I think there is some void in the definition of extension methods. Its designers think that calling them on null instances should generally throw an exception, so why do I need to check for the parameter and throw the exception myself?

Unindexed Foreign Keys

A guy called Jordi Ramot, puts it in these words:

To decide if a foreign key needs to be indexed or not, I follow a simple rule:

I always/only create an index on a foreign key whether:

1 - A deletion on the parent table is allowed and it triggers a cascade delete on the child table

2 - There's need to perform JOIN queries from the parent to the child

In the first situation, an unindexed foreign key will force a full table scan for each parent record deleted. In the second situation, a lack of the foreign key index in the child table will slow down join queries.

I rarely find suitable to create indexes on foreign keys in other situations though.

I think the question is not only “to index or not to index” on foreign keys.

I have been debating this subject with my boss (a hardcore Informix believer) all day. We found that Informix creates indexes on foreign keys automatically, while Oracle, DB2, and SQL Server don't.

So, why did some engineers decide to go one way and others in the opposite? I think this is an interesting design issue.

Informix takes all responsibility in optimizing JOINs and CASCADING operations on the foreign key.

Instead, Oracle, DB2 and SQL Server will happily leave the burden of tuning indexes for JOINs and CASCADING operations on your shoulders.

So, even if an Informix DBA fails to tune the indexes, the database will probably show acceptable performance on JOIN operations.

Oracle, IBM and Microsoft/Sybase on the other side, apparently decided that tuning was an nonnegotiable duty of the DBA. However, there are many reasons to want the higher level of control those database engines provide:

First, each index you create comes with a cost. Not only it will use storage space, but once you created it, the database engine has to maintain it on every table update.

Also, 98% of all SELECT and UPDATE queries will probably include a WHERE clause or will involve more than one JOIN operation.

There is also an opportunity for index coverage, meaning that if the index contains certain columns, some SELECT could be resolved entirely by reading the indexes, and never touching the real table.

To get all those benefits at the same time, it is necessary a composite index that is headed by the foreign key but also includes other columns relevant to frequent queries.

So, the Informix approach is a winner for the most basic cases, but the higher level of control the other engines give you, could show better performance if tuned adequately (obviously, my boss won’t swallow that pill!).

If you want to distill a best practice from this, I think that creating indexes on your foreign keys is a a good first approach, but you should later tune your indexes globally, by using real profiling data.

Fortunately, for those of us using mostly SQL Server, Index Tuning Wizard exists.

UPDATE: You also have to consider how foreign keys are actually implemented. My boss found some articles that mention that in some RDBMs foreign keys are internally implemented as "pointer chains".

Help Find Jim Gray

I know I don't manage any significant traffic here, but anyway: If you know of Jim Gray and how he has been missing in the sea from last Sunday, there is a way you can help in finding him.

The Coast Guard already called off their search effort and so friends and colleagues have taken the challenge.

Amazon set up a job in their site Mechanical Turk. So you can go there, login to you Amazon account and start visually scanning recent satellite images of the search area.

Update: You can go read on Werner Vogel's blog how they do it.

Please, join!

Jim Gray Is Missing in the Sea

Just found via Scoble that Jim Gray has been missing in the sea since Sunday.

I admire Jim Gray, and I hope he will return safe soon. I am also thinking that I really have and want a chance to meet him in person, but this can wait.

Update 1/31: No news is sad news. I do hope he is still alive, but my mind keeps trying to imagine what actually happened.

Vacations, Sweet Vacations

I have been on vacations since December 21st, visiting family and friends in my hometown. I am happy to see that most people here are doing better than two years ago. I haven´t met as many friends as I would have liked, but this is because many of them are out of town on their own vacations.

Besides all the happiness that is being here, watching my son interact with his grandfathers and grandmothers, etc., on January 3rd I had some wonderful news:

I got an offer from my favorite company to work with my favorite technology. Still, I can hardly express how it feels!

Broken Windows (Part 1)

Today at work, someone I respect much told me that I write very well (in Spanish) and that I am both inspiring and motivating to others. She also said that those were very rare features in a developer.

Seldom I have been aware of this. Actually I believe that the real underlying fact is that I cannot go to work every morning if I am bored of it and I don't feel at ease. I think I could not keep a boring job for a week, just for the money. So, when I am becoming bored, I have to do whatever it takes to make it fun again. People that see me, understand very quickly that I work for the fun, and that I love what I do. I don't know for sure, but when I see someone that seems to love what she or he does, it helps me keep my own fire alive.

During Tuesday afternoon, I talked on the phone with a very special group of people. Now I wish i could have a second chance to tell them:

See guys? Someone I work with thinks I am inspiring and motivating! John? Are you there? At least you have my blog address... :)

Well, the writing my office mate was referring to is an email I wrote to many developers in our organization about a rather basic topic that we needed to reinforce. I have been involved a lot in code quality initiatives lately. I need to translate the text in order to publish it, so please wait until my next post.

Jon Udell gets assimilated! ;)

I have been a fan of Jon for 18 years. This is great news. I think he will have a very important role in the ongoing Microsoft change.

With Rory and Jon on board, Channel 9/10 looks like the dream team.

Now, let's see how I do it this week...

Transactional File System in Windows Vista (Part 2)

In my last post I tried to describe what I think is the single most important issue with the implementation of TxF (and TxR).

Now I want to explain how I think the pre-existing file name-based APIs could have been changed (or could still be changed in future versions) to allow for opting-in Transactions for virtualy any code that works with files, including things like System.Data.DataSet.WriteXml(String).

I take this just as an exercise for my mind, as I am almost sure that someone thought about this solution, but then discarded it for a reason I cannot discern.

First of all, let’s say that all code running inside or on top of Windows, when it needs to access the file system, ends up invoking on of a relatively small set of Win32 APIs. Of those, some are file-name based (like CreateFile, DeleteFile, SetFileAttributes, etc.) and some are file-handle based (like GetFileSize, ReadFile, WriteFile, etc.).

Second, you always have to call first a name-based API to get a handle you can use with a handle-based function.

Third, no new Transacted versions of handle-based APIs were created. Instead, handle-based APIs get transactional behavior only if the handle passed is already associated with a transaction.

(Still, it bugs me why some name-based APIs were not replicated but instead behave like in the beta 2 model, becoming transactional depending on the ambient transaction).

So, you only signal that want to participate in an existing transaction at the time you call name-based APIs. In the current model, the way to do it is calling the transactional version of the function (like CreateFileTransacted, DeleteFileTransacted, SetFileAttributesTransacted, etc).

But what all those functions have in common in the first place, is that they receive a file name as a parameter!

(Edit: What follows is my proposition, not how TxF/TxR works in Vista. Somehow, after some editing I got the text wrong. )

What if we change rules a little: If the file name is prefixed with a moniker like, for instance “txf:” or “txfile:”, then it becomes transactional. You can understand it as designating a new namespace for TxF, one that points to the same file system store, but behaves differently.

There are already many rules about file names. However, for me it makes sense to add just one more in this case. Of course the exact form of the prefix is not important (NTFS seems to prefer other kind of prefixes).

By this plan, calling CreateFile(“txf:foo.txt”…) would be equivalent to calling CreateFileTransacted(“foo.txt”…). Something similar could be done with TxR.

I think this change would integrate perfectly with the model that shipped in Vista, meaning that you could mix and match calls to the Transacted functions with calls to the “normal” functions with the moniker prepended to the file name.

Also, existing client code and code “hidden” across the programming stack would not need to opt-out. It would be automatically be not transactional because its hard-coded file names (or registry key names) would not contain the transactional moniker.

But the real benefit would be that this could enable us to do things like myDataSet.WriteXml(“txf:foo.xml”) or [insert any other function that takes a file name as a parameter](“txf:bar.ext”), without waiting for a revision of them. Virtually all the programming stack could enjoy the benefits of TxF, without requiring modifications.

I can only think of a few functions that try to parse the file name and that could fail on the presence of the prefix, but I think those would be rare exceptions.

What do you think? If you like it, you can vote for this as I entered a closely related suggestion to the .NET Framework team in Microsoft Connect.

Transactional File System in Windows Vista

In November, 1998, Microsoft Transaction Server was about a year old and SQL Server 7 was just arriving. I had at hand the task of coding a small CRM-like application in Visual Basic 5. Among other features, it had to upload unstructured documents and keep them linked to rows in a database.

I had one important decision to make: Should those documents be stored in the database itself or in the server file system?

SQL Server 6.5 had a lot of limitations with its lack of row locking and some performance issues with BLOB columns.

On the other side, the file system lacked transactional capabilities, and I lacked the ability to create a Compensating Resource Manager.

A transactional file system would have been super useful.

In November 2006, eight years later, Windows Vista is available. Transactions were introduced as a new feature of NTFS, named TxF. The Windows Registry is also getting support for transactions in Vista, under the name of TxR.

Before TxF, for instance, if you wanted to get ACID-like behavior from multiple file system operations, you could, but you had to fiddle a lot with temporary files, renaming, etc. With TxF you just issue something like a "begin transaction", then do your stuff in NTFS, and last, you commit or roll back the whole thing.

This way, TxF pushes best practices under the rug, and pushes the developer one level of abstraction up regarding files.

Surendra Verma, Developer Manager in the CFS group, explained how TxF/TxR works in Channel 9 some months ago. But it is interesting to note that after the video was recorded, there were major design changes to TxF/TxR.

As Jim Johnson explains in this first, second and third posts, from Beta 2 to RC1, TxF API changed from "implicit transaction enlistment" model that worked with the existing Win32 file APIs to a more explicit model for which new "Transacted" versions of some APIs were added.

In the first version, you just did something like:

EnterTransactionScope();
// do whatever file work with your favorite file APIs
ExitTransactionScope();

Everything you did in the middle got automatically enlisted in a thread specific ambient transaction.

In the new model, you have to do something like this (some function names were invented):

hTransaction = GetTransactionHandle();
hFile = CreateFileTransacted(... hTransaction ...);
// do whatever, but now using new *Transacted APIs
CloseHandle(hTransaction );

The complete listing of APIs that were affected by TxF is here.:

If you take a look at it, all the APIs for which a new Transacted version were created are file name-based. Besides, some existing APIs were updated and are now transaction aware, meaning that they acquire transactional behavior in the presence of a file handle that is associated to a transaction (for file handle-based APIs) or in the presence of a thread level ambient transaction (for yet another group of file name-based APIs).

The reason Microsoft change models, as explained by Surendra in the discussion of the video in Channel 9, is that the more simple original version, had a major drawback:

Between any pair of EnterTransactionScope()/ExitTransactionScope(), every single file or registry operation made by any code, even code lost in the middle of the programming stack was automatically and forcefully enlisted in the ambient transaction, acquiring a behavior that was often not intended at the time such code was created.

You could not opt-out.

So, if implicit transactions means that current code will break or misbehave, it is good that they abandoned this path.

On the other side, the main tradeoff of the new version, in my opinion, is that it is "too explicit":

Only those new APIs and those that have been changed will get transactional behavior. So, the hundreds, if not thousands, of higher level APIs that somehow affect the file system, won't get the possibility of having transactional behavior until the whole stack gets updated.

Now, you cannot opt-in.

For a .NET developer like me, this means that I have to use a lot of interop, or wait until new versions of System.IO.FileStream, methods like System.File.Delete, and even that the brand new APIs in System.IO.Packaging get revised to include the option of using transactions.

I have been thinking of a deceptively simple change they could do to the existing file name-based APIs, that could solve this issue. I must be missing something, or they would have implemented it on Vista.

I tried to discuss my idea con Surendra, but he is probably having vacation after shipping Vista :)

I will try to explain the idea in my next post...

Arbitrary precision types for .NET (and some musings)

I came up today to this article that describes two new arbitrary precision types for .NET. One is an Integer, and the other a Decimal (I think it is not correct to call it a Double as in the title).

Although I don't often use arbitrary precision math, this is a welcome addition. It is also interesting to think how nicely such a feature can integrate in the existing framework, given generics and the unified type system.

Speaking of which... I found recently that the Parse and TryParse methods that numeric value types share do not belong to a common generic interface, reducing its usefulness in some scenarios.

I hope we will someday see something like IParseable<T> together with IComparable, IEquatable, IConvertible and IFormattable.

And what about the MinValue and MaxValue constants? Shouldn't they belong to a common type?

There is also this internal class called Number that seems to hold the implementation of methods of every numeric type. I wonder if they could somehow introduce a Number class as a common ancestor without breaking value types rules.

Ashamed

I got the call, I went there, but then I disappointed myself.

No matter if I was trying to create a recursive function that wouldn't suck as much as the usual f(n)=if(n<=2,1,f(n-1)+f(n-2)). I should have just written this in a couple of seconds as I did now! Or maybe the more correct f(n)=if(n<2,n,f(n-1)+f(n-2)) (that is still only correct for natural numbers, but doesn't check it of course).

But I was too much out of shape.

How can this have happened to me? It should have had some impact that as a kid I was fascinated by the Fibonacci Series, Tartaglia's (Pascal's) Triangle and the Golden Number. I even discovered (or reinvented) myself the Binet formula when I was a teen. What for then? Maybe I should have applied for my dream job 15 years earlier.

I have never felt so uneducated before. I should probably go back to the University and get a Master in Computer Science, if my brain can still take the challenge. I cannot forgive myself for getting stuck with something so simple as a recursive Fibonacci.