This would not normally be a huge concern since the members of the components, the files, are not going to change. However, I did run into an interesting issue. Because my company’s versioning scheme is MAJOR.MINOR.REVISION.BUILD where BUILD is a value auto-incremented by the build server, as far as MSI is concerned 4.0.0.123 is the same version as 4.0.0.187. And because our patch strategy is to simply release a hot-fixed MSI, we have to allow same-version upgrades.

Anyone familiar with MSI upgrades knows the dirty secret that allowing same version upgrades allows same version downgrades. Except these downgrades would result in files disappearing from the filesystem. After much research I determined the issue was because of my stupid decision to group multiple files into single Components. MSI was not applying the standard file versioning rules correctly, and thus during a downgrade the following steps were occurring:

1. Version X is installed on a system and user attempts to install version X-1.
2. The installer appears to calculate which Components need to be installed by version and date. It does not include the X-1′s Components because they are already on the system.
3. The installer removes X’s Components from the file system.
4. Finally, the installer chooses not to install X-1′s Components because the installer has already calculated that the Components on the filesystem, at version X, were newer.

I found myself in quite a pickle. How do I prevent same version downgrades when the only varying component of the version is the 4th component (which again, MSI ignores)?

First of all, let me clear the air and say that I’m using the original WiX method of handling major upgrades.

<Upgrade Id="$(var.UpgradeGuid)">
  <UpgradeVersion Minimum="3.9.9" IncludeMinimum="yes" Maximum="$(var.Version)" IncludeMaximum="yes" Property="OLDERVERSIONBEINGUPGRADED" />
  <UpgradeVersion Minimum="$(var.Version)" IncludeMinimum="no" OnlyDetect="yes" Property="NEWERVERSIONDETECTED" />
</Upgrade>
<CustomActionRef Id="WixExitEarlyWithSuccess"/>

This bit of code uses the WiX’s custom action WixExitEarlyWithSuccess in conjunction with the NEWERVERSIONDETECTED property to detect whether or not a new version is detected, and thus the installer will exist early with a successful error code (ERROR_NO_MORE_ITEMS) if a newer version of the product is already installed. I know that more recent versions of WiX include a MajorUpgrade element that makes this process more concise (cleaner), but this new method does not work in conjunction with the WixExitEarlyWithSuccess custom action.

Since the WixExitEarlyWithSuccess custom action works by reading the NEWERVERSIONDETECTED property, I thought I’d be tricky and try populating that property myself. To that end I used a FileSearch element to look for a key file that I knew would always have the same MAJOR.MINOR.REVISION.BUILD version that I needed to compare against.

<Property Id="NEWERVERSIONDETECTED">
  <DirectorySearch Id="NewerFileVersionDirSearch" Path="[INSTALLDIR]">
    <FileSearch Name="KeyFile.dll" MinVersion="$(var.Version4)"/>
  </DirectorySearch>
</Property>

This worked perfectly! Or so I thought. While this fix worked in conjunction with WiX’s own WixExitEarlyWithSuccess custom action during UI-enabled installations, it failed when the MSI was run in any type of quiet mode. A quick examination of the MSI log files with the verbose option enabled revealed that it was because the WixExitEarlyWithSuccess custom action is scheduled after FindRelatedProducts and my instance of the NEWERVERSIONDETECTED property is not set yet when run in silent mode when the WixExitEarlyWithSuccess custom action is processed.

My solution was to create my own custom action: ExitEarlyWithSuccess.

<CustomAction Id="ExitEarlyWithSuccess" VBScriptCall="Main" Property="ExitEarlyWithSuccessScript" />

<InstallExecuteSequence>
  <Custom Action="ExitEarlyWithSuccess" After="InstallInitialize">NOT Installed AND NEWERFILEVERSIONDETECTED</Custom>
</InstallExecuteSequence>

<Property Id="ExitEarlyWithSuccessScript">
  <![CDATA[
    Function Main()
      Main = 5
    End Function
  ]]>
</Property>

<Property Id="NEWERFILEVERSIONDETECTED">
  <DirectorySearch Id="NewerFileVersionDirSearch" Path="[INSTALLDIR]">
    <FileSearch Name="KeyFile.dll" MinVersion="$(var.Version4)"/>
  </DirectorySearch>
</Property>

The secret to my custom action is that it is scheduled after InstallInitialize giving the normal WiX properties, such as Installed a chance to be populated. The ExitEarlyWithSuccessScript is an embedded VBScript that returns the ERROR_NO_MORE_ITEMS code to indicate to the installer that it should exit immediately with a successful exit code. Also, the custom action is set to run only when the product is not installed since you don’t want this action to occur during maintenance or removal (not that it matters too much since the file version comparison wouldn’t match — the MinVersion attribute in a File element matches the next version that it is compared to. For example, 4.0.0.0 would match 4.0.0.1, but not 4.0.0.0. In this way, MinVersion will never match the currently installed version).

I’ve tested this method of exiting an installer early over several MSI files now, and it works for each one of them.

Hope this helps!

Broken Grove Bamboo iPhone4 Case

One cannot argue that the Grove made iPhone4 case is nothing less than perfect. I have received more comments on how amazing the case is then I can count. However, for $89, a case should not fall apart after taking the bottom half of it off an on on a regular basis just so you can dock it or plug it into a car charger. That said, I may buy another one eventually, if Grove has strengthened their manufacturing process (and glue that holds the black trim to the bamboo). For now I’ll probably invest in a Vapor case that is all aluminum. Should help block the attenuation issues with my phone anyway.

The answer to this problem is usually creating wireframe mockups of the eventual UI. However, I’ve yet to find a mockup tool that I like. I’ve tried tools like Visio and OmniGraffle which are good tools, but they focus more on diagraming and process flow than wireframe design. Not to mention the aforementioned applications share a particular trait with an almost standard for creating mockups — Axure — complexity. The barrier to entry for these applications for doing something as simple as creating mockups is relatively high. I have longed for something simple, powerful, and straight-forward.

And then I found Balsamiq Mockups.

Balsamiq Mockups is exactly what I’ve been looking for in an application for creating wireframe mockups. It:

• Has zero learning curve
• Is fun to use
• Creates elegant, to-the-point wireframes
• Runs on Windows, OS X, and Linux
• Has an active stencil community (much like OmniGraffle’s Graffletopia site)
• Is free to use for 7 days and is only $79 after that
• Saves all files in BMML (Balsamiq’s XML markup language) so versioning is a snap!

I’m currently working on a functional specification where I have to mock up several UI screens that involve the VMware vSphere Client. Here are some screenshots of the actual VMware client with EMC VSI 4.0 versus my mockups:

VSI 4.0 - Feature Manager

VSI 4.0 - Feature Manager - Mockup

VSI 4.0 - Symmetrix SRA Utilities

VSI 4.0 - Symmetrix SRA Utilities - Mockup

I created the above mockups from scratch, having never used Balsamiq before, in just a few hours. And since then I’ve created several more templates that rest of my team can use for their own specifications.

I just can’t say enough good things about Balsamiq Mockups. So far it has been fast, reliable, and makes working on this current specification a breeze. I do have some suggestions for their team that I would like to see implemented at some point in the future.

• Create a tool to turn any icon into a sketch-like stencil. This way I can re-use existing icon libraries. (This feature already exists. Check the comments!)
• I’m sure you’ve heard it before, but layers would be nice.
• As would templates (real ones — not your save-as-image work-a-round) (Another feature that I missed in my few hours with Mockups. Awesome work guys!)
• It would be nice to set background colors for containers or text boxes
• The concept of multiple worksheets in a single project

Thanks again to Balsamiq for making such a fantastic tool. Keep up the good work!

(Hopefully our team’s licenses will come in soon so I can stop worrying about my demo expiring!)

i18n

For those of you not familiar with the process of creating internationally-useable applications, there are two distinct steps: internationalization (commonly abbreviated as i18n) and localization. The first step, i18n, is the process by which you prepare your application for the second step, localization. Preparing an application for localization involves several steps:

• You must move all of your application’s strings, images, and even some visual formatting elements to one or more resource files. This decouples your application’s or assembly’s codebase from the information that may need to differ based on the culture of the user’s system running the application.
• Additionally, not all cultures read from left-to-right. While it may be better to use a box/flow layout for designing user interfaces (UIs) in order to more easily support right-to-left layouts, this alternative to a fixed layout takes a substantially greater amount of development time.
• While many cultures employ a phonetic alphabet, there are still many that rely on logographic languages. One such example is Nipponese where the word tree may be represented by a single symbol. So a warning message that consumes 100 pixels in England may in fact take up 300 pixels in Japan due to the width-additive nature of logographic scripts.

The above steps are just three of the many areas where developers must be conscious of i18n compatibility. And making an application i18n compatible is not even half of the story. Next comes the process of localization.

Localization

Localizing an application is straight-forward if the i18n process was handled with care (although there are some nasty issues you can run into). Ideally localization involves creating resource files for each locale that you want to support. Each of these resource files should contain the appropriate, localized values for the same keys in your default resource file. For example, if the key “WelcomeText” contains a string value of “Hello”, then the key “WelcomeText” in the file Resources.fr.resx would contain a string value of “Bonjour.” However, while the process of creating the localized resource files is fairly trivial, it adds a substantial amount of testing time.

For every locale supported, a full test regiment must be developed to ensure that the correct strings are in place, layouts are correct, and that all of the formats used do not cause any exceptions. So while moving localizable resources into resource files does make localization much easier, it does nothing to relieve the burden of the testing that must occur for every locale.

The Cost

As you can see, i18n has a fairly set cost, but the cost of localization increases with every locale you support. The resources must be translated and tested. This means having a specialized team of people who speak many different languages and have knowledge of many different cultures in order to properly handle localization. Or you can out source the task to teams around the world. Regardless of how it is accomplished, it is not free.

And yet, VSI 4.0 is free. Free to all EMC customers in fact. We have tried extraordinarily hard to produce software that will make managing your EMC storage devices from the vSphere Client as simple as possible. And as much as it pains us to not support all of the locales of the world, it is not something we had the bandwidth to accomplish in tis one release cycle. It is the stated goal of our team to continue to not only produce outstanding software, but also move forward with localization efforts. We are working with management to make sure that EMC customers on every continent can enjoy the wonderful software that is VSI 4.0!

As most .NET developers know, the System.Globalization.CultureInfo class is the starting point at which you determine the locale of a process. The CultureInfo class has two properties: CurrentCulture and CurrentUICulture.

CurrentCulture

The CurrentCulture property reflects the locale of the executing thread. The current culture is controlled through the Control Panel’s “Regional and Language Options” screen:

Regional and Language Options

When you set the system’s regional settings you are affecting the date and time, number, currency, and various other formats that the system uses. However, an important distinction to make is that changing a system’s region does not affect the language of the operating system (OS). The value of the regional setting is reflected in the CurrentCulture property in the .NET Framework. This value is important because as I mentioned it is used to determine many formats. One such example is the date/time format. Parsing a date/time string in the en-US (English) locale is different than parsing a date/time string using the nl-BE (Dutch) locale.

However, the CurrentCulture property has nothing to do with the most common aspect of .NET i18n preparation, the Resource Manager.

CurrentUICulture

The CurrentUICulture property is set to the locale returned by the Win32 API function GetUserDefaultUILanguage when the process is first started. The kicker is, though, that this low-level Windows function will return en-US despite what is configured in the “Regional and Language Options” control panel. The CurrentUICulture is only set to a non-English locale if:

• The Windows OS in question is a completely localized version of Windows built for a specific culture.
• If it is using the Multi-Language User Interface (MUI) pack and the MUI is set to use a language pack other than English.

The CurrentUICulture is what determines what visual elements are shown to a user in Windows forms-based applications. More importantly than that, however, the CurrentUICulture property is what the .NET ResourceManager uses to load locale-appropriate resource files.

Remember, the CurrentCulture can be completely different than the CurrentUICulture, and the latter is only set under very specific circumstances. Maybe you can see where this is going.

Th3 F4n P8rt

It bears repeating:

• The CurrentCulture property determines what formats are used when displaying and parsing things like dates, currency, numbers, etc. It is per-thread and controlled via the “Regional and Language Options” control panel.
• A completely separate property, CurrentUICulture, controls what visual elements are displayed to the user as well as what resource files are loaded for an assembly. This property is only set if the MUI is installed or a fully-localized version of Windows is being used.

So what’s the problem? I’ll tell you. Imagine you’re a good, internationally-concious developer (like me) and you place a lot of color codes in a resource file so that localization teams can create satellite assemblies later on to customize an application’s appearance for a specific culture.

Color formats in a resources file

Well, as it turns out this is a quick way to crash an application.

Since the color codes are formatted as “R, G, B”, and some cultures, such as nl-BE, use a comma instead of a decimal point, these values are interpreted as 32-bit integers when using the ColorConverter class to convert them to color objects.

So a person with their “Regional and Language Options” locale set to “Dutch (Belgium)” will quickly receive an unhandled exception about “122, 122, 122″ not being a valid 32-bit integer.

Ugh.

The solution seems simple enough, right? Create a file called “Resources.nl-be.resx” and override the offending color formats with locale-appropriate ones:

Resources.nl-BE.resx

Note: For the record, the appropriate way to separate numerical values for locales that forgo decimal points is with a semi-colon instead of a comma.

The problem is apparent: I’m storing values affected by CurrentCulture in a resource file which is only loaded when its culture matches that of CurrentUICulture. And since the CurrentCulture can easily be different than CurrentUICulture, the result is that in Belgium this particular application will crash even though there is a perfectly good resource file. The reason is that many Windows systems in Belgium that use the nl-BE locale simply use the English version of Windows and set the regional settings, which in no way affects the CurrentUICulture and thus prevents any locale-specific resource files from loading.

So what’s the solution?

The easiest solution is to take advantage of the generated code that Visual Studio creates when you enable the default Resource file for an assembly. The Resources class that is created contains a property named Culture. In your assembly that contains the Resources add a method that allows external callers to set the Resources class’s Culture property (the Resource class itself is scoped as internal). This will allow callers to set the culture that is used by the ResourceManger (which again, uses CurrentUICulture by default) when resources are queried. Essentially you’re allowing others to synchronize the CurrentCulture and CurrentUICulture values for that resource file. However, while this solution does apply to all threads it does not apply to all resource files.

Of course, you can also set the static property System.Threading.Thread.CurrentThread.CurrentUICulture to ensure that all ResourceManagers use the specified culture, but you’ll need to make sure that if you do that all threads have this set, not just the one on which the code is currently executing.

It’s Worth It

This may seem like a lot of work to support i18n preparation and localization efforts, but it’s worth it. Remember, we live in a big world, and you want to use software that you’re friends in Germany, Japan, and even Latveria create. So do unto others as you’d have them do unto you!

The Blog-Health-o-Meter™ reads Wow.

Crunchy numbers

The average container ship can carry about 4,500 containers. This blog was viewed about 21,000 times in 2010. If each view were a shipping container, your blog would have filled about 5 fully loaded ships.

In 2010, there were 18 new posts, growing the total archive of this blog to 62 posts. There were 30 pictures uploaded, taking up a total of 3mb. That’s about 3 pictures per month.

The busiest day of the year was March 10th with 767 views. The most popular post that day was SimDK.

Where did they come from?

The top referring sites in 2010 were communities.vmware.com, ntpro.nl, serverfault.com, virtualizationadmin.com, and blogs.msdn.com.

Some visitors came searching, mostly for simdk, andrew kutz, backuppc archive, snow leopard ipfw rules example, and stylecopcmd.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

SimDK March 2010
11 comments

Software June 2008
22 comments

Building a Better OS X Firewall (or How I solved the NAT problem for VirtualBox) August 2009
21 comments

A SimDK Virtual Appliance March 2010
27 comments

How To: vSphere Client to vCenter Using HTTP March 2010
3 comments

At the university where I used to work, my colleagues and I would joke that the university would be a great place to work if it weren’t for the students. Obviously we saw the irony, and it was more of a comment on the behaviors we did not care for that the younger undergraduates typically displayed. On that note, World of Warcraft would be a fun game if it weren’t for the other players.

Hey Blizzard, is there some way you can add an option so that I only have to interact with players who are older than 25, have a full time job, families, and don’t live with their mom? You could call it a maturity or politeness filter. C’est la guerre I suppose, but it really does stink that an otherwise interesting experience is still ruined by people who have no room for those of us just trying to have fun. I guess when your life is so bad in the real world, it’s much easier to take a game so seriously.

Eight months later, I have my answer. I’m tremendously proud to introduce to the world, in all its gory detail:

EMC Virtual Storage Integrator (VSI) 4.0 for VMware vSphere

History

To understand how VSI 4.0 came into being requires a trip down memory lane. In 2008 one of EMCs most talented software engineers, Lee McColgan (whom I now have the pleasure of calling a colleague) decided he wanted to be able to identify the types of EMC devices attached to ESX hosts from within the vSphere Client. That is the moment that the VSI project was born. Initially known as StorageViewer, the 1.0 release of Lee’s software allowed users to see which Symmetrix and CLARiiON logical unit number (LUN) devices were members of a vSphere environment in vSphere contexts such as virtual machines (VMs), hosts, datastores, and raw device mappings (RDMs).

StorageViewer 1.0 was a hit among customers, and Lee released version 2.0 in 2009. The second release of StorageViewer added the ability to configure EMC’s Storage Replication Adapter (SRA) for VMware Site Recovery Manager (SRM) as well new PowerPath details such as the number of paths and path policy.

January 2010 marked a milestone for Lee’s once pet project. StorageViewer 2.0 became VSI 3.0, and transformed into more than just a read-only tool. With the help of Brian Partridge, a colleague of Lee’s, support was added for provisioning LUNs for datastores and RDMs, and storage quota management. But perhaps VSI 3.0′s greatest accomplishment was that it had become a portal with which other engineering groups wanted to integrate. Due to that desire, support for Celerra was added by a cross-group effort between Lee and EMC engineers Josh Hutt and Scott Penrose at Research Triangle Park.

Some time in early 2010 an idea was conceived by EMC Vice President (VP) Chad Sakac. Chad thought it would be really interesting if EMC could provide a tool to VMotion VMs across once forbidden VMotion boundaries — to “teleport” them if you will. That’s where I come in. Chad contacted me to see if I would be interested in coming on board developing such a tool. It seemed like an interesting challenge, so I said “Sure, why not?” And so in April of 2010 I joined EMC.

Late in the development stages of this “teleport” tool I was asked to provide integration with VSI, and I began to chat with Lee on the subject. Up until this point external functionality had been integrated with VSI by other engineers simply sending Lee libraries and he’d implement user interfaces (UIs) for them. Lee and I discussed the problems this approach presented: forced synchronization of release cycles, development choke-points, hit-by-a-bus scenario, and so on and so forth. It was clear to me that we needed a better way to integrate new functionality into VSI.

And it was only going to get worse.

Collaboration

I find that the vast majority of the public have an impression of large companies or universities that is in complete contradiction to the truth. When a cable appears on the Associated Press news wire about EMC or a major university most readers have in mind that an entire organization is being represented. While this may be technically true, there may in fact be only a handful of people responsible for the new item. Or in a lot of cases, there can be several groups spread out across the organization that came together in a collaborative effort to produce something newsworthy. Whatever the case may be, an organization is the sum of its parts, and those parts are not part of a single hive mind. There is politicking. People have to work together. While technology can’t solve all problems, the right tools can make collaboration easier.

And collaborate VSI must. Unlike some other information-focused organizations, EMC is responsible for a wide range of products that integrate with vSphere, from Small to Medium Businesses (SMBs) all the way up to the largest of enterprises. Our family of storage products includes the aforementioned CLARiiON, Celerra, and Symmetrix devices, and in 2010 EMC introduced the the world to its next-generation storage architecture, VPLEX. Not to mention one of the industry’s most talked about back-up solutions, Avamar. And VSI needs to expose all of this technology through the vSphere client.

Clearly one or two engineers cannot be domain experts when it comes to such a plethora of information technologies, so the burden must be shouldered by those who do have the expertise. But as I mentioned, the previous integration model would simply not scale. A new approach was needed.

Enter VSI 4.0!

Framework

The vSphere client already provides an extensive plug-in framework for partners whom wish to integrate with VMware’s enterprise management client. However, in order to facilitate easy, stable, and rapid collaboration amongst the many groups at EMC, VSI had to become more than a plug-in; it needed to itself be a framework. And so, that is what VSI has transformed into: a framework for other EMC engineers to leverage in order to develop their Features.

VSI 4.0 Framework

VSI 4.0 includes all of the previous functionality of VSI, but this time each piece of functionality is its own Feature that can be updated on completely different release schedules. At the same time, while the VSI 4.0 Framework was designed to decouple discreet units of functionality, it also brings them closer together than ever before by providing a common UI control library, logging and persistence managers, and an inter-Feature communication protocol. Where once many separate context menu items and tabs appeared on behalf of EMC, VSI 4.0 organizes all EMC-provided functionality under a single context menu item “EMC” and a single tab “EMC VSI”.

The Framework itself stays out of the way for the most part. VSI is all about the Features. There are occasions where you will encounter the core components of VSI, such as the Feature Manager, a tool that can be used to enable, disable, and uninstall Features.

VSI 4.0 - Feature Manager

If a Feature or the Framework is not behaving properly a user can adjust logging directly in the vSphere client for the entire Framework or a particular Feature.

VSI 4.0 - Logging

The VSI 4.0 Framework is even backwards compatible with older vSphere 4.x plug-ins through a process I refer to as “adoption.” Customers using vSphere 4.x plug-ins designed by EMC that have not yet been fully reengineered for VSI 4.0 can easily load the plug-ins into the VSI 4.0 Framework. Simply move the plug-in’s folder from the vSphere Plugins directory to the VSI 4.0 Framework’s Features directory, located by default at %ProgramFiles%\EMC\Virtual Storage Integrator\vSphere4\Features\.

Features

Along with a shiny new engine under the hood, VSI 4.0 includes all of its previous functionality and then some!

• Path Management (Lee McColgan) – Enables the user to set VMware vSphere native multipath driver (NMP) and PowerPath/VE multipath policies for hosts and clusters.
  
  

  VSI 4.0 - Path Management

• Storage Viewer (Lee McColgan) – Provides storage mapping and reporting for VMware vSphere objects across all EMC storage platforms.
  

  VSI 4.0 - Storage Viewer

• Storage Pool Management (Brian Partridge & Lee McColgan) – Brings EMC Symmetrix storage resources closer to being a first class citizen in a VMware vSphere environment by allowing the user to provision storage (LUNs for datastores and RDMs) as a shared resource within the virtual datacenter. Quota management and automated provisioning allow storage resources to appear like the default compute resources (CPU & RAM) in a vSphere Client.
  
  

  VSI 4.0 - Storage Pool Management

• Symmetrix SRA Utilities (Brian Partridge, Vibhu Bontala, & Joe Russo) – Aids in the setup and configuration of EMC Symmetrix device pairs for VMware SRM failover and provides VMware SRM with its missing failback functionality. Failback gets your production environment up and running again quickly and easily.
  
  

  VSI 4.0 - Symmetrix SRA Utilities

• Unified Storage Management (Josh Hutt, Scott Penrose, & Noel Curtis) – With support for both the CLARiiON and Celerra families, this Feature enables efficient provisioning of datastores, hardware accelerated VM cloning, storage optimization up to 50%, hardware assisted VM locking, data protection, and disaster recovery.
  
  

  VSI 4.0 - Unified Storage Management

Symphony

If you’ve ever experienced the auditory delicacy of basking in the sounds of a symphony orchestra you know that during the performance you tried, at least once, to listen to a single instrument — and found it nearly impossible to do so. A quality orchestra is so in sync that it is almost impossible to distinguish the parts from the whole. This type of unity is what we’ve tried to achieve with EMC’s VSI 4.0. While its Features are still built by many different engineers, we think that through our unique approach to collaboration, both away from and in front of our integrated development environments (IDEs), has resulted in a software package that is as beautiful as a symphony. VSI 4.0 may be a sum of its parts, but we hope you can’t tell.

If you have any questions about EMC VSI 4.0 please don’t hesitate to drop me an e-mail.

This post, however, is all about two issue we ran into when we released VSI 4.0 to industry insiders. Right off the bat people had problems downloading the installers. One person even had an issue running the installer because of the operating system’s restrictive security settings. This is known as “The Caper of the Internet Sandboxes and Digital Signatures!”

Internet Sandboxes

The first problem that we encountered prevented some users from downloading our installers from PowerLink, EMC’s software distribution platform. Our installers are built using a .NET installer technology and are were (now they are zipped up to prevent the discussed issue from occurring) portable execution images (EXE files). When attempting to download the installers customers found that Internet Explorer (IE) would simply hang and eventually crash. Other browsers, however, worked without issue. I managed to reproduce the issue using the following systems:

Download Hangs

• Windows 2003 Server x86 with IE8
• Windows 2008 Server R2 x64 with IE9
• Windows XP SP3

Download Succeeds

• Windows XP SP3 x86 with IE8
• Windows 2003 Server x86 with IE8
• Windows 2008 Server R2 x64 with IE8

As you can see there does not seem to be a commonality between OS or browser versions. The downloads simply hung on some systems and not others. I even moved the file to multiple web servers (both IIS and Apache) to rule out a server issue, and attempting to download the EXE from any web server on a system that failed would always fail. So it’s definitely a client-side issue. Not only that, but it is an issue that is specific to .NET EXEs, as existing EXEs produced with C++ were not exhibiting the same issue.

At this point I decided to take our own EXEs out of the equation since they are both digitally signed (more on this later) as well as embedded with a special manifest which demands User Access Control (UAC) privilege elevation. I created a brand new .NET 3.5 Console Application using Visual Studio 2010 and added no additional code beyond the boiler-plate code inserted by Visual Studio. I compiled the application as TestExe.exe and placed it on a web server. It too exhibited the same symptoms as the VSI 4.0 EXEs.

On Windows 2008 Server an actual error would occur as opposed to the browser simply crashing on Windows 2003 Server, and I was able to gain further insight into the underlying problem. We were receiving errors such as System.BadImageFormatException and System.Security.SecurityException. It was then that I noticed the errors were not being thrown from the iexplore.exe process (as expected, since that is the image that spawns the IE process), but rather an image that I had not heard of until Saturday, ieexec.exe.

It turns out that ieexec.exe is a program that IE uses to host remote programs in an “Internet Sandbox”. This is where my debugging of the issue spun off into many unproductive tangents. The most promising one (at first) centered around the idea the type of processor architecture that ieexec.exe was compiled for versus the EXEs that were being download. The platform architecture angle turned out to be a bust, specifically since I was able to reproduce the issue on x86 systems.

However, investigating the aforementioned possibility did cause me to use Process Explorer to filter ieexec.exe during a download attempt. And what I found surprised me! The files were in fact being downloaded in their entirety and then ieexec.exe was crashing.

It dawned on me that ieexec.exe was treating these EXEs as remote programs that should be launched directly instead of saved to disk. And of course the programs would throw exceptions upon launch because executing an image from a remote web server would violate just about every .NET security policy known to man!

To prove my theory I relaxed the security on the system by turning it off with the caspol.exe tool. And voila! Clicking on a link to my TestExe.exe application caused a command prompt to briefly appear and vanish — the application had been successfully hosted from within ieexec.exe. Not only that, but I could also launch my installers. Of course, this was not a permanent solution for customers, and it certainly did not explain why IE was treating these EXEs as remote programs instead of simply downloading them. Isn’t an EXE on a web server simply an octet stream that is saved to disk?

Luckily, thinking along these lines led me to the following StackOverflow post. It turns out that .NET installs a client plug-in into IE that examines the stream of remote EXE images to determine if they are CLR code. If so they then are not treated like regular downloads, instead they are hosted as remote programs inside of ieexec.exe. Just one more reason not to use IE in my opinion.

So what is the solution? There are a few:

• Don’t use IE. Although IE8 and 9 are pretty good browsers, they are still too intertwined with the OS to treat the web as a remote resource. Issues like this one illustrate that well. Chrome is a great, light-weight browser. I use Safari myself.
• As the aforementioned StackOverflow article mentions, add a Content-disposition header to all .NET EXEs. You can test this out on my web server. I created a page where you can download TestExe.exe with the header and TestExe2.exe without the header. On systems that fail, the second link will not work while the first one will.
• Zip any .NET EXEs you post for download. This is probably the best solution.

I’m still not sure why this bug? occurs on some systems when it doesn’t on others. It seems to me that the MIME type filter that is installed inside of IE by .NET should be present on all Windows systems, but clearly there is some specific set of settings that cause .NET EXEs to be treated differently on different systems. For what it’s worth, I was also able to resolve the issue by playing around with client-side security settings, but never found a universal answer.

Digital Signatures

Once we rectified the download issue we were faced with yet another fun problem. It seems that the digital signature we signed our installers with was not valid according to one of our early adopters.

Now, I have to take a minute and point out that we have a fairly rigorous testing process. Before our software is ever distributed to customers it runs on all of the engineers’ systems, goes through Design and Verification Testing (DVT), Quality Assurance (QA), and finally some of our sales staff and vSpecialists take it for a spin. Not to mention that some of the functionality we develop is also certified by VMware. So you can see, we don’t develop in a vacuum. The fact that we ran into the download issue caught us off guard, but I think the issue with digital signatures did even more so.

Regardless, the evidence was explicit, the signature chain did not appear valid on one of customer’s systems. Normally this isn’t a problem, as only Windows drivers by default require valid digital signatures to be installed. However, on this particular system the security was restrictive enough to prevent the user from installing VSI 4.0. Obviously this is not a good thing.

It turns out that the Thawte-provided code signing certificate is issued by an intermediate certificate authority (CA) with a common name (CN) of Thawte Code Signing CA which is not present on Windows systems by default. The code signing CA is issued by the Thawte Premium Server CA, which is installed by default on Windows systems via Microsoft security updates. Thus, the digital signatures appear as invalid because the intermediate CA in the chain is not present on most users’ systems.

Why wasn’t this caught in testing? Simply put, most of the time an invalid signature chain won’t prevent you from installing software, but given the right security constraints it can. That singular case managed to pop up. What can we/you do about it? Glad you asked.

• Customers who encounter issues with the digital signature with VSI 4.0 should follow the instructions posted at the following Thawte knowledge base (KB) article. This particular article also states that the absence of the intermediate CA shouldn’t prevent the EXE from running, but clearly it can if the security is restrictive enough.
• Future versions of VSI 4.0 will include the Thawte Code Signing CA as part of their digital signature’s certificate chain. This will cause the digital signature to be valid without any action taken on the part of a user since the chain will be valid on their system since they already have the root certificate that issued the intermediate code signing certificate.

Both solutions aside, most customers won’t have this issue. As I mentioned above, the installers were tested quite thoroughly prior to release. This particular error will only occur on a very small sub-set of uber security-conscious users’ systems (I’m looking at you Ed Haletky!).

If you’re curious as to why the intermediate CA was not included in 4.0 release’s digital signatures, the answer is simple — the signtool.exe utility from Microsoft does not include an option to do so from the command line. It turns out, however, if the PFX container contains the intermediate CA in addition to the public and private key of the signing certificate, or if the Local Computer’s Intermediate Certificate Authorities store contains the Thawte Code Signing CA at the time of the signing, the intermediate CA is automatically included in the signature. We’ve added the intermediate CA to the Intermediate Certification Authorities store on our build agent, so future installers will ship with enough of the certificate chain to validate it.

Conclusion

Having my first release out the door feels great. Hopefully the lessons I learned from it can benefit others in the future. I had a heck of a time finding the answers to these two problems in particular, so maybe with this blog post out there, Google has been kinder to you if you’re reading this now.

Hope this helps!