When I agreed to join EMC I was not sure what to expect. I’d worked in academia, a start-up, dabbled in reverse engineering, and contributed many projects to the open source community. However, EMC is unique with respect to all of the positions I’ve previously held or projects I’ve created. Higher education is a lot like open source, there is an implicit freedom — you get to play. A start-up is small and fast, and reverse engineering is all about figuring out how to gleam the rubik’s cube. EMC is the titan, the behemoth, the first IBM-like company that I decided to work for. Would I be consumed, or would I manage to stay relevant even as employee number 123456?
Eight months later, I have my answer. I’m tremendously proud to introduce to the world, in all its gory detail:
To understand how VSI 4.0 came into being requires a trip down memory lane. In 2008 one of EMCs most talented software engineers, Lee McColgan (whom I now have the pleasure of calling a colleague) decided he wanted to be able to identify the types of EMC devices attached to ESX hosts from within the vSphere Client. That is the moment that the VSI project was born. Initially known as StorageViewer, the 1.0 release of Lee’s software allowed users to see which Symmetrix and CLARiiON logical unit number (LUN) devices were members of a vSphere environment in vSphere contexts such as virtual machines (VMs), hosts, datastores, and raw device mappings (RDMs).
StorageViewer 1.0 was a hit among customers, and Lee released version 2.0 in 2009. The second release of StorageViewer added the ability to configure EMC’s Storage Replication Adapter (SRA) for VMware Site Recovery Manager (SRM) as well new PowerPath details such as the number of paths and path policy.
January 2010 marked a milestone for Lee’s once pet project. StorageViewer 2.0 became VSI 3.0, and transformed into more than just a read-only tool. With the help of Brian Partridge, a colleague of Lee’s, support was added for provisioning LUNs for datastores and RDMs, and storage quota management. But perhaps VSI 3.0’s greatest accomplishment was that it had become a portal with which other engineering groups wanted to integrate. Due to that desire, support for Celerra was added by a cross-group effort between Lee and EMC engineers Josh Hutt and Scott Penrose at Research Triangle Park.
Some time in early 2010 an idea was conceived by EMC Vice President (VP) Chad Sakac. Chad thought it would be really interesting if EMC could provide a tool to VMotion VMs across once forbidden VMotion boundaries — to “teleport” them if you will. That’s where I come in. Chad contacted me to see if I would be interested in coming on board developing such a tool. It seemed like an interesting challenge, so I said “Sure, why not?” And so in April of 2010 I joined EMC.
Late in the development stages of this “teleport” tool I was asked to provide integration with VSI, and I began to chat with Lee on the subject. Up until this point external functionality had been integrated with VSI by other engineers simply sending Lee libraries and he’d implement user interfaces (UIs) for them. Lee and I discussed the problems this approach presented: forced synchronization of release cycles, development choke-points, hit-by-a-bus scenario, and so on and so forth. It was clear to me that we needed a better way to integrate new functionality into VSI.
And it was only going to get worse.
I find that the vast majority of the public have an impression of large companies or universities that is in complete contradiction to the truth. When a cable appears on the Associated Press news wire about EMC or a major university most readers have in mind that an entire organization is being represented. While this may be technically true, there may in fact be only a handful of people responsible for the new item. Or in a lot of cases, there can be several groups spread out across the organization that came together in a collaborative effort to produce something newsworthy. Whatever the case may be, an organization is the sum of its parts, and those parts are not part of a single hive mind. There is politicking. People have to work together. While technology can’t solve all problems, the right tools can make collaboration easier.
And collaborate VSI must. Unlike some other information-focused organizations, EMC is responsible for a wide range of products that integrate with vSphere, from Small to Medium Businesses (SMBs) all the way up to the largest of enterprises. Our family of storage products includes the aforementioned CLARiiON, Celerra, and Symmetrix devices, and in 2010 EMC introduced the the world to its next-generation storage architecture, VPLEX. Not to mention one of the industry’s most talked about back-up solutions, Avamar. And VSI needs to expose all of this technology through the vSphere client.
Clearly one or two engineers cannot be domain experts when it comes to such a plethora of information technologies, so the burden must be shouldered by those who do have the expertise. But as I mentioned, the previous integration model would simply not scale. A new approach was needed.
Enter VSI 4.0!
The vSphere client already provides an extensive plug-in framework for partners whom wish to integrate with VMware’s enterprise management client. However, in order to facilitate easy, stable, and rapid collaboration amongst the many groups at EMC, VSI had to become more than a plug-in; it needed to itself be a framework. And so, that is what VSI has transformed into: a framework for other EMC engineers to leverage in order to develop their Features.
VSI 4.0 includes all of the previous functionality of VSI, but this time each piece of functionality is its own Feature that can be updated on completely different release schedules. At the same time, while the VSI 4.0 Framework was designed to decouple discreet units of functionality, it also brings them closer together than ever before by providing a common UI control library, logging and persistence managers, and an inter-Feature communication protocol. Where once many separate context menu items and tabs appeared on behalf of EMC, VSI 4.0 organizes all EMC-provided functionality under a single context menu item “EMC” and a single tab “EMC VSI”.
The Framework itself stays out of the way for the most part. VSI is all about the Features. There are occasions where you will encounter the core components of VSI, such as the Feature Manager, a tool that can be used to enable, disable, and uninstall Features.
If a Feature or the Framework is not behaving properly a user can adjust logging directly in the vSphere client for the entire Framework or a particular Feature.
The VSI 4.0 Framework is even backwards compatible with older vSphere 4.x plug-ins through a process I refer to as “adoption.” Customers using vSphere 4.x plug-ins designed by EMC that have not yet been fully reengineered for VSI 4.0 can easily load the plug-ins into the VSI 4.0 Framework. Simply move the plug-in’s folder from the vSphere Plugins directory to the VSI 4.0 Framework’s Features directory, located by default at %ProgramFiles%\EMC\Virtual Storage Integrator\vSphere4\Features\.
Along with a shiny new engine under the hood, VSI 4.0 includes all of its previous functionality and then some!
- Path Management (Lee McColgan) – Enables the user to set VMware vSphere native multipath driver (NMP) and PowerPath/VE multipath policies for hosts and clusters.
- Storage Viewer (Lee McColgan) – Provides storage mapping and reporting for VMware vSphere objects across all EMC storage platforms.
- Storage Pool Management (Brian Partridge & Lee McColgan) – Brings EMC Symmetrix storage resources closer to being a first class citizen in a VMware vSphere environment by allowing the user to provision storage (LUNs for datastores and RDMs) as a shared resource within the virtual datacenter. Quota management and automated provisioning allow storage resources to appear like the default compute resources (CPU & RAM) in a vSphere Client.
- Symmetrix SRA Utilities (Brian Partridge, Vibhu Bontala, & Joe Russo) – Aids in the setup and configuration of EMC Symmetrix device pairs for VMware SRM failover and provides VMware SRM with its missing failback functionality. Failback gets your production environment up and running again quickly and easily.
- Unified Storage Management (Josh Hutt, Scott Penrose, & Noel Curtis) – With support for both the CLARiiON and Celerra families, this Feature enables efficient provisioning of datastores, hardware accelerated VM cloning, storage optimization up to 50%, hardware assisted VM locking, data protection, and disaster recovery.
If you’ve ever experienced the auditory delicacy of basking in the sounds of a symphony orchestra you know that during the performance you tried, at least once, to listen to a single instrument — and found it nearly impossible to do so. A quality orchestra is so in sync that it is almost impossible to distinguish the parts from the whole. This type of unity is what we’ve tried to achieve with EMC’s VSI 4.0. While its Features are still built by many different engineers, we think that through our unique approach to collaboration, both away from and in front of our integrated development environments (IDEs), has resulted in a software package that is as beautiful as a symphony. VSI 4.0 may be a sum of its parts, but we hope you can’t tell.
If you have any questions about EMC VSI 4.0 please don’t hesitate to drop me an e-mail.