akutz’s blog

Introducing GuessMyOSToo!

15Sep09

GuessMyOSToo is the world’s first third-party plug-in that works with both VMware Infrastructure 3 and vSphere 4 clients! Following on the success of GuessMyOS, the sequel replaces generic VM icons in the VI3 and vSphere inventory trees with OS-specific icons. Created by Andrew Kutz at Hyper9 GuessMyOSToo is free and available for download today!

Filed under: software development, virtualization | 3 Comments
Tags: guessmyos, guessmyostoo, vi3, vmware, vsphere4

The OVM^2 Initiative

30Aug09

Some of you have been e-mailing me recently, asking why I’ve been so uncharacteristically silent these past few months. Well, I’m finally ready to do the big reveal, to pull the covers off what I’ve been working on all this time. I’m pleased to announce the Open Virtualization Management and Monitoring (OVM^2) Initiative.

An ambitious project by Hyper9, OVM^2 aims to provide a completely free and open software stack for managing and monitoring leading virtualization platforms such as:

VMware ESX 3.x
VMware VirtualCenter 2.x
VMware ESX 4.x
VMware vCenter4
Microsoft Hyper-V
Citrix XenServer 5.x
*Amazon EC2
*KVM

Overview

The OVM^2 Initiative is supported by a series of open source software projects that Hyper9 is announcing and making available today under the BSD license:

Vangaea – Provides an open, RESTful interface and single sign-on access for communicating with leading hypervisors.
UVAPI – A Java library that provides a universal model and API for accessing different virtualization platforms.
JWbem – An on-going Java implementation of Microsoft’s Scripting Web-Based Enterprise Management (SWbem) API. JWbem allows Java developers to easily monitor and manage Microsoft Windows via Windows Management Instrumentation (WMI).
Hyper9-Common – Binds together all the other modules; a set of utilities providing common support.

When combined with supporting hypervisor SDKs, these projects fit together to form the basis of the OVM^2 Initiative.

OVM^2 Overview

Availability

I’ve been putting the finishing touches on the different parts of OVM^2 for a weeks now, and I’m finally satisfied enough with the various bits and pieces to approve the first public release of OVM^2, just in time for VMworld 2009.

In conjunction with the release, I recorded a quick demonstration highlighting some of the features of Vangaea, the RESTful API component of OVM^2. Take a look!

Support

Hyper9 needs your support! If you are interested in working on any component of OVM^2 please e-mail me at akutz at hyper9 dot com, and let me know for which project you think you can be a positive contributor.

*Coming Soon

Filed under: software development, virtualization | 9 Comments
Tags: akutz, citrix, hyper-v, jwbem, microsoft, ovm^2, uvapi, vangaea, virtualization, vmware, xen

Building a Better OS X Firewall (or How I solved the NAT problem for VirtualBox)

20Aug09

My Macbook Pro’s hard drive died. Ugh. Since I was forced to rebuild my system I decided to do a clean install and not copy over any preferences from my backups, just the data that I needed (documents, secure shell (SSH) keys, music, etc.). Another decision I made was to forgo VMware Fusion for a more light-weight desktop hypervisor. Fusion has a nice feature set, but I have never known a program to be more of a resource hog, or to be so annoying (always grabbing my input, yanking me into another space) as VMware Fusion for OS X.

So the decision that had to be made was, “Which hypervisor do I use?” It really boiled down to two choices: Parallels Desktop for Mac or VirtualBox. As I’m a fan of open source, and VirtualBox is free compared to Parallel’s 79 United States Dollars (USD) price tag, I chose VirtualBox. VirtualBox is a wonderful hypervisor, it has active community forums and members waiting to assist you on its internet relay chat (IRC) channel. Albeit, there are a few annoying bugs such as it always wanting me to re-authenticate whenever I perform a seemingly innocuous action (clicking on the help menu for example). But, to the credit of the community, one of the VirtualBox developers mentioned that this is a known bug and will be fixed when the final version of Snow Leopard is released (I’m running the latest developer’s build of Snow Leopard). Right now it is just too much of a moving target for them to concern themselves with.

VirtualBox does have one major issue that was almost a non-starter for me — its terrible network address translation (NAT) implementation. The creators of VirtualBox are incredibly helpful and intelligent, but much like the wonderful people behind the Milwaukee Sawzall (which despite several incarnations, still throws waste back into the eye of the user) need to use their creation a bit more in the real world. To understand what I am harking on about, you need to understand how both the VMware and Parallels NAT implementations work. The following image illustrates a high-level view of how NAT is generally implemented by desktop hypervisors:

A Standard NAT Implementation

As you can see, all of the virtual machines (VMs) are connected to an independent NAT process. This implementation has some pros and cons associated with it.

Pros

VMs are protected – VMs are protected behind the NAT daemon and while they can see the internet, the internet cannot see them. This is a perfect set up for testing new operating systems (OSs) or other software in a beta state.
VMs are connected – Because all of the VMs belong to the same NATd network segment, all of the VMs can communicate with each other. This is perfect for testing client-to-server functionality between two VMs.

Cons

NATd is persistent – The disadvantage to this implementation is that the NAT daemon is always running, taking up precious resources.

With the standard NAT implementation the pros definitely outweigh the cons in my opinion. I’d rather have a persistent NAT process hanging around then my VMs not being able to communicate with each other behind a firewall. Unfortunately for us, the creators of VirtualBox do not see it this way. Let’s take a look at VirtualBox’s NAT implementation:

VirtualBox NAT Implementation

As the above image depicts, with VirtualBox each VM’s process hosts its own NAT engine. This method also has its own pros and cons.

Pros

NATd is not persistent – When there are no VMs around to need a NAT daemon, no NAT daemon is present.
VMs are protected – VMs are protected behind the NAT daemon and while they can see the internet, the internet cannot see them. This is a perfect set up for testing new operating systems (OSs) or other software in a beta state.

Cons

VMs are not connected – Because all of the VMs belong to different NATd network segments, none of the VMs can communicate with each other. This is prevents testing client-to-server functionality between two VMs.

There is a thread on VirtualBox’s forums discussing their unique implementation of NAT. My response is currently the last in the thread, and I simply point out that sometimes doing things differently is not always the best solution since people trying out VirtualBox are likely to be used to the way things work in VMware and Parallels.

I will say it again, the absence of VMs being able to communicate with each other, protected by a NAT, just baffles me. The VirtualBox developers do have two solutions, neither of which are things I recommend implementing:

Create a Virtual Router – The first solution is to create two or more VMs on a host-only network and then dual home the first VM on a bridged network and have it act as a router for the other VM(s). This is such a bad design that it makes me upset to think about it. First of all, this was the way ESX 3.x worked and people did not like it. We asked VMware for a proper, built-in NAT, and they finally gave us one with vSphere (hell, VMware Server, or GSX, always had a NAT implementation built-in). Not to mention that there is no way that you can tell me having a VM as a dedicated router uses fewer resources than running NATd.
Bridge Your VMs – Dumb. Dumb. Dumb. The reason we like NAT is to keep our VMs off of the public internet so we can get away with all sorts of things we would never dream of doing on a system open to the world.

I have been told that NAT really isn’t a security measure, and to that I respond with “For all intents and purposes it is.” Some people insist that MySQL is not a free and open database, but to these people, as well to the ones who contend NAT’s security benefits, I simply say that you are living in a 1% world and the other 99% of us do not have the time nor inclination to get into prickly details of what constitutes free, open, or ultimately 100% secure. NAT provides some measure of added security for the rest of us — deal with it.

Luckily for me, I understand just enough about networking to be dangerous to myself and others. When you place a VirtualBox VM into the networking mode, “Host Only,” you are actually able to give the VM access to outside world through NAT using nothing more than the tools that ship with OS X — all without placing VMs on a bridged network or creating a resource-hogging router VM. Here’s how I did it.

vboxnet*

When you place a VirtualBox into the networking mode, “Host Only,” you are actually attaching the VM to a virtual networking interface VirtualBox creates on the host. The first VirtualBox interface is called vboxnet0, the second vboxnet1, and so on. Since the VirtualBox documentation states that all traffic from a VM in “Host Only” mode passes through the virtual interface it occurred to me that I should be able to route that traffic to and from the outside world using IP tables/chains and NATd.

IPFW and NATD

IPFW is BSD’s (on which OS X is built) tool for implementing the Unix equivalent of IP tables/chains. You can block incoming and outgoing network as well as diverting said traffic. NATD is a popular *Nix daemon that is used to NAT outgoing and incoming packets for internal, private networks. OS X ships with both of these tools, it is just a matter of getting them set up correctly to talk to VirtualBox’s virtual network interfaces.

NATD

The first step to getting this to work was to launch NATD so that it would listen to traffic on my public-facing adapter. Since I am working on a Macbook Pro and my public-facing adapter is an AirPort, the adapter is en1. I start NAT like so:

sudo sysctl -w net.inet.ip.forwarding=1
sudo natd -use_sockets -same_ports -dynamic -interface en1

The first line enables OS X to forward packets from one adapter to another. Normally this is prohibited. The second line starts the NAT daemon on my airport. The flag -dynamic is important since my AirPort’s IP address could change. This flag will cause the NAT daemon to handle a changing IP address.

IPFW

Now that a NAT daemon is up and running we simply need to set up some IP tables about how to divert the appropriate traffic. IP tables are rather daunting and not for the faint of heart, so I am going to simply glaze over my implementation here. The important thing to notice in my IP chains is that I divert all incoming and outgoing traffic to the NAT socket. However, my rules are complicated further by the fact that I don’t just use static rules, I set up a dynamic state table to improve their performance. Additionally, I’m not just redirecting traffic to the VirtualBox interfaces, I’m also setting up a very secure firewall. For example, my rules allow specific outbound services, such as HTTP, IRC, Bittorrent, E-mail, etc. If the service is not on the list then you cannot talk to it once my firewall is enabled. You will be able to edit these on your own accord. The inbound rules in my firewall allows for even fewer services. Only DHCP, SSH, and pings (from certain addresses) are allowed. Again, feel free to edit these to suit your own needs. Here are my rules as they exist right now:

00010   1569    255611 allow ip from any to any via lo*
00011      0         0 allow ip from any to any via vboxnet*
00100      0         0 deny ip from 127.0.0.0/8 to any in
00110      0         0 deny ip from any to 127.0.0.0/8 in
00120      0         0 deny ip from 224.0.0.0/3 to any in
00130      0         0 deny tcp from any to 224.0.0.0/3 in
10000 348166 427961532 divert 8668 ip from any to any in via en1
10010      0         0 check-state
10020    256     38154 deny ip from any to any established in
10030      0         0 deny ip from any to any frag in
20000   1151     64456 skipto 50000 icmp from any to any out keep-state
20010 473220 413906904 skipto 50000 tcp from any to any dst-port 80,8080 out setup keep-state
20011  41930  12064801 skipto 50000 tcp from any to any dst-port 443,8443 out setup keep-state
20020      0         0 skipto 50000 tcp from any to any dst-port 53 out setup keep-state
20020   6996    741940 skipto 50000 udp from any to any dst-port 53 out keep-state
20030  32996  23308966 skipto 50000 tcp from any to any dst-port 110,993 out setup keep-state
20031   1290    806595 skipto 50000 tcp from any to any dst-port 25,465,587 out setup keep-state
20040      0         0 skipto 50000 tcp from any to any dst-port 22 out setup keep-state
20050      0         0 skipto 50000 tcp from any to any dst-port 21 out setup keep-state
20060      0         0 skipto 50000 tcp from any to any dst-port 123 out setup keep-state
20060    156     11856 skipto 50000 udp from any to any dst-port 123 out keep-state
20070      0         0 skipto 50000 tcp from any to any dst-port 113 out setup keep-state
20070      0         0 skipto 50000 udp from any to any dst-port 113 out keep-state
20080      0         0 skipto 50000 tcp from any to any dst-port 194 out setup keep-state
20080      0         0 skipto 50000 udp from any to any dst-port 194 out keep-state
20090      0         0 skipto 50000 tcp from any to any dst-port 137-139,445 out setup keep-state
20090    576     44928 skipto 50000 udp from any to any dst-port 137-139,445 out keep-state
20100      0         0 skipto 50000 tcp from any to any dst-port 1194 out setup keep-state
20100      0         0 skipto 50000 udp from any to any dst-port 1194 out keep-state
20110      0         0 skipto 50000 tcp from any to any dst-port 111,2049 out setup keep-state
20110      0         0 skipto 50000 udp from any to any dst-port 111,2049 out keep-state
20120      0         0 skipto 50000 tcp from any to any dst-port 6881-6999 out setup keep-state
30000      0         0 allow tcp from any to me dst-port 22 in setup keep-state
30010      0         0 allow udp from any to me dst-port 67,68 in keep-state
30020      0         0 allow icmp from 192.168.0.2 to any in keep-state
40000  13796   1511970 deny ip from any to any
50000 222149  24513546 divert 8668 ip from any to any out via en1
65535 663694 524799538 allow ip from any to any

That’s a lot, I know. But once they’re in place you will have an incredibly secure OS X box (you will no longer need to use the built-in, crappy OS X firewall) and a true NAT system for VirtualBox.

AKIN

Everything is not hoorific in hooville however. It turns out that NATD really doesn’t like when you sleep and wake your laptop. It just isn’t designed for it. And wouldn’t it be nice if your firewall started when you started your computer? Or how about if the NAT daemon would figure out if you disabled your network or you switched from using your built-in AirPort to the built-in Ethernet port? Yes, these things would be nice, wouldn’t they? Luckily for you I’ve got a solution in the form of akin.

akin stands for akutz’s ipfw and natd controller (10.5+ only) and is a project on SourceForge at http://sourceforge.net/projects/akin. Simply download the latest tar ball from SourceForge, deflate, and use the included install script to get your firewall and router up and going.

Details

So what exactly does akin do? A lot actually. There are several files included with akin:

akin (src) – This is an OS X binary I wrote in C using Xcode that acts as a launchd daemon to monitor sleep and wake states and when your active network adapter changes.
akinctl (src) – This is the heart of akin, a BASH script that can be used to start, stop, and restart IPFW and NATD. It will automatically detect if you have VirtualBox installed and only start NATD if you do. It adjusts the IPFW rules in the case that you do not. akinctl can also print out its status and the name of the active NIC using the status and nic arguments.
ipfw.rules (src) – These are some of the IPFW rules that akin installs. Not all though — some are dynamically generated depending on whether or not you need NAT and which of you network interfaces is active.
com.wordpress.akutz.launchd.akin.plist (src) – This is the launchd plist that will start akin on boot and daemonizes it so that it can listen for sleep and wake events as well as detect if your active network interface changes.
install (src) – A script that installs akin in /opt/akin. This script also registers akin as a launch daemon and starts it immediately.
uninstall (src) – A script that uninstalls akin from /opt/akin. This script also unregisters akin as a launch daemon and stops it immediately.

Logging

You can use Console.app to monitor the status of akin.

8/20/09 7:52:00 AM	com.wordpress.akutz.launchd.akin[53]	active nic changed from disabled to en1
8/20/09 7:52:00 AM	com.wordpress.akutz.launchd.akin[53]	executing '/opt/akin/akinctl start'
8/20/09 7:52:00 AM	com.wordpress.akutz.launchd.akin[53]	enabled ip forwarding
8/20/09 7:52:00 AM	com.wordpress.akutz.launchd.akin[53]	enabled natd
8/20/09 7:52:00 AM	com.wordpress.akutz.launchd.akin[53]	enabled ipfw

Salve

Well, I hope that with this blog and akin I’ve accomplished two things. 1) I hope I’ve showed that it is possible to do NAT correctly with VirtualBox and 2) I hope that I’ve made it easy and secure to do so.

Update 1 (2009/08/20)

I forgot to give credit to the articles which helped me learn how to make a dynamic state table play nice with NATd. In no specific order of usefulness:

Update 2 (2009/08/21)

Matt from the standalone-sysadmin.com makes a great point in the comments. One way to achieve a NATd VM that can see other VMs using VirtualBox is to dual home it with a NAT interface and a private interface. However, after thinking on it some more, this too has its own problems. One con I forgot to mention above about VirtualBox’s NAT implementation is that because their NAT engine is running as a low-privileged user account (likely your account), it has multiple failings:

A Ping to Nowhere – The ping command operates over the ICMP protocol and thus requires root access. Because VirtualBox’s NAT engine is running as a normal user and not root, you cannot use the ping command in VMs that use VirtualBox’s NAT engine. Using the method I describe above with akin it is possible to use ping because the NAT daemon that I set up is running as root.
Forwarding Ports – Again, because VirtualBox’s NAT engine is not running as root you cannot forward ports less than 1024 because these are considered privileged ports. Using my implementation you can forward away!

Filed under: apple, security, software development | 16 Comments
Tags: diy, firewall, nat, os x, virtualbox

VIJMark

05Aug09

I benchmarked the performance of three Java toolkits for working with VMware Infrastructure and vSphere: VI (vSphere) Java, the VI SDK for Java, and the vSphere SDK. You can read more at http://vijmark.sourceforge.net/.

(Hint – VI (vSphere) Java wins)

Filed under: software development, virtualization | Leave a Comment
Tags: vijava, vmware

VMM 1.0 Released!

08Jun09

VMM without the host

I am very pleased to announce the immediate availability of Hyper9 VMM 1.0! Since creating VMM earlier this year, being acquired by Hyper9, and continuing its development, I have always believed that VMM will become the industry standard for heterogeneous, mobile virtualization management. With its first public release, VMM is another step closer to that reality. So what features are new in version one dot zero? Well, let’s take a look:

VMM backed down its Java requirements to Java 5. For enterprises who have not yet made the leap to Java 6, VMM will plug right into your existing environments!
A brand new properties file format has been introduced that allows VMM administrators to customize VMM to their heart’s content. Customizations include:
- Whether or not to show the host information.
- Whether or not to show the host’s CPU statistics.
- Whether or not to show the host’s memory statistics.
- Whether or not to show a VM’s CPU statistics.
- Whether or not to show a VM’s memory statistics.
- Whether or not to show a VM’s power operations.
- Whether or not to show a VM’s network operations.

Administrators can customize these settings and more based on the name of the server VMM connects to, the user logged into VMM, and even the type of browser the user is connecting from. When we say VMM is mobile, we mean mobile! You simply cannot find this level of flexibility anywhere else.

This flexibility builds on VMM’s existing feature set:

Monitor a host’s or data center’s CPU and memory
Monitor a VM’s CPU and memory
Manage a VM’s power state
Manage a VM’s network state

VMM no host stats
And we’re not talking just VMware ESX. Nope, VMM can do all of the above for VMware Infrastructure 3.5 and above, VMware vCenter 4, VMware Server 2, Microsoft Hyper-V, and Citrix XenServer 5. And with a AJAX-rich full mode for modern smart phones such as Apple’s iPhone and Google’s Android platform, and a lite mode for other devices, VMM is sure to support your favorite mobile device.

With VMM 1.0 monitoring and managing your favorite hypervisor is as close as the mobile phone in your pocket. Get it today!

Filed under: software development, virtualization | Leave a Comment

Getting Started with Hyper9 VMM Beta 2.2

15May09

FYI, if you find yourself just getting started with Hyper9 VMM Beta 2.2, be SURE to read the documentation on the Hyper9 VMM WIKI. The first thing I will do when you file a bug report is refer you to the documentation if you have not read it.

Also, for instructions on how to file a bug report please see this forum post.

Thanks!

–
-a

Filed under: software development, virtualization | Leave a Comment
Tags: vmm

A recorded demo of VMM Beta 2.2

14May09

I recorded a demo of VMM Beta 2.2 on my Mac using the Apple iPhone simulator. In this demo I log into a live VMware VirtualCenter 2.5 server in full mode, then I log into a Microsoft Hyper-V serve in full mode, and while logged into the Hyper-V server I transition into lite mode and then back to full. Please let me know if you have any questions.

Filed under: software development, virtualization | Leave a Comment
Tags: demo, mobile, video, vmm

Watch Hyper9 VMM Beta 2.2 – As Mobile As Your VMs

14May09

Just a little presentation I put together to explain what VMM is and how it works.

Filed under: this | Leave a Comment
Tags: iphone, mobile, virutalization, vmm

Read Hyper9 VMM Beta 2.2 – As Mobile As Your VMs

13May09

Here is a this slick presentation I put together that reviews VMM Beta 2.2. I welcome your feedback and comments. Thanks!

Download “Hyper9 VMM Beta 2.2 - As Mobile As Your VMs“

Filed under: software development, virtualization | 2 Comments
Tags: akutz, iphone, mobile, pdf, vmm

VMM Beta 2.2 is available!