UPDATE – This topic is being discussed on the Openswan Users mailing list as well as the OS X forums:
Jacco maintains the definitive site for connecting to Openswan with the various client OSs. However, his site has not been updated for connecting to Openswan with clients running the Mac OS X Leopard (10.5.1) operating system. Based on my experiences getting this configuration to work, there are three notes worth mentioning.
Testing on VMware Fusion
A lot of OS X users are taking advantage of VMware Fusion for testing various server configurations. My initial OS X-to-Openswan testing used Fusion to facilitate an Ubuntu 7.04 Feisty Fawn x86_64 system. However, it turns out that VMware Fusion has a bug in its NAT implementation that causes network packets to fragment. The result of which is GRE errors in a VM that is running VPN software, whether it is pptpd or Openswan.
The solution is to either move your VM to ESX or VMware Server or to change your VMware Fusion VM’s networking type to Bridged instead of NATd.
OS X and VPN certifications
Jacco mentions the requirements that OS X places on the server SSL certificate: subjectAltName and the extendedAttributes bit. However, I found out the hard way that not all implementations of OpenSSL (on Ubuntu for example) will add the usr_cert attributes (which subjectAltName is under) to the certificate. You have to tell it to when you sign your certificate, e.g. -extfile $SSL_DIR/openssl.cnf -extensions usr_cert. The “extfile” option tells it which file to grep the extensions from and the -extensions option tells it which section to read (in this case, usr_cert).
If you do not do this and are running Tiger you will get an INVALID_CERTIFICATE error on your VPN server. The error is slightly different and far less useful if you are running Leopard. The error Leopard causes is PAYLOAD_MALFORMED. Yeah, that’s useful alright.
Leopard (10.5.1) requires MPPE-128 when negotiating L2TP/IPsec connections (you can change this by directly editing /Library/Preferences/SystemConfiguration/preferences.plist). However, if you attempt such a connection following a reboot or clean boot it will fail with a warning that MPPE is required but the kernel does not support it.
At first I thought I had misconfigured openswan, xl2tpd, and the ppp options file on my Ubuntu VPN server, but that turned out not to be the case as I could make a successful PPTP connection with MPPE turned on at 128 (you can edit the PPTP MPPE encryption strength in the GUI, just not the strength for L2TP over IPSEC). Seeing this I tried connecting with L2TP over IPSEC again and it worked. That bothered me all day because it started working for no reason.
Then at lunch I rebooted my Mac after VMware Fusion caused a Kernel panic. I then attempted to reconnect to my VPN with L2TP over IPSEC and no joy! Very odd. At this point I was still clueless. After a while it started working again. Then last night I had to replace my battery necessitating a cold boot. And the VPN failed again! I finally caught on. I initiated a PPTP connection, disconnected, and then reconnected with L2TP over IPSEC and it worked. I rebooted 3 times and did various tests ad proved conclusively that you have to initiate a successful PPTP connection to load the MPPE module. Doing a L2TP over IPSEC connection doesn’t get the job done.
For testing purposes I was actually initiating a PPTP connection to a different server than my VPN server (a VM) just to prove it was not my L2TP IPSEC configuration on my server at fault (my VPN server stayed alive during all of these reboots).
I did try calling /usr/sbin/pppd manually from Terminal with “require-mppe-128” to see if that would load the MPPE module: nope. If you look at your processes you will see that when you initiate a PPTP or L2TP over IPSEC connection from Network Manager pppd is actually called with some special flags that do not appear in its man page:
serviceid 6FD5995A-D9A3-4272-90F9-DBC9E4B85F7E controlled
The serviceid is the UUID of the VPN configuration that is stored in the aforementioned plist file. I do not know what controlled means. I do know that you cannot call this manually from Terminal because it will give you an error that the launch daemon attempting this function is not authenticated (I assume launchd calls this and authenticates it somehow).
The solution is to either always make a successful PPTP connection first (POP before SMTP anyone?) or edit the preferences plist file so that L2TP over IPSEC does not require MPPE. The IPSEC tunnel is already encrypted, so the PPP connection that L2TP calls does not need to be encrypted with MPPE, because it’s just double-encryption.
Hope this helps!