install a VPN server on a Mac

In this guide we will install SoftEther VPN server that is available for Linux, macOS and Windows.

One of the most popular VPN server is without a doubt the open source OpenVPN server, which we can install via homebrew but that has no GUI. Other important factor is that most VPN servers operate trough the TCP protocol. Unfortunately TCP ports are blocked in public places, hotels etc. and only the essential TCP port 80 for https is open and 443 for https. So if your boss wants to connect to your corporate network and she/he is in a hotel room where firewall blocks majority of TCP traffic, then you’ll be called quite quickly why the VPN server is not working when it actually does…

A nice thing about SoftEther server is that it runs over UDP protocol that is less probably blocked by firewalls.

SoftEther server can be run on Mac OS X 10.4 and up – you can use use any old mac as a dedicated VPN server, even an a G5 !

For the purpose of the article we are installing it on macOS 10.4.6 Mojave

First, let’s go to softether.org and download the installer matching our system. We also need to download the VPN Server Manager that is a separate application. That means we can manage our server remotely from other OSes too. After unpacking the server install we get the following folder that we placed on the desktop:

Let’s open the Terminal and go to this folder:

cd /Users/admin/Desktop/vpnserver/

in order to install it we need the .install.sh script which is in this folder, but invisible, because its name starts with dot. So let’s switch to sudo in order to see everything what we have in this folder:

sudo su

…and hit Enter. Now let’s list the content of the folder with:

ls

…and hit Enter again. This is the output we get:

.install.sh
Authors.txt
Makefile
ReadMeFirst_Important_Notices_cn.txt
ReadMeFirst_Important_Notices_en.txt
ReadMeFirst_Important_Notices_ja.txt
ReadMeFirst_License.txt
code
hamcore.se2
lib

let’s start the install with:

./.install.sh

after answering the questions the following windows will appear if we don’t have the command line tools installed already:

let’s hit Install… if you see this window, you will have to start the installation again by repeating the ./.install.sh command

After the VPN server is successfully installed we have to start it:

./vpnserver start

or in the future you can stop it with:

./vpnserver stop

Let’s install the manager app that comes as a .pkg file.

after the installation is complete look for VPN Server Manager in the Applications folder

after opening, the system is reminding us, that this app will not work on future versions on macOS starting with Catalina that runs only 64 bit apps. Hit OK to ignore the message…

In the windows that appears click on New Setting… name the settings as ‘Test VPN Server’, enter the IP of your server, or if you are running the Manager locally, then simply check the Connect to Localhost setting. Leave the password field empty and close the window with OK.

in the main window select the ‘Test VPN Server’ from the list and hit the Connect button. A window will prompt you to give a password for the server – don’t forget it as you will need it when changing its settings.

check the Remote Access VPN Server – although SoftEther has an exciting feature to connect multiple corporate sites with bridge function, we are not going to explore it now. Let’s just note that with this function you can avoid purchasing expensive Cisco routers.

in the next windows you can leave VPN as a name for the Virtual Hub – read the documentation to understand what hubs are.

A priceless surprise awaits you here: a built-in dynamic DNS client for FREE! Usually it’s something you have to pay about 50 EUR / year. The name can be customised:

Enable the L2TP Server Function which provides LPTP over IPsec that is required to connect all modern OSes as macOS, iOS, Android or Windows. Also specify a Pre-Shared key.

For our testing we will disable the Azure option in the next screen – see the documentation for what it is…

Let’s create our first user by entering a name and a password – we are not going to cover how to use a certificate in this guide.

after defining our test user and returning to the previous windows as a third step choose en0 as a local bridge – without enabling this our VPN server will not provide a DHCP service and although users will be able to authenticate, they will get no IP address and the connection will be ended.

On the firewall / router we have to port-forward UDP 500 and 4500 ports to the IP of our VPN server!

Finally we are ready to configure the VPN connection on a client machine. Go to Apple / System Preferences / Network… click the plus sign, choose VPN from the Interface list and L2TP over IPSec for the VPN Type. You can give any name for the Service.

The beautiful thing about SoftEther is that you don’t need a third-party vpn client app, you can use the built-in of the OS. Same is true for iOS.

Hit the Create button, then enter the IP address of the server and the user name. By clicking the Authentication button you can enter the password and the shared key

and finally click the Connect to initiate a VPN connection to the server!

In order to start the server at startup automatically, we need a LaunchDaemon plist.

1) stop the server, and lets move the ‘vpnserver’ folder inside Applications folder as a final destination

2) we are going to create  a wrapper script first:

sudo pico /Applications/vpnserver/vpnserver.start.sh

paste the following and save the file:

/Applications/vpnserver/vpnserver start


now, we have to make it executable:

chmod +x /Applications/vpnserver/vpnserver.start.sh


as next, we are going to create the plist that will call this script:

sudo pico /Library/LaunchDaemons/org.SoftEther.VPN.plist

paste the following and save it:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>org.SoftEther.VPN.plist</string>
	<key>ProgramArguments</key>
	<array>
		<string>/Applications/vpnserver/vpnserver.start.sh</string>
		<string>start</string>
	</array>
	<key>RunAtLoad</key>
	<true/>
</dict>
</plist>

let’s see if it works:

sudo launchctl load /Library/LaunchDaemons/org.softether.vpn.plist

Open the Activity Monitor and look if you see a vpnserver process. The number of processes you see depends on the number of CPU cores you have.