Fortinet SD-WAN Lab Setup

As a Fortinet Systems Engineer, I will often run SD-WAN proof of concepts for my customers and have a need to create solid configs for these as well. To have this always-on lab, I utilized my VMware ESXi server to run two FortiGate VMs as Hubs and connected two FortiGate 80Fs as Spokes. I also setup WANem to mimic WAN conditions and introduce latency, packet loss and jitter. This post will share how I setup my virtual networking, VLANs, etc. to get everything up and connected. In a future post, I plan to share more of my BGP, VPN and SD-WAN configs, but it would be much too long of a post to share all that here.

Lab Components

Most SD-WAN deployments utilize a Hub and Spoke design. The Hub can be an active/passive or active/active high-availability (HA) cluster or two separate Hubs operating independently (possibly at separate data centers). In my lab my Hub is utilizing one active/passive HA cluster with multiple ISPs, illustrating the design of one data center:

• 2 FortiGate VMs deployed from OVA template
• vmnic1 = WAN1
• vmnic2 = WAN2
• vmnic3 = LAN
• vmnic4 = HA Heartbeat
• vmnic5 = Management

A typical Spoke design will be a branch or remote location with either one FortiGate or two setup in a HA cluster; they often will have two ISPs to take advantage of SD-WAN steering traffic optimally. In my lab my Spokes is utilizing one active/passive HA cluster with multiple ISPs:

• 2 FortiGate 80Fs + 1 FortiSwitch 108E downstream connected to both FortiGates
• WAN1 = ISP1
• WAN2 = ISP2
• internal2 = HA Heartbeat
• A = Fortilink connection to downstream FortiSwitch 108E

---

Underlying Networking

Network Diagram

Below is a diagram of the VLANs, subnets and components I used. The WANem WAN emulator in the middle is the router between the Hub WAN networks and Spoke WAN networks; it’s meant to mimic the Internet backbone. WANem will then route all Internet-bound up to my Home Lab FortiGates to reach the Internet so that I can illustrate Direct Internet Access at the Hub and Spoke. I use the Infrastructure 192.168.5.0/24 network to manage all of my VMs, including FortiManager; this helps me login to each FortiGate and VM over the network from my desk, but you might not need it in your lab. I know it’s complex, but we’ll get to each component shortly and you’ll see its relative simplicity…

Setup vSwitch

First off, I couldn’t get my FortiGate VMs to communicate with anything (including each other to setup HA) because they have a permanent MAC address and a HA hardware address.  I had to edit the vSwitch to permit MAC address changes in order for communication to flow:

Create Port-Groups in ESXi

These VLANs I created in ESXi are unique and either don’t route outside the Hypervisor (in the case of the Hub VLANs) or are only used on the physical FortiSwitch the Branch/HQ hangs off of.  Only the Infrastructure (192.168.5.0/24) and WANem Uplink (192.168.99.0/24) will route outside this lab environment.

Setup Branch WAN VLANs on Home Lab FortiSwitches

If you’re using FortiGate VMs for your Spoke/Branch, then you can skip this section. But I wanted to utilize physical FortiGates for my Spoke/Branch so I could connect devices, illustrate Zero Touch Provisioning, etc. I first created the Spoke/Branch WAN VLANs on my Home Lab FortiSwitches:

Then assigned these VLANs to the Home Lab FortiSwitch interfaces that the Branch FortiGate 80Fs WAN interfaces connect to:

Create WANem Uplink Interface on Home Lab FortiGates

Remember how WANem is my Internet backbone? Well to give this lab environment Internet access, we need to create the WANem uplink’s interface on the Home Lab FortiGates that connect to the Internet (and make sure the VLAN ID matches what we’re using in ESXi):

---

Virtual Machines

Now that we have our VLANs created in ESXi and on the physical FortiSwitches connecting between ESXi and the Branch FortiGates, we’re ready to build your VMs.

WANem

WANem is a simple tool created and maintained by TATA Consultancy Services in India that we will use to emulate WAN conditions of latency, jitter and packet loss. I downloaded the 3.0 beta 2 release since it was released in 2014 (a long, long time ago) and I figured it was stable enough.

I created a new WANem VM in ESXi as Linux Other 64-bit (because it runs Knoppix) with the following settings:

Note that I hit some kind of bug where the WANem-Uplink port-group wasn’t selectable for Network adapter 6… I had to download the .vmx file from ESXi, edit it (ethernet5.networkName = “WANem-Uplink”), upload it, then power the VM on.  Later on after I rebooted the hypervisor, I couldn’t get that NIC to work so I had to redo that process with the .vmx file.

Also note that WANem doesn’t like the SCSI controller for our hard drive, so I changed it to “IDE controller 1”:

Upon booting the VM up, select Start Menu > Preferences > WANem HD Installer:

And follow the prompts to use the whole disk and install GRUB to the MBR. Then disconnect the ISO and reboot.

At the next boot, type “startx” to bring up the GUI and the Network Adapter settings.  Assign addresses to your interfaces to match what you had configured in the VM’s Hardware Config:

• eth0 = Network Adapter 1 = Infrastructure = 192.168.5.102/24
• eth1 = Network Adapter 2 = Hub-WAN1 = 100.100.100.1/24
• eth2 = Network Adapter 3 = Hub-WAN2 = 200.200.200.1/24
• eth3 = Network Adapter 4 = Branch-WAN1 = 10.231.224.1/29
• eth4 = Network Adapter 4 = Branch-WAN2 = 10.231.224.9/29
• eth5 = Network Adapter 5 = WANem-Uplink = 192.168.99.2/24

And modify the default route to use the WANem Uplink by bringing up a terminal locally:

• exit2shell
• route delete default gw 192.168.5.1 eth0 (if specified a default gateway before)
• route add default gw 192.168.99.1 eth5 (use WANem-Uplink as the default route to the Internet)
• route add -net 192.168.x.0/24 gw 192.168.5.1 dev eth0 (so that I can reach WANem from my desk)

Now WANem will be accessible across the network http://192.168.5.102/WANem/. But we still need to NAT all outbound connections to the WANem Uplink interface so that our FortiGates (virtual and physical) can reach the Internet:

Branch WAN DHCP Server

This VM is really only useful if you want to run a DHCP server to provide addresses to the WAN interfaces of the Branch/Spoke; if you plan to use static IP addresses on your Spoke/Branch FortiGates, you can skip this section.  It also allows us to specify DHCP options to point FortiDeploy’d FortiGates to FortiManager without making FortiManager public (or using FortiGate Cloud).

• Network Adapter 1 = Infrastructure (192.168.5.107)
• Network Adapter 2 = Branch-WAN1 (10.231.224.2/29)
• Network Adapter 3 = Branch-WAN2 (10.231.224.10/29)

After the OS is installed, install dhcpd:

sudo apt install isc-dhcp-server

Then edit the config to create the WAN1 and WAN2 DHCP scopes and specify the FortiManager IP address (if using ZTP):

sudo vim /etc/dhcp/dhcpd.conf
option fmg code 240 = ip-address;
subnet 10.231.224.0 netmask 255.255.255.248 {
range 10.231.224.3 10.231.224.6;
option routers                10.231.224.1;
option domain-name-servers    10.231.224.1;
option subnet-mask            255.255.255.248;
option broadcast-address      10.131.224.7;
option fmg                    192.168.5.105;
}
subnet 10.231.224.8 netmask 255.255.255.248 {
range 10.231.224.11 10.231.224.14;
option routers                10.231.224.9;
option domain-name-servers    10.231.224.9;
option subnet-mask            255.255.255.248;
option broadcast-address      10.131.224.15;
option fmg                    192.168.5.105;
}
sudo vim /etc/default/isc-dhcp-server
INTERFACESv4=”ens192 ens224″
sudo service isc-dhcp-server restart
sudo service isc-dhcp-server status

Make sure service is running and Active, also shows DHCP request and acknowledgements:

cat /var/lib/dhcp/dhcpd.lease:

Hub FortiGates

Deploy your Hub FortiGate VMs from the OVF you can download from FortiCare and map the network adapters like so:

Boot the VM and configure the IP addresses:

config system interface
edit “port1”
set ip 100.100.100.2 255.255.255.0
set allowaccess ping
set alias “WAN1”
next
edit “port2”
set ip 200.200.200.2 255.255.255.0
set allowaccess ping
set alias “WAN2”
next
edit “port3”
set ip 10.0.0.1 255.255.255.0
set allowaccess ping
set alias “LAN”
next
edit “port4”
set ip 1.1.1.1 255.255.255.252
set alias “HA”
next
edit “port5”
set ip 192.168.5.113 255.255.255.0
set allowaccess ping https ssh http fgfm
set alias “MGMT”
next
end

Set the static routes so you can manage the FortiGate from your desk and it can reach out to FortiGuard to activate its license:

config router static
edit 1
set dst 192.168.x.0 255.255.255.0
set gateway 192.168.5.1
set device “port5”
next
edit 2
set gateway 100.100.100.1
set device “port1”
next
edit 3
set gateway 200.200.200.1
set device “port2”
next
end

Go ahead and apply the license, it will reboot.  It has to be able to reach FortiGuard though to activate before you can login.

To get HA to work, you’ll need to specify IP addresses for the hbdev interface:

Hub1:
config system interface
edit port4
set ip 1.1.1.1/30

Then configure HA:

config system ha
set group-id 100
set group-name “DataCenter”
set mode a-p
set password _______________
set route-ttl 60
set session-pickup enable
set session-pickup-connectionless enable
set ha-uptime-diff-margin 180
set override disable
set priority 129
set monitor “port1” “port2” “port3”
set unicast-hb enable
set unicast-hb-peerip 1.1.1.2
end

Repeat the prior process to deploy Hub2 from OVF and use the same HA config (except priority 128 and unicast-hb-peerip 1.1.1.1). Also configure port4 on Hub2 to use ip 1.1.1.2/30.

Hub2:

config system interface
edit port4
set ip 1.1.1.2/30

Now, confirm HA is good:

get sys ha status

Apply the license to Hub2 and let it reboot.

Wrap Up

At this point we have our two Hub FortiGate VMs in active-passive HA, two Spoke FortiGate 80Fs in active/passive HA, WANem VM acting as the Internet backbone and the DHCP VM serving up WAN DHCP addresses for the Spoke FortiGates. I also joined the two FortiGate clusters to FortiManager in a separate administrative domain (ADOM) so that I can build everything out in templates and standardize my configurations. Definitely more to come on this as I setup BGP, VPN, SD-WAN and ZTNA. Stay tuned!