So you have a load balanced internet setup using multiple WAN Links that provide internet to your network. This is great except that it can often hide issues with WAN links by spreading the load across working links. Here is a simple solution to monitoring load balanced WAN links using Smokeping and a little bit of routing.
Setup
You have multiple WAN links such as ADSL, Wireless, Fibre, Ethernet over copper that you want to load balance for any number of reasons. Because the load balancing connections will automatically load balance around a failure you may want to be able to monitor each WAN link to identify issues on that link. For this example I will use the following information.
Network Setup.
- 3 x WAN interfaces that are terminated onto a Sonicwall
- A Sonicwall with a LAN IP of 192.168.50.250
- A Linux (I use Ubuntu) server on the network with an IP address of 192.168.50.5. This doesn’t need to be powerful, most monitoring machines that I build are VMs with 512MB Memory and a 10GB Hard drive so any old machine will be suitable.
- Sendmail and smokeping are already setup and configured correctly to be working.
You will also need one reliable and pingable IP on the internet for each WAN interface. In this example I have selected the following three.
- 8.8.8.8
- 8.8.4.4
- 4.2.2.2
Smokeping Setup
Now that we have all the information we setup Smokeping to monitor those IP Addresses. Add the following code to your Targets configuration file to create a ‘Control’ Menu with the three graphs, one for each WAN Interface.
+ Remote ++ Google menu = Google title = Google host = www.google.com ++ Gateway menu = Gateway title = Gateway (192.168.50.250) host = 192.168.50.250 ++ X1Ping menu = X1 Ping title = X1 Ping Test (8.8.8.8) host = 8.8.8.8 alerts = startloss,someloss,bigloss,rttdetect,hostdown ++ X2Ping menu = X2 Ping title = X2 Ping (8.8.4.4) host = 8.8.4.4 alerts = startloss,someloss,bigloss,rttdetect,hostdown ++ X3Ping menu = X3 Ping title = X3 Ping (4.2.2.2) host = 4.2.2.2 alerts = startloss,someloss,bigloss,rttdetect,hostdown
Notice that there are a few other entries in this Targets other then the targets that we discussed earlier. This is to assist in identifying if an issue is on a connection or a WAN interface. For example if there is packet loss on the Gateway connection then the issue is more likely on the LAN. if the LAN is solid but the WAN is reporting loss then the issue is most likely on the WAN.
Now add the following to the Alerts configuration file
*** Alerts *** to = [email protected] from = [email protected] +bigloss type = loss # in percent pattern = ==0%,==0%,==0%,==0%,>0%,>0%,>0% comment = suddenly there is packet loss +someloss type = loss # in percent pattern = >0%,*12*,>0%,*12*,>0% comment = loss 3 times in a row +startloss type = loss # in percent pattern = ==S,>0%,>0%,>0% comment = loss at startup +rttdetect type = rtt # in milli seconds pattern = <10,<10,<10,<10,<10,<100,>100,>100,>100 comment = routing messed up again ? +hostdown type = loss # in percent pattern = ==0%,==0%,==0%, ==U comment = no reply +lossdetect type = loss # in percent pattern = ==0%,==0%,==0%,==0%,>20%,>20%,>20% comment = suddenly there is packet loss
Once setup and working you will start to see information appear in your smokeping setup for the monitored interfaces that should look something similar to this.
A configured and working X1 and X2 ping test
Now that setup alone will not correctly monitor the individual WAN ports because the traffic will be automatically balanced to a working WAN interface in the event that an outage or issue does occur. The next steps allows the Sonicwall (or any load balancing device) to send the traffic down the appropriate WAN interface.
Sonicwall Setup
Example X1 route policy for Sonicwall (Click for larger view)
Once Smokeping is setup and working and you are receiving information for the probes the next step is to setup the routing that will allow the probes to monitor the correct interface.
- Once in your Sonicwall interface select Routing under Network (I am going to perform this setup in a Sonicwall but it could be done with any load balancing device.)
- Add a route policy with the following settings.
Source = LAN Subnets
Destination = Create a new Object and call it X1 Ping Test (or X2 etc) This will be a WAN object with an IP of 8.8.8.8 (or appropriate IP)
Service = ICMP
Gateway = X1 Default Gateway (or appropriate interface Gateway)
Interface = X1 (or appropriate interface e.g. X2)
Metric = 10
Comment = X1 Interface monitoring
Disable route when interface disconnected = unchecked (This will mean that if the interface is down the probe will stop)
Leave Allow VPN Path to take precedence,Permit TCP acceleration and Probe to None / unchecked - Repeat this for each WAN interface that you have using the appropriate information in each option (Gateway, Interface and Probe IP) until you have added all the routes required
X1 and X2 Routes configured ready for monitoring
And thats all that is required. This setup will allow each probe to be routed correctly down the appropriate interface. By un-checking the option to disable the route when interface is disconnected means that the probe will start to fail once an interface is disconnected like the image below. This will also send an email alert advising that there is an issue on X1 interface.
- During a test the X1 interface was disconnected to test that the reporting was working correctly
While this type of monitoring works great to identify issues with a WAN interface it can also be adapted to catch high packet loss which might indicate that a connection is under load or something is wrong with the connection itself.