Load Balancing two ISPs
Today I finally managed to use my two ISPs together on my desktop, combining both bandwidths into one big (almost 3Mbit!) pipe.
My setup:
- ADSL modem 1Mb down, 320Kb up
- GSM modem 2Mb down, 512Kb up
Configuring both at the same time is simple, but then we have two default gateways and our packets always go out through the first one found (or with the lower cost as defined in the ‘metric’ parameter.)
So how can we divide our packets over both links?
Googling around I found several suggestions to use iptables.
The general idea is:
- use -m statistic in a chain to choose packets to use on of two links (either the ‘nth’-method or the ‘average’-method
- set a mark on the packet
- use an ‘ip rule’ to select a routing table for mark 1, mark 2, etc.
That sounded like a perfect solution. This way I could really balance my two links like 40%/60% or whatever.
But it didn’t work…
My desktop is not a router, so I have to treat the packets in the OUTPUT chain, where routing has already taken place. The above-described method works on Linux routers treating the PREROUTING chain in iptables, where we can mark a package before routing.
So I studied IP ROUTE and IP RULES a bit more, browsing through the fantastic Linux Advanced Routing & Traffic Control site.
I discovered that we can use ‘nexthop’ to ‘hop’ between several routes.
After experimenting a bit I wrote the following script::
#!/bin/bash
#
# bal_local Load-balance internet connection over two local links
#
# Version: 1.0.0 - Fri, Sep 26, 2008
#
# Author: Niels Horn
#
# Set devices:
DEV1=${1-eth0} # default eth0
DEV2=${2-ppp0} # default ppp0
# Get IP addresses of our devices:
ip1=`ifconfig $DEV1 | grep inet | awk '{ print $2 }' | awk -F: '{ print $2 }'`
ip2=`ifconfig $DEV2 | grep inet | awk '{ print $2 }' | awk -F: '{ print $2 }'`
# Get default gateway for our devices:
gw1=`route -n | grep $DEV1 | grep '^0.0.0.0' | awk '{ print $2 }'`
gw2=`route -n | grep $DEV2 | grep '^0.0.0.0' | awk '{ print $2 }'`
echo "$DEV1: IP=$ip1 GW=$gw1"
echo "$DEV2: IP=$ip2 GW=$gw2"
### Definition of routes ###
# Check if tables exists, if not -> create them:
if [ -z "`cat /etc/iproute2/rt_tables | grep '^251'`" ] ; then
echo "251 rt_dev1" >> /etc/iproute2/rt_tables
fi
if [ -z "`cat /etc/iproute2/rt_tables | grep '^252'`" ] ; then
echo "252 rt_dev2" >> /etc/iproute2/rt_tables
fi
# Define routing tables:
ip route add default via $gw1 table rt_dev1
ip route add default via $gw2 table rt_dev2
# Create rules:
ip rule add from $ip1 table rt_dev1
ip rule add from $ip2 table rt_dev2
# If we already have a 'nexthop' route, delete it:
if [ ! -z "`ip route show table main | grep 'nexthop'`" ] ; then
ip route del default scope global
fi
# Balance links based on routes:
ip route add default scope global nexthop via $gw1 dev $DEV1 weight 1 \
nexthop via $gw2 dev $DEV2 weight 1
# Flush cache table:
ip route flush cache
# All done...
You can download the script here from my homepage.
This is not the perfect solution, as routes are cached, so once you connected to an external site, it will continue to use the linkt hat was originally selected.
So an FTP download won’t benefit from this solution, but torrent downloads will, as they use several parallel connections.
I tested the result and managed to download using BitTorrent with the incredible speed of 250KBytes/sec:
Update: This script is now also part of an article on slackwiki. 
This entry was posted on Friday, September 26th, 2008 at 23:56 and is filed under Linux, networking. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
Dutch guy living in Brazil. Has worked in the IT business since it was invented (well, almost...). Interests: Slackware, Lego, Star Wars.
December 8th, 2008 at 20:24
great post, thanks
but what is that utility you are watching interface bandwith with?
December 9th, 2008 at 9:02
I used ‘iptraf’ as it comes with Slackware. It’s a nice tool that works from the console.
July 4th, 2009 at 21:22
Niels, interesting article. I am experimenting with Opera Unite. I have 2 wwan accounts (Verizon). How do I serve up a web page thru both connections simultaneously?
July 23rd, 2009 at 22:18
Sorry for the delay in responding – I have been busy with some other projects.
If I understand correctly, you want your server to be accessible from both internet connections?
This is not impossible, but then the load-balancing is not done on your side.
When a user accesses your web page like "http://yourserver/page.html" the address of your server is translated by the DNS server to an IP address.
Since two internet connections means two external IP addresses, you cannot have incoming requests from both connections.
Professional site resolve this at the DNS level.
Try for instance "dig http://www.uol.com.br".
If you repeat the command several times, you can get different answers for the IP address…
Hope this answered your question at least partially!
October 12th, 2009 at 5:08
Hello, Niels.
Thank you for this great post.
But, even though you are far more experienced than me in scripting, I dare to tell you that awk can make that IP address extraction in one run (I'm referring to your load balance script):
# Get IP addresses of our devices:
ip1=`ifconfig $DEV1 | awk '/inet addr:/ { split($2,inetline,":"); print inetline[2]}'`
ip2=`ifconfig $DEV2 | awk '/inet addr:/ { split($2,inetline,":"); print inetline[2]}'`
As you work with heavy-weight servers, perhaps this can make some difference (for a SOHO user speed and footprint improvements are unnoticeable).
October 12th, 2009 at 10:12
@gmbastos: You're right, it can be done in a shorter line. Like they say "there are many ways to Rome".
I kept the script as simple & clear as possible to make it easier to understand. But everyone is free to improve it like you did.
Awk if an extremely powerful tool and I sometimes think I only use like 10% of its power…
January 7th, 2010 at 9:36
hi,
could you please help me out where in i get even (same)speed across two ip’s say 100kbits/sec through two ett0 and ppp0.
January 7th, 2010 at 12:10
Both links should be equally used, but only if you have several simultaneous connections (like my example downloading with some torrent software). If you are downloading a large file from a single IP, there will be only one IP-to-IP connection that cannot be balanced between two links, as the sending IP can only send data to one destination for a single download.
With torrent software you will have many simultaneous connections, downloading different parts from different peers. In this case the balancing works fine.
March 24th, 2010 at 8:54
Hello Niels….This is a wonderful post. Thanks.
I was following your link.
My linux machine is behind a NAT router and it is connected to the router/switch using eth0. For the other connection I use a GSM broadband modem (ppp0). I could get it working after making one change from your script, for now I did that manually.
For my case, “ifconfig ppp0″ is
ppp0 Link encap:Point-to-Point Protocol
inet addr:115.117.135.93 P-t-P:172.29.243.129 Mask:255.255.255.255
……
route -n | grep ppp0
……
172.29.243.129 0.0.0.0 255.255.255.255 UH 0 0 0 ppp0
0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 ppp0
……
From your script.. gw2=`route -n | grep $DEV2 | grep ‘^0.0.0.0′ | awk ‘{ print $2 }’`
gw2 becomes ‘0.0.0.0′ and this was not working.
But instead of that when I changed gw2 to 172.29.243.129, then things were working.
March 24th, 2010 at 9:08
@aspnair:
It seems that when you use your GSM modem, it is not recognized as a gateway in your routing table.
I use a script to “dial” my ISP with my GSM modem that automatically adds it as a gateway, so the balancing script finds it when I start that.
Setting the gw2 variable manually works, but it would be best to set the gateway on your ppp connection.
Anyway, thanks for your feedback!
March 24th, 2010 at 12:03
I just modified the script to find out the ppp0 gateway from ifconfig itself.
Following is my modification.
gw2=$(route -n | grep $DEV2 | grep ‘^0.0.0.0′ | awk ‘{print $2}’)
if [ "$gw2" == "" ]
then
gw2=$(ifconfig $DEV2 | grep P-t-P | awk ‘{print $3}’ | awk -F: ‘{print $2}’)
fi
I tried dialing using pon with pppd and wvdial. It looks like, they dont add a default route
when it sees another active default route, eth0 interface in this case.
March 27th, 2010 at 13:09
@aspnair:
IIRC, pppd should create the gateway. You might have to set the “defaultroute” parameter in your pppd configuration file, or use it as an option when calling pppd.
March 21st, 2011 at 22:25
Hi,
I tried this script but under my circumstances does not work. I have 2 adsl bridged modems from the same ISP and I connected it via pppoe. The problem is that I have the same default gw for ppp0 and ppp1. When only one link is up it works perfectly but when I start the second and run the script the two links goes “down”(not really because when I shut down one the other begin to work). Do you need more details? Any help will be appreciated.
P.S. I use Slackware too.
March 22nd, 2011 at 10:07
Hello,
In this case – having the same gateway – my solution won’t work for you…
But it should be possible to adapt it to your setup. The problem is I can’t test it for you here
You might try using “via a.b.c.d” where a.b.c.d is the internal IP address of the NIC connected to your adsl modem, instead of the gateway address.
But – as I said – I have not tried this here…