Sunday 27 August 2017

linux - Load balancing with multiple gateways


I have to different ISPs, each on each own network. The main connects via ethernet and the secondary via wifi. The two networks have no relation at all. I just connect to them simultaneously. The reason I want to load balance between them is to achieve higher Internet speeds. Note: I have no advanced network hardware. Just my pc and the two routers that I have no access... main network:


if: eth0
gw: 192.168.178.1
my ip: 192.168.178.95
speed: 400 kbit/s

secondary network:


if: wlan0
gw: 192.168.1.1
my ip: 192.168.1.95
speed: 300 kbit/s

A diagram to explain the situation:


enter image description here


I'm on Arch Linux x64. I use netcfg to configure the interfaces Configs:


# /etc/network.d/main
CONNECTION='ethernet'
DESCRIPTION='A basic static ethernet connection using iproute'
INTERFACE='eth0'
IP='static'
ADDR='192.168.178.95'

# /etc/network.d/second
CONNECTION='wireless'
DESCRIPTION='A simple WEP encrypted wireless connection'
INTERFACE='wlan0'
SECURITY='wep'
ESSID='wifi_essid'
KEY='the_password'
IP="static"
ADDR='192.168.1.95'

And I use iptables to load balance, rules:


#!/bin/bash
/usr/sbin/ip route flush table ISP1 2>/dev/null
/usr/sbin/ip rule del fwmark 101 table ISP1 2>/dev/null
/usr/sbin/ip route add table ISP1 192.168.178.0/24 dev eth0 proto kernel scope link src 192.168.178.95 metric 202
/usr/sbin/ip route add table ISP1 default via 192.168.178.1 dev eth0
/usr/sbin/ip rule add fwmark 101 table ISP1
/usr/sbin/ip route flush table ISP2 2>/dev/null
/usr/sbin/ip rule del fwmark 102 table ISP2 2>/dev/null
/usr/sbin/ip route add table ISP2 192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.95 metric 202
/usr/sbin/ip route add table ISP2 default via 192.168.1.1 dev wlan0
/usr/sbin/ip rule add fwmark 102 table ISP2
/usr/sbin/iptables -t mangle -F
/usr/sbin/iptables -t mangle -X
/usr/sbin/iptables -t mangle -N MARK-gw1
/usr/sbin/iptables -t mangle -A MARK-gw1 -m comment --comment 'send via 192.168.178.1' -j MARK --set-mark 101
/usr/sbin/iptables -t mangle -A MARK-gw1 -j CONNMARK --save-mark
/usr/sbin/iptables -t mangle -A MARK-gw1 -j RETURN
/usr/sbin/iptables -t mangle -N MARK-gw2
/usr/sbin/iptables -t mangle -A MARK-gw2 -m comment --comment 'send via 192.168.1.1' -j MARK --set-mark 102
/usr/sbin/iptables -t mangle -A MARK-gw2 -j CONNMARK --save-mark
/usr/sbin/iptables -t mangle -A MARK-gw2 -j RETURN
/usr/sbin/iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
/usr/sbin/iptables -t mangle -A PREROUTING -m comment --comment "this stream is already marked; escape early" -m mark ! --mark 0 -j ACCEPT
/usr/sbin/iptables -t mangle -A PREROUTING -m comment --comment 'prevent asynchronous routing' -i eth0 -m conntrack --ctstate NEW -j MARK-gw1
/usr/sbin/iptables -t mangle -A PREROUTING -m comment --comment 'prevent asynchronous routing' -i wlan0 -m conntrack --ctstate NEW -j MARK-gw2
/usr/sbin/iptables -t mangle -N DEF_POL
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'default balancing' -p tcp -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'default balancing' -p udp -m conntrack --ctstate ESTABLISHED,RELATED -j CONNMARK --restore-mark
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw1 tcp' -p tcp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 0 -j MARK-gw1
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw1 tcp' -p tcp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 0 -j ACCEPT
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw2 tcp' -p tcp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 1 -j MARK-gw2
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw2 tcp' -p tcp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 1 -j ACCEPT
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw1 udp' -p udp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 0 -j MARK-gw1
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw1 udp' -p udp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 0 -j ACCEPT
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw2 udp' -p udp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 1 -j MARK-gw2
/usr/sbin/iptables -t mangle -A DEF_POL -m comment --comment 'balance gw2 udp' -p udp -m conntrack --ctstate NEW -m statistic --mode nth --every 2 --packet 1 -j ACCEPT
/usr/sbin/iptables -t mangle -A PREROUTING -j DEF_POL
/usr/sbin/iptables -t nat -A POSTROUTING -m comment --comment 'snat outbound eth0' -o eth0 -s 192.168.0.0/16 -m mark --mark 101 -j SNAT --to-source 192.168.178.95
/usr/sbin/iptables -t nat -A POSTROUTING -m comment --comment 'snat outbound wlan0' -o wlan0 -s 192.168.0.0/16 -m mark --mark 102 -j SNAT --to-source 192.168.1.95
/usr/sbin/ip route flush cache

(this script was made by fukawi2, I don't know how to use iptables) but I have no Internet connection...


output of iptables -t mangle -nvL


Chain PREROUTING (policy ACCEPT 1254K packets, 1519M bytes)
pkts bytes target prot opt in out source destination
1278K 1535M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore
21532 15M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* this stream is already marked; escape early */ mark match ! 0x0
582 72579 MARK-gw1 all -- eth0 * 0.0.0.0/0 0.0.0.0/0 /* prevent asynchronous routing */ ctstate NEW
2376 696K MARK-gw2 all -- wlan0 * 0.0.0.0/0 0.0.0.0/0 /* prevent asynchronous routing */ ctstate NEW
1257K 1520M DEF_POL all -- * * 0.0.0.0/0 0.0.0.0/0

Chain INPUT (policy ACCEPT 1276K packets, 1535M bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 870K packets, 97M bytes)
pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 870K packets, 97M bytes)
pkts bytes target prot opt in out source destination

Chain DEF_POL (1 references)
pkts bytes target prot opt in out source destination
1236K 1517M CONNMARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default balancing */ ctstate RELATED,ESTABLISHED CONNMARK restore
15163 2041K CONNMARK udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default balancing */ ctstate RELATED,ESTABLISHED CONNMARK restore
555 33176 MARK-gw1 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* balance gw1 tcp */ ctstate NEW statistic mode nth every 2
555 33176 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* balance gw1 tcp */ ctstate NEW statistic mode nth every 2
277 16516 MARK-gw2 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* balance gw2 tcp */ ctstate NEW statistic mode nth every 2 packet 1
277 16516 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* balance gw2 tcp */ ctstate NEW statistic mode nth every 2 packet 1
1442 384K MARK-gw1 udp -- * * 0.0.0.0/0 0.0.0.0/0 /* balance gw1 udp */ ctstate NEW statistic mode nth every 2
1442 384K ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* balance gw1 udp */ ctstate NEW statistic mode nth every 2
720 189K MARK-gw2 udp -- * * 0.0.0.0/0 0.0.0.0/0 /* balance gw2 udp */ ctstate NEW statistic mode nth every 2 packet 1
720 189K ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* balance gw2 udp */ ctstate NEW statistic mode nth every 2 packet 1

Chain MARK-gw1 (3 references)
pkts bytes target prot opt in out source destination
2579 490K MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* send via 192.168.178.1 */ MARK set 0x65
2579 490K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save
2579 490K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0

Chain MARK-gw2 (3 references)
pkts bytes target prot opt in out source destination
3373 901K MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* send via 192.168.1.1 */ MARK set 0x66
3373 901K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save
3373 901K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0

Answer



Unless you can somehow split up the traffic based on their local parameters (like LAN IP), using this solution you will end up having a lot of random errors because a lot of sites don't allow you to use the same session from radically different IP addresses, not to mention other protocols like FTP, DNS, etc.


If you really want to do this, you will need to rent a (virtual) server with at least 2 IP addresses and build a VPN on each of your connections, then set up an OSPF load balancing to utilize both connections equally. Your server will need to have at least the double of your two connections' speed together.


In summary: without network support failover works well enough, load balancing however is going to be a constant pain.


Update: What you need to do:



  1. Set up the server and make sure your local machine sees the two server IP's through different connections.

  2. Start two OpenVPN servers on the server, listening on each of the addresses.

  3. Start two OpenVPN clients on the client.

  4. Install Quagga on both the server and the client.

  5. Make sure the server and the client see each other through the OpenVPN without advertising any routes yet.

  6. Set up routing advertisement on the server so the default route (0.0.0.0/0) gets advertised over OSPF one link to the client via redistributing static routes. (This is hard.)

  7. Add the second link as a backup link over OSPF.

  8. Change the configuration so the two links are used in load balancing modes.


OSPF is a whole new world, it's an integral part of how the internet out there works. If you really want to do this, I recommend you give yourself time and read some books as it is way out of scope for this description. I've done this once and I might get back to writing a tutorial on this, but it will take considerable time so I'll make no promises.


No comments:

Post a Comment

Where does Skype save my contact's avatars in Linux?

I'm using Skype on Linux. Where can I find images cached by skype of my contact's avatars? Answer I wanted to get those Skype avat...