Desired Config: An openVPN client connection running from a Tomato/OpenWRT/DDWRT router, routing traffic for only some of our LAN clients through the VPN, while all others route through the default ISP connection.

Enter “ip route” (I've always just used the default “route” in the past). “Ip route” allows the creation of multiple routing tables and “rules” to direct the traffic to a specific table.

Details on everything are below, or cut to the chase.

My VPN provider sets a bunch of routes when making the connection, if you let it. If you don't want that you need to set the openVPN directive “route-noexec”. In Tomato, there's an option called “Create NAT on tunnel”, that is checked. Under Advanced settings, my openVPN client settings are like so:

persist-key
persist-tun
tls-client
auth-user-pass /tmp/auth.txt
comp-lzo
verb 1
reneg-sec 0
route-noexec
route-up /tmp/vpnrouteup.sh
down /tmp/vpnroutedown.sh

The contents of /tmp/auth.txt are simply VPN provider username on the top line, password on the second line. The contents of the route-up and route-down scripts are at the bottom of this page.

For those interested, the “Create NAT on tunnel” option executes the /tmp/etc/openvpn/fw/client1-fw.sh script, which looks like:

#!/bin/sh
iptables -I INPUT -i tun11 -j ACCEPT
iptables -I FORWARD -i tun11 -j ACCEPT
iptables -t nat -I POSTROUTING -s 192.168.1.0/255.255.255.0 -o tun11 -j MASQUERADE

From what I've seen there appear to be 3 default tables:

• main
• local
• default

This can be confirmed via ip rule list:

root@wifi1:/tmp/home/root# ip rule list
0:	from all lookup local 
32766:	from all lookup main 
32767:	from all lookup default

Other tables are defined with integer names. To create a new table simply add a route to it with ip route add [route] table #tablename#:

ip route add 192.168.100.100 via 127.0.0.1 dev lo table 100

To see the table, ip route list table 100:

root@wifi1:/tmp/home/root# ip route list table 100
192.168.100.100 via 127.0.0.1 dev lo 

To add a rule to affect traffic and direct it via routes in your new table 100, ip rule add [blah] table 100, and to see the rule in place, ip rule list:

root@wifi1:/tmp/home/root# ip rule add from 9.9.9.9 to 192.168.100.100 dev lo table 100
root@wifi1:/tmp/home/root# ip rule list
0:	from all lookup local 
32765:	from 9.9.9.9 to 192.168.100.100 iif lo lookup 100 
32766:	from all lookup main 
32767:	from all lookup default

That rule (as far as I can tell) tells the kernel that when traffic arrives on interface lo from ip 9.9.9.9 with destination address 192.168.100.100, then the pertinent routes are found in routing table 100. If you want to direct traffic both ways to and from specific ip's, then you must do so explicitly:

root@wifi1:/tmp/home/root# ip rule add to 9.9.9.9 from 192.168.100.100 dev lo table 100
root@wifi1:/tmp/home/root# ip rule list
0:	from all lookup local 
32764:	from 192.168.100.100 to 9.9.9.9 iif lo lookup 100 
32765:	from 9.9.9.9 to 192.168.100.100 iif lo lookup 100 
32766:	from all lookup main 
32767:	from all lookup default 

If you make a mistake and want to delete those rules, ip rule delete pref #rule#:

root@wifi1:/tmp/home/root# ip rule delete pref 32764
root@wifi1:/tmp/home/root# ip rule delete pref 32765
root@wifi1:/tmp/home/root# ip rule list
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

In my case, the routes that get set by my provider look like so:

root@wifi1:/tmp/home/root# ip route show |grep tun11
10.173.1.5 dev tun11 proto kernel scope link src 10.173.1.6
10.173.1.1 via 10.173.1.5 dev tun11
0.0.0.0/1 via 10.173.1.5 dev tun11
128.0.0.0/1 via 10.173.1.5 dev tun11

My TUN ifconfig in that case looks like:

root@wifi1:/tmp/home/root# ifconfig tun11
tun11 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.173.1.6 P-t-P:10.173.1.5 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:17428 errors:0 dropped:0 overruns:0 frame:0
TX packets:19874 errors:0 dropped:16 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:9120474 (8.6 MiB) TX bytes:3519467 (3.3 MiB)

We can see then that our tun11 IP is .6, our gateway (which is “via” in the table) is .5, and then some default routes are set.

To make these routes work for only some of our LAN clients, we need to add them to a new routing table, and then specify rules to make use of them.

In my case, I'm going to route only traffic from 192.168.1.253 through the VPN.

Manually, that would go like this:

ip route add 10.173.1.5 dev tun11 proto kernel scope link src 10.173.1.6
ip route add 0.0.0.0/1 via 10.173.1.5 dev tun11 table 10
ip route add 128.0.0.0/1 via 10.173.1.5 dev tun11 table 10
ip route add 10.173.1.1 via 10.173.1.5 dev tun11 metric 1 table 10
ip rule add from 10.173.1.0/24 table 10
ip rule add to 10.173.1.0/24 table 10
ip rule add from 192.168.1.253 table 10

Now, since I also have devices getting their DNS servers from DHCP, we'll also need to add rules for those internal devices to make use of the ISPs default route to access DNS servers. To do that, the rules simply address traffic to and from the internal client ip to the DNS servers via the “main” table instead of our new table “10”:

ip rule add from 192.168.1.253 to 89.150.129.10 lookup main
ip rule add from 192.168.1.253 to 89.150.129.22 lookup main
ip rule add to 192.168.1.253 from 89.150.129.10 lookup main
ip rule add to 192.168.1.253 from 89.150.129.22 lookup main

Disclaimer: I have never made use of awk before doing this configuration, so this may be the dumbest awk code ever written. It works, so that's good enough for me.

A basic script to set the routes in my case looks like this:

vpndhosts="192.168.1.253"
dnsservers="89.150.129.10 89.150.129.22"
tun_iface=$(ifconfig|grep tun|awk '{ print $1 }')
tun_inet=$(ifconfig $tun_iface|grep P-t-P|awk -F ':' '{ print $2 }'|awk -F ' ' '{ print $1 }')
tun_ptp=$(ifconfig $tun_iface|grep P-t-P|awk -F ':' '{ print $3 }'|awk -F ' ' '{ print $1 }')
tun_gw=$(echo $tun_inet |awk -F '.' '{print $1"."$2"."$3".1"}')
tun_net=$(echo $tun_inet |awk -F '.' '{print $1"."$2"."$3".0/24"}')
ip route add 0.0.0.0/1 via $tun_ptp dev $tun_iface table 10
ip route add 128.0.0.0/1 via $tun_ptp dev $tun_iface table 10
ip route add $tun_gw via $tun_ptp dev $tun_iface metric 1 table 10
ip rule add from $tun_net table 10
ip rule add to $tun_net table 10
for host in $vpndhosts;do ip rule add from $host table 10;done
for host in $vpndhosts;do for server in $dnsservers;do ip rule add from $host to $server lookup main;ip rule add from $server to $host lookup main;done;done

To add other hosts, vpndhosts is a space delimited list of local ip's.

Unfortunately, getting port-forwarding working from my chosen provider is another issue. Optware here I come.

To delete all custom rules and return to default (in my case at least):

for rule in $(ip rule list |grep -v "all lookup"|awk -F ":" '{ print $1 }');do ip rule delete pref $rule;done

So, add the two code sections above to /tmp/vpnrouteup.sh and /tmp/vpnroutedown.sh and chmod them via the Administration→Scripts→Init script:

cat > /tmp/vpnrouteup.sh <<EOF
#!/bin/sh 
vpndhosts="192.168.1.253"
dnsservers="89.150.129.10 89.150.129.22"
tun_iface=\$(ifconfig|grep tun|awk '{ print \$1 }')
tun_inet=\$(ifconfig \$tun_iface|grep P-t-P|awk -F ':' '{ print \$2 }'|awk -F ' ' '{ print \$1 }')
tun_ptp=\$(ifconfig \$tun_iface|grep P-t-P|awk -F ':' '{ print \$3 }'|awk -F ' ' '{ print \$1 }')
tun_gw=\$(echo \$tun_inet |awk -F '.' '{print \$1"."\$2"."\$3".1"}')
tun_net=\$(echo \$tun_inet |awk -F '.' '{print \$1"."\$2"."\$3".0/24"}')
ip route add 0.0.0.0/1 via \$tun_ptp dev \$tun_iface table 10
ip route add 128.0.0.0/1 via \$tun_ptp dev \$tun_iface table 10
ip route add \$tun_gw via \$tun_ptp dev \$tun_iface metric 1 table 10
ip rule add from \$tun_net table 10
ip rule add to \$tun_net table 10
for host in \$vpndhosts;do ip rule add from \$host table 10;done
for host in \$vpndhosts;do for server in \$dnsservers;do ip rule add from \$host to \$server lookup main;ip rule add from \$server to \$host lookup main;done;done
EOF

cat > /tmp/vpnroutedown.sh <<EOF
#!/bin/sh
for rule in \$(ip rule list |grep -v "all lookup"|awk -F ":" '{ print \$1 }');do ip rule delete pref \$rule;done
EOF

chmod 700 /tmp/vpnrouteup.sh
chmod 700 /tmp/vpnroutedown.sh

bzzt.

Here's the link with basic info.

Setup Optware via the very handy tutorial here.

Install curl via ipkg install libcurl.

Add the following to Administration→Scripts→Init:

#Setup port forwarding client-ID
head -n 100 /dev/urandom | md5sum | tr -d " -" > /tmp/pia_client_id

Setup a scheduled task that runs at least once per hour that contains:

pia_client_id=$(cat /tmp/pia_client_id)
pia_user=$(head -1 /tmp/pia.txt)
pia_pw=$(tail -1 /tmp/pia.txt)
tun_inet=$(ifconfig $tun_iface|grep P-t-P|awk -F ':' '{ print $2 }'|awk -F ' ' '{ print $1 }')

curl -d "user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" https://www.privateinternetaccess.com/vpninfo/port_forward_assignment >> /tmp/port_forward.txt

What's missing is the traffic needs to go out and back over the tunnel- ip rule add?

root@wifi1:/tmp/home/root# ip rule add to 75.52.9.107 table 10
root@wifi1:/tmp/home/root# ip rule add from 75.52.9.107 table 10
root@wifi1:/tmp/home/root# ip rule add to 209.222.18.222 table 10
root@wifi1:/tmp/home/root# ip rule add from 209.222.18.222 table 10
root@wifi1:/tmp/home/root# traceroute 209.222.18.222
traceroute to 209.222.18.222 (209.222.18.222), 30 hops max, 38 byte packets
 1  10.127.1.1 (10.127.1.1)  64.245 ms  64.217 ms  64.661 ms
 2  buc-ird-14sw.voxility.net (93.115.85.33)  64.751 ms  64.891 ms  64.757 ms
 3  buc-ird-02c.voxility.net (93.115.87.29)  64.335 ms  64.456 ms  75.212 ms
 4  buc-ird-01gw.voxility.net (109.163.234.53)  70.970 ms  buc-ird-04gw.voxility.net (195.60.77.13)  65.213 ms
root@wifi1:/tmp/home/root# curl -d "user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" https://www.privateinternetaccess.com/vpninfo/port_
forward_assignment 
curl: (35) error:0D0890A1:asn1 encoding routines:ASN1_verify:unknown message digest algorithm

Hmmmmmmmmmm. Apparently this means curl has an SSL issue.

Ran ipkg install wget-ssl, then /opt/bin/wget –post-data=“user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet” –no-check-certificate https://ww w.privateinternetaccess.com/vpninfo/port_forward_assignment and got:

2014-02-19 21:13:51 (633 KB/s) - `port_forward_assignment' saved [57]
root@wifi1:/tmp/home/root# more port_forward_assignment 
{"error":"port forwarding not available for this region"}
root@wifi1:/tmp/home/root# 

Looks like somehow my ip rule add's got dropped or aren't working? Indeed a traceroute shows traffic heading out the wrong interface.

Tomorrow…

I wrote down the wrong IP somehow and then had been copy & pasting that into my rules:

root@wifi1:/tmp/home/root# ip rule add to 75.52.9.107 lookup 10
root@wifi1:/tmp/home/root# ip rule add from 75.52.9.107 lookup 10
root@wifi1:/tmp/home/root# traceroute www.privateinternetaccess.com
traceroute to www.privateinternetaccess.com (72.52.9.107), 30 hops max, 38 byte packets
 1  89.150.184.1 (89.150.184.1)  22.198 ms  22.421 ms  22.421 ms
 2  te1-1-0.mr1.fb.dk.ip.fullrate.dk (90.185.4.138)  19.302 ms  19.245 ms  19.104 ms
 3  xe-7-3-0-0.boanqp7.dk.ip.tdc.net (188.180.68.81)  19.264 ms  xe-3-0-0-0.boanqp7.dk.ip.tdc.net (195.215.109.229)  19.227 ms  xe-7-3-0-0.boanqp7.dk.ip.tdc.net (188.180.68.81)  19.556 ms
 4  ae4-0.asd9nqp1.nl.ip.tdc.net (83.88.30.13)  38.615 ms  37.569 ms  38.085 ms
 5  amsix2.teleglobe.net (195.69.145.167)  38.603 ms  37.433 ms  37.635 ms
 6
root@wifi1:/tmp/home/root# ip rule add from 72.52.9.107 lookup 10
root@wifi1:/tmp/home/root# ip rule add to 72.52.9.107 lookup 10
root@wifi1:/tmp/home/root# traceroute www.privateinternetaccess.com
traceroute to www.privateinternetaccess.com (72.52.9.107), 30 hops max, 38 byte packets
 1  10.115.51.1 (10.115.51.1)  67.864 ms  117.204 ms  66.705 ms
 2  buc-ird-01c.voxility.net (93.114.43.193)  66.965 ms  67.535 ms  66.999 ms
 3  buc-ird-01gw.voxility.net (109.163.235.49)  67.726 ms  buc-ird-02gw.voxility.net (109.163.235.57)  123.000 ms  buc-ird-04gw.voxility.net (109.163.235.61)  67.645 ms
 4

Fixed. The return data now is simply blank:

root@wifi1:/tmp/home/root# /opt/bin/wget --post-data="user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" --no-check-certificate https://www.privateinternetaccess.com/vpninfo/port_forward_assignment
--2014-02-20 07:53:49--  https://www.privateinternetaccess.com/vpninfo/port_forward_assignment
Resolving www.privateinternetaccess.com (www.privateinternetaccess.com)... 72.52.9.107
Connecting to www.privateinternetaccess.com (www.privateinternetaccess.com)|72.52.9.107|:443... connected.
WARNING: cannot verify www.privateinternetaccess.com's certificate, issued by `/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `port_forward_assignment'

    [ <=>                                                                                                                                                                               ] 2           --.-K/s   in 0s      

2014-02-20 07:53:51 (23.3 KB/s) - `port_forward_assignment' saved [2]

root@wifi1:/tmp/home/root# more port_forward_assignment 
{}

Just a bit of duh I'm guessing, I was using variables in that command that this morning are all un-set:

root@wifi1:/tmp/home/root# pia_user=$(head -1 /tmp/pia.txt)
root@wifi1:/tmp/home/root# pia_pw=$(tail -1 /tmp/pia.txt)
root@wifi1:/tmp/home/root# pia_client_id=$(head -1 /tmp/pia_client_id)
root@wifi1:/tmp/home/root# tun_inet=$(ifconfig $tun_iface|grep P-t-P|awk -F ':' '{ print $2 }'|awk -F ' ' '{ print $1 }')
root@wifi1:/tmp/home/root# /opt/bin/wget --post-data="user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" --no-check-certificate -q -O - https://www.privateinternetaccess.com/vpninfo/port_forward_assignment
{"port":22172}root@wifi1:/tmp/home/root# 

Ta-da! So up next is setting a port forward rule in the router along with testing a connection to the LAN client port. I just realize that my “ip rule add from” may not be good enough to get unprompted incoming traffic through to the LAN client, probably need an “ip rule add to”. Then of course there's also the on-the-fly update to transmission on the NAS to tell it what the listening port should be, or can I just update the port forward rule with the external port set to the pre-configured internal listening port? Probably that.

And in this testing I find I can't really seem to connect to/from anything via the LAN ip. I get no route to host if there's no “ip rule to” rule in place, if I add that rule, ICMP stop working. Blech.

On a hunch, restarted everything, and with the default setup, and connections out from the LAN client worked just fine. I might have messed some rules up along the way.

root@wifi1:/tmp/home/root# ip rule add to 72.52.9.107 lookup 10
root@wifi1:/tmp/home/root# /opt/bin/wget --post-data="user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" --no-check-certificate -q -O - https://www.privateinternetaccess.com/vpninfo/port_forward_assignment
{"port":22172}

Okay, and confirmed via use of ipkg installed netcat on the router that the forwarded port itself does function, so it's probably just a matter of getting iptables w/ip route to do the right thing to pass connections on to the LAN client.

[root@ragingdouche ~]#  nc -v 37.221.165.196 22172
Connection to 37.221.165.196 22172 port [tcp/*] succeeded!
tet
^C

root@wifi1:/tmp/home/root# /opt/bin/nc -l -p 22172
tet
^C

Found some iptables possibilities here, http://strongvpn.com/forum/viewtopic.php?id=3174:

iptables -t nat -A PREROUTING -p <tcp OR udp> --dport <port you want to forward> -j DNAT --to-destination <IP you want to forward to>
iptables -A FORWARD -s <your TUN IP> -p <tcp OR udp> --dport <port you want to forward> -j ACCEPT

After playing with variations on those commands I don't get connection refused from the outside, it times out instead.

If I delete all the PREROUTING rules via iptables -t nat -D PREROUTING #rule I can at least get to the tun interface listening and responding via netcat:

root@wifi1:/tmp/home/root# /opt/bin/nc -v -l -p 22172
listening on [any] 22172 ...
connect to [10.106.1.6] from li245-253.members.linode.com [173.255.235.253] 56825
test

With http://www.dd-wrt.com/phpBB2/viewtopic.php?p=782506 - GOT IT!!!

root@wifi1:/tmp/home/root# iptables -t nat -I PREROUTING -p tcp -i tun11 --dport 22172 -j DNAT --to 192.168.33.253:1024
root@wifi1:/tmp/home/root# iptables -I FORWARD -p tcp -d 192.168.33.253 --dport 1024 -j ACCEPT

On the NAS:

-bash-3.2# while : ;do ./nc -v -l -p 1024;done
Connection from 173.255.235.253:60984
test

From an external host:

[root@ragingdouche ~]#  nc -v 37.221.165.196 22172
Connection to 37.221.165.196 22172 port [tcp/*] succeeded!
test

So- the vpnrouteup.sh script will need to set rules to access www.pia.com for port forward requests, and then the port forward script (/opt/etc/port_forward_update.sh) will need to get all the PIA client info together, get the IP of the www host, add rules to table 10 for that host, make the port forwarding request, retrieve the port, then run those last two iptables commands to forward the port to the NAS appropriately. Woot.

Current contents of port_forward_update.sh:

#!/bin/sh

#to-do: get current nat preroute rule and compare port, if it's the same do nothing, if it's different, delete the old and add the new

pia_user=$(head -1 /tmp/pia.txt)
pia_pw=$(tail -1 /tmp/pia.txt)
pia_client_id=$(head -1 /tmp/pia_client_id)
tun_inet=$(ifconfig $tun_iface|grep P-t-P|awk -F ':' '{ print $2 }'|awk -F ' ' '{ print $1 }')

#get our forwarding port from PIA
forwarded_port=$(/opt/bin/wget --post-data="user=$pia_user&pass=$pia_pw&client_id=$pia_client_id&local_ip=$tun_inet" \
	--no-check-certificate \
	-q -O - https://www.privateinternetaccess.com/vpninfo/port_forward_assignment \
	| awk -F ':' '{ print $2 }'| awk -F '}' '{ print $1 }')

echo $forwarded_port
#set forwarding to our NAS host
#iptables -t nat -I PREROUTING -p tcp -i tun11 --dport $forwarded_port -j DNAT --to 192.168.33.253:1024
#iptables -I FORWARD -p tcp -d 192.168.33.253 --dport 1024 -j ACCEPT

Made some more changes, but found that transmission wants to report it's peer-port despite the router port forwarding to a different outside port. Boo. That can be mitigated by adding an update script to the port-forward script to ssh to the nas box, use sed to update the json settings file, then killall -HUP transmission-daemon. That's a to-do. Also have yet to find out if that will kill our open file handles on files that have been moved from the dl directory by sickbeard. Confirmed, open handles stay open through a -HUP.

Last step is creating the script to do the HUP. Key created with dropbearkey -t rsa -f ~/.ssh/id_rsa, ssh-keygen does not exist in busybox.

Here's what I came up with:

root@wifi1:/opt/etc/scripts# more transmission_port_update.sh 
#!/bin/sh
echo $1 to $2
ssh -i /opt/etc/pia/id_rsa root@nas.benhall.com cp /etc/transmission-daemon/settings.json /etc/transmission-daemon/settings.json.bak
ssh -i /opt/etc/pia/id_rsa root@nas.benhall.com "cat /etc/transmission-daemon/settings.json.bak | sed -e s#\:\ $1,#\:\ $2,#g > /etc/transmission-daemon/settings.json"
ssh -i /opt/etc/pia/id_rsa root@nas.benhall.com killall -HUP transmission-daemon