EdgeRouter: Routing through OpenVPN and IPSEC with OSPF

By | May 9, 2016

Open Shortest Path First (OSPF) is a routing protocol for Internet Protocol (IP) networks. Because I have site-to-site OpenVPN and IPSEC tunnels between my primary and remote sites I ended up defining lots of static routes. The environment started to be complex enough to justify the use of OSPF that is supported by EdgeRouter.

The current setup of my environment looks like:

routers

Primary Site:

ER-8 (with load-balancing WAN1 and WAN 2):
– WAN 1: eth0 linked to the ISP 1 through a Hitron cable modem in bridge mode. Thus the ER-8 gets the IP from the ISP.
– WAN 2: eth1 not linked.
– LAN 7: eth7 to internal LAN 2
– LAN 11: eth2 internal LAN 11

D-Link DWR-921 LTE:
– WAN 1: LTE link to Mobile service ISP.

EdgeRouter POE:
– WAN1: etho, IP = 192.168.7.10 linked to EdgeRouter ER-8 eth7 with gateway 192.168.7.1
– WAN2: eth1, IP = 192.168.0.50 link to D-Link DWR-921 LTE eth4 with gateway 192.168.0.1
– LAN 2: switch0, all the internal LAN

Remote Site:

UPC Cable Modem:
– WAN 1: eth0 linked to the ISP 1

EdgeRouter Lite:
– WAN 1: eth0, link to UPC Cable Modem eth1 with gateway 192.168.0.1
– LAN 9: eth1, local service LAN
– LAN 10: eth2, local management LAN

OpenVPN site-to-site between primary site (EdgeRouter POE) and remote site (EdgeRouter Lite). The tunnel is used to communicate between networks 192.168.2.0/24 from primary site and 192.168.9.0/24 from remote site.

IPSEC site-to-site with vti between primary site (ER-8) and remote site (EdgeRouter Lite).

By using IPSEC site-to-site with vti see https://blog.voina.org/edgerouter-ipsec-site-to-site-with-virtual-tunnel-interface/ I have the advantage of being able to tret the IPSEC associated virtual interface as any other routable interface, so OSPF can be applied for IPSEC also.

By using OSPF I am able to get rid of all the static routes necessary to be able to route through OpenVPN and IPSEC tunnels. This will make it very easy to add new local network on primary and remote sites without the need to add additional static routes.

Note that OSPF works by discovering routes from the neighbour. In the digram of the network we can see we have a loop made of the three EdgeRouter routers. Very important in this loop is the direction in which the OSPF will discover routes.

(EdgeRouter POE)(eth0) —uplink—> (eth7)(ER-8)(vti0) —IPSEC—> (vti0)(EdgeRouter Lite)(vtun0) —OpenVPN—> (vtun0)(EdgeRouter POE)

I am choosing the above direction because I want some networks to be routed through IPSEC connection that is much faster than the OpenVPN. Because OpenVPN is not using any hardware acceleration on EdgeRouter has much lower bandwidth than IPSEC.

The configuration will be done in the same order.

STEP 1: Configure OSPF on EdgeRouter POE

STEP 1.1: Allow OSPF traffic through the firewall

Add a new allow rule to the firewall for OSPF protocol:

name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 1 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 2 {
            action accept
            description "Allow icmp"
            log disable
            protocol icmp
        }
        rule 3 {
            action accept
            description "Allow ospf"
            log disable
            protocol ospf
        }
        rule 4 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }

STEP 1.2: Modify interfaces for OSPF

We have to declare in each interface definition that will be involved with OSPF, the OSPF parameters. Note that currently we will not use vtun0 for OSPF as is not necessary to “close” the loop. To route directly through the OpenVPN tunnel we will use static routes. See the static route used to force going to 192.168.9.0/24 from the remote site.

...
static {
        interface-route 192.168.9.0/24 {
            next-hop-interface vtun0 {
            }
        }
....
}

So we add to the eth0 interface the ospf routing parameters:

interfaces {
    ethernet eth0 {
        address 192.168.7.10/24
        description Internet
        dhcp-options {
            default-route update
            default-route-distance 210
            name-server no-update
        }
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        ip {
            ospf {
                dead-interval 40
                hello-interval 10
                priority 1
                retransmit-interval 5
                transmit-delay 1
            }
        }
        poe {
            output off
        }
        speed auto
    }
...

STEP 1.3: Advertise networks for ospf

Under protocols add an ospf section:

protocols {
    ospf {
        area 0.0.0.0 {
            network 192.168.2.0/24
            network 192.168.7.0/24
        }
        parameters {
            abr-type cisco
            router-id 192.168.7.10
        }
    }
...
}

Note:
1. network 192.168.2.0/24 = with this we advertise a network for ospf. You can add any network that is connected to the router.
2. router-id 192.168.7.10 = this is important to be the router IP that is linked to the next neighbour. The IP of the immediate neighbour will be an IP from the same network (in our case 192.168.7.1 the IP on eth7 on the ER-8).

STEP 1.4: Commit changes
After we commit the changes osfp daemon will be automatically started and the router will start route discovery. Nothing will happen yet as there is no active neighbour.

STEP 2: Configure OSPF on ER-8

STEP 2.1: Allow ospf traffic through the firewall

Add a new allow rule to the firewall for OSPF protocol:

name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 1 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 2 {
            action accept
            description "Allow IPSEC ESP"
            log disable
            protocol esp
        }
        rule 3 {
            action accept
            description "Allow IPSEC IKE"
            destination {
                port 500
            }
            log disable
            protocol udp
        }
        rule 4 {
            action accept
            description "Allow IPSEC NAT-T"
            destination {
                port 4500
            }
            log disable
            protocol udp
        }
        rule 5 {
            action accept
            description "Allow icmp"
            log disable
            protocol icmp
        }
        rule 6 {
            action accept
            description "Allow ospf"
            log disable
            protocol ospf
        }
        rule 7 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }

STEP 2.2: Modify interfaces for OSPF

We have to declare in each interface definition that will be involved with ospf, the ospf parameters. This is the middle device so we have two interfaces that advertise for ospf: eth7 and vti0

So we add to the eth7 and vti0 interface the ospf routing parameters:
Note that vti0 has a special setting: “network point-to-point” to indicate that this is a special kind of OSPF.

interfaces {
...
       ethernet eth7 {
        address 192.168.7.1/24
        description Internal
        duplex auto
        ip {
            ospf {
                dead-interval 40
                hello-interval 10
                priority 1
                retransmit-interval 5
                transmit-delay 1
            }
        }
        speed auto
    }
...
    vti vti0 {
        address 40.0.0.1/30
        ip {
            ospf {
                dead-interval 40
                hello-interval 10
                network point-to-point
                priority 1
                retransmit-interval 5
                transmit-delay 1
            }
        }
        mtu 1436
    }
...

STEP 2.3: Advertise networks for OSPF

Under protocols add an ospf section:

protocols {
    ospf {
        area 0.0.0.0 {
            network 40.0.0.0/30
            network 192.168.7.0/24
            network 192.168.11.0/24
        }
        parameters {
            abr-type cisco
            router-id 192.168.7.1
        }
    }
...
}

Note:
1. network 192.168.7.0/24 = with this we advertise a network for ospf. You can add any network that is connected to the router. Note that we must add the vti0 network 40.0.0.0/30
2. router-id 192.168.7.1 = this is important to be the router IP that is linked to the previous neighbour. The IP of the immediate neighbour will be an IP from the same network (in our case 192.168.7.10 the IP on eth0 on the EdgeRouter-POE).
3. because the vti0 interface is declared as having ospf as “network point-to-point” we are able to see the next neighbour directly through this interface. In this case it does not matter that we have “router-id 192.168.7.1”.

STEP 2.4: Commit changes
After we commit the changes OSPF daemon will be automatically started and the router will start route discovery.

STEP 3: Configure OSPF on ER-Lite

STEP 3.1: Allow ospf traffic through the firewall

Add a new allow rule to the firewall for OSPF protocol:

name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 1 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 2 {
            action accept
            description "Allow IPSEC ESP"
            log disable
            protocol esp
        }
        rule 3 {
            action accept
            description "Allow IPSEC IKE"
            destination {
                port 500
            }
            log disable
            protocol udp
        }
        rule 4 {
            action accept
            description "Allow IPSEC NAT-T"
            destination {
                port 4500
            }
            log disable
            protocol udp
        }
        rule 5 {
            action accept
            description "Allow icmp"
            log disable
            protocol icmp
        }
        rule 6 {
            action accept
            description "Allow ospf"
            log disable
            protocol ospf
        }
        rule 7 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }

STEP 3.2: Modify interfaces for OSPF

We have to declare in each interface definition that will be involved with ospf, the ospf parameters. This is the last device so we have one interface that advertises for ospf: vti0

So we add to the vti0 interface the ospf routing parameters:
Note that vti0 has a special setting: “network point-to-point” to indicate that this is a special kind of OSPF.

interfaces {
...
    vti vti0 {
        address 40.0.0.2/30
        ip {
            ospf {
                dead-interval 40
                hello-interval 10
                network point-to-point
                priority 1
                retransmit-interval 5
                transmit-delay 1
            }
        }
        mtu 1436
    }
...

STEP 3.3: Advertise networks for OSPF

Under protocols add an ospf section:

protocols {
    ospf {
        area 0.0.0.0 {
            network 40.0.0.0/30
            network 192.168.9.0/24
            network 192.168.10.0/24
        }
        parameters {
            abr-type cisco
            router-id yy.yy.yy.yy
        }
    }
    static {
        interface-route 192.168.2.0/24 {
            next-hop-interface vtun0 {
            }
        }
    }
}
}

Note:
1. network 192.168.9.0/24 = with this we advertise a network for ospf. You can add any network that is connected to the router. We must add the vti0 network 40.0.0.0/30
2. router-id yy.yy.yy.yy = this is the external WAN IP or the WAN IP. Because our immediate neighbour is the neighbour through the vti0 interface this router-id is not used to detect the neighbour. Because the vti0 interface is declared as having ospf as “network point-to-point” we are able to see the next neighbour directly through this interface.

STEP 3.4: Commit changes
After we commit the changes OSPF daemon will be automatically started and the router will start route discovery.

STEP 4:
Because the OSPF daemons were started in the order the configurations were done not everything was discovered. Let’s clear the ip tables and force discovery.
Execute on all the routers:

clear ip ospf process

Other useful ospf commands:

Show IP information for OSPF:

show ip ospf

Show IP database for OSPF network:

show ip ospf database
show ip ospf neighbor

Show OSPF interface :

show ip ospf interface eth0
show ip ospf interface vti0

STEP 5: List the discovered routes on all routers

Note that there are a lot of routes discovered across networks and routers. There are even multiple routes for different networks from the remote site, through vtun0 and also through vti0.
The distance of OSPF detected routes is 110 so lower than static defined routes. This means that we can overwrite at any time an OSPF route with a static one.

Router 1: EdgeRouter POE

ubnt@ubnt:~$ show ip route
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       > - selected route, * - FIB route, p - stale info
IP Route Table for VRF "default"
S    *> 0.0.0.0/0 [1/0] via 192.168.7.1, eth0
S       0.0.0.0/0 [200/0] via 192.168.0.1, eth1
C    *> 10.99.99.1/32 is directly connected, vtun0
C    *> 10.99.99.2/32 is directly connected, vtun0
O    *> 40.0.0.0/30 [110/11] via 192.168.7.1, eth0, 21:23:23
C    *> 127.0.0.0/8 is directly connected, lo
C    *> 192.168.0.0/24 is directly connected, eth1
C    *> 192.168.2.0/24 is directly connected, switch0
C    *> 192.168.7.0/24 is directly connected, eth0
S    *> 192.168.9.0/24 [1/0] is directly connected, vtun0
O       192.168.9.0/24 [110/12] via 192.168.7.1, eth0, 21:23:23
O    *> 192.168.10.0/24 [110/12] via 192.168.7.1, eth0, 21:23:23
O    *> 192.168.11.0/24 [110/11] via 192.168.7.1, eth0, 21:23:23

Router 2: ER-8

ubnt@ubnt:~$ show ip route
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       > - selected route, * - FIB route, p - stale info
IP Route Table for VRF "default"
S    *> 0.0.0.0/0 [210/0] via 78.97.202.1, eth0
C    *> 40.0.0.0/30 is directly connected, vti0
C    *> xx.xx.xx.xx/23 is directly connected, eth0
C    *> 127.0.0.0/8 is directly connected, lo
O    *> 192.168.2.0/24 [110/11] via 192.168.7.10, eth7, 21:20:28
C    *> 192.168.7.0/24 is directly connected, eth7
O    *> 192.168.9.0/24 [110/11] via 40.0.0.2, vti0, 21:20:48
O    *> 192.168.10.0/24 [110/11] via 40.0.0.2, vti0, 21:20:48
C    *> 192.168.11.0/24 is directly connected, eth2

Router 3: EdgeRouter Lite

ubnt@ubnt:~$ show ip route
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       > - selected route, * - FIB route, p - stale info
IP Route Table for VRF "default"
S    *> 0.0.0.0/0 [210/0] via 192.168.0.1, eth0
S       0.0.0.0/0 [1/0] via 10.0.0.1 inactive
C    *> 10.99.99.1/32 is directly connected, vtun0
C    *> 10.99.99.2/32 is directly connected, vtun0
C    *> 40.0.0.0/30 is directly connected, vti0
C    *> 127.0.0.0/8 is directly connected, lo
C    *> 192.168.0.0/24 is directly connected, eth0
S    *> 192.168.2.0/24 [1/0] is directly connected, vtun0
O       192.168.2.0/24 [110/21] via 40.0.0.1, vti0 inactive, 21:25:10
O    *> 192.168.7.0/24 [110/20] via 40.0.0.1, vti0, 21:25:35
C    *> 192.168.9.0/24 is directly connected, eth1
C    *> 192.168.10.0/24 is directly connected, eth2
O    *> 192.168.11.0/24 [110/20] via 40.0.0.1, vti0, 21:25:35

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.