MPLS Implementing L3 VPN’s

The goal of this document is to show the implementation steps of MPLS L3 VPNs on Mikrotik Router OS. For this we are going to be using OSPF within our service provider network as an IGP and OSPF as our CE-PE protocol .
We will be using the following topology:


(Click image to enlarge)

Routers PR1 through PR4 are in the service provider network with PR1 and PR3 as Provider Edge Routers (PE) and CR1 and CR2 as Customer Edge Routers (CE).

First setup basic IP addressing as per topology:

Next is to setup normal OSPF within the service provider network:

Note the above OSPF configuration assumes that the default Mikrotik OSPF settings below are on the Routers:

Next is to make sure our routing table in our service provider network looks correct:

Next is to setup MPLS within our service provider network. Note that we only setup MPLS on PR router interfaces facing other PR routers I.E. there is no MPLS running on the interfaces facing the CR routers.

Now we must confirm that our service provider MPLS network looks correct:

So at this stage we have setup our service provider MPLS cloud.

Our goal is to allow our customer edge routers to peer with our provider edge routers. Our customer edge routers should only see routes from other customer routers. They should be completely oblivious to anything within the service provider network. Our provider edge routers will have to maintain two routing tables. One routing table for customer routes and another for its operation within the service provider network. Our provider edge routers will also have to run two instances of OSPF. One for connecting to our customer and another for LDP within the service provider network.

To maintain scalability our service provider routers PR2 and PR4 should also not have any routes from our customer network within its routing table.

From our customer perspective our network should look like this:

mplsCustPer(Click image to enlarge)

Our provider MPLS network is hidden and just appears as one router from the perspective of the routing tables within our customer routers.

On our provider edge routers we must setup Virtual Router Forwarding instances to maintain a separate routing table for all customer routes advertised to them.

We also define a route distinguisher and import and export route targets. The route distinguisher can be any arbitrary value. It is used to maintain uniqueness in our routing tables. You could have many customers all using the same private IP address space and the route distinguisher allows the provider edge routers to make each route unique even if the same route is used by multiple different customers. There are various best practices associated with picking a route distinguisher for a customer. In this case I have just used the arbitrary value 666:777. For simplicity I have used the same value for our import and export targets.

From the above configuration you can see that all routes learned from our customer are going to have the routing mark ‘CustomerRoutes’ all of our other routes from within our service provider network will have no routing mark.

Next we must setup an IBGP session between our two provider edge routers. We use this IBGP (not standard BGP but MBGP [multiprotocol-BGP]) session to redistribute routes learned from our customer from OSPF then into BGP then across to our other provider edge router then back into OSPF again for our customer. (See notes on BGP at the end)



(Click image to enlarge)

For this configuration we are just using the private ASN 65530

We are also redistributing any routes from OSPF that also have the routing mark ‘CustomerRoutes’

Note the above configuration assumes that the provider edge routers have the default configuration:

Now we must confirm our IBGP peering between PR1 and PR3:

Next we must setup our separate instance of OSPF on our provider edge routers facing our customer edge routers.

We also redistribute and routes from BGP into this OSPF instance and assign the routes the routing mark of ‘CustomerRoutes’

Now for the OSPF on our customer routers using bog standard OSPF:

So that is our implementation complete!

We run a ping from CR2 to CR1 to see if we are successful.

You can see for the routing table of CR2 it is only aware of the route advertised by CR1

A traceroute from CR2 to CR1:

Look at the second hop. What’s going on here? This is down to the provider routers decrementing the TTL field of the IP header of the packet, and the provider router (in this case PR2 or PR4) doesn’t have a path back into the customer’s network so we have this unknown hop. So this does not look ideal from the customer’s perspective. To solve this on all of our PR routers we run:

This tells the provider router not to decrement the TTL field of the inner IP header.

We run the traceroute again:

Now the provider network is completely invisible to the customer routers.

Routing table of CR1. Showing only the route advertised by CR2:


Routing table of PR1 showing both routing tables the main routing table and a separate routing table for the customer’s routes:


OSPF routes from PR1 showing both instances of OSPF:

OSPFRoutesPR1Problems with this implementation

This implementation is extremely basic with only one customer with two sites. Once you start to scale up to more than 3 or 4 sites the IBGP configuration starts to become much more complex. Each provider edge router that the customer is connected to must establish an IBGP session with every other provider edge router that the customer is connected to in a full mesh configuration. Needless to say the administrative over head of this becomes prohibitively complex once you have more than 3 or 4 sites.

The solution to this is to use BGP route reflectors. Have all your provider edge routers peer with IBGP to your route reflector rather than directly with each other. This vastly simplifies configuration. Of course having only one route reflector introduces a common point of failure. So you should have one main route reflector and at least one backup one. Then you should have each provider edge router peer with your primary route reflector and your backup one.

Posted in Network Engineering