MPLS: Creating an Internet Off-ramp

Yes, MPLS circuits are still being used by companies out there. Shocking! Well, actually it’s not. Even though there continues to be growth in low-cost broadband connections, some companies still utilize MPLS circuits back to their data centers from their branches around the world. Companies continue to use these circuits to connect their sites to each other. The reasons for this varies from company to company. Perhaps it is security, stability or even just a long provider contract that keeps them out there. Either way, MPLS will still be in play for some time. Many companies have not fully embraced the cloud. They host important applications in-house in a data center. Some companies funnel their branch traffic through their data center as its heads out to the internet. This has its benefits. You might only need centralized firewalls or other appliances since all traffic exits via the same egress. Companies still do this as it does save money; however as the adoption of cloud grows, this method starts to see issues.

A company that funnels their world-wide users through a centralized data center and also subscribes to cloud services will run into issues at some point. If you are a global company, lets say in the US, users in Asia already have to face a bit of latency to reach your US-based services. That continues to grow if they have to go out to the internet through the US to a website that is actually hosted back in Asia. Remote Branches might also run into geo-locked websites. A certain website might only be accessible through IP addresses from that country. Some cloud apps give recommendations on latency for best-use of its applications. Instead of back-hauling internet traffic to another country, it is usually best to use a local provider to get you to the internet much faster. Companies like Microsoft, Apple, Google, Amazon have data centers all over the world. This helps users reach their services instantly, which will be much faster than sending the request around the world.

So what can be done about balancing an MPLS connection as well as a local off-ramp for internet services? There appears to be a plethora of SD-WAN solutions out on the market now. A Software Defined Wide Area Network or SD-WAN allows company branches to reach each other, applications and other services in an intelligent manner. Multiple links can be utilized to load balance your traffic to destinations. SD-WAN appliances will see issues on a link and can dynamically move the traffic over another link. SD-WANs support multiple connection types, like an MPLS, broadband and even wireless communications. This all sounds great, so why not jump on something like SD-WAN right away? Perhaps the company does not have the budget to add the SD-WAN appliances to their sites. Some solutions would like to replace your branch WAN router with an appliance in the long-run. You might be running some voice services like a PRI on these routers, so you still need to utilize them in some fashion. You might be busy with so many other projects and simply do not have time for a major change to connectivity. Either way, you still want to take advantage of a local link for internet services and still keep your MPLS as the path to the data center until those doors open.

Scenario: Utilizing a local connection to the internet as well as an MPLS circuit to your other branch sites and data center. It must also be a redundant solution.
Devices: Cisco Routers, Cisco Switch, Palo Alto firewall.

The above image is a GNS3 lab I threw together accompanied with some of my real lab firewall configs, but represents the common scenario seen in different companies where internet traffic is back-hauled to the remote data center, wherever it is in the world.
Usually, the WAN01 and the Core01 would be the only devices at a small branch. Since all the traffic heads back to the data center, the firewalls and other services might be centralized there.
The data center and all other branches would connect to a provider’s MPLS using BGP as the routing protocol of choice. It is not the only choice, but usually the most common. The data center would provide a default route via BGP to bring back all traffic from the branches.

At this point, all works well for a company until local branch connectivity is needed or even required.
Here we would introduce a local firewall. Security should always be at the top of the list for a company, yet sometimes it is not. Budgeting everything a company needs in the IT world is not possible sometimes. Having a local connection to the internet from an enterprise does require at least this layer of security. The local connection to the provider would terminate on the firewall as shown in the network diagram above. Outside the scope of this entry are the finer details of the firewall config, such as needing rules and NAT. Now how do route our users to the internet through the local connection and continue to have access to services across the MPLS?
This is where we utilize our Layer 3 core switch a bit more. The L3 core switch is just that, the center of the network. All gateways reside on it and so will the path to the firewall. The core switch also has a default static route pointing up towards the WAN router. This default does not have the normal Administrative Distance. The AD here is 220. This will be explained in a bit.

When users head out to the internet, it will be through the core and out through the firewall. How? The magic that is routing will help us here. OSPF would be the choice between the WAN router, the core switch and the firewall. You might have a different choice for your favorite routing protocol, but because I am using a Palo Alto firewall, EIGRP is not an option. The Palo Alto would provide another default route, this one out to the local provider. Palo Alto also gives me a Path Monitoring feature that allows me to ping one or more destinations. This Path Monitoring feature is enabled on our new default route. The destinations to choose should be destinations you know will not change. These destinations should also be IPs that will hopefully not be throttled. In my example below, I used Google’s DNS as well as Cisco’s Umbrella DNS. Keep in mind, these are not the best destinations to choose as someday they might be throttled. Many people say the Internet Root DNS servers are good ones to use. I would love to hear feed back on what you might use. My default route is set to remove itself from the Palo Alto routing table whenever both destinations disappear. This ensures the default route is not removed just because a single destination is not reachable.

My example above does not show a public source IP, but the source of the pings would be the public IP your provider has given you.
As this default route has a smaller Administrative Distance than my original Default Route, all traffic heads out in the direction of the firewall now:

Well, that ensured my internet browsing, Skype traffic and all other cloud services are using my local connection, but that pretty much broke everything out to the MPLS; in other words, my data center. In production, the next section would occur prior to the “going live” with the local internet. I still need a way to ensure my branch can reach my other branches and the data center/s. The easiest way might not be a way many are fans of, but it works for me. At my WAN router, I would redistribute BGP into OSPF. Now, before you start throwing tomatoes, this is would apply to a small environment. My core switch would also not be something I purchased from the local convenience store. A 2960XR would be just fine with this occurring. Keep in mind, I am not redistributing the internet into my core switch, the routes the WAN router can get to are simply routes on the MPLS: other branches and the data center. A couple hundred routes from several sites would not affect the switch or the firewall for that matter. This would allow the branch to continue reaching anything else out in the MPLS.

As with any major change, I hope you have some CPU baselines. Always compare the before and after.
So now, I maintain a connection to my subnets in the rest of the enterprise, as well as utilize my local link for internet connectivity. If my local provider runs into issues and both of the routes Path Monitoring is observing fail, the internet would still be reachable via the MPLS. This will not save you if the local connection is reachable, but perhaps a bit slow and unstable. That is where you might need to utilize an SD-WAN solution. There are probably some IP-SLA tricks we can add into this as well, but it also continues to complicate the solution. If you are still not a fan of BGP redistribution, there are other ways to accomplish this. Additional static routes can be created and Cisco’s EEM can be utilized to add/remove entries. I’ve had to utilize Cisco EEM for that purpose since PPPoE is not a valid connection option that works with Palo Alto’s Path Monitoring feature. I am sure I’ll have a separate blog entry for EEM. We also have not discussed redundancy for that MPLS connection. Here I would utilize a VPN through the local link. I’ll save that one for later as well.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at

Up ↑

%d bloggers like this: