Thursday, June 16, 2016

Cisco Routers: Easy Hair-Pin NAT for Internal Guest Network

Hey all! 

Recently I've been pouring myself into one particular configuration issue that is remarkably hard to solve on Cisco's IOS platform: Hairpin NAT. 

I've been tasked with designing a complete architecture for almost 20 sites. They want DMZs, Guest networks, lots of static and complex NATs, the works. And guess how many ASAs I can use? None

Which isn't that big of a deal, right? IOS has caught up to most security features over the past 5 or so years. NAT NVI (Nat Virtual Interface) can handle even complex NATs, ZBF (Zone-Based Firewall) is a nuanced and fantastic way to handle access control, etc. Hell, you can even build AnyConnect on IOS these days! 

But there's one problem that is intractable on Cisco's IOS platform: Hairpin-NAT

Because this problem is called so many things, let's define it. It's the issue that is seen when an internal host wants to access an internal server using the server's public (NAT'd) IP. Usually you don't see this with internal hosts, but the issue frequently comes up in designs when there is a "guest" network. Usually the guest network uses the same network infrastructure and internet connection as your other internal servers, but you use public DNS to make network segmentation clean and prevent as much internal access as possible. 
Picture from ServerFault
On an ASA, this problem is easy to solve - in fact, it's a single command.

same-security-traffic permit intra-interface

Surely it's this easy to solve on a router?

I tried to solve the problem in a half-dozen ways, and each has a "gotcha" that prevents it being useful in production. 

Potential Solution #1: Nat Virtual Interface (NVI) NAT'ing

This has the benefit of immediate results. Turn on NVI NAT'ing, modify the NAT statements and "ip nat enable" on a few interfaces, and you're good, all the guest network can use public IPs to access the DMZ servers. This technique is widely recommended as THE solution to this problem on the Internet. However, when there is more than an "inside" and "outside" interface, this type of NAT'ing gets very complex. 

As Alex points out in the comments below, you can make this work by creating exemptions in your NAT traffic selection. However, the complexity of this solution makes it less than ideal. 

Major drawback: Complexity with >2 interfaces and WAN NATing, high router utilization as NVI traffic is punted to the CPU for most ISR platforms, all NATs would have to be rewritten in the new style. 

Verdict: This is a no-go.

Potential Solution #2: Tons of Route-Maps

A widely cited solution from Tassos (who is a gosh-darn genius, and you should read his blog) nearly 8 years ago is to use regular NAT, but use route-maps to push traffic through an "ip nat outside" loopback address with policy routing on several interfaces. 

Major drawback: Super complex, high router utilization doing policy routing on several/all interfaces, I was not smart enough to get it working in a lab. 

Verdict: Not going to fly in a real production network (if I could even build it)

Potential Solution #3: Put the guest network on a separate internet connection

This is the solution often employed by Cisco shops that have become frustrated with the complexity and lack of manageability of Guest networks on the inside of the network. Shops either policy route the guest network out the other internet connection on their firewall or segment the network with a separate SSID/switch for guest that isn't connected to the internet network. 

Major drawback: As a consultant, I can't in good conscience ask the client to turn up 20 new consumer-class Internet circuits because I can't figure this issue out. 

VerdictDammit, I'm a good network engineer, I should be able to do this!

Potential Solution #4: Mix Up Some NATs

At this point I'm pulling my hair out, and really starting to believe that this just isn't possible, an answer that is widely circulated on the internet. I really just wish I could put the "guest" on the outside of my network. So I did. And dammit, it felt good. But of course, with both the "WAN" and "guest" in the same "ip nat outside" zone, there's no overload NAT to give the poor guys some Internets. 

And then I think of a funny joke. I wonder if I could use BOTH zoned NAT (ip nat inside/ip nat outside) AND stateless NAT (NVI) at the same time. I could just turn on stateless NAT on the guest and WAN zones, then write a single NAT statement to give it Internet access. I mean... there's no way Cisco would let you run both types of NAT at the same time, but I'm at wit's end. And low and behold, it flipping works. 

Major drawback: Configuration is slightly unorthodox. Using both NAT types looks funny, and someone might absent-mindedly 'fix' this in the future. Cisco might remove the older-style inside/outside NAT in future releases. 

VerdictSolution! (Short of Cisco actually supporting hairpin-nat, this'll do) 

Final Configuration

! Inside Host (vlan 1)
int eth 0/0
 ip add 192.168.1.50 255.255.255.0
ip route 0.0.0.0 0.0.0.0 192.168.1.1

! Guest Host (vlan 10)
int eth 0/0
 ip add 192.168.10.50 255.255.255.0
ip route 0.0.0.0 0.0.0.0 192.168.50.1

! DMZ Host (vlan 99)
int eth 0/0
 ip add 192.168.99.50 255.255.255.0
ip route 0.0.0.0 0.0.0.0 192.168.99.50

! Edge Router with router on a stick

! Interface configuration
interface Ethernet1/0.1
 encapsulation dot1Q 1 native
 ip address 192.168.1.1 255.255.255.0
 ip nat inside  <-- Most interfaces remain 'ip nat inside,' and NAT statements don't have to be rewritten

interface Ethernet1/0.10
 description GUEST
 encapsulation dot1Q 10
 ip address 192.168.10.1 255.255.255.0
 ip nat outside  <-- Note that 'guest' SVI is in outside zone. This enable all existing zoned-NATs to behave properly, without any config rewrites. So 'guest' users can access DMZ hosts using their public addresses just fine. 
 ip nat enable  <-- Stateless NAT also created on this interface

interface Ethernet1/0.99
 description DMZ
 encapsulation dot1Q 99
 ip address 192.168.99.1 255.255.255.0
 ip nat inside

interface Ethernet1/1
 description WAN
 ip address 20.0.0.2 255.255.255.0
 ip nat outside
 ip nat enable

! Route
ip route 0.0.0.0 0.0.0.0 20.0.0.1

! Object-groups and ACL

object-group network LocalGuest  <-- Highly recommend using object-groups to minimize configuration and simplify
 192.168.10.0 255.255.255.0
object-group network RFC1918Private 
 10.0.0.0 255.0.0.0
 172.16.0.0 255.240.0.0
 192.168.0.0 255.255.0.0

ip access-list extended Guest_2_WAN
 permit ip object-group LocalGuest any

ip access-list extended privateToPublic
 permit ip object-group RFC1918Private any

ip access-list extended siteToSite
 permit ip object-group RFC1918Private object-group RFC1918Private

route-map natOverload deny 10
 match ip address siteToSite
route-map natOverload permit 20
 match ip address privateToPublic

! NAT Configuration

ip nat inside source route-map natOverload interface Ethernet1/1 overload  <-- Regular inside/outside overload NAT for internet traffic

ip nat source list Guest_2_WAN interface Ethernet1/1 overload  <-- The new "NVI" style of NAT. Note the lack of "inside" in this command. Simply provides internet access to "guest" network segment. 

ip nat inside source static 192.168.99.50 20.0.0.50 extendable  <-- Hosts can still be exposed to the internet with regular "ip nat inside source static" commands. 

Lock it Down: ZBFW and Reflexive ACLs

I initially attempted to use ZBFW rules to lock down traffic from WAN-->Self and Self-->WAN, which works perfectly for zoned NAT (inside/outside). However, with NVI-style NAT, return internet traffic is evaluated by the edge router as destined for the edge router itself. Even when the outbound traffic is inspect on leaving this behavior occurs. 

But in engineering, there is always more than one way to skin that darn cat. So let's build some reflexive ACLs to permit return traffic. 

! Configuration: 


ip access-list extended In_2_Out
 permit ip any any reflect StatefulInbound  <-- Allow all traffic outbound, and remember it (default 300 seconds)

ip access-list extended Out_2_In
 evaluate StatefulInbound  <-- Allow all traffic that exited to return, using the stateful ACL table. 
 permit ip any host 20.0.0.50  <-- Permit inbound traffic for this particular DMZ services host


interface Ethernet1/1
 ip access-group Out_2_In in
 ip access-group In_2_Out out

In fact, you can still use ZBFW for most traffic, just make sure to permit it past the interface ACL so it can be picked up by NAT and then ZBF and delivered to internal hosts. 

Pro tip: If you're using this reflexive interface ACL as well as ZBF, remember your order of operations. The interface ACL is evaluated pre-NAT (so inbound traffic destinations will be public IPs), and ZBF is evaluted post-NAT (so inbound traffic dests will be your private IPs)

Download the GNS3 and Do It Yourself

There's nothing better for learning than building the thing yourself. Here's a completed GNS3 file with all features deployed. Please download and play with it yourself!

Files are here: 
https://1drv.ms/f/s!AliOPzHSO-GngrhpfKDb-YNfu1uLXQ

Good luck out there. 
Kyler

Friday, April 15, 2016

GNS3: Redundant Phase 2 (P2P) DMVPN with Certificate Authentication

I was recently tasked with converting a VPN WAN mesh with 19 nodes on ASAs over to routers. The client initially wanted statically defined VPNs, similar to what they were used to in the ASA scheme. However, that is a lot of keys and configurations. Thankfully, we were able to convince them to allow us to build a DMVPN mesh to avoid all of that static assignment and just let the protocol sort it all out.

As an aside, the calculation to find out how many connections there would be in a VPN mesh is n*(n-1)/2. For 19, that means we were nearly tasked with manually configuring 171 static profiles. Not something I’d relish.

Anyway, the client’s security team asked us to investigate building the dmvpn mesh with certificates to authenticate peers, rather than passwords. A CA infrastructure can be much more secure and manageable than passwords, so we said sure, how hard can it be?

Here’s what the design looks like:

Because DMVPN relies on a “hub and spoke” design, let’s make sure the hub is redundant. Unfortunately there isn’t a good way on Cisco routers to define two hubs under a single interface, so let’s use two tunnel interfaces – tunnel1 (DC1 router is the hub) and tunnel2 (DC2 router is the hub). To allow the hubs to be aware of each other, let’s establish a ‘backplane’ connection between the routers via DMVPN over the WAN. Effectively, this creates two redundant dmvpn networks over the same WAN infrastructure, and then connects them – a redundant, self-healing WAN infrastructure. 

So let’s build it!

1. Crypto – let’s use some SuiteB strong encryption for our WAN (all routers)

crypto isakmp policy 5
 encr aes 256
 hash sha512
! authentication rsa-sig   <-- Note, this is the default, no need to configure it
 group 16

crypto ipsec transform-set dmvpnPhase2 esp-aes 256 esp-sha512-hmac
 mode tunnel

crypto ipsec profile dmvpnCrypto
 set transform-set dmvpnPhase2

2. DMVPN

IP information:
Tunnel1 subnet: 10.255.254.0/24
Tunnel2 subnet: 10.255.255.0/24
Tunnel subnets, host IP: DC1: X.X.X.1 / DC2: X.X.X.2 / Site1: X.X.X.3 / Site2: X.X.X.4
DC1 router public IP: 1.1.1.1
DC2 router public IP: 2.2.2.2


! --- DC1 router – Hub for Tunnel1 DMVPN cloud
! Hub tunnel
int tun 1
 description Tunnel1 - Hub
 ip add 10.255.254.1 255.255.255.0
 ip mtu 1440
 ip nhrp map multicast dynamic
 ip nhrp network-id 1
 tunnel key 1
 tunnel source Ethernet1/1
 tunnel mode gre multipoint
 tunnel protection ipsec profile dmvpnCrypto shared

! Client tunnel
int tun 2
 description Tunnel2 - Client
 ip add 10.255.255.1 255.255.255.0
 ip mtu 1440
 ip nhrp map multicast dynamic
 ip nhrp network-id 2
 ip nhrp map 10.255.255.2 2.2.2.2
 ip nhrp map multicast 2.2.2.2
 ip nhrp nhs 10.255.255.2
 tunnel key 2
 tunnel source Ethernet1/1
 tunnel mode gre multipoint
 tunnel protection ipsec profile dmvpnCrypto shared


! --- DC2 router – Hub for Tunnel2 DMVPN cloud
! Client tunnel
int tun 1
 description Tunnel1 - Client
 ip add 10.255.254.2 255.255.255.0
 ip mtu 1440
 ip nhrp map 10.255.255.1 1.1.1.1
 ip nhrp map multicast 1.1.1.1
 ip nhrp nhs 10.255.255.1
 ip nhrp map multicast dynamic
 ip nhrp network-id 1
 tunnel key 1
 tunnel source Ethernet1/1
 tunnel mode gre multipoint
 tunnel protection ipsec profile dmvpnCrypto shared

! Hub tunnel
int tun 2
 description Tunnel2 - Hub
 ip add 10.255.255.2 255.255.255.0
 ip mtu 1440
 ip nhrp map multicast dynamic
 ip nhrp network-id 2
 tunnel key 2
 tunnel source Ethernet1/1
 tunnel mode gre multipoint
 tunnel protection ipsec profile dmvpnCrypto shared


! --- Client site routers
int tun 1
 description Tunnel1 - Client
 ip add 10.255.254.XXX 255.255.255.0
 ip mtu 1440
 ip nhrp map 10.255.255.1 1.1.1.1
 ip nhrp map multicast 1.1.1.1
 ip nhrp nhs 10.255.255.1
 ip nhrp map multicast dynamic
 ip nhrp network-id 1
 tunnel key 1
 tunnel source Ethernet1/1
 tunnel mode gre multipoint
 tunnel protection ipsec profile dmvpnCrypto shared

! Client tunnel
int tun 2
 description Tunnel2 - Client
 ip add 10.255.255.XXX 255.255.255.0
 ip mtu 1440
 ip nhrp map multicast dynamic
 ip nhrp network-id 2
 ip nhrp map 10.255.255.2 2.2.2.2
 ip nhrp map multicast 2.2.2.2
 ip nhrp nhs 10.255.255.2
 tunnel key 2
 tunnel source Ethernet1/1
 tunnel mode gre multipoint
 tunnel protection ipsec profile dmvpnCrypto shared

3. Routing – EIGRP

Pro tip #1: Make sure to “no next-hop-self” (to allow phase2 direct pathing to occur) and “no-split-horizon” (to allow DMVPN clients to learn about other other client site routes)

Pro tip #2: If you want to redistribute ‘connected’ routes into the routing process, use a route-map to filter the tunnel’s “source” interface, or you’ll end up with eigrp looping errors due to stacked routers – the neighbors will think the best way to get to the tunnel source interface is through the tunnel, which doesn’t make any sense and will break things.

!--- Hub sites
router eigrp ECorp
 !
 address-family ipv4 unicast autonomous-system 5
  !
  af-interface TunnelXX 
   no next-hop-self
   no split-horizon
  exit-af-interface
  !
  topology base
   redistribute static
  exit-af-topology
  network 10.0.0.0
 exit-address-family

!--- Client sites
router eigrp ECorp
 !
 address-family ipv4 unicast autonomous-system 5
  !
topology base
   redistribute static
  exit-af-topology
  network 10.0.0.0
 exit-address-family

4. Certificate infrastructure

Okay, now we have a respectable DMVPN infrastructure. We could either add some static pre-shared keys, or we could build the certificate infrastructure on the routers to allow dynamic, flexible management. Let’s do it. 

Pro tip: On all routers, make sure to set up ntp to an accurate time-source. Certificates are time-stamped, and if your clocks drift too far, your infrastructure might fall apart for what appears to be no reason.

! – NTP, clocks (all routers)
ntp server 132.163.4.101
clock timezone MST -6
clock summer-time MDT recurring

!-- To build certs we need a hostname and domain (all routers)
hostname Hub1
ip domain-name ecorp.com

!-- To allow clients to download their certs (CA server only)
ip http server

!—Certificate server (Only on CA server – we’ll use Hub1 router)
Pro tip: Because of a limitation of GNS3 to store files in flash, I’m storing the certs in nvram. You’re able to store them with ftp, tftp, direct to flash, and many other ways if you’d like.

crypto key generate rsa label hub1-ca modulus 2048 exportable
crypto pki server hub1-ca
 issuer-name CN=Hub1.ecorp.com
 grant auto
 hash sha512
 auto-rollover
 database url nvram:
 database level complete
 no shut

!—Register the router to the CA and get a signed identity certificate.

Pro tip #1: Even the CA server (If participating in DMVPN, like we are doing here) needs to check in with the CA (itself) to get an identity certificate.

Pro tip #2: This doesn’t have to occur over a local network or over a VPN link on the internet – it works just fine over the internet. And the routers are only exchanging public keys, never their private signing keys, so there is little risk here.

crypto key generate rsa label Hub1-dmvpn modulus 2048 exportable
crypto pki trustpoint dmvpn-client
 enrollment url http://1.1.1.1:80
 subject-name CN=XXXX.ecorp.com  <-- Use a unique name, like the router’s hostname
 revocation-check crl
 rsakeypair Hub1-dmvpn
 storage nvram:
 exit

!—Once the trustpoint is built, we need to activate it to accept the CA as valid and get our own router identity certificate to be used for dmvpn authentication.

crypto pki authenticate dmvpn-client
! Provided your router can reach the IP of the CA (public IPs work just fine), the router will download the CA certificate and store it locally. You are asked to accept it.
crypto pki enroll dmvpn-client
! You are asked several questions about downloading your identity certificate. Make sure to say “yes” to downloading your own cert, and the routers will negotiate the rest. 

5. Debugging

If you have any issues, you can use the following commands to troubleshoot the issues. In my experience, these aren’t incredibly useful, but it might help! 
! Debug hub
debug crypto pki messages
debug crypto pki server
debug crypto pki transactions

! Debug spokes
debug crypto pki messages
debug crypto pki server
debug crypto pki transactions
debug crypto isakmp
debug crypto ipsec
debug crypto engine

Profit! 

And that’s it. Clients will be able to authenticate to the hubs and to each other. Clients will automatically roll-over to new certificates when the certs reach their expiration point, and no more user interaction is required at any point.

Also, any routes injected into the routing domain by any client or hub devices will propagate through the dmvpn clouds and direct routing will take over nearly immediately.

Possible venues for making this situation more redundant are more hubs, redundant CA infrastructures, and regionalizing initial routing. 

To regionalize WAN routing so the shortest path is used for the initial connection (before phase2 routing utilized p2p connections), lessen the delay (default 50,000u sec) on the tunnel interface pointed at the local DMVPN hub site. 

Download and play with the topology here. 

Have fun and good luck!

Kyler