Deploy Cisco ASAv in AWS for Infrastructure VPN with TONS of pictures

The AWS Virtual Private Gateway (VGW) is flexible, and integral to many of the AWS-native network backbone features exposed to users. However, it has some significant throughput drawbacks - in testing, we only see about about 640Mbps, or about 80MBps. And beyond that, flows seem to be limited to 400Mbps each, so utilizing the full 640Mbps requires multi-threading on the part of the application. The documentation claims the VGWs are capable of pushing 1.25Gbps, but we have yet to see close to that in our day to day testing.

To be fair to AWS, they do offer a pretty significant asterisk on the 1.25Gbps throughput claim.

If you're paying for a 10Gbps Direct Connect (DX), you're probably not too happy about being limited to 6.4% of the speed you thought you were getting. Or even 12.5%, if you get closer to the theoretical 1.25Gbps speed Amazon claims the VGW can push.

The real answer here is to use a Private Virtual InterFace (PVIF). But PVIFs have security drawbacks - they have none. The AWS security model is based around endpoint security ONLY - Network ACLs (NACLs) and Security Groups (SGs) are all that you get. Other features are coming online over time, but there are many reasons network interconnect filtering might be required. Here are a few examples, by no means an exhaustive list

• You need high bandwidth, requiring a PVIF - no interconnect filtering for you
• You're connecting to another cloud via a VPN - no interconnect filtering for you
• You're connecting to a partner's AWS VPC, via VPN or VPC peering - no interconnect filtering for you

You're noticing the trend here. The AWS-native toolkit isn't great here, so partners have stepped up, Cisco among them. The Cisco ASAv is a virtual machine that emulates the feature-set of the eponymous Cisco ASA series. Right now the highest throughput supported in AWS (or any other cloud) is 2Gbps with an ASAv30. It's still only 20% of the 10Gbps DX your company is springing for, but it's closer, and the security gains from deploying it are oceans away from the nothing-nada-none interconnect security on the AWS-native options. 

The Cisco ASAv50, which would help us utilize 100% of the 10Gbps DX at our hypothetical company, isn't yet available on the public cloud. True to form, the AWS Marketplace entry for the ASAv shows 2Gbps throughput as the highest yet available. Fingers crossed it becomes available over time.

None, enough prologue. Let's build this thing. First, go the AWS Markplace page for the ASAv and subscribe. There are two options:

• Pay-as-you-go: You're charged in an ec2-like way for per-minute. The reviews on this version aren't great, mostly from folks misunderstanding billing, or maybe from Cisco over-billing. Regardless, I don't recommend this one. 
• BYOL: Bring Your Own License. The ASAv doesn't actually REQUIRE a license. However, you'll be locked in at 100Kbps throughput until you purchase a license from a VAR, register it with Cisco, and get it install - Cisco Smart licenses only. 

Pick your poison, then hit "Continue to Subscribe". It'll take a few minutes, then you'll get an email that shows you are now subscribed. You DON'T need to continue to Configuration - it's actually easier if you don't. 

Now, let's get this really moving. Log into the AWS console --> ec2, and then click "Launch Instance" in the top right.

Search for ASAv, and find the BYOL device and hit "Select" next to it to start configuring the device.

Take note of the next screen, particularly the right side that shows rates for the different ec2 instance types. These costs vary by region (and possibly by contracted rate with AWS?), so yours might be slightly different.

Now compare the rates and sizings with the AWS Marketplace entry for the ASAv - it shows the suggested sizings for the different ASAv types and throughputs.

For me in this region, it's clear that the c4.xlarge is my best bet to get 2Gbps (ASAv30) with the cheapest rate. So let's do it. Hit "continue" on the ec2 deployment wizard, check the box and continue on to "Next: Configure Instance Details".

Here you get to decide where to put your ASAv device. Which is an interesting question, because this is a transitive device, not very common in cloud-land. The first interface built on an ASAv is called the "mgmt0/0" interface when you log into the ASA. Cisco recommends you use this as an actual management interface, but I like to use it as an internal interface, so let's put the device into a "private" subnet. Private doesn't mean much here - just that it is reachable from the inside of your network so you can route traffic through it.

Remember to also hit the "Enable termination protection" option to prevent someone from accidentally terminating this instance and deleting all your hard work.

Scroll down, and let's add our second interface to the ASAv. This one will be called "Gigabit0/0" locally on the ASAv. I like to put this in the "public" zone, and use it for inbound traffic. That requires some very specific network items, such as:

• This interface has to be in a subnet with a default route through an Internet GateWay (IGW)
• The NACL on the subnet has to permit inbound traffic on VPN ports: 
• udp/500: ISAKMP
• udp/4500: ESP, used for nat-traversal, highly relevant in the AWS environment where all hosts have private IPs and are behind 1:1 NAT to their Elastic IPs (EIPs)

Let's add an interface to our ec2 instance and put it in the "public" subnet. 

Now, it seems like you're done with this page, but this is where you'll find the first gotcha of this deployment. When the Cisco ASAv comes up, its configuration will be blank, which prevents anyone to connect to it over the network. There is no console to connect in cloud-land, so we need a way to give the device its initial configuration to permit us to connect to it. Cisco has an example of what they call a "0-day Configuration" posted on their docs page. It'll get the job done, but let's modify it for our use-case. I'll highlight the information that I modified:

hostname NewASAvThanksKyler
int gi 0/0
 nameif outside
 security-level 0
 ip address dhcp setroute
 no shut
interface management0/0
 management-only  <-- remove this line if you want to use it as your internal interface 
 nameif inside
 security-level 100
 ip address dhcp
 no shut
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
crypto key generate rsa modulus 2048
ssh 10.0.0.0 255.0.0.0 inside
ssh 10.0.0.0 255.0.0.0 outside    <-- Set this one to "ssh 0 0 outside" if you want to connect to it from the internet - say, if you aren't privately connected to this VPC. 
ssh timeout 30
ssh version 2
username admin password SuperSecretPassword privilege 15   <-- Set your own password
username admin attributes
service-type admin

Customize it however you like, then expand the "Advanced Details" panel at the bottom of the screen and paste in the configuration you just created. Leave the radio button on "As text". Then click "Next: Add Storage".

Click through the storage panel with no changes, which brings us to tags. Tags can be used for all sorts of organizational and billing purposes. Here, we'll just give our hosts a name. Either click "Add a Name tag" or just create a new tag with Key of "Name" and call it whatever you want. It makes sense to have the AWS name match the Cisco configuration name, something we didn't do here.

The next panel asks us for a SG to assign to our host. Every host in AWS has a SG that controls the ingress and egress of traffic to it. The default permits inbound tcp/22 (SSH) from the internet, which is a terrible, bad, awful idea. The host will be protected by an SSH key, but why permit any schmoe on the internet to connect to it? Update the rules shown below. You CAN put "0.0.0.0/0" on the tcp/500 and tcp/4500 rules, which would permit anyone on the internet to try and establish a VPN to your host, but I don't recommend it. No reason to expose your devices more than you need to, even when the device "Security" in the name.

Now you can review. Make sure everything looks like you expect. Then hit Launch! And we're off to the races! Oh wait, we're not. We need to select an SSH key to control access to the box. Either select an existing SSH key or build a new one. In either case, make sure you have the private key, then check the box and hit "Launch Instances".

The ASAv will build. It takes about 10 minutes to build and come up to where you can reach it, but that time will fly - there's a few more steps we have to take before this device is reachable. Let's add an EIP - a public Amazon-owned IP so this device can reach (and be reachable from) the internet. Before we go and grab the EIP, we need to learn where we're going to put it. Usually you'd associated an EIP with an ec2 instance, which is simple with an instance that has a single interface. However, we have a few, and could potentially have many more. So we need to learn exactly which Elastic Network Interface (ENI) we want to attach it to. In the ec2 panel, click on the interface, then go to Actions --> Networking --> Manage IP Addresses. We don't care about the actual IPs yet - what we're interested in the ENI of eth1, the SECOND interface on this device. Remember, the second interface is gi0/0, and the one we set in the "outside" zone in our Cisco config. Copy the ENI-XXX to your clipboard or write down the last few.

Now let's go get the real EIP. Go to the ec2 panel, then on the left side, under "Network & Security", click on Elastic IPs.

In the top left, click "Allocate new address", then "Amazon pool," then click Allocate to see your new IP. I'd recommend writing the IP down, especially if you have a bunch of other EIPs, so you don't get it mixed up. Click close, then click on the EIP in your list, Actions at the top, then Associate. Bam, your device is now on the internet.

Cisco AWS ec2 instances have a built-in security measure to prevent hosts from becoming transit devices and siphoning data from your network. That security measure is called the source/destination check, and makes sure that only traffic destined TO or sourced FROM an IP the host owns passes to or from the host. Which is exactly the rule we need to break for this ASAv to act as a transit device, so let's turn it off. On the ec2 panel, under Network & Security, click on Network Interfaces.

As an aside, if you need to find an IP in your org, regardless of what service is consuming it, that service has to create an ENI to slurp up traffic. So if you want to find out if an IP belongs to RDS, Batch, Directory Services, or a plane-jane ec2 instance, look for the ENI as a shortcut.

An easy way to find both ENIs assigned to our host is to search for the SG name that we created. Bam, found. We only need to disable this check on the "inside" interface of the ASAv, which will be the first interface we assign. You can find it by looking for "Primary Network Interface" in the description.

Click on the Primary network interface and click Actions --> Change Source/Dest. Check. Then turn it off and hit save.

Once the host boots, ssh to it using the SSH key you selected when building the host and you are in! If you get an error message from a mac or linux client when attempting to use the SSH key that the key is inaccessible even though you know it is, update the permissions to be more specific:
computer:~ kyler$ chmod 400 Downloads/KylerASAv.pem 

Now that the ASA is built, reachable, and all interfaces are in the right subnets, verify your internet access. If you can reach out, the ASAv is done. Now you just need to build some VPNs!

The one major next step is to update any routing tables in your VPC to point at the internal interface (ENI) of the ASAv so it can carry the traffic across. Find the ENI of the Primary network Interface just as above, then go the VPC panel in the AWS web console. Click on the route table you want to edit, then on "Routes," then on "Edit Routes". Add a new route to whatever the partner's network is that you want to route towards, then click in the "Target" field. You'll need to select "Network Interface," then you can paste int the entire ENI-XXX string where you want to send this traffic. Hit "Save routes" and your changes are live - you're sending your internal traffic to the internal ENI of the ASAv device.

Good luck out there! 

kyler

Stitching Clouds - Azure to AWS Cisco CSRs behind IGW (static NAT)

The Microsoft Azure cloud has made some tremendous strides forward in the past few years. Despite entering the market years after the market leader (AWS) had established a dominance, they have quickly built their market share. The playbook was classic Microsoft - court enterprises, and leverage the overwhelming dominance of the Windows operating system with packaged and discounted OS licensing costs.

Because of this and more, Azure now holds a strong second place in the cloud environment market. Most infrastructure abstractions copy the AWS model, but use different names. Cisco has provided a solid mapping of names between the services.

Most enterprises begin with a single cloud, and quickly realize that each cloud has its own benefits and potentially unique features, and almost all enterprises are now what's called in industry as "multi-cloud". That means linking the network of these cloud together so all services can communicate.

A good place to start is Azure, since it will help us build our Cisco CSR IOS configuration once built. There are a half-dozen steps to building the VNG (Virtual Network Gateway, the parallel for the AWS VGW - Virtual Network Gateway), and which Azure's documentation covers very well, so I won't rehash it. This doc will walk you through creating:

• Gateway Network (where Azure-build public-facing services, like the VNG, will live)
• Virtual Network Gateway (VNG), the device which terminates infrastructure VPNs
• Local Network Gateways, the reference object which contains public IPs and other context information for your public VPN endpoints, similar to "Customer Gateways" in AWS-land
• Connections, finally, the real VPN between your Azure tenant and the non-Azure VPN endpoint

Now, the docs from $MSFT are excellent, but there are a few low-hanging fruit and gotchas, and I'll talk through them here. For the VNG, here's a few gotchas: 

• HA - High Availability. If you're an enterprise, you want HA. However, the VNG doesn't launch with HA enabled by default, like it would in AWS. So let's go into the VNG and add HA. Flip "Active-active mode" to "Enabled". 
• NOTE: When you save after updating the active-active mode, this will cause the VNG to be torn down and non-functional for about 20-30 minutes while it is being rebuilt in HA-mode. All your Local Network Gateways and Connections will endure, but the configuration of each will be affected, so it's a good idea to do this before moving on to the other items. 
• Set the BGP ASN - it has to be a public ASN (non-registered), but can be an other. Make sure it doesn't overlap with your CSRs or other ASNs, or your BGP atribute routing will get complex. 

Go through and build all of these items for your CSR or CSRs. For the Local Network Gateway, make sure to add the BGP configuration and point at the public (Elastic IP) of your CSR. For the BGP peer IP address, this IP will be inside your VPN, so it will be a local loopback on your device. I created a local loopback on my CSR with IP 10.255.255.1. This interface and IP will be what you tell BPG to source your connection from when you build BGP across this tunnel. 

Once all that configuration information is saved, switch to the "Connection" configuration item. At this point, it knows everything it should be doing, but there is a gotcha, where you need to enable BGP across this connection. It's a simple on/off, so flip it to "Enabled" and hit save. It'll take a minute to update. 

Now let's let Azure do the work for us. Navigate into your Connection to your CSR, click on "Overview" in the left column, and then click "Download Configuration." There's lots of configuration types, pick the one most relevant to you. For our Cisco CSRs, the one that is easiest for me to read and works is the IOS (ISR, ASR) template. 

My advice is to copy all the configuration out to a notepad and make sure you save a copy. Save as you go, we'll be making a few very important changes in order for this to work. 

Right at the top, there's a few items to check. First, make sure there are two public IPs listed. If there aren't, your HA mode isn't finished rebuilding or isn't enabled. I don't recommend any enterprise of any size move forward without HA. It's always worth the investment. 

!   > Public IP addresses:   

!     + Public IP 1:         1.2.3.4

!     + Public IP 2:         5.6.7.8

Second, make sure this connection is built with BGP enabled. If you don't see "True" here, or if the line is missing, go back and double-check your Connection, Local Network Gateway, and Virtual Network Gateway config - one of them will be missing the BGP=Enabled section or was rebuilding after a save when you downloaded config. 

!   > Azure virtual network

!     + Enable BGP:            True

In the IKE section, make sure to update the local IP address to your PRIVATE IP address. Hosts in AWS aren't directly on the internet, they almost always use an EIP to do 1:1 NAT. 

crypto ikev2 policy AzureCSR1

  proposal AzureCSR1-proposal

  match address local 10.20.30.40

  exit

The IKEv2 policy has the same issue, matching on the public (EIP) address, rather than local, and needs to be updated: 

crypto ikev2 profile AzureCSR1-profile

  match address  local 10.20.30.40

The tunnels that are built require a slight modification - when HA is used, there are two destinations, requiring two tunnels. In Cisco-land, when two tunnels share a crypto profile, they require the "shared" keyword. I have no idea why this is, but Azure's auto-config builder misses this, and that means the CSR can only bring up one tunnel at a time... unless you make this change: 

int tun 90

 tunnel protection ipsec profile AzureCSR1-IPsecProfile shared

Remember that all tunnels in Cisco's transitCSR product for AWS live in VRFs, and it's a good idea for these tunnels to also exist in VRFs. First, let's build the VRF. 

ip vrf AzureCSR1_VRF

 rd 64518:200

 route-target export 64518:0

 route-target import 64518:0

Then put the new loopback interface in the VRF

int loopback 90
 ip vrf forwarding AzureCSR1_VRF
 ip address 10.255.255.1
 no shut

Make sure to update the BGP config to source traffic from your new interface, and make sure to configure the BGP neighbor in the Azure connectivity VRF:

router bgp 64512
 address-family ipv4 vrf AzureCSR1_VRF
  neighbor 10.9.255.228 remote-as 65555
  neighbor 10.9.255.228 activate
  neighbor 10.9.255.228 update-source loopback90

And there's one final oddity with Azure's provided config - they recommend you use an APIPA reserved address (169.254.X.X) for your tunnel, and a /32 to boot. Which means the router can't inherently understand which traffic to send over the tunnel, even though all the items are in place. the trick to kick-start it all is to add a static route over the tunnel towards the BGP neighbor so they can establish a neighborship and start routing. If you are using HA mode (again, highly recommended, make sure to send each BGP neighbor over the appropriate tunnel).

ip route vrf AzureCSR1_VRF 10.9.255.228 255.255.255.255 tun 90
ip route vrf AzureCSR1_VRF 10.9.255.229 255.255.255.255 tun 91

And bam, you now have routing between your Cisco transit CSRs and an Azure tenant. There's no fancy lambda to configure connectivity that I'm aware of - but if you stumble across any Azure-focused Cisco automation here, please link in the comments and we can share with the community.

Thanks all. Good luck out there!
kyler

Cisco TransitVPC in AWS - How to Use Lambda To Update CSR Config Automatically

Cisco in the Cloud

Cisco has had a hard time adjusting to cloud-centric enterprises. It doesn't help that these cloud providers aren't too keen on third parties providing "core" services in their environments. Both AWS and $MSFT Azure have taken the Microsoft model where popular third-party services are emulated and built into the core product, providing services that lack customization but are often cheaper, better supported, and scale better than these third party services.

Something that AWS and Azure don't do well yet is something I'm calling mid-network filtering. What I mean by that is that both cloud providers have taken to an endpoint-focused security model, and build their security tools around locking down the endpoints. There's nothing inherently wrong with this model - in fact, if you need to choose a model, endpoint-adjacent network filtering is excellent. But in my experience something that's lost with cloud is flexibility - the ability to build the network and access controls in a way that makes sense for your business. Many network and virtualization vendors, including Cisco, have focused on providing the most flexible and extensible methods of building... whatever you want, in whichever want you want it! Because of that, their products can be confusing and sometimes esoteric - try writing a universally valid CoS policy for Cisco switches across their different small biz to core enterprise switches - it's impossible. But I digress.

Cisco's foray into covering one of the current gaps in AWS cloud computing is called the "Transit Network VPC." It's a pre-packaged VPC environment with all the accoutrements, including subnets, routing tables, NACLs, security groups, etc. all built to serve one purpose - hosting 2x redundant Cisco CSR routers which operate as a hub for the VPNs to every spoke cloud VPC. Because of their positioning in the network, and because they're a full fledged enterprise network platform, they're able to perform ingress and egress filtering (including ZBFW) on each and every tunnel (VPN) interface to every VPC. That is world's away from the endpoint-security-or-nothing approach that AWS has implemented in the rest of the network.

Here's a link to the marketplace entry: Cisco Cloud Services Router (CSR) 1000v - Transit Network VPC - BYOL

For now, AWS supports this product and even hosts documentation for how to deploy it. As an aside, it can't last - the CSR routers, even fully built, can only push 4.5Gbps, and the VPN tunnels they're terminating can only push 400Mbps of throughput due to VGW capacity limitations. Native tooling, including non-ec2 based appliances, as well as native backhaul constructions like VPC peering, DX gateways, transit gateways, etc., are going to push nearly every customer away from devices like this.

But let's talk about something that these devices do well. In order to build the VPNs from every VPC you control and want to communicate together, each VPC gets a CloudFormation (CF) stack that polls the hub and pushes the local VGW configuration to an S3 "config" bucket in the hub VPC where the CSRs live. There are a series of lambdas there that are triggered by this upload, and read the information uploaded. This connection info is transformed into Cisco IOS configuration and pushed automatically via Lambda automation to the CSRs.

The real power here is that the Lambda doing this pushing is written in python, and is extractable from the vanilla Lambda provided by Cisco. The file is called "cisco-configurator.py", and looks a little like this:

Standard python code - cryptic, and not very useful. However, scroll down and you'll see some human-readable code showing what commands are being executed:

These commands are executed in order on both CSRs, and it's easy enough to add a custom ACL to each tunnel automatically when it's built:

And remember that every VGW forward all the networks it knows about to every BGP VPN peer, which can very VERY quickly lead to routing conflicts and shut your network down. To prevent, this, how about we add an inbound prefix-list to the BGP peer to only allow the networks we'd like to receive, and no others.

And keep in mind that VGWs will only accept 100 routes from a BGP neighbor before shutting down the neighbor for a 5-minute hold-down timer. So let's prevent more than say, 20 routes from coming into our BGP transit network before shutting the neighbor down. I'd rather shut down a single spoke than forward more than 100 routes to all the peers and shutting down the entire global network. You can do this with "max-prefix" command, like this:

Save all your work and your updated lambda is now good to go. Zip up the entire folder structure into a .zip file - be careful on macs, their default right click "compress" option will ignore any files and folders with a leading period, which will skip a required library and cause your lambda to fail to run when it's uploaded.

Remember that .libs_cffi_backend folder and file! You need it!

Now log into your AWS Lambda console and open up the "cisco configurator" lambda. Scroll down to "Function code" and click the upload button. Find your updated and zipped up Cisco Configurator lambda and click upload.

Bam, you're done. Make sure to check out the logs to verify the lambda is properly running. The most frequent problem I ran into was skipping the libs_cffi_backend folder and library file, which caused it to error out saying it couldn't find that exact file.

If you have any issues after uploading the file to AWS, turn up the logging to "DEBUG" and hit save. The monitoring logs will generate WAY more information, and are very verbose about what's wrong.

And that's all - your lambda will run automatically each time a new spoke VPN with the "poller" lambda updates its VGW tags for the VPN transit network - either turning it off or on will generate a new log file upload, which will trip this lambda to run. For extra fun, turn on SSH debugging and watch your lambda get to work.

Good luck out there,
kyler

Next up: How to bring all the Cisco lambda calls in-house, to avoid vendor source control of these files

Cisco ASA - L2L VPN Filters

Hey all,

Suppose you have a Cisco ASA with a L2L VPN to an untrusted partner, and you have a requirement that large swaths of the network are included in the crypto map to this partner. However, you want to only permit particular ports, to particular hosts. Thankfully, Cisco ASAs support what's called a vpn-filter that can do exactly that.

VPN-filters are an excellent tool, and can be used in this exactly same way for Cisco AnyConnect client VPNs. If you want to learn more about how to use these filtering in a dynamic system, where login is handled by external auth, like AD, but hosts are filtered, read this. Here, though, we'll focus on L2L VPNs.

First of all, create your objects, object-groups, and access list that you'll apply to the VPN. These type of "vpn-filter" ACLs are applied inbound, so the source of traffic is remote.

! Create partner objects
object network RemotePartner_Subnet1
 subnet 10.200.0.0 255.255.0.0
object-group network RemotePartner_VPNGroup
 network-object object RemotePartner_Subnet1

! Create objects for our network

object network MyNetwork_Subnet1
 subnet 10.100.0.0 255.255.0.0
object-group network MyNetwork_VPNGroup
 network-object object MyNetwork_Subnet1

! Create services/port filtering object

object-group service web tcp

 port-object eq www

 port-object eq https

! Create inbound ACL to use as a vpn-filter, include icmp for testing purposes

access-list Remote_2_MyNetwork_vpnFilter extended permit tcp object RemotePartner_VPNGroup object-group MyNetwork_VPNGroup object-group web
access-list Remote_2_MyNetwork_vpnFilter extended permit icmp any any

! Create a group policy to reference the vpn filter
group-policy MyNetwork_2_RemotePartner attributes
 vpn-filter value Remote_2_MyNetwork_vpnFilter
 vpn-tunnel-protocol ikev1

! Update the tunnel group for the remote peer to map to the group policy
tunnel-group 50.50.50.50 general-attributes
 default-group-policy MyNetwork_2_RemotePartner

Done! Now your L2L VPN is filtered inbound on your side, and doesn't require any action by the remote partner in the event you need to add a few hosts or further restrict the access.

Good luck out there.
kyler

Cisco ASA - Strong SuiteB Encryption - ECDHE

Hey all!

So you have Cisco ASAs that handle your client remote access, and you want to be sure you're using the strongest technical security you can configure. Cisco ASAs make this pretty easy to do, but you need to be aware of a few things:

1. This change is global, to all SSL cryptography. This means you can't just switch over a couple of group policies for testing. It's recommended to make this change during off-hours so you can test yourself
2. The change won't kick all the SSL tunnels that are built, so you can certainly make the change, test a connection, then switch back mid-day without an impact to your users
3. This change also affects the web page that users access pre-VPN to download the AnyConnect client and access any bookmarks and other features exposed over clientless VPN. This does include the Cisco ASDM, which requires a Java update to work properly after this change. I'll cover this more down below. 

TLS Version

First of all, use only TLS version 1.2. SSL 3.0/TLS1.0 has been found to have significant weaknesses, and TLS 1.1 is not longer recommended due to the additional features added in version 1.2. To make this change, run the following command. This limits all SSL connections to the ASA to use only version 1.2 (or greater, when available). When version 1.3 becomes standard, you'll want to update to that, but 1.3 is not yet implemented in ASA code.

ASAHostname(config)# ssl server-version ?

configure mode commands/options:
  tlsv1    Enter this keyword to accept SSLv2 ClientHellos and negotiate TLSv1
           (or greater)
  tlsv1.1  Enter this keyword to accept SSLv2 ClientHellos and negotiate
           TLSv1.1 (or greater)
  tlsv1.2  Enter this keyword to accept SSLv2 ClientHellos and negotiate
           TLSv1.2 (or greater)

ASAHostname(config)# ssl server-version tlsv1.2

Ciphers

Second, make sure that the ciphers used to encrypt the data are classified by Cisco as "High" security.
ASAHostname(config)# ssl cipher tlsv1.2 high

This classification includes:
ASAHostname# show ssl ciphers high 
  ECDHE-ECDSA-AES256-GCM-SHA384 (tlsv1.2)
  ECDHE-RSA-AES256-GCM-SHA384 (tlsv1.2)
  DHE-RSA-AES256-GCM-SHA384 (tlsv1.2)
  AES256-GCM-SHA384 (tlsv1.2)
  ECDHE-ECDSA-AES256-SHA384 (tlsv1.2)
  ECDHE-RSA-AES256-SHA384 (tlsv1.2)
  DHE-RSA-AES256-SHA256 (tlsv1.2)
  AES256-SHA256 (tlsv1.2)

DH Group

Third, make sure that the Diffie-Hellman Group used to exchange data uses larger moduli, which should keep the key exchange tamper resistant against more powerful computers.

ASAHostname(config)# ssl dh-group ?    

configure mode commands/options:
  group1   Configure DH group 1 - 768-bit modulus
  group2   Configure DH group 2 - 1024-bit modulus
  group5   Configure DH group 5 - 1536-bit modulus
  group14  Configure DH group 14 - 2048-bit modulus, 224-bit prime order
           subgroup (FIPS)
  group24  Configure DH group 24 - 2048-bit modulus, 256-bit prime order
           subgroup (FIPS)

ASAHostname(config)# ssl dh-group group24

Testing

Your change is done, you can now test your VPN with AnyConnect. Also visit the page using a browser from the outside to view the clientless page and verify that security is classes as solid using Chrome.

I also recommend Qualsys' SSL Labs testing toolkit, which is free. Make sure to check the box "do not show the results on the board" to keep your results private.

ASDM Fix ("Unable to launch device manager from...")

Because of the enforced strong encryption on the ASA, the default download of Java doesn’t contain the encryption libraries required to run the ASDM GUI application to manage the firewall. If Java is not updated with these libraries, the following error will be seen:

To allow Java to connect, the Java Cryptography Extension pack must be downloaded and pasted over the existing files in the local Java install. The JCE pack for Java family 8 can be found here: http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

Both new library files must be dropped into this path: C:\Program Files (x86)\Java\(Java version path)\lib\security\. For instance, on a test machine this exact path is used: C:\Program Files (x86)\Java\jre1.8.0_131\lib\security\.

Keep an eye on this folder, though - every time Java updates itself, it will wipe out the JCE packs that you added, and you'll need to re-drop the JCE into the folder again.

Happy encrypting,
kyler

GNS3: DMVPN with FVRF (Front Door VRF)

Hey all! 


I'm building up my iWan skills, and in the most recent CVD (Cisco Validated Design) for iWan, Cisco recommends setting up an FVRF (Front Door VRF) for each internet connection you have at a site. 


FVRFs allow your WAN routing to exist in a separate VRF from your internal network. There is security there, but it also allows for your default routes to the internet to not even risk being redistributed into your internal routing domain.

Now, I don't have access to any images in GNS3 to run the full iWan, but I do have images capable of running FVRFs, so that's what I've built here.

Download the GNS3 and Do It Yourself

There's nothing better for learning than building the thing yourself. Here's a completed GNS3 file with all features deployed. Please download and play with it yourself!

Files are here: https://1drv.ms/f/s!AliOPzHSO-Gng6BvZdB1wkuBgPnrFA

Good luck out there. 
Kyler

Cisco Routers: Easy Hair-Pin NAT for Internal Guest Network

Hey all! 

Recently I've been pouring myself into one particular configuration issue that is remarkably hard to solve on Cisco's IOS platform: Hairpin NAT. 

I've been tasked with designing a complete architecture for almost 20 sites. They want DMZs, Guest networks, lots of static and complex NATs, the works. And guess how many ASAs I can use? None. 

Which isn't that big of a deal, right? IOS has caught up to most security features over the past 5 or so years. NAT NVI (Nat Virtual Interface) can handle even complex NATs, ZBF (Zone-Based Firewall) is a nuanced and fantastic way to handle access control, etc. Hell, you can even build AnyConnect on IOS these days! 

But there's one problem that is intractable on Cisco's IOS platform: Hairpin-NAT. 

Because this problem is called so many things, let's define it. It's the issue that is seen when an internal host wants to access an internal server using the server's public (NAT'd) IP. Usually you don't see this with internal hosts, but the issue frequently comes up in designs when there is a "guest" network. Usually the guest network uses the same network infrastructure and internet connection as your other internal servers, but you use public DNS to make network segmentation clean and prevent as much internal access as possible. 

Picture from ServerFault

On an ASA, this problem is easy to solve - in fact, it's a single command.

same-security-traffic permit intra-interface

Surely it's this easy to solve on a router?

I tried to solve the problem in a half-dozen ways, and each has a "gotcha" that prevents it being useful in production. 

Potential Solution #1: Nat Virtual Interface (NVI) NAT'ing

This has the benefit of immediate results. Turn on NVI NAT'ing, modify the NAT statements and "ip nat enable" on a few interfaces, and you're good, all the guest network can use public IPs to access the DMZ servers. This technique is widely recommended as THE solution to this problem on the Internet. However, when there is more than an "inside" and "outside" interface, this type of NAT'ing gets very complex. 

As Alex points out in the comments below, you can make this work by creating exemptions in your NAT traffic selection. However, the complexity of this solution makes it less than ideal. 

Major drawback: Complexity with >2 interfaces and WAN NATing, high router utilization as NVI traffic is punted to the CPU for most ISR platforms, all NATs would have to be rewritten in the new style. 

Verdict: This is a no-go.

Potential Solution #2: Tons of Route-Maps

A widely cited solution from Tassos (who is a gosh-darn genius, and you should read his blog) nearly 8 years ago is to use regular NAT, but use route-maps to push traffic through an "ip nat outside" loopback address with policy routing on several interfaces. 

Major drawback: Super complex, high router utilization doing policy routing on several/all interfaces, I was not smart enough to get it working in a lab. 

Verdict: Not going to fly in a real production network (if I could even build it)

Potential Solution #3: Put the guest network on a separate internet connection

This is the solution often employed by Cisco shops that have become frustrated with the complexity and lack of manageability of Guest networks on the inside of the network. Shops either policy route the guest network out the other internet connection on their firewall or segment the network with a separate SSID/switch for guest that isn't connected to the internet network. 

Major drawback: As a consultant, I can't in good conscience ask the client to turn up 20 new consumer-class Internet circuits because I can't figure this issue out. 

Verdict: Dammit, I'm a good network engineer, I should be able to do this!

Potential Solution #4: Mix Up Some NATs

At this point I'm pulling my hair out, and really starting to believe that this just isn't possible, an answer that is widely circulated on the internet. I really just wish I could put the "guest" on the outside of my network. So I did. And dammit, it felt good. But of course, with both the "WAN" and "guest" in the same "ip nat outside" zone, there's no overload NAT to give the poor guys some Internets. 

And then I think of a funny joke. I wonder if I could use BOTH zoned NAT (ip nat inside/ip nat outside) AND stateless NAT (NVI) at the same time. I could just turn on stateless NAT on the guest and WAN zones, then write a single NAT statement to give it Internet access. I mean... there's no way Cisco would let you run both types of NAT at the same time, but I'm at wit's end. And low and behold, it flipping works. 

Major drawback: Configuration is slightly unorthodox. Using both NAT types looks funny, and someone might absent-mindedly 'fix' this in the future. Cisco might remove the older-style inside/outside NAT in future releases. 

Verdict: Solution! (Short of Cisco actually supporting hairpin-nat, this'll do) 

Final Configuration

! Inside Host (vlan 1)

int eth 0/0

 ip add 192.168.1.50 255.255.255.0

ip route 0.0.0.0 0.0.0.0 192.168.1.1

! Guest Host (vlan 10)

int eth 0/0

 ip add 192.168.10.50 255.255.255.0

ip route 0.0.0.0 0.0.0.0 192.168.50.1

! DMZ Host (vlan 99)

int eth 0/0

 ip add 192.168.99.50 255.255.255.0

ip route 0.0.0.0 0.0.0.0 192.168.99.50

! Edge Router with router on a stick

! Interface configuration

interface Ethernet1/0.1

 encapsulation dot1Q 1 native

 ip address 192.168.1.1 255.255.255.0

 ip nat inside  <-- Most interfaces remain 'ip nat inside,' and NAT statements don't have to be rewritten

interface Ethernet1/0.10

 description GUEST

 encapsulation dot1Q 10

 ip address 192.168.10.1 255.255.255.0

 ip nat outside  <-- Note that 'guest' SVI is in outside zone. This enable all existing zoned-NATs to behave properly, without any config rewrites. So 'guest' users can access DMZ hosts using their public addresses just fine. 

 ip nat enable  <-- Stateless NAT also created on this interface

interface Ethernet1/0.99

 description DMZ

 encapsulation dot1Q 99

 ip address 192.168.99.1 255.255.255.0

 ip nat inside

interface Ethernet1/1

 description WAN

 ip address 20.0.0.2 255.255.255.0

 ip nat outside

 ip nat enable

! Route

ip route 0.0.0.0 0.0.0.0 20.0.0.1

! Object-groups and ACL

object-group network LocalGuest  <-- Highly recommend using object-groups to minimize configuration and simplify

 192.168.10.0 255.255.255.0

object-group network RFC1918Private 

 10.0.0.0 255.0.0.0

 172.16.0.0 255.240.0.0

 192.168.0.0 255.255.0.0

ip access-list extended Guest_2_WAN

 permit ip object-group LocalGuest any

ip access-list extended privateToPublic

 permit ip object-group RFC1918Private any

ip access-list extended siteToSite

 permit ip object-group RFC1918Private object-group RFC1918Private

route-map natOverload deny 10

 match ip address siteToSite

route-map natOverload permit 20

 match ip address privateToPublic

! NAT Configuration

ip nat inside source route-map natOverload interface Ethernet1/1 overload  <-- Regular inside/outside overload NAT for internet traffic

ip nat source list Guest_2_WAN interface Ethernet1/1 overload  <-- The new "NVI" style of NAT. Note the lack of "inside" in this command. Simply provides internet access to "guest" network segment. 

ip nat inside source static 192.168.99.50 20.0.0.50 extendable  <-- Hosts can still be exposed to the internet with regular "ip nat inside source static" commands. 

Lock it Down: ZBFW and Reflexive ACLs

I initially attempted to use ZBFW rules to lock down traffic from WAN-->Self and Self-->WAN, which works perfectly for zoned NAT (inside/outside). However, with NVI-style NAT, return internet traffic is evaluated by the edge router as destined for the edge router itself. Even when the outbound traffic is inspect on leaving this behavior occurs. 

But in engineering, there is always more than one way to skin that darn cat. So let's build some reflexive ACLs to permit return traffic. 

! Configuration: 

ip access-list extended In_2_Out

 permit ip any any reflect StatefulInbound  <-- Allow all traffic outbound, and remember it (default 300 seconds)

ip access-list extended Out_2_In

 evaluate StatefulInbound  <-- Allow all traffic that exited to return, using the stateful ACL table. 

 permit ip any host 20.0.0.50  <-- Permit inbound traffic for this particular DMZ services host

interface Ethernet1/1

 ip access-group Out_2_In in

 ip access-group In_2_Out out

In fact, you can still use ZBFW for most traffic, just make sure to permit it past the interface ACL so it can be picked up by NAT and then ZBF and delivered to internal hosts. 

Pro tip: If you're using this reflexive interface ACL as well as ZBF, remember your order of operations. The interface ACL is evaluated pre-NAT (so inbound traffic destinations will be public IPs), and ZBF is evaluted post-NAT (so inbound traffic dests will be your private IPs)

Download the GNS3 and Do It Yourself

There's nothing better for learning than building the thing yourself. Here's a completed GNS3 file with all features deployed. Please download and play with it yourself!

Files are here: https://1drv.ms/f/s!AliOPzHSO-GngrhpfKDb-YNfu1uLXQ

Good luck out there. 
Kyler

GNS3: Redundant Phase 2 (P2P) DMVPN with Certificate Authentication

I was recently tasked with converting a VPN WAN mesh with 19 nodes on ASAs over to routers. The client initially wanted statically defined VPNs, similar to what they were used to in the ASA scheme. However, that is a lot of keys and configurations. Thankfully, we were able to convince them to allow us to build a DMVPN mesh to avoid all of that static assignment and just let the protocol sort it all out.

As an aside, the calculation to find out how many connections there would be in a VPN mesh is n*(n-1)/2. For 19, that means we were nearly tasked with manually configuring 171 static profiles. Not something I’d relish.

Anyway, the client’s security team asked us to investigate building the dmvpn mesh with certificates to authenticate peers, rather than passwords. A CA infrastructure can be much more secure and manageable than passwords, so we said sure, how hard can it be?

Here’s what the design looks like:

Because DMVPN relies on a “hub and spoke” design, let’s make sure the hub is redundant. Unfortunately there isn’t a good way on Cisco routers to define two hubs under a single interface, so let’s use two tunnel interfaces – tunnel1 (DC1 router is the hub) and tunnel2 (DC2 router is the hub). To allow the hubs to be aware of each other, let’s establish a ‘backplane’ connection between the routers via DMVPN over the WAN. Effectively, this creates two redundant dmvpn networks over the same WAN infrastructure, and then connects them – a redundant, self-healing WAN infrastructure. 

So let’s build it!

1. Crypto – let’s use some SuiteB strong encryption for our WAN (all routers)

crypto isakmp policy 5

 encr aes 256

 hash sha512

! authentication rsa-sig   <-- Note, this is the default, no need to configure it

 group 16

crypto ipsec transform-set dmvpnPhase2 esp-aes 256 esp-sha512-hmac

 mode tunnel

crypto ipsec profile dmvpnCrypto

 set transform-set dmvpnPhase2

2. DMVPN

IP information:

Tunnel1 subnet: 10.255.254.0/24

Tunnel2 subnet: 10.255.255.0/24

Tunnel subnets, host IP: DC1: X.X.X.1 / DC2: X.X.X.2 / Site1: X.X.X.3 / Site2: X.X.X.4

DC1 router public IP: 1.1.1.1

DC2 router public IP: 2.2.2.2

! --- DC1 router – Hub for Tunnel1 DMVPN cloud

! Hub tunnel

int tun 1

 description Tunnel1 - Hub

 ip add 10.255.254.1 255.255.255.0

 ip mtu 1440

 ip nhrp map multicast dynamic

 ip nhrp network-id 1

 tunnel key 1

 tunnel source Ethernet1/1

 tunnel mode gre multipoint

 tunnel protection ipsec profile dmvpnCrypto shared

! Client tunnel

int tun 2

 description Tunnel2 - Client

 ip add 10.255.255.1 255.255.255.0

 ip mtu 1440

 ip nhrp map multicast dynamic

 ip nhrp network-id 2

 ip nhrp map 10.255.255.2 2.2.2.2

 ip nhrp map multicast 2.2.2.2

 ip nhrp nhs 10.255.255.2

 tunnel key 2

 tunnel source Ethernet1/1

 tunnel mode gre multipoint

 tunnel protection ipsec profile dmvpnCrypto shared

! --- DC2 router – Hub for Tunnel2 DMVPN cloud

! Client tunnel

int tun 1

 description Tunnel1 - Client

 ip add 10.255.254.2 255.255.255.0

 ip mtu 1440

 ip nhrp map 10.255.255.1 1.1.1.1

 ip nhrp map multicast 1.1.1.1

 ip nhrp nhs 10.255.255.1

 ip nhrp map multicast dynamic

 ip nhrp network-id 1

 tunnel key 1

 tunnel source Ethernet1/1

 tunnel mode gre multipoint

 tunnel protection ipsec profile dmvpnCrypto shared

! Hub tunnel

int tun 2

 description Tunnel2 - Hub

 ip add 10.255.255.2 255.255.255.0

 ip mtu 1440

 ip nhrp map multicast dynamic

 ip nhrp network-id 2

 tunnel key 2

 tunnel source Ethernet1/1

 tunnel mode gre multipoint

 tunnel protection ipsec profile dmvpnCrypto shared

! --- Client site routers

int tun 1

 description Tunnel1 - Client

 ip add 10.255.254.XXX 255.255.255.0

 ip mtu 1440

 ip nhrp map 10.255.255.1 1.1.1.1

 ip nhrp map multicast 1.1.1.1

 ip nhrp nhs 10.255.255.1

 ip nhrp map multicast dynamic

 ip nhrp network-id 1

 tunnel key 1

 tunnel source Ethernet1/1

 tunnel mode gre multipoint

 tunnel protection ipsec profile dmvpnCrypto shared

! Client tunnel

int tun 2

 description Tunnel2 - Client

 ip add 10.255.255.XXX 255.255.255.0

 ip mtu 1440

 ip nhrp map multicast dynamic

 ip nhrp network-id 2

 ip nhrp map 10.255.255.2 2.2.2.2

 ip nhrp map multicast 2.2.2.2

 ip nhrp nhs 10.255.255.2

 tunnel key 2

 tunnel source Ethernet1/1

 tunnel mode gre multipoint

 tunnel protection ipsec profile dmvpnCrypto shared

3. Routing – EIGRP

Pro tip #1: Make sure to “no next-hop-self” (to allow phase2 direct pathing to occur) and “no-split-horizon” (to allow DMVPN clients to learn about other other client site routes)

Pro tip #2: If you want to redistribute ‘connected’ routes into the routing process, use a route-map to filter the tunnel’s “source” interface, or you’ll end up with eigrp looping errors due to stacked routers – the neighbors will think the best way to get to the tunnel source interface is through the tunnel, which doesn’t make any sense and will break things.

!--- Hub sites

router eigrp ECorp

 !

 address-family ipv4 unicast autonomous-system 5

  !

  af-interface TunnelXX 

   no next-hop-self

   no split-horizon

  exit-af-interface

  !

  topology base

   redistribute static

  exit-af-topology

  network 10.0.0.0

 exit-address-family

!--- Client sites

router eigrp ECorp

 !

 address-family ipv4 unicast autonomous-system 5

  !

topology base

   redistribute static

  exit-af-topology

  network 10.0.0.0

 exit-address-family

4. Certificate infrastructure

Okay, now we have a respectable DMVPN infrastructure. We could either add some static pre-shared keys, or we could build the certificate infrastructure on the routers to allow dynamic, flexible management. Let’s do it. 

Pro tip: On all routers, make sure to set up ntp to an accurate time-source. Certificates are time-stamped, and if your clocks drift too far, your infrastructure might fall apart for what appears to be no reason.

! – NTP, clocks (all routers)

ntp server 132.163.4.101

clock timezone MST -6

clock summer-time MDT recurring

!-- To build certs we need a hostname and domain (all routers)

hostname Hub1

ip domain-name ecorp.com

!-- To allow clients to download their certs (CA server only)

ip http server

!—Certificate server (Only on CA server – we’ll use Hub1 router)

Pro tip: Because of a limitation of GNS3 to store files in flash, I’m storing the certs in nvram. You’re able to store them with ftp, tftp, direct to flash, and many other ways if you’d like.

crypto key generate rsa label hub1-ca modulus 2048 exportable

crypto pki server hub1-ca

 issuer-name CN=Hub1.ecorp.com

 grant auto

 hash sha512

 auto-rollover

 database url nvram:

 database level complete

 no shut

!—Register the router to the CA and get a signed identity certificate.

Pro tip #1: Even the CA server (If participating in DMVPN, like we are doing here) needs to check in with the CA (itself) to get an identity certificate.

Pro tip #2: This doesn’t have to occur over a local network or over a VPN link on the internet – it works just fine over the internet. And the routers are only exchanging public keys, never their private signing keys, so there is little risk here.

crypto key generate rsa label Hub1-dmvpn modulus 2048 exportable

crypto pki trustpoint dmvpn-client

 enrollment url http://1.1.1.1:80

 subject-name CN=XXXX.ecorp.com  <-- Use a unique name, like the router’s hostname

 revocation-check crl

 rsakeypair Hub1-dmvpn

 storage nvram:

 exit

!—Once the trustpoint is built, we need to activate it to accept the CA as valid and get our own router identity certificate to be used for dmvpn authentication.

crypto pki authenticate dmvpn-client

! Provided your router can reach the IP of the CA (public IPs work just fine), the router will download the CA certificate and store it locally. You are asked to accept it.

crypto pki enroll dmvpn-client

! You are asked several questions about downloading your identity certificate. Make sure to say “yes” to downloading your own cert, and the routers will negotiate the rest. 

5. Debugging

If you have any issues, you can use the following commands to troubleshoot the issues. In my experience, these aren’t incredibly useful, but it might help! 

! Debug hub

debug crypto pki messages

debug crypto pki server

debug crypto pki transactions

! Debug spokes

debug crypto pki messages

debug crypto pki server

debug crypto pki transactions

debug crypto isakmp

debug crypto ipsec

debug crypto engine

Profit! 

And that’s it. Clients will be able to authenticate to the hubs and to each other. Clients will automatically roll-over to new certificates when the certs reach their expiration point, and no more user interaction is required at any point.

Also, any routes injected into the routing domain by any client or hub devices will propagate through the dmvpn clouds and direct routing will take over nearly immediately.

Possible venues for making this situation more redundant are more hubs, redundant CA infrastructures, and regionalizing initial routing. 

To regionalize WAN routing so the shortest path is used for the initial connection (before phase2 routing utilized p2p connections), lessen the delay (default 50,000u sec) on the tunnel interface pointed at the local DMVPN hub site. 

Download and play with the topology here. 

Have fun and good luck!

Kyler