Software-Defined Networking in the Physical World

I just finished a short lunch-hour style training on the Cisco Prime platform, and I've got to say - they get it. No one wants to play with switches all day.

That's a hard thing to admit.I've spent 500+ hours studying for Cisco certifications, and I'm currently prepping for a CCIE, a certification solely in route/switch that costs over $1,500 per attempt in a far-away state. But I'm confident saying it:

I don't want to work on switches all day. 

I want to work on a management platform that can completely configure, control, and monitor my route/switch infrastructure. I want to use this platform to correlate threat data and alert me when my ports are filling up, let me know when to plug in another cable between my switches so it can configure a port-channel, and I want it to have the smarts to shut down a network storm.

If Cisco enabled CDP to allow for simple management, the management appliance could reach new devices by chaining through CDP-enabled devices to set up the management connection on new devices. It could also allow for recovery if a command renders a device unreachable.

In short, I want my network devices to act like lightweight APs - they have some ability to operate themselves, but they are able to act intelligently and seamlessly when controlled by a central management station.

This is entirely possible, and appears to be where Cisco is going.

Let me start over - Cisco's Prime platform is a super-charged monitoring platform. It's able to discover and monitor all your routers, switches, ASA, etc with great reporting. It's able to apply templates that you've built to a switch once you've given the switch a base config and a routable IP that the management device can reach.

But still, you need to know the configs. You need to know which things to turn on and turn off and why. I'm sure there's an argument there that only the experts in the field, who've invested significant time and money (wink wink), should be doing this anyway, so who cares?

If Cisco is able to realize this dream, they'll have brought the fantastic benefits of Software Defined Networking (SDN) into the physical world. 

They'd allow existing businesses with millions invested in 'heavy,' self-managed switches to upgrade their switches software and enable central, intelligent management. Costs for expertise in networking will go down, and the cost of running a business will go down. Constantly mutating security attacks can be more quickly identified by taking the aggregate data and correlating it in a central place.

More importantly, if Cisco doesn't move to do this quickly, someone else will. 

You can read Cisco's information on Cisco Prime here: http://www.cisco.com/c/en/us/products/cloud-systems-management/prime.html
kyler

CCIE Lab - Back To Business

My post to finish out my CCIE journey is here.

Alright -- I passed the CCIE route/switch written. Which is awesome. There is a great deal of information and concepts in that test, and all the tests and study that've brought me to this point. I'm proud of myself and my accomplishment. My total is around 1600 hours of study thus far.

Damn right I'm proud

But that's behind me. I've taken two weeks off, and now it's time to start getting ready to face the dragon. The CCIE route/switch lab is a heck of a test. 8 hour long practical in a distant state, with many different rules and requirements that cost points and can fail an exam.

The failure rate on the first attempt is 90-95%
The failure rate for any attempt is around 80%

I've been using video on demand through CBTNuggets, and I've bought several books that cover the certification. Every single one says to expect to fail. The odds are so harsh, and it's so hurtful to a tester's confidence to expect to succeed and then fail, they all say that the first try is a 'practice' attempt.

A $1600 practice attempt!

I don't intend to take this test twice, but I'm not going to let that eat at me. The goal is to get the certification. If I have to test twice, 3x, 4x, I'm going to get it. But my wallet and my lovely wife would be thankful if, on this first attempt, I'd fall into that exceedingly qualified 5-10% that pass on the first try.

So it's back to studying. My studying schedule for around the past 18 months has been unchanged:
* M-F - study over my lunch hour, then 7-9:30 after work
* Saturday: Study 9-3, date night with Lindsey
* Sunday: 9-3, house chores, groceries, cooking

A part-time job would consume less time

It's almost 30 hours each week if I stick to my schedule. I invariably skimp on a single night, or take some extra time off to spend time with friends, but it's close.

The goal is still to get the CCIE number by Thanksgiving of this year. It's an aggressive, challenging goal. If I make it, I'll be around CCIE #50,000 in the world.

Wish me luck!
-kyler

Solution: EAP Wireless Failure, "Network authentication failed due to a problem with the user account"

Earlier today I set up an 802.1X EAP wireless network with a Ruckus WLAN controller.

Here's the setup:
Ruckus WLAN controller with a few dozen headless Ruckus WAPs
Domain CA that has enrolled in AD and is pushing certificates to all client pcs via group policy
Windows 2008R2 NAP server authenticating clients based on AD group "Domain Users"

I set up the NAP server and ruckus according to this guide - http://forums-archive.ruckuswireless.com/forums/8/topics/1278

Everything looked perfect, but I was getting a strange error on my test client when attempting to join the wireless network. Here's the error message:

A request was made to authenticate to a wireless network.

Subject:
Security ID:
Account Name: kyler.middleton
Account Domain: (removed)
Logon ID: 0x517503

Network Information:
Name (SSID): Test-RadiusWireless
Interface GUID: {b02ab000-4120-483a-8633-9c473d7a5004}
Local MAC Address: 40:F0:2F:4B:FA:33
Peer MAC Address: C4:01:7C:96:FA:18

Additional Information:
Reason Code: Explicit Eap failure received (0x50005)
Error Code: 0x40420110
EAP Reason Code: 0x40420110
EAP Root Cause String: Network authentication failed due to a problem with the user account

EAP Error Code: 0x40420110

I tore every piece of my NAP setup apart -- went through every line of relevant group policy, certificates, Ruckus wireless configuration, when I found the issue.

I had set the NAP server (Windows 2008R2) to use the certificate of the trusted domain CA to sign the EAP messages. Though the client had the same trusted CA installed locally, a trusted CA is not a certificate, so the client was rejecting the two-way EAP authentication.

The issue was fixed when I changed the certificate the Windows NAP server was using to sign the EAP messages. When I updated it to a certificate signed by the CA (rather than the CA's own certificate), everything started working immediately.

Good luck to you!
Kyler

P2P Open-Sourced Comm Infrastructure

1. People are everywhere.

2. Most (at least in first-world countries) own a smartphone that they bring everywhere.

3. Because of this, there exists a 'living' network of programmable mobile computers with long-range data-capable antennas.

4. An application could be written that uses these data antennas to communicate p2p.

5. If enough users opt-in, the need for established ISPs and TelCos is eliminated.

If non-repudiation and reliable encryption could be written as an open-source software tool that's easy enough for non-technical, crypto-smart people to use, we could:

* Seriously disrupt the business model of highly entrenched ISPs and TelCos

* Strengthen the communications infrastructure of the populated world

This is my new project. Suggestions welcome.

Backup Certs from Cisco ASA

Understand and Back up Certs from Cisco ASA

Backing up your certificates and private keys is a vital part of Cisco ASA administration. Here is a walkthrough to help even someone who isn't familiar with certificate types get their private key and certificate in clear-text.

This walk-through assumes someone has installed a certificate on your Cisco ASA. This means that the private and public keys and certificate are stored on the ASA, and we can go and get them!

Step 1: Get the Right Tools

Certificates and cryptography are sometimes arcane, because they are done by such a small subset of people. As such, the required tools to backup and convert certificates are not included by default on many linux distributions, and certainly not Windows.

The de facto standard in this area is called OpenSSL. You can read about it here: http://www.openssl.org/

Here's how to download and extract/install it on Windows:
1. There is a version of OpenSSL that is compiled for Windows. Click here to visit the download page.

2. Look down the list and download the appropriate OpenSSL.
Windows x32: Win32 OpenSSL v1.0.1e
Windows x64: Win64 OpenSSL v1.0.1e

3. Unzip the file to a location of your choosing. I put it directly on the C: drive for quick access.

Step 2: Get the Certificate from your Cisco ASA/Firewall

1. Open the Cisco ASDM

2. In the top left, click "Configuration"

3. In the bottom left, click "Device Management."

4. In the left column, expand "Certificate Management" and click "Identity Certificates"

5. Click on the certificate you'd like to export and click "Export" in the right column.

6. Choose where to save it. The file type is a .PFX, though Cisco's ASDM won't add that for you. Also add an encryption passphrase for two reasons:
a) Some versions of ASDM won't allow this export to succeed without an encryption passphrase due to a bug.
b) These are the keys to your crypto kingdom. If someone gets this file in cleartext, they can listen to any transmissions from you and pretend to be you on the internet -- not something you want to have to clean up.

7. Copy the certificate to a known location on your local computer.

Step 3: OpenSSL Magicks!, or: How to Convert from Base64 Encoding to Binary

Now's the time for some arcane OpenSSL commands. Cisco stores and exports PFX certificates in base64 encoding, which Windows (and most other applications) can't read. Let's convert it: 

1. Open the command line - WITH ADMIN privs. Else it won't be able to write to your computer, and you'll get all sorts of errors. 

2. Navigate to the root directory for OpenSSL. This is where you stored your files when you unzipped OpenSSL. You'll want to find the directory where OpenSSL.exe is. For me, that's: C:\OpenSSL-Win64\bin

3. Type your version of this command to convert the cert to binary. : 

openssl.exe base64 -in "C:\TempCerts\CiscoASACertExportBase64.pfx" -d -out "C:\TempCerts\CiscoASACertBinary.pfx"

If that all works, you'll see something like this:
WARNING: can't open config file: C:\OpenSSL\bin\openssl.cfg

Not to worry - check your output directory and make sure the certificate has some size other than 0kb. If so, you're golden.

Step 4 (optional): Get your Private Key and Cert in ClearText

WARNING: This is not required. Do this only if you need to get your private key and certificate in clear-text. Again, these are the keys to your kingdom and should not be distributed. 

1. Run a single command: 

openssl.exe pkcs12 -in "C:\TempCerts\CiscoASACertBinary.pfx" -out "C:\TempCerts\CiscoASACertBinaryPEM.pem" -nodes

Remember to add "-nodes" at the end to add in the private key information. You'll get a new file in your target directory called CiscoASACertBinaryPEM.pem. Open that file with notepad (or NotePad++, a great text editor for all kinds of stuff) and you'll see it all in clear-text. 

Good luck!

Ongoing troubleshooting and fixes for RANCID on Fedora

Ongoing troubleshooting and fixes for RANCID on Fedora

Keeping RANCID running is an ongoing process, and I'll document fixes as I go on here. Hope it helps others!

Replacing or Updating Equipment can Break RANCID
RANCID is no longer updating and won't run anymore!
How to add a new switch or Router to RANCID
cvs update

This is a security feature of SSH on Linux that can break RANCID's ability to check in with an IP when it lives on a new device (hardware upgrade) or has had its crypto keys regenerated. The logs for Rancid will read: "clogin error: Error: The host key for 10.10.10.1 has changed. Update the SSH known_hosts file accordingly."

Here's how to fix it:

su
#Enter super-user mode
su - rancid
#Changes the terminal context to the local rancid user. Change "rancid" to your specific local user if you use another account to run the rancid service.
ssh-keygen -R 10.10.10.1
#replace the IP with any other IPs which are failing to check in

The local saved SSH key for the device will be flushed, and the next time the device is reached out to, the linux environment will save the new key and remember it. This retains the great security feature, but resets the specific IP.

RANCID has a strange habit of not updating its databases occasionally. There are a few reasons for this including normal permissions issues, database credentials, but there's one issue that is unique to RANCID -- a lock file that will block all instances of rancid from running. 

The error message you'll see in your log files is self-explanatory: "Config diffs failed: /tmp/.networking.run.lock" You'll also see a manual run of rancid-run end immediately without writing any diffs to the cvs database. 

This file appears when rancid-run unexpectedly stops -- like if you ctrl+c interrupt a test run of rancid-run, like I did. Or if the server reboots mid rancid-run. 

The fix is to delete the lock file: 

su

rm /tmp/.networking.run.lock

Rancid will run normally, and you can test it immediately by: 

su

# To enter super-user mode. 

su - rancid

# To change the terminal to rancid user context. 

cd /usr/local/rancid/bin

./rancid-run

Once RANCID is fully set up and functioning, you'll invariably need to add more devices to it. Here's how:

su 
#to enter super-user mode which enables you to switch to other users
su - rancid
#Switches to the local rancid user, whom runs the local rancid service
vi /usr/local/rancid/var/nteworking/router.db
#this opens VI to edit the router.db text
Add the IP of the router/switch
:w
#Writes the changes
:q
#Quits vi and return to terminal
cd /usr/local/rancid/var/networking/configs
#tells the cvs change management tool to re-read the router.db list and start monitoring the new IP

RANCID on Fedora FIX- "Rlog output ended early. Expected RCS file..."

RANCID is Great!

RANCID, Really Awesome New Cisco confIg Differ (link) is an absolutely fantastic Cisco configuration backup system. It has native support for Cisco switches, routers, and firewalls, and stores revision history (through Subversion (link)). This allows SysAdmins to pull up the changes which were made, say, between last Tuesday when SystemX broke and today to pinpoint the problem update.

Rancid can be configured to reach out all the time through cron, although the official documentation from Rancid's admins has it check for changes every hour. If no changes have been made, no version is checked into the Rancid database.

I use Fedora 17/18 (link) for most all the linux tools on our network. It's an environment I'm comfortable with, and it's supported natively for almost every tool I've set up on it, RANCID included.

Except For A Strange Error Message...

I set up Rancid according to the official documentation here, but ran into a specific problem when accessing the web page of the 'networking' folder. The error message follows: Error: Rlog output ended early. Expected RCS file "/opt/cvs/project/file,v".

This error is documented on RANCID's web page (link), but there is no fix documented there or anywhere else I could easily find.

The Fix!

After using my google-foo for several hours, I stumbled upon a possible fix (link) - to add a configuration line to the viewvc.conf file that controls how ViewVC uses subversion and other tools. Here's how:

1. Follow the instructions for RANCID up to step 24.

2. Modify the following in the viewvc.conf file at /etc/viewvc/viewvc.conf
#cvs_roots = cvs:
root_parents = /usr/local/rancid/var/CVS : cvs
rcs_path = /usr/local/bin
address = <a href=mailto:helpdesk@mydomain.com>Email Helpdesk</a>
#use_enscript = 1
#enscript_path = /usr/bin/
#use_highlight = 1
#highlight_path = /usr/bin

3. Add the following line to the same file:
use_rcsparse = 1

4. Follow the rest of the instructions as listed. 

Enjoy!

RANCID really is a fantastic tool. I hope with this fix you're able to enjoy it also!

k