Monday, October 21, 2013

Backup Certs from Cisco ASA

Understand and Back up Certs from Cisco ASA

Backing up your certificates and private keys is a vital part of Cisco ASA administration. Here is a walkthrough to help even someone who isn't familiar with certificate types get their private key and certificate in clear-text.

This walk-through assumes someone has installed a certificate on your Cisco ASA. This means that the private and public keys and certificate are stored on the ASA, and we can go and get them!

Step 1: Get the Right Tools

Certificates and cryptography are sometimes arcane, because they are done by such a small subset of people. As such, the required tools to backup and convert certificates are not included by default on many linux distributions, and certainly not Windows.

The de facto standard in this area is called OpenSSL. You can read about it here:

Here's how to download and extract/install it on Windows:
1. There is a version of OpenSSL that is compiled for Windows. Click here to visit the download page.

2. Look down the list and download the appropriate OpenSSL.
Windows x32: Win32 OpenSSL v1.0.1e
Windows x64: Win64 OpenSSL v1.0.1e

3. Unzip the file to a location of your choosing. I put it directly on the C: drive for quick access.

Step 2: Get the Certificate from your Cisco ASA/Firewall

1. Open the Cisco ASDM

2. In the top left, click "Configuration"

3. In the bottom left, click "Device Management."

4. In the left column, expand "Certificate Management" and click "Identity Certificates"

5. Click on the certificate you'd like to export and click "Export" in the right column.

6. Choose where to save it. The file type is a .PFX, though Cisco's ASDM won't add that for you. Also add an encryption passphrase for two reasons:
a) Some versions of ASDM won't allow this export to succeed without an encryption passphrase due to a bug.
b) These are the keys to your crypto kingdom. If someone gets this file in cleartext, they can listen to any transmissions from you and pretend to be you on the internet -- not something you want to have to clean up.

7. Copy the certificate to a known location on your local computer.

Step 3: OpenSSL Magicks!, or: How to Convert from Base64 Encoding to Binary

Now's the time for some arcane OpenSSL commands. Cisco stores and exports PFX certificates in base64 encoding, which Windows (and most other applications) can't read. Let's convert it: 

1. Open the command line - WITH ADMIN privs. Else it won't be able to write to your computer, and you'll get all sorts of errors. 

2. Navigate to the root directory for OpenSSL. This is where you stored your files when you unzipped OpenSSL. You'll want to find the directory where OpenSSL.exe is. For me, that's: C:\OpenSSL-Win64\bin

3. Type your version of this command to convert the cert to binary. : 
openssl.exe base64 -in "C:\TempCerts\CiscoASACertExportBase64.pfx" -d -out "C:\TempCerts\CiscoASACertBinary.pfx"

If that all works, you'll see something like this:
WARNING: can't open config file: C:\OpenSSL\bin\openssl.cfg

Not to worry - check your output directory and make sure the certificate has some size other than 0kb. If so, you're golden.

Step 4 (optional): Get your Private Key and Cert in ClearText

WARNING: This is not required. Do this only if you need to get your private key and certificate in clear-text. Again, these are the keys to your kingdom and should not be distributed. 

1. Run a single command: 
openssl.exe pkcs12 -in "C:\TempCerts\CiscoASACertBinary.pfx" -out "C:\TempCerts\CiscoASACertBinaryPEM.pem" -nodes

Remember to add "-nodes" at the end to add in the private key information. You'll get a new file in your target directory called CiscoASACertBinaryPEM.pem. Open that file with notepad (or NotePad++, a great text editor for all kinds of stuff) and you'll see it all in clear-text. 

Good luck!

Friday, October 18, 2013

Ongoing troubleshooting and fixes for RANCID on Fedora

Ongoing troubleshooting and fixes for RANCID on Fedora

Keeping RANCID running is an ongoing process, and I'll document fixes as I go on here. Hope it helps others!

Replacing or Updating Equipment can Break RANCID
RANCID is no longer updating and won't run anymore!
How to add a new switch or Router to RANCID
cvs update

This is a security feature of SSH on Linux that can break RANCID's ability to check in with an IP when it lives on a new device (hardware upgrade) or has had its crypto keys regenerated. The logs for Rancid will read: "clogin error: Error: The host key for has changed. Update the SSH known_hosts file accordingly."

Here's how to fix it:

#Enter super-user mode
su - rancid
#Changes the terminal context to the local rancid user. Change "rancid" to your specific local user if you use another account to run the rancid service.
ssh-keygen -R
#replace the IP with any other IPs which are failing to check in
The local saved SSH key for the device will be flushed, and the next time the device is reached out to, the linux environment will save the new key and remember it. This retains the great security feature, but resets the specific IP.

RANCID has a strange habit of not updating its databases occasionally. There are a few reasons for this including normal permissions issues, database credentials, but there's one issue that is unique to RANCID -- a lock file that will block all instances of rancid from running. 

The error message you'll see in your log files is self-explanatory: "Config diffs failed: /tmp/" You'll also see a manual run of rancid-run end immediately without writing any diffs to the cvs database. 

This file appears when rancid-run unexpectedly stops -- like if you ctrl+c interrupt a test run of rancid-run, like I did. Or if the server reboots mid rancid-run. 

The fix is to delete the lock file: 
rm /tmp/

Rancid will run normally, and you can test it immediately by: 
# To enter super-user mode. 
su - rancid
# To change the terminal to rancid user context. 
cd /usr/local/rancid/bin
Once RANCID is fully set up and functioning, you'll invariably need to add more devices to it. Here's how:
#to enter super-user mode which enables you to switch to other users
su - rancid
#Switches to the local rancid user, whom runs the local rancid service
vi /usr/local/rancid/var/nteworking/router.db
#this opens VI to edit the router.db text
Add the IP of the router/switch
#Writes the changes
#Quits vi and return to terminal
cd /usr/local/rancid/var/networking/configs
#tells the cvs change management tool to re-read the router.db list and start monitoring the new IP

Saturday, October 5, 2013

RANCID on Fedora FIX- "Rlog output ended early. Expected RCS file..."

RANCID is Great!

RANCID, Really Awesome New Cisco confIg Differ (link) is an absolutely fantastic Cisco configuration backup system. It has native support for Cisco switches, routers, and firewalls, and stores revision history (through Subversion (link)). This allows SysAdmins to pull up the changes which were made, say, between last Tuesday when SystemX broke and today to pinpoint the problem update.

Rancid can be configured to reach out all the time through cron, although the official documentation from Rancid's admins has it check for changes every hour. If no changes have been made, no version is checked into the Rancid database.

I use Fedora 17/18 (link) for most all the linux tools on our network. It's an environment I'm comfortable with, and it's supported natively for almost every tool I've set up on it, RANCID included.

Except For A Strange Error Message...

I set up Rancid according to the official documentation here, but ran into a specific problem when accessing the web page of the 'networking' folder. The error message follows: Error: Rlog output ended early. Expected RCS file "/opt/cvs/project/file,v".

This error is documented on RANCID's web page (link), but there is no fix documented there or anywhere else I could easily find.

The Fix!

After using my google-foo for several hours, I stumbled upon a possible fix (link) - to add a configuration line to the viewvc.conf file that controls how ViewVC uses subversion and other tools. Here's how:

1. Follow the instructions for RANCID up to step 24.

2. Modify the following in the viewvc.conf file at /etc/viewvc/viewvc.conf
#cvs_roots = cvs:
root_parents = /usr/local/rancid/var/CVS : cvs
rcs_path = /usr/local/bin
address = <a>Email Helpdesk</a>
#use_enscript = 1
#enscript_path = /usr/bin/
#use_highlight = 1
#highlight_path = /usr/bin

3. Add the following line to the same file:
use_rcsparse = 1

4. Follow the rest of the instructions as listed. 


RANCID really is a fantastic tool. I hope with this fix you're able to enjoy it also!

Monday, January 21, 2013

Ping fails, NSLookup works fine - Windows Quirk

Hey all!

So I've been fighting tooth and nail with a problem on my network with DNS. The problem is this: users couldn't access some things by domain name. Sometimes. Not all the time.

When I'd use NSLookup on their machines, everything would resolve fine. And then ping would work -- probably due to the name being added to the cached network names on the computer.

The problem only affected Windows -- my Mac machines were able to resolve all things fine.

The DNS servers on these affected machines (handed out via DHCP) were as follows:
Primary DNS:
Secondary DNS:

We added the public DNS as the secondary DNS in the event that our company's DNS servers were ever down -- the machines could still connect to the internet, even if internal queries wouldn't be answered.

And that's where the problem lies. The Windows implementation of ping uses a round-robin selection method of DNS server (random), whereas NSLookup uses the ordered lookup method (ordered - 1, then 2).

When requests were randomly using the public DNS server, they had no idea what was, and failed the pings.

So in the future -- make sure you have all valid internal DNS servers on your company machines - Windows' non-standard implementation of "Ping" will cause you great pain and intermittent DNS failures.


Thursday, January 3, 2013

Physical Access is Data Access

I have a good friend that is focused on data security within networks, and he is constantly telling me: "Physical access is data access."

And I never believed it. We have encryption, firewalls, many of our servers are virtualized, etc. - I thought we were reasonably secure.

But today I got the opportunity to eat my own words -- I got the opportunity to break into a fully patched Windows 2008R2 server which was acting as a primary domain controller, ftp server, file-server, etc. for a small, very private network segment.

The scenario is this: I inherited a server set up for some very secure data. The local administrator password was lost, but it never mattered much -- until someone really needed some private data off of it.

In the course of 45 minutes -- 15 minutes of research and 30 minutes of hacking (if you could even call it that), I was able to gain complete administrative access to this (very private) windows server.

I followed instructions here:

Check it out for yourself -- but I would certainly recommend using a non-production machine, unlike this SysAdmin.