Cloud, DevOps: In Defense of Doing It Wrong

I am just awful at watching training videos and remembering the content. Which feels like a fair trade-off from the world for my ability to remember most things that I read with good fidelity. A place where this problem comes to an (unexpected) head is in DevOps and cloud architectures.

AWS and Azure are moving so fast they have trouble keeping up with written documentation. The written documentation usually exists, and isn't terrible, but is usually missing the most recent iterative developments.

Especially as a network engineer, the old model for architecting a system was to sit and read the architectural books. OSI standards don't really go and change on you, so the information can be presented in many different ways, boiled down by excellent authors, and presented in an accurate way that'll stand the test of time.

For instance, I know when I started studying for a CCIE (Cisco expert networking certification) one of the most recommended books to purchase and study was a volume that came out in the mid 90s, when I was learning how to tie my shoes and starting to read chapter books.

...the pace of iteration and invention within the technology space is increasing.

But I'm sure most of the folks reading this from technology fields feel this - the pace of iteration and invention within the technology space is increasing. Every week it seems like a new cloud functionality is released that layers up what that provider is offering, or a new devops framework, module, or practice is developed that can help increase the efficiency of what you and your team are doing.

That's not to say that the old methods and tools you're using will be deprecated (although it sometimes does!), but usually that you're designing for the state of the art from weeks, months, or years ago, and your competitors might be designing for the state of the art today.

It has a tremendously negative effective on the "tribal knowledge" of a team as new tools and practices are implemented. No longer does the cisco networking guru stay at the top of their game just by renewing their CCIE every few years with the same knowledge they had years ago - now we're trying to level up as fast as we can, and so is everyone else.

...we're trying to level up as fast as we can, and so is everyone else.

So Let's Do It Wrong

Which brings me to my point. The tools and technologies we use aren't going to slow down, and the documentation around them will continue to be sub-par. There's no way to become a paper expert at "cloud" or "devops" - the only way to get there as an expert is to DO it.

So deploy your own cloud, learn how your devops tools work by doing it wrong, and then iterating, and then doing it a bit less wrong, etc. Each time you do it wrong you learn a valuable lesson that can't have been learned elsewhere.

Each time you do it wrong you learn a value lesson that couldn't have been learned elsewhere.

So GO - build, break, iterate. Let's build this thing.
kyler

Network Engineering is Dying (Except at Cloud Providers)

Hey all!

This past week I spoke to a recruiter for one of the gang of 4 largest companies in tech. That term refers to Google, Amazon, Facebook, Apple (and sometimes Microsoft). The recruiter pitched me on a network engineering role - something that I've happily done for years now.

For the past 20+ years, network engineering teams from most companies have maintained the networks that connect computers which serve up every internet service we interact with each day. Network engineers make sure redundancies exist for the inevitable failures of a network that spans the globe, and they verify the health of all the hardware devices and interfaces which run the network.

A common analogy for network engineering is building the roads for the application "cars" to drive upon.

These jobs have been stable and profitable, integral to the growth and stability of any company that wants to use the internet to drive its business (read: all of them). Most would jump at the opportunity to take any job at these companies. These jobs sparkle on resumes, and even if the day-to-day is similar to most other jobs in the industry, the looming profile these companies have in the news cycle mean it'd be foolish to write off an employment opportunity like this one.

However, the world of network engineering is changing. Many would say dying.

With the exception of maybe a dozen companies on the planet, nearly every company is moving away from physical data centers. IT orgs struggle with the long lead times required to make changes in physical data centers. Purchasing hardware, organizing cabling standards, cooling, 24x7 staffing, and dozens of other concerns are simply avoided by moving to the cloud.

Ironically, the only companies who aren't decreasing their data center footprint? Cloud providers. 

Because of the increased demand, cloud providers are growing their physical data centers at an incredible rate. This requires hiring network engineers, data center engineers, and others with the skillsets to grow them in a scalable way.

The gilded cage of skill-set lock-in

The problem, of course, is skillset lock-in. Not only do most of the gang of 4 famously build their own tooling, but their business model is shared by almost no other company on the globe - to build world-spanning data centers and massive internet-scale networks.

Only cloud providers still invest in physical data centers - and the skillsets required to run them.

Spending time in your career at one of these companies in a department focused on these legacy networks is a dead-end in a career because of this skillset lock-in. It'll be difficult for the folks locked into these positions to leave the very small network of a dozen or so companies that provide these massive clouds and take just a job at just about anywhere else, because these other companies are looking for reliability engineers (SREs), DevOps engineers, and any number of other software-defined cloud computing experts that need entirely different skillsets than those harbored within these divisions at the gang of 4.

If you have the opportunity to work in these divisions at cloud providers, good luck to you! Their famously great pay and benefits are nothing to scoff at. But I hope you consider my points above about career lock-in. Your career must be played as a strategic long-game, and I worry these jobs might be the wrong move.

Best of luck out there.
kyler

I'm An Engineer (I Think)

I spend a great deal of time thinking about where my career is headed. Am I making the right decisions, what steps are required to get to the 'next level,' etc. It's symptomatic of my age - the late 20-something disease of 'careers are hard.'

How Did I Even Get Here?

I didn't intend to even DO InfoTech. I initially wanted to be a librarian at the college level, but when I did my research and saw that librarians with 25 years of experience made a salary equivalent to an IT member with less than 5, I realized that I had an opportunity.

I'm not one to do something solely for money. Money doesn't keep you warm at night, it's not something to live for, but I grew up without much of it. I realize that money represents more than a shiny new car. It can mean buying lunch each day, not having to worry about rent or a mortgage, or living month to month and desperately hoping that no one gets sick, or the car doesn't break down. Money matters, shallow as that topic sometimes seems.

But Everyone Knows How Computers Work!

I honestly didn't realize that I had what it takes to "make it" in IT. I assumed that everyone grew up with computers, and took electronics apart to see how they worked.

In my mind, computer competency and unquenchable curiosity is akin to literacy - it's something that I'm sure a few folks miss out on, but most people have it, right?

It turns out that most like modern gadgets, but when it comes to building and fixing them, they prefer not to know. It's more comforting to them to believe that some things in their life are just.. magical.

But I prefer to know. I must know. I think that's all that engineering is - curiosity that can't be turned off. Whether it's mechanical, structural, electrical - it just means that this particular group of people aren't satisfied to know that things work. They have to know how and why.

Hard Problems Vs. Wicked Problems

I enjoy the hell out of network engineering. It's fascinating to study the infrastructure in an environment and tweak it to do all the tricks that are required for complex integrations and automatic failover. There are lots of things to learn in this field, but I feel that most of the tasks here have a solution.

I'm drawn to wicked problems. Wicked problems are not just difficult problems, they are problems that likely don't have a good solution. I feel that computer security is one of those problems.

InfoSec Engineers challenge themselves to think of their environment from every angle. After all, threats come at any organization from many directions, and only one path has to be open to a hacker or disgruntled user to create an incident. And the good guys have to defend every single possible angle of attack, a wickedly unfair dynamic that is regardlessly true.

What's Next?

First, I certified in Security+ (one week of study), Certified Ethical Hacker (3 weeks of study), and now I'm studying for the CISSP (aiming to be done in January). I'm also using my free time to pick up programming languages to start building my fluency - Python, Bash, Perl. Structural languages to build functional tools. 

I'm preparing my tool belt for my next step. Let's get to it. 

kyler

I Might Be a Felon, and You Could Be, Too!

Inspired by this article in wired, I decided to put down this rant

Pres. Obama will recommend "hacking" be upgraded to a racketeering crime, which means associates of hackers and those which knowingly spread illegal information can be charged with up to 20 years in prison with a felony charge.

This sounds like a good thing, right? Hackers are bad guys. Well, the stuff that I post all the time (on software issues, company data leaks, etc.) falls under this law. If I were ever to come under scrutiny by any gov't entity, they could lean on me with racketeering or hacking charges.

And if you have ever reposted anything I put on here about software vulnerabilities, guess what? You can be charged with felony racketeering. Maybe our prison cells will be close together!

Please, vote against this law if it comes up to that. Exploring and helping fix software does not deserve a felony charge. We cannot continue to persecute our computer security experts and expect to remain a secure country in a globally connected world.

Kyler Middleton, Computer Security Enthusiast and Potential Felon

Software-Defined Networking in the Physical World

I just finished a short lunch-hour style training on the Cisco Prime platform, and I've got to say - they get it. No one wants to play with switches all day.

That's a hard thing to admit.I've spent 500+ hours studying for Cisco certifications, and I'm currently prepping for a CCIE, a certification solely in route/switch that costs over $1,500 per attempt in a far-away state. But I'm confident saying it:

I don't want to work on switches all day. 

I want to work on a management platform that can completely configure, control, and monitor my route/switch infrastructure. I want to use this platform to correlate threat data and alert me when my ports are filling up, let me know when to plug in another cable between my switches so it can configure a port-channel, and I want it to have the smarts to shut down a network storm.

If Cisco enabled CDP to allow for simple management, the management appliance could reach new devices by chaining through CDP-enabled devices to set up the management connection on new devices. It could also allow for recovery if a command renders a device unreachable.

In short, I want my network devices to act like lightweight APs - they have some ability to operate themselves, but they are able to act intelligently and seamlessly when controlled by a central management station.

This is entirely possible, and appears to be where Cisco is going.

Let me start over - Cisco's Prime platform is a super-charged monitoring platform. It's able to discover and monitor all your routers, switches, ASA, etc with great reporting. It's able to apply templates that you've built to a switch once you've given the switch a base config and a routable IP that the management device can reach.

But still, you need to know the configs. You need to know which things to turn on and turn off and why. I'm sure there's an argument there that only the experts in the field, who've invested significant time and money (wink wink), should be doing this anyway, so who cares?

If Cisco is able to realize this dream, they'll have brought the fantastic benefits of Software Defined Networking (SDN) into the physical world. 

They'd allow existing businesses with millions invested in 'heavy,' self-managed switches to upgrade their switches software and enable central, intelligent management. Costs for expertise in networking will go down, and the cost of running a business will go down. Constantly mutating security attacks can be more quickly identified by taking the aggregate data and correlating it in a central place.

More importantly, if Cisco doesn't move to do this quickly, someone else will. 

You can read Cisco's information on Cisco Prime here: http://www.cisco.com/c/en/us/products/cloud-systems-management/prime.html
kyler

CCIE Lab - Back To Business

My post to finish out my CCIE journey is here.

Alright -- I passed the CCIE route/switch written. Which is awesome. There is a great deal of information and concepts in that test, and all the tests and study that've brought me to this point. I'm proud of myself and my accomplishment. My total is around 1600 hours of study thus far.

Damn right I'm proud

But that's behind me. I've taken two weeks off, and now it's time to start getting ready to face the dragon. The CCIE route/switch lab is a heck of a test. 8 hour long practical in a distant state, with many different rules and requirements that cost points and can fail an exam.

The failure rate on the first attempt is 90-95%
The failure rate for any attempt is around 80%

I've been using video on demand through CBTNuggets, and I've bought several books that cover the certification. Every single one says to expect to fail. The odds are so harsh, and it's so hurtful to a tester's confidence to expect to succeed and then fail, they all say that the first try is a 'practice' attempt.

A $1600 practice attempt!

I don't intend to take this test twice, but I'm not going to let that eat at me. The goal is to get the certification. If I have to test twice, 3x, 4x, I'm going to get it. But my wallet and my lovely wife would be thankful if, on this first attempt, I'd fall into that exceedingly qualified 5-10% that pass on the first try.

So it's back to studying. My studying schedule for around the past 18 months has been unchanged:
* M-F - study over my lunch hour, then 7-9:30 after work
* Saturday: Study 9-3, date night with Lindsey
* Sunday: 9-3, house chores, groceries, cooking

A part-time job would consume less time

It's almost 30 hours each week if I stick to my schedule. I invariably skimp on a single night, or take some extra time off to spend time with friends, but it's close.

The goal is still to get the CCIE number by Thanksgiving of this year. It's an aggressive, challenging goal. If I make it, I'll be around CCIE #50,000 in the world.

Wish me luck!
-kyler

P2P Open-Sourced Comm Infrastructure

1. People are everywhere.

2. Most (at least in first-world countries) own a smartphone that they bring everywhere.

3. Because of this, there exists a 'living' network of programmable mobile computers with long-range data-capable antennas.

4. An application could be written that uses these data antennas to communicate p2p.

5. If enough users opt-in, the need for established ISPs and TelCos is eliminated.

If non-repudiation and reliable encryption could be written as an open-source software tool that's easy enough for non-technical, crypto-smart people to use, we could:

* Seriously disrupt the business model of highly entrenched ISPs and TelCos

* Strengthen the communications infrastructure of the populated world

This is my new project. Suggestions welcome.

The Open Source Company

When setting up a company, one of the primary concerns we face is that our income must exceed our cost of doing business. For many businesses, that cost is difficult to calculate -- any single user touches a half dozen services per day at minimum -- probably more, especially for smaller companies where users wear so many hats.

Open-source software can be a great answer to that question. For those outside of the software/computer world, open-source software is software whose authors have donated their time and work for the better good. Anyone may use and improve the product as long as they attribute the previous authors. That allows great tools to be freely available to smaller companies with shallower pockets than larger companies.

So how far can open-source software take you? Paid software from full-time development companies must be superior, right? Well -- yes and no. Paid software generally has 5 times as many features, but the core features are typically available and solid on both.

Here's what I've built for my companies:

Open Source/Free:

Wiki - MediaWiki
The same source-code that Wikipedia uses, most people are familiar with the layout.

Instant Messaging - OpenFire XMPP server with Pidgin desktop client
Both OpenFire from Jive Software and Pidgin are open-source software which are freely available. They both support the open-source XMPP chat protocol which allows us to federate (link up to) other software and protocols. OpenFire in particular is just fantastic server software, and pidgin is an old favorite of mine.

Server virtualization platform - ESXi 
Though not open-source, their base-level server virtualization is available free of charge. This version allows up to 32gb of memory to be dynamically divvied up between any number of machines. This dynamic sharing of resources is both efficient for enterprises and green for the environment - fewer machines gobbling up power. Just fantastic software.

Voice conferencing: Asterisk software (AsteriskNow prepackaged)
Asterisk is an open-source framework for communication -- telephony, conferencing, and (I believe) video telephony. To ease set-up, we installed a pre-packaged kit which includes most features and involves little customization...at least in theory. Setting it up has been a beast, and its still not entirely working -- not a recommended product. If you know a great open-source conferencing platform, send it my way!

Application/Network monitoring: Nagios with a Monarch GUI front-end. 
Nagios is the monitoring framework which is fully SNMP compatible and includes many pre-packaged monitoring tools which makes it fantastic. That said -- it's just the framework. Unless you want to do all your configuration in CLI, you need a GUI. We selected Monarch because a partner highly recommended it, and it is extensive. Recommended, at least for a business of our size (small to medium).

IT Ticketing: GLPI
GLPI is an open-source ticketing system geared towards small to (small-)medium businesses. It can also do cataloging of computers and other data with an OCS module that fully integrates. We compared a few other products which were much more complex and harder for users to ... use. GLPI is a form -- category, urgency, title and summary. Submit. Users get it. And that's what we're looking for here. Highly recommended for smaller businesses.

Server OS: Fedora from Red Hat software
Red Hat has an interesting business model -- release great software, open-source, free of charge. Then charge for support. As a small-business, I'm interested. Great, stable, freely available software. As a medium business, my software is highly integrated with their platform and I need a support contract. I'm snared. And everybody wins. Highly stable, great platform -- Recommended. VM that and play today.

Paid Software:

There is always that paid software which cannot be avoided without significant risk or detriment to your company. Here are some of the examples and reasons why.

Desktop OS: Windows 7 x64
Almost always the most polarized software, the Windows OS has come a long way. I'm (obviously) a big proponent of open-source software, but Microsoft has done a great job of patching security holes and adding features to their desktop OS. We're still evaluating Win8, but it looks like something that could really take off if some usability tweaks are implemented.

Server OS: Windows Server 2008R2
With the stability issues of the past few releases behind it, 2008/2008R2 is pretty good. Very graphical, it's a great option for file-serving to the Windows OS. That's the primary reason we run it. When something goes wrong, it takes quite a bit of experience and digging to find the logs which reflect the problem, and even then troubleshooting is difficult. When the code is closed, you are forced to turn to the company which wrote it (Microsoft) for answers. And they're not too easy to reach, either. If we could effectively run file-serving from a *nix, we'd be planning a migration today.

Document Editing/Email Client: Microsoft Office 2010
I've spent hours and hours playing with Thunderbird and OpenOffice/LibreOffice and they simply do not stand up to Microsoft Office in terms of features (in a huge way) and compatibility with Microsoft Office (which, of course, is everywhere).
A wildcard here is Google Docs, but it simply isn't there yet in terms of features or compatibility either. That said, it's the most interesting product in the bunch here, and I sincerely hope they pick up their game. There's at least one interested party here.

Email Server: Microsoft Exchange 2010
The most graphical and user-friendly email server I've seen is also the most popular and therefore the most techs have experience with. Honestly haven't even looked at other services (all cloud-based services are out due to security concerns) because once it is configured, it simply runs.

Switching/ASA: Cisco
When a company practically (literally) invents the category, it's difficult to find great software which can compete. And especially in the security/backbone category, its important to use software which others are experienced with for quick troubleshooting if (when) problems arise. This puts us firmly in Cisco's (very costly) grasp.

Telephony: Cisco
Call-Manager Express (and at some point Call Manager Business Edition) is the platform that carries our voice traffic. The platform is stable and somewhat feature-rich, but difficult to configure, at least for us on CME. Entirely CLI, we're able to easily dig into difficult issues, but even the simplest of changes can cause serious problems, so there's a difficult trade-off here. If we can find a better product that can provide stable, feature-rich telephony in an open-source package, we'll consider migration.

I hope my thoughts and ramblings can help you select better tools for your projects.
Thanks!
kyler

About Me

Hello, World.

So.. why a blog?

I started this blog for two reasons. The most important of which is -- I have learned so much from many, many random posts online as I work on different systems. I feel I should give back and post my own revelations, tribulations, and solutions to tough and confusing issues so others can benefit from the time and effort I've put in. Not doing good -- just doing my best to even the score a little bit.

The least important of the why is so I can remember my own solutions.

Information wants to be Free

It's my deep belief that information wants to be free, and this is a small way that I can help. The world has become flat and cheap in terms of accessibility of information and the ability for someone to both learn a great deal and improve the lives of others, even at great distance. I hope to do that, and I hope this blog encourages others to do that, also.

Open Source Software Rocks

The sense of community, flexibility, and general do-gooder-ness of open-source software encourages me to use it and contribute when I can. I also work for a small company with a careful budget, so that type of software is great for our situation.

Please, contribute!

We all help one another by contributing. I'll doubtlessly post information that's misleading, wrong, etc. -- tell me! We, as a group, improve one another. I'll do my best to listen to posts and update my entries to provide correct information (and I'll do my best to attribute corrections to you!)

How to find me

LinkedIn: www.linkedin.com/in/kylermiddleton
Twitter: https://twitter.com/kymidd
GitHub: https://github.com/KyMidd
Or use the contact me form on the right sidebar!