Tuesday, September 17, 2019

Cloud, DevOps: In Defense of Doing It Wrong

I am just awful at watching training videos and remembering the content. Which feels like a fair trade-off from the world for my ability to remember most things that I read with good fidelity. A place where this problem comes to an (unexpected) head is in DevOps and cloud architectures.

AWS and Azure are moving so fast they have trouble keeping up with written documentation. The written documentation usually exists, and isn't terrible, but is usually missing the most recent iterative developments.

Especially as a network engineer, the old model for architecting a system was to sit and read the architectural books. OSI standards don't really go and change on you, so the information can be presented in many different ways, boiled down by excellent authors, and presented in an accurate way that'll stand the test of time.

For instance, I know when I started studying for a CCIE (Cisco expert networking certification) one of the most recommended books to purchase and study was a volume that came out in the mid 90s, when I was learning how to tie my shoes and starting to read chapter books.
...the pace of iteration and invention within the technology space is increasing.
But I'm sure most of the folks reading this from technology fields feel this - the pace of iteration and invention within the technology space is increasing. Every week it seems like a new cloud functionality is released that layers up what that provider is offering, or a new devops framework, module, or practice is developed that can help increase the efficiency of what you and your team are doing.

That's not to say that the old methods and tools you're using will be deprecated (although it sometimes does!), but usually that you're designing for the state of the art from weeks, months, or years ago, and your competitors might be designing for the state of the art today.

It has a tremendously negative effective on the "tribal knowledge" of a team as new tools and practices are implemented. No longer does the cisco networking guru stay at the top of their game just by renewing their CCIE every few years with the same knowledge they had years ago - now we're trying to level up as fast as we can, and so is everyone else.
...we're trying to level up as fast as we can, and so is everyone else.

So Let's Do It Wrong

Which brings me to my point. The tools and technologies we use aren't going to slow down, and the documentation around them will continue to be sub-par. There's no way to become a paper expert at "cloud" or "devops" - the only way to get there as an expert is to DO it.

So deploy your own cloud, learn how your devops tools work by doing it wrong, and then iterating, and then doing it a bit less wrong, etc. Each time you do it wrong you learn a valuable lesson that can't have been learned elsewhere.
Each time you do it wrong you learn a value lesson that couldn't have been learned elsewhere.
So GO - build, break, iterate. Let's build this thing.

Sunday, September 15, 2019

Terraform - Iterative Subnet Module - AWS

Hey all!

Terraform has the ability to call modules, which are snippets of terraform code that can be passed information to build resources. Generally these modules enshrine best practices, and help to keep your DevOps teams on-track in terms of resource nomenclature, structure, and security guidelines.

Modules also have the ability to contain multiple resources, and be passed a "count" variable, which will lead to several similar resources being constructed.

In this blog I'll share code for an AWS subnet and route-table association module that can accept a count variable, and build "n" number of subnets. New subnets are as easy as updating the count variable and updating the list of subnets passed to the module.

But enough talking about the cool thing, let's build it.

Modules Overview

Creating a module is as easy as saying "hey, terraform, call this module, here's where it lives, like this: 

That'd work just fine, however there's much more power in modules when we pass information to them. Imagine calling an ec2 module and passing the subnetID, AMI, size of subnet, etc to it. That makes the module a lot more powerful.

You can imagine how extensible this solution is. For example, here's the finish module call we'll be building today:

We're calling a module that builds subnets, we're telling it a "group" to use in naming of the subnets, availability zones, subnet addresses, route table, oh my! The module has to be written to accept and use all these values, which is the tricky part. So let's jump into that.

Iterative Modules - The Secret Sauce

Writing modules isn't terribly hard - you tell it what values to accept from the caller, and assign those values to fields that terraform accepts, and boom, you have a functional module. However, that module can only build a single resource. Telling it to build several resources in a cogent way is some engineering, some creativity, and some luck. It starts with the "count" parameter.

Count is a built-in terraform variable that terraform uses to know how many times to loop over the same resource and build it several times. This built-in loop within terraform is exposed to the resource via an "index" attribute that tells the loop how many times the loop has run.

If that's made your head spin, that's okay - we'll walk through it with examples. First, let's take the subnet module a few lines at a time. Here's the start of our subnet module, and we're building a subnet resource. We're also using the count attribute we just talked about, but rather than assigning it by hand (which would work fine, as long as we remember to update it for each new subnet!), we're going to pass the number of items in the subnet_addresses list instead using the length function. Basically, we're saying to terraform, "If there are 3 subnets in the subnet_addresses list, iterate 3 times and build 3 subnets."

You can also see that the VPC ID is just specified like you would in any normal non-iterative module. That's because that value will remains static across all the subnets we build.

Let's add one more line - the variable that tells terraform the CIDR address of this subnet. Now, this item needs to change depending on which iteration of the loop we're on. So we tell terraform to pick up the variable passed to this module called subnet_addresses using the element function, of index whatever number of the loop we're on. So if we pass this module an array of "1, 2, 3" and the loop is on iteration 3, it'll pick out the 3rd item in the list, and use the value "3".

You can use those index values to interpolate also. It makes a lot of sense to me to name the subnets starting at 1, rather than 0 where a computer starts counting, so we interpolate the loop index value, and add 1 (so 0 becomes 1, and 1 becomes 2).

And boom, that's our iterative module. However, we also need to associate the subnet to a route-table. Check out the second half of this module and see if you can pick out where iterative looping is done, where interpolation and value modification is done, and follow along.

Notice also that when we're calling the subnet module above, we're only 2 availability zones and 2 route tables. How is the subnet module dealing with only having 2 values when it iterates 5 times? The answer is that lists inherently loop. So if a list contains value "A, B" and the loop is called 5 times, loop 1 will be A, loop 2 will be B, loop 3 will be A, and so on.

Outputs - Splat!

Normally, a module can output a static number of resources, so outputs are easy to write. However, in an iterative module, any number of resources can be created. Outputs don't support the "count" parameter in the same way resources do, so we have to use another creation of Terraform's - the Splat expression.

Here's what our output looks like in this subnet module:

So within the subnet module, that's how you'd export ALL subnet IDs. But say you wanted to reference one of the subnets from the main.tf? It'd look like this - referencing the array value and module name from within the main.tf file:

Go Build It Yourself!

You can find all the functional code at GitHub here: https://github.com/KyMidd/TerraformAwsSubnetModule

Go build some cool iterative modules! Imagine building a dozen ec2 instances, and standardizing their names, security settings, and managing them as a single flexible unit. The possibilities are endless.

Good luck out there.