Network Engineering is Dying (Except at Cloud Providers)

Hey all!

This past week I spoke to a recruiter for one of the gang of 4 largest companies in tech. That term refers to Google, Amazon, Facebook, Apple (and sometimes Microsoft). The recruiter pitched me on a network engineering role - something that I've happily done for years now.

For the past 20+ years, network engineering teams from most companies have maintained the networks that connect computers which serve up every internet service we interact with each day. Network engineers make sure redundancies exist for the inevitable failures of a network that spans the globe, and they verify the health of all the hardware devices and interfaces which run the network.

A common analogy for network engineering is building the roads for the application "cars" to drive upon.

These jobs have been stable and profitable, integral to the growth and stability of any company that wants to use the internet to drive its business (read: all of them). Most would jump at the opportunity to take any job at these companies. These jobs sparkle on resumes, and even if the day-to-day is similar to most other jobs in the industry, the looming profile these companies have in the news cycle mean it'd be foolish to write off an employment opportunity like this one.

However, the world of network engineering is changing. Many would say dying.

With the exception of maybe a dozen companies on the planet, nearly every company is moving away from physical data centers. IT orgs struggle with the long lead times required to make changes in physical data centers. Purchasing hardware, organizing cabling standards, cooling, 24x7 staffing, and dozens of other concerns are simply avoided by moving to the cloud.

Ironically, the only companies who aren't decreasing their data center footprint? Cloud providers. 

Because of the increased demand, cloud providers are growing their physical data centers at an incredible rate. This requires hiring network engineers, data center engineers, and others with the skillsets to grow them in a scalable way.

The gilded cage of skill-set lock-in

The problem, of course, is skillset lock-in. Not only do most of the gang of 4 famously build their own tooling, but their business model is shared by almost no other company on the globe - to build world-spanning data centers and massive internet-scale networks.

Only cloud providers still invest in physical data centers - and the skillsets required to run them.

Spending time in your career at one of these companies in a department focused on these legacy networks is a dead-end in a career because of this skillset lock-in. It'll be difficult for the folks locked into these positions to leave the very small network of a dozen or so companies that provide these massive clouds and take just a job at just about anywhere else, because these other companies are looking for reliability engineers (SREs), DevOps engineers, and any number of other software-defined cloud computing experts that need entirely different skillsets than those harbored within these divisions at the gang of 4.

If you have the opportunity to work in these divisions at cloud providers, good luck to you! Their famously great pay and benefits are nothing to scoff at. But I hope you consider my points above about career lock-in. Your career must be played as a strategic long-game, and I worry these jobs might be the wrong move.

Best of luck out there.
kyler

Sync Terraform Config and .tfstate for Existing AWS Resources

Hey all!

Terraform is a great (and dominant) infrastructure automation tool. It is multi-cloud, can build all sorts of resources, and in some cases supports API calls to build resources before the native tooling from cloud providers does.

However, it's dependent on a state file that is local, and only reflects resources created by terraform, and a local configuration file to describe resources. It's not able to reach out to a cloud account and create a configuration and .tfstate file based on the existing resources that were built via another method. Or at least, it isn't able to yet. The scaffolding for this functionality exists within Terraform for the AWS cloud, and is called the "import" functionality. It's able to map a single existing resource to a single configuration block for the same resource type and fill in the info, which is of course a manual and tedious process. And imagine if you have hundreds (or thousands!) of resources. It isn't a feasible way to move forward.

Terraforming (link) is a wrapper around terraform and is able to map multiple resources at the same time to configuration blocks, as well as build .tfstate files for multiple existing resource types. Still, it's a little awkward to use - only a single resource type is able to be imported at the same time, and if a command is run against a non-existing resource type (say you don't have a batch configuration, and run a sync against the batch resource), it wipes out the existing .tfstate entirely, removing your progress.

Clearly, the tools could use some help. So I wrote some. I imagine both of these tools (terraform import & terraforming) will eventually get this same functionality. In fact, both of these tools are open source, and I'll work on adding this functionality natively to both of these tools via PRs.

However, for the time being, I'm publishing my code which permits:

• Creating from scratch a .tfstate file for every terraforming supported resource in an AWS region
• Creating a single (monolithic) configuration file for each existing resource in an AWS region

This code assumes you don't have an existing .tfstate file - in fact, it wipes out your existing local .tfstate file and builds a new one. So please back up your .tfstate and configuration files before running this tool. 

However, if your'e new to terraform and want to sync the configuration to an existing AWS region and look at the config for all the resources that exist there, this is a neat shortcut. 

Rather than post the code here and update each time I (or you! The code is open source) update it, I'll post a link to the public github repo. 

https://github.com/KyMidd/TerraformingAllResources

I hope it's useful to you. Please add any corrections, comments, and feature additions you'd like via pull requests. And if you know how to update the terraform or terraforming source code to add these functionalities and make my code obsolete, please do so! That would be the best case scenario. 

Thanks all. Good luck out there! 

kyler

Azure DevOps, Terraform Validation and Linting

Hey all!

This post is part of a series on Azure DevOps CI/CD, which we've used to integrate with Azure Cloud, build a terraform deployment, and commit code to build resources.

In part 1 of this series, we:

• Learned several DevOps and Azure Cloud terms
• Signed up for an Azure Cloud and Azure DevOps (ADO) account
• Created an Azure Cloud Service Connection to connect Azure Cloud and ADO
• Initialized a new git repo in ADO
• Installed git on our machine (if it didn't have it already)
• Created an SSH key and associated it with our user account
• Cloned our (mostly empty) git repo to our computers

In part 2 of this series, we: 

• Create terraform base code and a .gitignore file for terraform
• Used git on our local computer to create a branch, add the files to it, and push the files and branch to our Azure DevOps git repository
• Merged the code changes into our master branch
• Created a build and release pipeline for terraform to validate and push code out to our cloud environment

This blog will continue from where part 2 ended, so if you're following along, walk through part 1 and 2 to build all the above from scratch. Or just read along and watch me do it - I've included lots of pictures. 

Despite all of these neat cloud pipelines, git branches, and terraform automation, most of our pipeline is still manual. Code must be: 

1. Build locally on a machine and tested before committing to the master - code isn't tested before committing to master and running a build and release
2. Running the build pipeline manually once new code is committed to master
3. Running the release pipeline to test code

There's also significant parallelization drawbacks to this method. Say I write a bad terraform commit and push it to master. The release pipeline starts failing for everyone, for any reason, until this particular problem is sorted. This might work for a few developers who have time to commit to this project, but imagine dozens of devs working on a project. There would constantly be faulty code that blocks others - production would grind to a halt.

Software engineering, from which DevOps borrows many of its practices and methodologies, handles this problem by integrating testing before a merge, during the pull request phase of a commit. Then if problems are created, it's only in the individual workspace of a branch, rather than the master that is shared by all developers. 

We're able to build something similar in Azure DevOps, and this blog will also walk through that process. So what are we waiting for? Let's get started! 

Break (Up) the Release Process

Right now our release pipeline has a single stage, and that stage does everything in Terraform - it stages the code, it init(ializes) the environment, validates the terraform, and applies it. 

That's not going to scale, so let's go into the release and edit the first single stage. What we'll do is remove all the "apply" steps, so the first stage is all about testing and validation. So let's update the first stage's title to match. 

We also want to remove the "terraform apply" step - don't worry, we'll add it to a future stage. Click on the apply step in the left column and then in the top right, click on the trash can labelled "remove" to delete that step. Hit save, and click "Pipeline" in the top left. 

We could go ahead and build an entirely new stage with just what we need, but there's an easier way. In the stages area, hover your mouse over the "Terraform Validate" step that we just updated. Click on "Clone" and you'll see a new stage appear. 

Boom, you are a master of efficiency. Click on the "1 job, 5 tasks" in that second (far-right) stage, that starts with "Copy of..."

This second stage is all about applying the Terraform. Before we get to this stage, the Terraform code will be validated by the first stage, so no need to validate again. However, we do need to "init" the environment again - each stage is handled by an entirely new container. So let's pop in there and remove that pesky testing. 

What you have left will look something like this - make sure the last step is Apply. It's also a good idea to name this step "Terraform Apply" to help keep track of what each stage is doing. 

Hit "Save" in the top right and we now have two discrete stages - the first stage tests, and the second stage applies. Which doesn't do a lot of good yet, because the stages are automatically linked and apply without any input from anyone! So let's tell stage 2 (Apply) to wait for our ok before continuing. 

In the top left click on pipeline again. Find the "Terraform Apply" step and hover your mouse over it. Click on the lightning bolt - that's the "Pre-Deployment Conditions" instructions. 

There's a lot going on here, but for now feel free to minimize the "Triggers" section to make this place a lot less busy. Find the "Pre-deployment approvals" section and slide the slider to the Enabled position. Azure DevOps will require that someone approve this step before continuing, and you can lock down who has the authority. In our imaginary business we're the only current employee, so add yourself as an approve. Then click save in the top right. 

Let's test it! In the top right, click on "Create Release" to run this release again. The first step will run normally - we triggered it to run, but something new will happen. The second stage WON'T run immediately. It'll wait patiently for us to "Approve" it. 

This is important for several different use cases. For one, we can just plain validate the "Terraform Plan" is only showing the actions we expect it to do. Second, you could assign more senior resources the ability to approve rollouts, or potentially an InfoSec team, platform team, etc. 

Click into the running release pipeline and the GUI is very clear - the "Terraform Validation" (and plan!) stage ran successfully, and the second stage "Terraform Apply" is pending approval, and won't run without our say so. Feel free to click into the Logs on the Terraform Validation step and make sure plan is only doing what we say, then click "Approve" on the second step, and the apply will continue on. 

Woot! We've broken out our flow of work so there is a separate testing stage from an apply stage. That'll be important for the next few automated linting test items we add. 

Limit the Blast Radius - Pull Requests

Anyone who's worked in infrastructure for any length of time is familiar with the phrase "limiting the blast radius." It's a way to frame changes or processes in an adversarial light - if it all goes wrong, how wrong can it go? The idea is to build processes, protections, and to train the team so that WHEN things go wrong, they don't go terribly wrong. 

We can take that principal to heart here and limit someone's ability to commit changes directly to our master branch, where it is both harder to back-out and avoids the normal review process that code (and any infrastructure changes) should go through. 

In most cases, it shouldn't be allowed except in unusual circumstance. So let's require Pull Requests (PRs). Click on Repos --> Branches and find your "master" branch. It's created automatically when the repo is initialized. Click on the 3 dots to pull up all the settings that apply to that branch, and click on "Branch Policies". 

Check the box next to "Require a minimum number of reviews" and change the minimum number to 1 (since our company only has a single employee!). If you are building this for a larger team, set the number wherever you'd like (limit 10). Since we'll be approving our own changes, make sure to also check the box next to "Allow users to approve their own changes." That's not a best practice for a real enterprise, but for our lab it'll do just fine. 

Hit save to make the policy live. But before we leave this page, we should make one other change. 

Automatic Branch Builds

Normally when you run a "build" job, it'll stage artifacts for all the release jobs from the master branch. It is possible to stage artifacts from a branch, or a particular commit, but it's complex to manage, and can go wrong (rolling out changes from an out-dated version of code, for instance). 

However, that's exactly what we need to do for branches - stage their code in a special artifacts area where it can be tested without messing with our master branch, or with any of the other branches where folks might be working in parallel. 

So it's convenient we're already in this settings page for the master branch. In it you'll find this exact setting that we need to enable. Find the section labelled "Build validation" and click the "Add build policy" button. 

Find your "Build pipeline" in the drop-down menu for Terraform building and select it. Leave the other options on their defaults. The one we're interested in in particular is the "Trigger" being set to automatic. 

What we've just done is made sure that every time a Pull Request targeting our master terraform branch is created or even updated, a new build process will run. That by itself won't test our code, but it's a step in the right direction. Now we just need to tell our release testing stages to also run when the code is staged in a pull request. So let's do that. 

Automatic Release Testing

What we'd like to happen when a PR is created or updated is for our terraform validation stages to run, but not our apply stages. Running an apply stage against a branch that might differ from the master branch is a recipe for chaos. We can do that by tagging a flag in two places - on the "artifacts gather" part of our release pipeline, as well as on each stage to include it in our automated build processes. 

First, go to Pipelines --> Releases and click edit on our terraform release. Find the Artifacts section and click on the lightning bolt - it's labelled the "Continuous deployment trigger" menu. 

In the menu that pops up we're going to change a few things. First, we need to make sure that this release pipeline automatically grabs the new build artifacts when a PR builds them, and then executes this pull request. 

Target branch filter is required to be set. It's asking when to trigger this automated pull request release. Sometimes code is staged in a branch to be ready for a release date. However in our case, most PRs will be created against the master branch directly, so select it as the target branch. 

At the very bottom of this picture above, note something interesting in yellow. Though we've turned on an automatic build and release, it's telling us that neither stage of this release pipeline is going to be executed. 

ADO is cautious - it doesn't want us accidentally over-writing our production infrastructure. So we'll need to go into each stage we want to enable for this automated release process and flip a flag to say "yes, include this stage in the process." Click on the lightning bolt on the "Terraform Validation" / planning step. 

Under the Triggers section, find the "Pull request deployment" slider and move it to Enabled. This is the flag to include this stage in our automated PR testing process. 

Hit save to make the changes live. 

If you were to head back to the Release pipeline's Artifact continuous deployment settings, where you last saw "0 of 2 stages are enabled..." you'll now see "1 of 2" stages are enabled for pull request deployment. 

Did it Work? 

Let's test it so we can see it in action! On our local machine let's make sure we've checkout out our master git branch, then pull any changes we might have missed. This is silly in our lab - we're the only ones in it! But it's a good practice to get into for production environments. 

Let's start a new branch (make sure to not use any spaces in your branch name). 

Edit our main.tf terraform file to add something new - say a virtual network and a new subnet. Something like this would do the trick: 

provider "azurerm" {}

terraform {

backend "azurerm" {}

}

resource "azurerm_resource_group" "rg" {

name = "testResourceGroup"

location = "westus"

}

resource "azurerm_virtual_network" "WeAreAwesome" {

  name                = "vNetNewWeAreAwesome1"

  address_space       = ["10.0.0.0/16"]

  location            = "westus"

  resource_group_name = "${azurerm_resource_group.rg.name}"

}

resource "azurerm_subnet" "test" {

  name                 = "testsubnet"

  resource_group_name  = "${azurerm_resource_group.rg.name}"

  virtual_network_name = "${azurerm_virtual_network.WeAreAwesome.name}"

  address_prefix       = "10.0.1.0/24"

}

Copy and paste the above and save it over your main.tf file. Then head back to your command line and add the updated file to your change (git add .), commit the change to your new branch (git commit -m "adding a vNet and subnet" and then push your new branch, commits and files in tow, to your git master, with "git push origin testing-unit-testing". 

Back on ADO, head to Repos --> Branches to find your new branch. All the way at the right there is a column called "Pull Request." If you hover over it, you'll see the option to create a new pull request, called "New pull request". Click it, and then click "Create" on the next page. 

And we'll see the build process go, but nothing else! What did we miss?

Interestingly, the build process has to run a single time to initialize the ADO back-end for build status. But we can fix it! Head back to Repo --> Branches and find the master branch. Hover over it to find the three dots and then click on Branch Policies. Click it and then look for a section called "Require approval from additional services." This vaguely named section is able to receive input from other processes and give a go/no-go on whether a PR can proceed to be merged. It's exactly what the doctor ordered here. Click on "Add status policy". 

Under "Status to check", you'll now see your release pipeline name. Select it, and check either "Required" (which I heartily recommend, even in a lab environment) or "Optional". The selections are literal - if a required build shows a failed state (like a terraform init or validate failed), the PR CANNOT be merged into master until the code is fixed and a release testing passes. 

Now head back to your PR (Repo --> Pull Requests in the left bar). The build status won't show up right away, but we can tell our build to run again, which will trigger our new required policy. Hover over the "Build succeeded" line and click "Queue build". The build will run once more (in the future this will happen automatically, this is just a first-time gremlin), and then the release pipeline's validate step will run automatically a report a status. 

If all goes well you'll see a couple of happy green check marks - 1 for the build status (staging artifacts) and 1 for the release status (doing linting, terraform init, validate, plan). 

Wait, What Did We Just Do? 

What we just did is build automated linting and validation. We set PRs to be required for all changes to our terraform codebase, and put in automated building and terraform initialization and validation to any PR creation or update. We also made it so that any failure in build or validation would block PRs from being merged into our master tree, where they might cause problems for others. 

Really what we did is enable a large team of developers to work on the same code concurrently without getting in each other's way, and permit developers of this infrastructure code to move fast and break things in their own branches, with their own testing pipelines, without breaking anything in production. Pretty good for a day's work. 

Good luck out there! 

kyler

Let's do DevOps: Build an Azure DevOps Terraform Pipeline Part 2/2

Hey all!

In part 1 of this series, we:

• Learned several DevOps and Azure Cloud terms
• Signed up for an Azure Cloud and Azure DevOps (ADO) account
• Created an Azure Cloud Service Connection to connect Azure Cloud and ADO
• Initialized a new git repo in ADO
• Installed git on our machine (if it didn't have it already)
• Created an SSH key and associated it with our user account
• Cloned our (mostly empty) git repo to our computers

Phew! That was a heck of a post. In this post, we'll get to do all the cool stuff our prep work from last time enabled. We're going to create a build and release terraform pipeline, check in code, permit staged deployments to validate what steps are going to be taken and approve them, then push real resources into our Azure Cloud from our terraform scripts. Let's get started. 

New Scary Terms Part Deux

Before we walk the walk, let's learn to talk the talk. 

• Azure DevOps (ADO): A Continuous Integration / Continuous Deployment tool, it will be the tool which executes our automation and actually "runs" the Terraform code. 
• Git terminology
• Master branch: The shared source of truth branch where finished code is committed. Usually code is iterated on in branches, and only "merged" into the master when it is ready. 
• Branch: When you want to make changes to a repo, you can push changes directly to the master. But you probably need some time to iterate on your code, and if many people are doing this simultaneously, the result is... chaos. To solve this problem, git has the concept of "branches" which is your own private workspace. You make any changes you want, push the code to the server as much as you like, and you won't affect others until it's time to merge the code. 
• Commit: Each time you make some changes to your code in a branch and want to save the code back to the server, you create a "commit". Generally descriptive notes are attached to commits so other team members know what you changed in each commit. 
• Pull Request: When a branch is ready to be merged into the master branch, we create a "pull request." This is a collection of commits that generally notifies the rest of your team that your changes are ready to be evaluated, commented on, and ultimately approved. Once approve, the code is ready to be merged. 
• Merge: When a pull request is ready, the code is "merged" into the master. Others will "pull" the code to sync their local repo to the master, and work from there. This keeps all code in sync. 
• ADO Build Pipeline: ADO has several different automation pipelines. A build pipeline fetches code from wherever it is stored and stages it where it can be consumed by "deployment" pipelines. This sounds (and is) boring when the code is stored locally in ADO, but build pipelines are capable of fetching code from all sorts of unlikely places, including GitHub, private git repos, public repos, etc. 
• Artifacts: When ADO runs a build pipeline to collect code, it puts it in a staging area and calls it an "artifact." 
• ADO Release Pipeline: These automation pipelines run the code that is staged as artifacts and can execute it in containers, and do just about any other automated action you can think of. These actions are arranged as pre-built tiles that can be dragged into sequence for an easy and graphical automation build experience.

Finally, Let's See Some Terraform

Terraform is a descriptive language based on HCL programming language and is intended to describe an end-state of your infrastructure. You say, "I want a server that looks like this" and Terraform's job is to work with a cloud API to make it happen. Terraform supports many different integrations, which Terraform docs call "providers". This includes things you'd expect like AWS Cloud, Azure Cloud, Google Cloud Platform, but also others you wouldn't expect MySql, F5, and VMware. 

Here's a sample script provided by HashiCorp to get Terraform started: 

provider "azurerm" { } terraform { backend "azurerm" {} } resource "azurerm_resource_group" "rg" { name = "testResourceGroup" location = "westus" }

Find the location on your computer where you synced the git repo to. Create a folder called "terraform" and save the above script into it as main.tf.

It's also a good idea to create a ".gitignore" (including the leading period) file in your repo. The recommended one is stored here, and looks like this:
# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log

# Ignore any .tfvars files that are generated automatically for each Terraform run. Most
# .tfvars files are managed as part of configuration and so should be included in
# version control.
#
# example.tfvars

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
#
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

Time to Git Moving

Once saved, go to the location of your repo in your command line and type "git status". You'll see a few interesting items: 

1. We're on branch master

2. Git recognizes there's a new "untracked" file we can add

So let's add it. Type "git add ." (git add period) and then "git status" again, and let's see if it's changed. Yeah, it has! It now shows the folder and file in green, and it's under the "changes to be committed" section. That's great. 

However, we're still on the "master" branch. You can technically commit and push this change from here, but best practice is to spin up a branch and push it from there. To do that, type "git checkout -b NewBranchName" and hit enter. Then hit "git status" again and we'll check how things look. 

Great, the file is added, and we now see we are on branch "NewBranchName". 

Okay, we have changes staged, and we're on a branch. Let's "commit" the staged changes to this branch so they're all packaged up and ready to be pushed to our server. To do that, type git commit -m "Initial main.tf commit". 

Up until this point, all changes have been only on our machine, but we're ready to do some cloud DevOps, so let's push these files to our git server, Azure DevOps. To do that, type "git push origin NewBranchName". The "NewBranchName" is the name of our local branch. You should see results like the below. 

Now we're done on our computer! Let's switch to Azure DevOps to check the file. 

Azure DevOps Pull Request and Merge

Head to ADO - https://dev.azure.com/ and get into your project. Then click on Repos --> Branches. You should see the master branch, as well as your new shiny branch! Others are now able to see your code, but you haven't opened a PR yet to get them to approve your new code so it can be used. Let's do that. 

Click into the new branch we just built, and you can see Azure DevOps is prompting us for our next step, so let's take it. Click on "Create a pull request" at the top. 

Here you can fill out all sorts of information about your proposed change. DevOps teams (and software engineering teams) do all sorts of cool stuff with git, but we'll keep it simple. 

At the top, ADO wants to know where we want to propose our code get merged into. We want to put it into the master. The title and description are free-form. For this exercise, you know exactly what you're doing, but imagine if there's 20 people (or way, way more) working concurrently - descriptions and titles are helpful stuff. 

There aren't any reviewers to approve our change, so we can leave that blank. Work items are automations in ADO that we won't use (yet!). Click "Create" 

Now the PR is opened, woot! Any reviewers added would be notified to review your code. In this case, we're confident in our change, so let's click "Complete". If this was someone else's change, or if a change requires multiple approvers, we'd only be able to click "Approve". But our policies are pretty lax in our lab environment, so we can skip and step and jump right to "Complete". Click it! Your code is now merged into the master branch. 

Click on Repos --> Files and you'll be able to see your code in the repo. 

Create an Azure DevOps Build Pipeline

All this cool new code can't be ingested by a release pipeline until it's built into an "artifact", and placed in a staging area. To do this, we need to create a "build" pipeline. Click on Pipelines --> Builds and then click on "New Pipeline". 

$Msft is pushing for these build pipelines to be built via code, which isn't terribly intuitive. What's more intuitive are draggable tiles to build actions. To do that, click on "Use the classic editor" at the bottom of the list. 

Everything here looks fine - we want to pull code from the local Azure Repo git, we're in our project, in the default repo (same name as the project), and we want to grab code from the master branch. Click on "Continue".

We're going to create our own build pipeline, so click "Empty job" and we'll be dropped into a drag-and-drop environment to add actions to our build pipeline. 

The pipeline we have pulled up is empty. It shows an "Agent job 1" which means a linux container will spin up and do... nothing. It's up to us to add some actions to our linux builder. We're only going to add two actions - a "copy files" action and a "Publish build artifacts" action. 

Click the plus sign on "Agent job 1" and find each of these actions in the right column. Make sure the names match, or your configuration for each will be different than what we'll walk through. 

Click on the "Copy files to:" job, and you'll see some information is being prompted on the right side. Add the source folder of "terraform" Contents can stay as the two asterisks - it'll copy all files. The target folder should have this string: "$(build.artifactstagingdirectory)terraform". That'll copy recursively our repo in the master branch into the root of where our release pipeline runs at. 

Also expand the Advanced options at the bottom and check both "Clean Target Folder" and "Overwrite". This'll keep the artifact staging directory from getting jumbled with all the artifacts we'll push in there in the future. 

The Publish Artifacts step can stay as it is. At the top, hit "save and queue", and then "save and queue" again on the pop-up that appears. The build will start running in the background. 

Click on "Builds" under pipelines in the left column to jump back to our list of builds. Under the History tab, you'll see the build. It only takes a few seconds, so it's probably completed. Hopefully you see the green check box as shown in this snapshot below. 

Click on the build if you'd like to see the steps that you configured being run. It'll look something like this: 

Release Pipeline - Let's run some Terraform!

Now that code is staged as artifacts, it can be consumed by our release pipelines. Let's get a new pipeline started and give it some terraform commands. In the left column, click on Pipelines --> Releases. Then click on the blue button that says "New Pipeline". 

ADO will offer to help us build it, but we're going to build it ourselves. At the very top, click on "Empty job" to start with an entirely empty release pipeline. 

Release pipelines have two phases. The first phase is gathering - it needs to know which files to operate on. We just created a build pipeline which staged some artifacts, so let's select it. Click on "Add an artifact". 

Under the "Source (build pipeline)" select our build pipeline in the dropdown. My was called the name of my project -CI. Then click on "Add". 

The second phase actually spins up containers, or runs code, or does all sorts of other cool stuff. These jobs are called "tasks" and they exist within stages. For now, let's click on "1 job, 0 task" - the blue underlined words there. 

Let's name our pipeline - simply click in the name area at the top and type a new name. I called mine "Terraform". Then save your pipeline by clicking the save button (floppy disk icon) in the top right.

Just as with the build pipelines, click the plus sign on the "Agent job". Search for terraform in the jobs at the right and find "Terraform Build and Release Tasks", an entry by Charles Zipp. Click "Get it Free". 

It'll pop you into a new window to the Azure Marketplace to accept this tool. Click "Get it free", then follow the workplace to sign up - it's free, no information or money changes hands, etc. At the end, it'll pop you back into Azure Devops. 

Go find your project again - click "edit" in the top right to jump back into our build mode - then click the plus sign on our agent to add some jobs. This time when you type "Terraform", you'll have a few new options! 

Add a "terraform installer" step first, then three "terraform cli" steps - 1 more step than I included in the picture below. 

Click through the terraform steps. The first step, Terraform Installer, that says "Use Terraform" and a version, defaults to 0.11.11, which is an older version of Terraform. It's a good idea to update it to the most recent release of terraform. You can find that on HashiCorp's main terraform page: https://www.terraform.io/

Looks like the most recent version is 0.12.3.

Update the version string on the terraform installer to 0.12.3 and then click on the first terraform validate CLI step, called "terraform validate". Terraform wouldn't actually let us do a validate yet - the first step in any terraform deployment is to do an "init". So let's change the command to that. 

The defaults work for the first few sections. The important one to change is the "backend type" away from local, which would mean store the state on the container, which is destroyed at the end of the step - that's not a great place to store our .tfstate. Change it to "azurerm" and we'll get a whole new section to configure - it'll store the .tfstate file in an Azure Cloud storage blob, where it can be referenced later. 

For the "Configuration Directory", click on the three dots and expand the folders - what you're viewing is the staged artifacts. You should see the "drop" folder where artifacts are built and stored. 

There's lots to configure in the AzureRM Backend Configuration, but we'll work through it together. The first step is the Backend Azure Subscription - this is the Service Connection we created in part 1. if you don't see anything here, try to hit the circle to the right of the drop-down. Select your Azure Cloud subscription. 

Check the box to build a backend if it doesn't exist - it doesn't, so we'll need ADO to build this storage blob for us. 

Use anything for the resource name - I used "azurerm_remote_storage". Set the other values as shown below except for the "Storage Account Name". That must be globally unique. I'd recommend throwing in some numbers or using your own name - remember that string must be all lower-case, no hyphens or underscores, 3-24 characters. 

Now click to the next Terraform CLI step - it still shows "Terraform Validate", and that's where we're going to leave it. The only option that needs to change here is the Configuration Directory - use the same value you used in the "terraform init" step. 

Click on the third step and let's update it to "Terraform plan". It's a good idea to have this step in your pipelines before any  "terraform apply" so you can make sure everything looks good before continuing. 

Make sure to set the same configuration directory as the last two steps, and to set your Azure cloud subscription again. Once complete, hit "save" at the top. 

Create Release!

We're ready to create our first release. This will run our release pipeline and test our steps. Expect a few things to be broken at first - that's normal for anything new! You can work through the descriptive error messages and fix it all. 

Click "Create Release" in the top right - the rocket ship, then click "Create" and our release will start running in the background. 

At the top, click on "Release-1" to jump to the release page, where we can view the results of our testing and see the response to our commands. 

Hover over "Stage 1" and click on "Logs" to view all the steps and watch them go through. 

 

Once it's complete, you'll see status. Here's what it looks like if everything went well. If you see any errors, click on the step to see the logs. 

Let's click on the "terraform plan" step so we can see terraform's output: 

That looks about like what we expected, so let's roll it out. Let's go back into our release pipeline and click edit, and add one more "terraform CLI" step to do a "terraform apply". 

Make sure to set the same configuration directory as the other steps, and to set your subscription again. 

Hit save, then re-run your release (remember the rocket in the top right?)

This time our release continues right on past the "terraform plan" step right to a "terraform apply" step, and builds us some resources. 

Here's the output of the build command. 

BOOM

What's Next? 

Now that all our machinery is built and confirmed working, we can start iterating on our terraform codebase. We can add some modules, define subnets, VPNs, servers, storage, security policies, and on and on. 

Each time you commit code, merge it into master via a PR (or commit directly to master), run a build, then run your release, and your resources will be built via Terraform in a cloud environment. 

There's lots more cool stuff to do - we can build in some safety checks where a "terraform plan" is executed, then waits for permission to continue to a "terraform apply", we can do automatic terraform builds and validations on PR opening, we can give you 3 magic wishes... Okay, maybe not that last one, but lots of cool stuff is coming. Check out future blog posts for more cool stuff.

EDIT: And it's been posted, check it out here: https://www.kylermiddleton.com/2019/06/azure-devops-terraform-unit-testing.html

Good luck out there! 

kyler