Friday, June 28, 2019

Let's do DevOps: Build an Azure DevOps Terraform Pipeline Part 2/2

Hey all!

In part 1 of this series, we:
  • Learned several DevOps and Azure Cloud terms
  • Signed up for an Azure Cloud and Azure DevOps (ADO) account
  • Created an Azure Cloud Service Connection to connect Azure Cloud and ADO
  • Initialized a new git repo in ADO
  • Installed git on our machine (if it didn't have it already)
  • Created an SSH key and associated it with our user account
  • Cloned our (mostly empty) git repo to our computers
Phew! That was a heck of a post. In this post, we'll get to do all the cool stuff our prep work from last time enabled. We're going to create a build and release terraform pipeline, check in code, permit staged deployments to validate what steps are going to be taken and approve them, then push real resources into our Azure Cloud from our terraform scripts. Let's get started. 

New Scary Terms Part Deux

Before we walk the walk, let's learn to talk the talk. 
  • Azure DevOps (ADO): A Continuous Integration / Continuous Deployment tool, it will be the tool which executes our automation and actually "runs" the Terraform code. 
  • Git terminology
    • Master branch: The shared source of truth branch where finished code is committed. Usually code is iterated on in branches, and only "merged" into the master when it is ready. 
    • Branch: When you want to make changes to a repo, you can push changes directly to the master. But you probably need some time to iterate on your code, and if many people are doing this simultaneously, the result is... chaos. To solve this problem, git has the concept of "branches" which is your own private workspace. You make any changes you want, push the code to the server as much as you like, and you won't affect others until it's time to merge the code. 
    • Commit: Each time you make some changes to your code in a branch and want to save the code back to the server, you create a "commit". Generally descriptive notes are attached to commits so other team members know what you changed in each commit. 
    • Pull Request: When a branch is ready to be merged into the master branch, we create a "pull request." This is a collection of commits that generally notifies the rest of your team that your changes are ready to be evaluated, commented on, and ultimately approved. Once approve, the code is ready to be merged. 
    • Merge: When a pull request is ready, the code is "merged" into the master. Others will "pull" the code to sync their local repo to the master, and work from there. This keeps all code in sync. 
  • ADO Build Pipeline: ADO has several different automation pipelines. A build pipeline fetches code from wherever it is stored and stages it where it can be consumed by "deployment" pipelines. This sounds (and is) boring when the code is stored locally in ADO, but build pipelines are capable of fetching code from all sorts of unlikely places, including GitHub, private git repos, public repos, etc. 
  • Artifacts: When ADO runs a build pipeline to collect code, it puts it in a staging area and calls it an "artifact." 
  • ADO Release Pipeline: These automation pipelines run the code that is staged as artifacts and can execute it in containers, and do just about any other automated action you can think of. These actions are arranged as pre-built tiles that can be dragged into sequence for an easy and graphical automation build experience.

Finally, Let's See Some Terraform

Terraform is a descriptive language based on HCL programming language and is intended to describe an end-state of your infrastructure. You say, "I want a server that looks like this" and Terraform's job is to work with a cloud API to make it happen. Terraform supports many different integrations, which Terraform docs call "providers". This includes things you'd expect like AWS Cloud, Azure Cloud, Google Cloud Platform, but also others you wouldn't expect MySql, F5, and VMware. 

Here's a sample script provided by HashiCorp to get Terraform started: 
provider "azurerm" { } terraform { backend "azurerm" {} } resource "azurerm_resource_group" "rg" { name = "testResourceGroup" location = "westus" }

Find the location on your computer where you synced the git repo to. Create a folder called "terraform" and save the above script into it as main.tf.

It's also a good idea to create a ".gitignore" (including the leading period) file in your repo. The recommended one is stored here, and looks like this:
# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log

# Ignore any .tfvars files that are generated automatically for each Terraform run. Most
# .tfvars files are managed as part of configuration and so should be included in
# version control.
#
# example.tfvars

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
#
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

Time to Git Moving

Once saved, go to the location of your repo in your command line and type "git status". You'll see a few interesting items: 
1. We're on branch master
2. Git recognizes there's a new "untracked" file we can add


So let's add it. Type "git add ." (git add period) and then "git status" again, and let's see if it's changed. Yeah, it has! It now shows the folder and file in green, and it's under the "changes to be committed" section. That's great. 


However, we're still on the "master" branch. You can technically commit and push this change from here, but best practice is to spin up a branch and push it from there. To do that, type "git checkout -b NewBranchName" and hit enter. Then hit "git status" again and we'll check how things look. 

Great, the file is added, and we now see we are on branch "NewBranchName". 


Okay, we have changes staged, and we're on a branch. Let's "commit" the staged changes to this branch so they're all packaged up and ready to be pushed to our server. To do that, type git commit -m "Initial main.tf commit". 


Up until this point, all changes have been only on our machine, but we're ready to do some cloud DevOps, so let's push these files to our git server, Azure DevOps. To do that, type "git push origin NewBranchName". The "NewBranchName" is the name of our local branch. You should see results like the below. 


Now we're done on our computer! Let's switch to Azure DevOps to check the file. 

Azure DevOps Pull Request and Merge

Head to ADO - https://dev.azure.com/ and get into your project. Then click on Repos --> Branches. You should see the master branch, as well as your new shiny branch! Others are now able to see your code, but you haven't opened a PR yet to get them to approve your new code so it can be used. Let's do that. 

Click into the new branch we just built, and you can see Azure DevOps is prompting us for our next step, so let's take it. Click on "Create a pull request" at the top. 


Here you can fill out all sorts of information about your proposed change. DevOps teams (and software engineering teams) do all sorts of cool stuff with git, but we'll keep it simple. 

At the top, ADO wants to know where we want to propose our code get merged into. We want to put it into the master. The title and description are free-form. For this exercise, you know exactly what you're doing, but imagine if there's 20 people (or way, way more) working concurrently - descriptions and titles are helpful stuff. 

There aren't any reviewers to approve our change, so we can leave that blank. Work items are automations in ADO that we won't use (yet!). Click "Create" 


Now the PR is opened, woot! Any reviewers added would be notified to review your code. In this case, we're confident in our change, so let's click "Complete". If this was someone else's change, or if a change requires multiple approvers, we'd only be able to click "Approve". But our policies are pretty lax in our lab environment, so we can skip and step and jump right to "Complete". Click it! Your code is now merged into the master branch. 


Click on Repos --> Files and you'll be able to see your code in the repo. 

Create an Azure DevOps Build Pipeline

All this cool new code can't be ingested by a release pipeline until it's built into an "artifact", and placed in a staging area. To do this, we need to create a "build" pipeline. Click on Pipelines --> Builds and then click on "New Pipeline". 


$Msft is pushing for these build pipelines to be built via code, which isn't terribly intuitive. What's more intuitive are draggable tiles to build actions. To do that, click on "Use the classic editor" at the bottom of the list. 


Everything here looks fine - we want to pull code from the local Azure Repo git, we're in our project, in the default repo (same name as the project), and we want to grab code from the master branch. Click on "Continue".


We're going to create our own build pipeline, so click "Empty job" and we'll be dropped into a drag-and-drop environment to add actions to our build pipeline. 


The pipeline we have pulled up is empty. It shows an "Agent job 1" which means a linux container will spin up and do... nothing. It's up to us to add some actions to our linux builder. We're only going to add two actions - a "copy files" action and a "Publish build artifacts" action. 

Click the plus sign on "Agent job 1" and find each of these actions in the right column. Make sure the names match, or your configuration for each will be different than what we'll walk through. 


Click on the "Copy files to:" job, and you'll see some information is being prompted on the right side. Add the source folder of "terraform" Contents can stay as the two asterisks - it'll copy all files. The target folder should have this string: "$(build.artifactstagingdirectory)terraform". That'll copy recursively our repo in the master branch into the root of where our release pipeline runs at. 

Also expand the Advanced options at the bottom and check both "Clean Target Folder" and "Overwrite". This'll keep the artifact staging directory from getting jumbled with all the artifacts we'll push in there in the future. 


The Publish Artifacts step can stay as it is. At the top, hit "save and queue", and then "save and queue" again on the pop-up that appears. The build will start running in the background. 

Click on "Builds" under pipelines in the left column to jump back to our list of builds. Under the History tab, you'll see the build. It only takes a few seconds, so it's probably completed. Hopefully you see the green check box as shown in this snapshot below. 


Click on the build if you'd like to see the steps that you configured being run. It'll look something like this: 


Release Pipeline - Let's run some Terraform!

Now that code is staged as artifacts, it can be consumed by our release pipelines. Let's get a new pipeline started and give it some terraform commands. In the left column, click on Pipelines --> Releases. Then click on the blue button that says "New Pipeline". 

ADO will offer to help us build it, but we're going to build it ourselves. At the very top, click on "Empty job" to start with an entirely empty release pipeline. 


Release pipelines have two phases. The first phase is gathering - it needs to know which files to operate on. We just created a build pipeline which staged some artifacts, so let's select it. Click on "Add an artifact". 

Under the "Source (build pipeline)" select our build pipeline in the dropdown. My was called the name of my project -CI. Then click on "Add". 


The second phase actually spins up containers, or runs code, or does all sorts of other cool stuff. These jobs are called "tasks" and they exist within stages. For now, let's click on "1 job, 0 task" - the blue underlined words there. 


Let's name our pipeline - simply click in the name area at the top and type a new name. I called mine "Terraform". Then save your pipeline by clicking the save button (floppy disk icon) in the top right.


Just as with the build pipelines, click the plus sign on the "Agent job". Search for terraform in the jobs at the right and find "Terraform Build and Release Tasks", an entry by Charles Zipp. Click "Get it Free". 


It'll pop you into a new window to the Azure Marketplace to accept this tool. Click "Get it free", then follow the workplace to sign up - it's free, no information or money changes hands, etc. At the end, it'll pop you back into Azure Devops. 


Go find your project again - click "edit" in the top right to jump back into our build mode - then click the plus sign on our agent to add some jobs. This time when you type "Terraform", you'll have a few new options! 

Add a "terraform installer" step first, then three "terraform cli" steps - 1 more step than I included in the picture below. 


Click through the terraform steps. The first step, Terraform Installer, that says "Use Terraform" and a version, defaults to 0.11.11, which is an older version of Terraform. It's a good idea to update it to the most recent release of terraform. You can find that on HashiCorp's main terraform page: https://www.terraform.io/

Looks like the most recent version is 0.12.3.

Update the version string on the terraform installer to 0.12.3 and then click on the first terraform validate CLI step, called "terraform validate". Terraform wouldn't actually let us do a validate yet - the first step in any terraform deployment is to do an "init". So let's change the command to that. 

The defaults work for the first few sections. The important one to change is the "backend type" away from local, which would mean store the state on the container, which is destroyed at the end of the step - that's not a great place to store our .tfstate. Change it to "azurerm" and we'll get a whole new section to configure - it'll store the .tfstate file in an Azure Cloud storage blob, where it can be referenced later. 


For the "Configuration Directory", click on the three dots and expand the folders - what you're viewing is the staged artifacts. You should see the "drop" folder where artifacts are built and stored. 


There's lots to configure in the AzureRM Backend Configuration, but we'll work through it together. The first step is the Backend Azure Subscription - this is the Service Connection we created in part 1. if you don't see anything here, try to hit the circle to the right of the drop-down. Select your Azure Cloud subscription. 

Check the box to build a backend if it doesn't exist - it doesn't, so we'll need ADO to build this storage blob for us. 

Use anything for the resource name - I used "azurerm_remote_storage". Set the other values as shown below except for the "Storage Account Name". That must be globally unique. I'd recommend throwing in some numbers or using your own name - remember that string must be all lower-case, no hyphens or underscores, 3-24 characters. 


Now click to the next Terraform CLI step - it still shows "Terraform Validate", and that's where we're going to leave it. The only option that needs to change here is the Configuration Directory - use the same value you used in the "terraform init" step. 

Click on the third step and let's update it to "Terraform plan". It's a good idea to have this step in your pipelines before any  "terraform apply" so you can make sure everything looks good before continuing. 

Make sure to set the same configuration directory as the last two steps, and to set your Azure cloud subscription again. Once complete, hit "save" at the top. 

Create Release!

We're ready to create our first release. This will run our release pipeline and test our steps. Expect a few things to be broken at first - that's normal for anything new! You can work through the descriptive error messages and fix it all. 

Click "Create Release" in the top right - the rocket ship, then click "Create" and our release will start running in the background. 

At the top, click on "Release-1" to jump to the release page, where we can view the results of our testing and see the response to our commands. 


Hover over "Stage 1" and click on "Logs" to view all the steps and watch them go through. 

 

Once it's complete, you'll see status. Here's what it looks like if everything went well. If you see any errors, click on the step to see the logs. 


Let's click on the "terraform plan" step so we can see terraform's output: 


That looks about like what we expected, so let's roll it out. Let's go back into our release pipeline and click edit, and add one more "terraform CLI" step to do a "terraform apply". 

Make sure to set the same configuration directory as the other steps, and to set your subscription again. 


Hit save, then re-run your release (remember the rocket in the top right?)

This time our release continues right on past the "terraform plan" step right to a "terraform apply" step, and builds us some resources. 


Here's the output of the build command. 


BOOM

What's Next? 

Now that all our machinery is built and confirmed working, we can start iterating on our terraform codebase. We can add some modules, define subnets, VPNs, servers, storage, security policies, and on and on. 

Each time you commit code, merge it into master via a PR (or commit directly to master), run a build, then run your release, and your resources will be built via Terraform in a cloud environment. 

There's lots more cool stuff to do - we can build in some safety checks where a "terraform plan" is executed, then waits for permission to continue to a "terraform apply", we can do automatic terraform builds and validations on PR opening, we can give you 3 magic wishes... Okay, maybe not that last one, but lots of cool stuff is coming. Check out future blog posts for more cool stuff.

EDIT: And it's been posted, check it out here: https://www.kylermiddleton.com/2019/06/azure-devops-terraform-unit-testing.html

Good luck out there! 
kyler

Sunday, June 23, 2019

Let's do DevOps: Build an Azure DevOps Terraform Pipeline Part 1/2

Hey all!

Cloud environments are entirely software-defined. Cloud has some inherent downsides (opaque hardware troubleshooting!), but the pros vastly outweigh the cons. This includes something very cool - automated, desired-state tooling. Which in non-buzzword-speak means you can write some text configs to define what you'd like your environment to look like, point your tool at a cloud environment, and bam, you have new resources.

Switching from the data center world to the cloud is a big undertaking, so I wanted to walk through a simple workflow that uses some automation tooling and makes it dead simple to start building your cloud skillset.

As an important aside - every tool and service we'll use in this walkthrough is entirely free. No better cost than free, right? Both of the Azure tools we're going to sign up for require a credit card, but won't charge you unless you take a further action - converting your account from "Free Trial" to "Pay as You Go."

I'm in infrastructure, not software engineering!

I know, I know. You have half a dozen Cisco certifications (at no small cost!), you're working on a Juniper certification, and you're great at this job. You don't feel like you need to learn this new and scary skillset. I get it, I was in those exact same shoes about a year ago.

I've got bad news and good news. The bad news is DevOps (software engineering + infrastructure) is eating the infrastructure world. It's coming for you, whether you like it or not. The good news is, it's incredibly fun to learn. Most of the tooling is open source, well documented, and hotly in demand. And who really wants to spend more time configuring CoS on a Catalyst switch when you can be learning iterative loops to deploy dozens of subnets across the globe?

And after taking this first step, you're no longer an infrastructure engineer - you're a DevOps engineer. You'll soon be talking about Terraform, SparkleFormation, CloudFormation, Ansible, and approving pull requests. It's an exciting time, so let's get started.

New Scary Terms

Welcome, nascent DevOps engineer! Read over the following terms. If you know them, skip to the fun stuff below. If not, these will be useful as we build some pipelines.
  • Cloud- Hosted IaaS - Infrastructure as a Service. These companies permit you to build an entire computer network, VMs, security, etc on their networks and manage it. The big 3 are:
    • AWS - Amazon Web Services
    • Azure - Microsoft Azure
    • GCP - Google Cloud Platform
  • Git - An open source code repository management tool for both servers and clients. Due to its open source nature, you'll see all sorts of practical implementations like:
    • GitHub - A very popular social-focused git repo. Code can be secured as private and accessed via keys. Many if not most of the open source projects that run the internet are checked in here, including much of the source code for Azure's Cloud product. 
    • Stash (now officially BitBucket Server) - Atlassian's git repo product is git on the back-end, but has a mature, user-flexible front-end for easy management. 
    • Azure DevOps - The $MSFT Azure DevOps product has its own git repo, and can do some cool CI/CD automation. See what CI/CD is below. 
  • Git Terminology
    • Branch - A workspace for you to make changes in without affecting anyone else. When your changes are ready for prime-time, you'll create a Pull Request
    • Pull Request - A request to merge some changes you've made on your computer into another branch, most frequently the "master" branch, where changes are collected and merged together into a single source of truth
  • CI/CD - Continuous Integration / Continuous Deployment
    • This concept is something which cloud embraces and enables - constantly making small tweaks to your environments in an automated, repeatable fashion. Rather than logging into an environment and clicking a few items, you can write code and push it in sequence to a dev --> qa --> stage, and finally, prod environments in sequence. 
    • Software engineers are more familiar with these integration steps, as this is a common practice in software development. 
    • In practice, this is the processing and tooling that permits code changes to be rolled out to environments in an ordered fashion to fit business needs

Sunny with a Chance of Clouds

First, let's sign up for the Azure Cloud.
  • Visit the Azure Portal (https://azure.microsoft.com/en-us/free/) and sign up for a free trial with your personal email. This process is mostly self-explanatory, so I'll skip it. 
    • This free trial lasts 30 days, or $200 in credits, whichever comes first. Even after the trial ends, you'll still be able to access your storage for up to a year at no cost. 

Now you can build Azure cloud resources by hand. Which is neat! Go nuts. But really, we want to do automation stuff, so let's move right into Azure DevOps.

Azure DevOps: Cloud Automation for your Cloud

Azure DevOps is a CI/CD (read: automation) platform that is also cloud-hosted. It can run all sorts of tools in a native, containerized environment, and has quick, easy hooks into the Azure Cloud. We'll use it to run Terraform code. 

However, Azure DevOps is a separate service from Azure Cloud, so we'll need to sign up for it separately. 
  • Visit the Azure DevOps landing page and click on the "Start free" button. Follow the steps to sign up for an account. The landing page is here: https://azure.microsoft.com/en-us/services/devops/?nav=min
    • Make sure to sign up with the same email address you used for your Azure Cloud account - that makes this drop-dead simple
    • When ADO asks for your "project name" feel free to give it anything. I used a project name "Kyler Middleton"

Once you're logged into Azure DevOps you'll be presented with a blank summary of your project. The first step we'll need to do is integrate our Azure DevOps account with our Azure account. 
  • In the bottom left, click on "Project Settings"
  • Then click on Pipelines --> Service Connections
  • Click on "+ New service connection" and choose the "Azure Resource Manager" type
  • On the page that opens to add a service connection, if all goes well, you'll see your subscriptions pop up! My free trial has expired so I'm on a "pay-as-you-go" plan. You'll probably select a Free Trial Subscription. 
    • For now, leave the "allow all pipelines to use this connection" checked
Now you have a working CI/CD pipeline account ready to go, and integrated with your cloud infrastructure. But we haven't really done anything yet. So let's build a terraform pipeline that can build us some cool stuff.

Initialize the Repo

A good first step in building our pipeline is setting up a repo. The local one in Azure DevOps will do just fine. It uses git, a tool that if you're not familiar with yet, you certainly will be. Git is an open source tool written by Linux Torvalds of Linux kernel fame, and is used to manage files and collaboration at scale. It's incredibly powerful, but not very intuitive. Thankfully, Azure DevOps can help us initialize our repo to get started. 

From Azure DevOps (https://dev.azure.com/), click on Repos --> Files

There are several ways to initialize this repo, but the easiest in this case is to click the "initialize" button at the far bottom. Feel free to leave the README checkbox on - we can remove it later. 


Click Initialize and your repo will be populated with some files on the "master" branch. The master branch is an informal (not required by git software) but common practice. Usually when you want to make a change to the codebase you're working with, you'll open what's called a Pull Request (see definitions above for a better explanation). 


The pull request is proposed combination of your "branch" of code into the master. Many branches can be created at the same time, and git helps merge the changes together in a meaningful way. Speaking of code, let's check a sample change into our shiny new git repo.

Let's Get Git

You'll need git installed on your computer in order to push code. If you're on linux, it's probably already installed. If not, you can "apt-get install git" or "yum install git". If you're on mac, brew is probably easiest:

If you're on the 90% of the world on Windows, you're not out of luck. Git can be installed from here: https://git-scm.com/downloads

Generate Some Keys

In order for the Azure DevOps repo to know who you are, you need to authenticate yourself with some SSH keys. SSH keys are commonly used rather than typing in a username and password. They're easier to manage and more secure than a username/password combo that can be guessed. Plus if you lose a key or have one stolen, you can just disable it.

You may already have some keys on your computer. If you're on linux or mac, look in the .ssh (the leading period is intended) for a file called id_rsa (the private key, don't share this) and id_rsa.pub (the public part, we'll upload this to Azure DevOps). If you find those keys, you're in luck, and you can skip down a bit.

If you don't have the keys yet, that's okay, we can build them. Microsoft has a great walkthrough on how to build keys. The gist of it is to go to your home directory and run the following command from bash. Bash is a native shell on linux or mac, but not on Windows - the Git For Windows package will install a windows-compatible bash shell when it installs - run this command from there.

Remember to change the email@whatever to your email address. It doesn't have to match the login you use for Azure or Azure DevOps - it's more for you to remember.
ssh-keygen -C "email@whatever.com"

It'll output the id_rsa and id_rsa.pub files you need. Copy the contents of the id_rsa.pub (make sure it's the .pub one, that's your public - shareable - key) to your clipboard.

In the top-right of your Azure DevOps, click on your initials, then on "Security".

Under Security on the left, click on "SSH Public Keys", then on "add" on the page that appears.

Type a description that'll help you remember the keypair you're uploading, then paste the contents of your id_rsa.pub into the "Key Data" field. 
Great, now your computer has an id badge to flash when it connects to your Azure DevOps git repo to identify itself!

Time to Clone the Repo

Once git is installed on your computer, let's "clone" (replicate) the files from our repo to our computer so we can start making changes.

First, let's start with Azure DevOps. Go to your repo by clicking on Repos --> Files in the left column. Then in the top right, click on "Clone" and change the tab to "SSH". There will be a "git@" string there. Copy that string to your clipboard - we'll need it!


Go to your profile's root location - your home directory. Your id_rsa private key should be in the .ssh folder here - on linux and mac it might be hidden because of the leading period. That's okay, the tool will still be able to find it. 

In that directory, run the "git clone" command followed by the string you copied from your repo. You might be prompted to accept the fingerprint of the repo you're connecting to. If you do, type "yes" and hit enter.



The files will replicate to your computer, and you'll see a new folder show up on your computer. It'll be called whatever your Azure DevOps repo is called. Mine is KylerMiddleton.



The files that we place in here will be accessible to git and can be pushed up to the git repo hosted in Azure DevOps.

Next Time

I intended to do all this in a single blog, but this is getting LONG! So I'll wrap up this one, and next time we'll discuss:
> Adding some terraform code
> Using remote storage for the state file
> Building a "Build Pipeline" to stage artifacts
> Building a "Release Pipeline" to execute jobs

UPDATE: Part 2 is now posted, you can view it here: https://www.kylermiddleton.com/2019/06/lets-do-devops-build-azure-devops_28.html

Thanks all. Good luck out there! 
kyler