Azure Multiple NICs or IPs through Terraform and DSC

A situation came up where I needed to have two HTTP bindings on port 80 on a web server residing in Azure. This would be 1 binding each on two different IIS sites within a single VM. The creation of this configuration isn’t as simple as one would initially expect, due to some Azure limitations.

There are two options in order to achieve this:

  • Add a secondary virtual network interface to the VM
  • Add a second IP configuration on the primary virtual network interface of the VM.

For each of these, I wanted to deploy the necessary configuration through Terraform and Desired State Configuration (DSC).

Second IP Configuration

In this scenario, its quite simple to add a second IP configuration in terraform:

resource "azurerm_network_interface" "testnic" {
  name                = "testnic"
  location            = "${var.location}"
  resource_group_name = "${azurerm_resource_group.test.name}"
  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = "${azurerm_subnet.test.id}"
    private_ip_address_allocation = "static"
    private_ip_address            = "10.0.1.4"
    primary                       = true
  }
  ip_configuration {
    name                          = "ipconfig2"
    subnet_id                     = "${azurerm_subnet.test.id}"
    private_ip_address_allocation = "static"
    private_ip_address            = "10.0.1.5"
  }
}

However, when using static IP addresses like this, Azure requires you to perform configuration within the VM as well. This is necessary for both the primary IP configuration, as well as the secondary.

Here’s how you can configure it within DSC:

Configuration dsctest 
{ 
    Import-DscResource -ModuleName PSDesiredStateConfiguration 
    Import-DscResource -ModuleName xPSDesiredStateConfiguration 
    Import-DscResource -moduleName NetworkingDSC
 
    Node localhost {
        ## Rename VMNic
        NetAdapterName RenameNetAdapter 
        { 
            NewName = "PrimaryNIC"  
            Status = "Up" 
            InterfaceNumber = 1 
        }
 
        DhcpClient DisabledDhcpClient
        {
            State          = 'Disabled'
            InterfaceAlias = 'PrimaryNIC'
            AddressFamily  = 'IPv4'
            DependsOn = "[NetAdapterName]RenameNetAdapter "
        }
 
        IPAddress NewIPv4Address
        {
            #Multiple IPs can be comma delimited like this
            IPAddress      = '10.2.0.5/24','10.2.0.6/24'
            InterfaceAlias = 'PrimaryNIC'
            AddressFamily  = 'IPV4'
            DependsOn = "[NetAdapterName]RenameNetAdapter "
        }
 
        # Skip as source on secondary IP address, in order to prevent DNS registration of this second IP
        IPAddressOption SetSkipAsSource
        {
            IPAddress    = '10.2.0.6'
            SkipAsSource = $true
            DependsOn = "[IPAddress]NewIPv4Address"
        } 
 
    }
 
}

Once the “SkipAsSource” runs, this sets the proper priority for the IP addresses and ensures that the un-mentioned IP (which defaults to source) is primary, matching the Azure configuration.

The second problem I have yet to solve is when later in the DSC configuration I want to use a Script resource to bind port 80 to a specific IP address, but I want to dynamically grab it within the Script resource without specifying it in DSC configuration data as a variable I can reference.

Second NIC

When adding a second NIC in Terraform, you have to add a property (“primary_network_interface_id”) on the VM resource for specifying the primary NIC.

resource "azurerm_network_interface" "testnic1" {
  name                = "testnic"
  location            = "${var.location}"
  resource_group_name = "${azurerm_resource_group.test.name}"
  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = "${azurerm_subnet.test.id}"
    private_ip_address_allocation = "static"
    private_ip_address            = "10.0.1.4"
  }
 
}
resource "azurerm_network_interface" "testnic2" {
  name                = "testnic"
  location            = "${var.location}"
  resource_group_name = "${azurerm_resource_group.test.name}"
  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = "${azurerm_subnet.test.id}"
    private_ip_address_allocation = "static"
    private_ip_address            = "10.0.1.5"
  }
 
}
 
resource "azurerm_virtual_machine" "test" {
  name                  = "helloworld"
  location              = "${var.location}"
  resource_group_name   = "${azurerm_resource_group.test.name}"
  network_interface_ids = ["${azurerm_network_interface.testnic1.id}","${azurerm_network_interface.testnic2.id}"]
  primary_network_interface_id = "${azurerm_network_interface.test.id}"
  vm_size               = "Standard_A1"
 
  storage_image_reference {
    publisher = "MicrosoftWindowsServer"
    offer     = "WindowsServer"
    sku       = "2016-Datacenter"
    version   = "latest"
  }
 
  storage_os_disk {
    name          = "myosdisk1"
    vhd_uri       = "${azurerm_storage_account.test.primary_blob_endpoint}${azurerm_storage_container.test.name}/myosdisk1.vhd"
    caching       = "ReadWrite"
    create_option = "FromImage"
  }
 
  os_profile {
    computer_name  = "helloworld"
    admin_username = "${var.username}"
    admin_password = "${var.password}"
  }
 
  os_profile_windows_config {
    provision_vm_agent = true
    enable_automatic_upgrades = true
  }
 
}

In this scenario, once again Azure requires additional configuration for this to work. The secondary NIC does not get a default gateway and cannot communicate out of it’s subnet.

One way to solve this with DSC is to apply a default route for that interface, with a higher metric than the primary interface. In this case, we don’t need to specify static IP addresses inside the OS through DSC, since there’s only one per virtual NIC.

Configuration dsctest 
{ 
    Import-DscResource -ModuleName PSDesiredStateConfiguration 
    Import-DscResource -ModuleName xPSDesiredStateConfiguration 
    Import-DscResource -moduleName NetworkingDSC
 
    Node localhost {
        ## Rename VMNic(s) 
        NetAdapterName RenameNetAdapter 
        { 
            NewName = "PrimaryNIC" 
            Status = "Up" 
            InterfaceNumber = 1 
            Name = "Ethernet 2"
        }
        NetAdapterName RenameNetAdapter_apps
        { 
            NewName = "SecondaryNIC"  
            Status = "Up" 
            InterfaceNumber = 2
            Name = "Ethernet 3"
        }
 
        DefaultGatewayAddress SetDefaultGateway
        {
            Address        = '10.2.0.1'
            InterfaceAlias = 'SecondaryNIC'
            AddressFamily  = 'IPv4'
        }
 
    }
 
}

Generally speaking, I’d rather add a second IP to a single NIC – having two NICs on the same subnet with the same effective default gateway might function, but doesn’t seem to be best practice to me.

DSC IIS bindings and SSL certificates

This was a tricky one that really didn’t leave me with an ideal solution.

In using DSC, I want my compiled node configurations to be generic, like “webserver” instead of “webserver01”, in order for them to be re-used by VMs sharing the same characteristics, and to avoid duplicating information like IP addressing and VM names which has already been specified in the Terraform configuration for deployment.

At the same time, I want to be able to deploy a web server with a site on port 443 and a self-signed certificate which is created by DSC.

Combining these two ideas was not something I found I could accomplish with pre-existing modules.

I first looked to the xWebsiteAdministration DSC module, which contains the functions xWebsite among others. With this, I could use the following syntax:

xWebSite Admin { 
            Name = "Admin"
            PhysicalPath = "E:\inetpub\AdminSite"
            State = "Started"
            ApplicationPool = "Admin"
            Ensure = "Present"
            BindingInfo     = @(
                MSFT_xWebBindingInformation
                {
                    Protocol              = "HTTPS"
                    Port                  = 443
                    CertificateThumbprint = ""
                    CertificateStore      = "MY"
                })
            LogPath = "E:\inetpub\logs\AdminSite"
            DependsOn = "[File]E_AdminSite"
         }

Here, I want to pass in the thumbprint of my previously generated self-signed certificate. However, I don’t know it’s thumbprint, and it will be unique when I deploy this node configuration between a Web01 and a Web02 VM.

I tried resolving the thumbprint like this: CertificateThumbprint = (Get-ChildItem Cert::\LocalMachine\My | where {$_.Subject -like “*$($Node.ClientCode).com*”}), however that continued to give me DSC errors when it was attempting to apply, about a null reference.

When I came across this StackOverflow question it was clear why this wasn’t working. Since the compilation happens on Azure servers, of course they won’t have a certificate matching my subject name, and thus it can’t generate a thumbprint.

So instead I thought, I can just create a script that applies the certificate after the website has been created. Here is where I ran into additional problems:

  • If I create the xWebsite with HTTPS and 443 with no certificate, it errors
  • If I create the xWebsite with no binding information, it default assigns HTTP with port 80 (conflicting with another website that I have)
  • If I create the xWebsite with HTTP and port 8080 as a placeholder value, now I have IIS listening on ports I don’t actually want open
  • If I create the xWebsite with HTTP and port 8080 and then cleanup that binding afterwards with a Script, on the next run DSC is going to try and re-apply that binding, since I’ve effectively said it is my desired state

Ultimately what I was left with was creating a script that deployed the whole website, and not using xWebsite at all. Like I said, not ideal but it does work to meet my requirements.

Here’s the script that I’ve worked out:

Script WebsiteApps          
        {            
            # Must return a hashtable with at least one key            
            # named 'Result' of type String            
            GetScript = {            
                Return @{            
                    Result = [string]$(Get-ChildItem "Cert:\LocalMachine\My")            
                }            
            }            
 
            # Must return a boolean: $true or $false            
            TestScript = {            
                Import-Module WebAdministration
                # Grab the IP based on the interface name, which is previously set in DSC
                $ip1  = (get-netipaddress -addressfamily ipv4 -InterfaceAlias $($Using:Node.VLAN)).IPAddress
                # Find out if we've got anything bound on this IP for port 443
                $bindcheck = get-webbinding -name "Apps" -IPAddress $ip1 -Port 443
                $bindcheckwildcard = get-webbinding -name "Apps" | where-object { $_.BindingInformation -eq "*:80:"}
                # If site exists
                    if (Test-Path "IIS:\Sites\Apps")
                    { 
                        Write-Verbose "Apps site exists."
                        # if log file setting correct
                        if ((get-itemproperty "IIS:\Sites\Apps" -name logfile).directory -ieq "E:\inetpub\logs\AppsSite")
                           {
                               Write-Verbose "Log file is set correctly."
                                   # if IP bound on port 443
                                   if ($bindcheckhttps)
                                   { 
                                       Write-Verbose "443 is bound for Apps."
                                       #if SSL certificate bound
                                       if (Test-path "IIS:\SslBindings\$ip1!443")
                                       {
                                            Write-Verbose "SSL Certificate is bound for Apps"
                                            # wildcard binding check for Apps
                                            if (-not ($bindcheckwildcard)) {
                                                Write-Verbose "* binding does not exist for Apps."
                                                Return $true
                                            }
                                            else
                                            {
                                                Write-Verbose "* binding exists for Apps."
                                                Return $false
                                            }
                                       }
                                       else
                                       {
                                           Write-Verbose "SSL Certificate is NOT bound for Apps"
                                           Return $false
                                       }
                                   }
                                   else
                                   {
                                       Write-Verbose "IP not bound on 443 for Apps."
                                       Return $false
                                   }
                           }
                            else 
                            {
                               Write-Verbose "Log file path is not set correctly"
                               Return $false
                            } 
                   }
                   else
                   {
                       Write-Verbose "Apps site does not exist"
                       Return $false
                   }
                }    
 
            # Returns nothing            
            SetScript = {
                $computerName = $Env:Computername
                $domainName = $Env:UserDnsDomain
 
                $apps = Get-Item "IIS:\Sites\Apps"
                $ip1  = (get-netipaddress -addressfamily ipv4 -InterfaceAlias $($Using:Node.VLAN)).IPAddress
                $bindcheckhttps = get-webbinding -name "Apps"  -IPAddress $ip1 -Port 443
                $bindcheckwildcard = get-webbinding -name "Apps" | where-object { $_.BindingInformation -eq "*:80:"}
 
                 # If site not exists
                    if (-not (Test-Path "IIS:\Sites\Apps"))
                    { 
                        Write-Verbose "Creating Apps site"
                        New-Website -Name "Apps" -PhysicalPath E:\inetpub\AppsSite -ApplicationPool "Apps"
                    }
 
                    # if port 443 not bound
                    if (-not ($bindcheckhttps))
                    { 
                        Write-Verbose "Binding port 443"
                        $apps = Get-Item "IIS:\Sites\Apps"
                        New-WebBinding -Name $apps.Name -protocol "https" -Port 443 -IPAddress $ip1
                    }
                    if ($bindcheckwildcard)
                    {
                        Write-Verbose "Removing wildcard binding for Apps"
                        get-webbinding -name "Apps" | where-object { $_.BindingInformation -eq "*:80:"} | Remove-Webbinding
                    } 
 
                    #if SSL certificate not bound        
                    if (-not (Test-path "IIS:\SslBindings\$ip1!443"))
                        {
                            Write-Verbose "Binding SSL certificate"
                            Get-ChildItem cert:\LocalMachine\My | where-object { $_.Subject -match "CN\=$Computername\.$DomainName" } | select -First 1 | New-Item IIS:\SslBindings\$ip1!443
                        }
 
                    # if log file setting correct
                    if (-not ((get-itemproperty "IIS:\Sites\Apps" -name logfile).directory -ieq "E:\inetpub\logs\AppsSite"))
                        {
                            Write-Verbose "Setting log file to the proper directory"
                            Set-ItemProperty "IIS:\Sites\Apps" -name logFile -value @{directory="E:\inetpub\logs\AppsSite"}
                        }
                } 
            DependsOn = "[xWebAppPool]Apps","[Script]GenerateSelfSignedCert"
        }

 

Terraform AzureRM Backend

One of the primary items I wanted to accomplish before my latest use of Terraform in production was storing the state file in a central location for shared use within my team.

This is controlled in Terraform by the “backend“. In my particular case, I was interested in the AzureRM backend.

There are a few ways to accomplish this configuration, but one of my requirements was to not actually store any key string in a file. Rather, I’m relying upon the Azure Cloud Shell as my deployment environment for Terraform, which I will have already authenticated to and can dynamically connect to resources within my subscription.

First I define a simple “backend.tf” file.

terraform {
    backend "azurerm" {}
}

Next, I wrote a wrapper script (“InitWrapper.ps1”) to actually run my “Terraform Init” command, passing in the variables for the backend as documented by Terraform.

 
$subscription_id = "e86a3dce" #
$resource_group = "Default"
$storage_account = "terraformstates"
$containerName = "Client1"
 
# Get the Storage Account Key for use in loading the backend file
Select-AzureRmSubscription -SubscriptionId $subscription_id
$accountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName $resource_group -Name $storage_account)[0].Value
 
# Perform init
terraform init `
    -backend-config="storage_account_name=$storage_account" `
    -backend-config="container_name=$containerName" `
    -backend-config="access_key=$accountKey" `
    -backend-config="key=prod.terraform.tfstate"

You can see here I’m referencing a storage account, and a blob container. This way you can isolate state files within a container for purposes of RBAC or management.
The last parameter I’m passing in is “key” which tells Terraform what to name the state file.

The end result of running this is a file in my Azure Cloud Shell share (/home/azureuser/clouddrive/client1 is where I would typically change directory to) named “terraform.tfstate” which is a pointer to my backend sitting in an Azure storage account blob container.

Now I no longer need to run the init file unless there are specific Terraform changes that need to be initialized in my cloud shell.

If another team member wishes to work with the same state file, they can run the Init wrapper, and their local state pointer will connect to the same “prod.terraform.tfstate” and be able to create/modify infrastructure in the same way.

Terraform Azure VM Extensions

Having recently gone through getting Terraform to deploy a virtual machine and a VM extension to register Desired State Configuration (DSC) with Azure Automation, I thought I’d note the method and code here for future reference.

This presumes a functioning Azure Automation account with a DSC configuration and generated node configurations.

 

First, I specify my variables in a file “Variables.tf”:

// For all VMs
 
variable "subscription" { type = "string" }
variable "location" { type= "string" }
variable "vmsize" { type = "map" }
variable "username" { type = "string" }
variable "clientcode" { type = "string" }
variable "password" { type = "map" }
variable "networkipaddress" { type = "map" }
variable "serveripaddress" { type = "map" }
variable "dnsservers" { type = "list" }
variable "dcdnsservers" { type = "list" }
//DSC related
# The key for the Azure Automation account
variable "dsc_key" { type = "string" }
# Endpoint, also referred to as the Registration URL
variable "dsc_endpoint" { type = "string" }
# This can be ApplyAndMonitor, ApplyandAutoCorrect, among others
variable "dsc_mode" { type = "string" }
# Heres where you define the node configuration that you actually want to apply to the VM
variable "dsc_nodeconfigname" { type = "map" }
 
variable "dsc_configfrequency" { type = "string" }
variable "dsc_refreshfrequency" { type = "string" }

Then I deploy the VM, along with the required dependencies:

# Initial Resource Group
resource "azurerm_resource_group" "Default" {
  name     = "az${var.clientcode}"
  location = "${var.location}"
}
 
resource "azurerm_virtual_network" "VirtualNetwork" {
  name                = "azcx${var.clientcode}"
  address_space       = ["${lookup(var.networkipaddress, "VMNet")}"]
  location            = "${var.location}"
  resource_group_name = "${azurerm_resource_group.Default.name}"
  dns_servers         = "${var.dnsservers}"
}
 
resource "azurerm_subnet" "subnet-lan" {
  name                 = "az${var.clientcode}-lan"
  resource_group_name  = "${azurerm_resource_group.Default.name}"
  virtual_network_name = "${azurerm_virtual_network.VirtualNetwork.name}"
  address_prefix       = "${lookup(var.networkipaddress, "subnet-lan")}"
}
resource "azurerm_network_interface" "dc1nic1" {
  name                = "az${var.clientcode}1nic1"
  location            = "${var.location}"
  resource_group_name = "${azurerm_resource_group.Default.name}"
  # reverse DNS for the domain controller
  dns_servers = "${var.dcdnsservers}"
 
  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = "${azurerm_subnet.subnet-lan.id}"
    private_ip_address_allocation = "static"
    private_ip_address            = "${lookup(var.serveripaddress, "az${var.clientcode}1")}"
 
  }
}
resource "azurerm_virtual_machine" "dc1" {
  name                             = "az${var.clientcode}1"
  location                         = "${var.location}"
  resource_group_name              = "${azurerm_resource_group.Default.name}"
  network_interface_ids            = ["${azurerm_network_interface.dc1nic1.id}"]
  vm_size                          = "${lookup(var.vmsize, "az${var.clientcode}1")}"
  delete_os_disk_on_termination    = true
  delete_data_disks_on_termination = true
 
  storage_image_reference {
    publisher = "MicrosoftWindowsServer"
    offer     = "WindowsServer"
    sku       = "2012-R2-Datacenter"
    version   = "latest"
  }
 
  storage_os_disk {
    name              = "az${var.clientcode}1_c"
    caching           = "ReadWrite"
    create_option     = "FromImage"
    managed_disk_type = "Standard_LRS"
    disk_size_gb      = "128"
  }
 
  storage_data_disk {
    name              = "az${var.clientcode}1_d"
    managed_disk_type = "Standard_LRS"
    caching           = "None"
    create_option     = "Empty"
    lun               = 0
    disk_size_gb      = "64"
  }
 
  os_profile {
    computer_name  = "az${var.clientcode}1"
    admin_username = "${var.username}"
    admin_password = "${lookup(var.password, "az${var.clientcode}1")}"
  }
 
  os_profile_windows_config {
    provision_vm_agent        = true
    enable_automatic_upgrades = false
  }
}

Now with the VM deployed, the Extension can be applied. This will pass in the proper variables defined, install the VM extension, and register with the Azure Automation account to begin the initial DSC deployment. Note, due to WordPress formatting, the line below that says “setting = SETTINGS” should look like this: “setting = <<SETTINGS”. The double bracket should exist for the PROTECTED_SETTINGS line too.

resource "azurerm_virtual_machine_extension" "dc1-dsc" {
  name                 = "Microsoft.Powershell.DSC"
  location             = "${var.location}"
  resource_group_name  = "${azurerm_resource_group.Default.name}"
  virtual_machine_name = "az${var.clientcode}1"
  publisher            = "Microsoft.Powershell"
  type                 = "DSC"
  auto_upgrade_minor_version = true
  type_handler_version = "2.76"
  depends_on           = ["azurerm_virtual_machine.dc1"]
 
  settings = SETTINGS
        {
            "WmfVersion": "latest",
            "advancedOptions": {
	              "forcePullAndApply": true 
                },
            "Properties": {
                "RegistrationKey": {
                  "UserName": "PLACEHOLDER_DONOTUSE",
                  "Password": "PrivateSettingsRef:registrationKeyPrivate"
                },
                "RegistrationUrl": "${var.dsc_endpoint}",
                "NodeConfigurationName": "${lookup(var.dsc_nodeconfigname, "dc1")}",
                "ConfigurationMode": "${var.dsc_mode}",
                "ConfigurationModeFrequencyMins": ${var.dsc_configfrequency},
                "RefreshFrequencyMins": ${var.dsc_refreshfrequency},
                "RebootNodeIfNeeded": true,
                "ActionAfterReboot": "continueConfiguration",
                "AllowModuleOverwrite": true
            }
        }
    SETTINGS
  protected_settings = PROTECTED_SETTINGS
    {
      "Items": {
        "registrationKeyPrivate" : "${var.dsc_key}"
      }
    }
PROTECTED_SETTINGS
}

For passing in the variable values, I use two files. One titled inputs.auto.tfvars with the following:

# This file contains input values for variables defined in "CLIENT_Variables.tf"
# As this file doesnt contain secrets, it can be committed to source control.
 
subscription = "bc5242b8"
 
location = "eastus2"
 
vmsize = {
  "azclient1" = "Standard_A1_v2"
  "azclientweb" = "Standard_A1_v2"
 }
 
username = "admin"
 
clientcode = "client"
 
serveripaddress = {
  "azclient1" = "10.0.0.211"
  "azclientweb" = "10.0.0.71"
}
 
networkipaddress = {
  "VMNet"                    = "10.0.0.0/23"
  "subnet-lan"               = "10.0.0.0/26"
}
 
resource_group_name = "azclient"
 
dnsservers = ["10.0.0.10", "10.0.0.111"]
dcdnsservers = ["10.0.0.111","10.0.0.10"]
 
#This is the registration URL of Azure Automation
dsc_endpoint = "https://eus2-agentservice-prod-1.azure-automation.net/accounts/guid"
dsc_mode = "ApplyandAutoCorrect"
dsc_configfrequency = "240"
dsc_refreshfrequency = "720"
dsc_nodeconfigname = {
  "dc1"   = "deploymentconfig.domaincontroller"
  "web1"   = "deploymentconfig.webserver"
}

And then another titled secrets.auto.tfvars which does not get uploaded to source control:

password = {
  "azclient1" = "password"
  "azclientweb" = "password"
}
 
dsc_key = "insert key here"

Get Inner Error from Azure RM command

Today I’m working on an ARM Template to deploy some resources into an Azure subscription. After building my JSON files and prepping parameters, I used the cmdlet “Test-AzureRmResourceGroupDeployment” in order to validate my template.

This failed with the error:

Code    : InvalidTemplateDeployment
Message : The template deployment 'e76887a9' is not valid according to the validation
          procedure. The tracking id is '6df0fffb'. See inner errors for details. Please
          see https://aka.ms/arm-deploy for usage details.
Details : {Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkModels.PSResourceManagerError}

I found that decidedly unhelpful, but found an effective way to determine the actual error message.

To retrieve the error details, use the following cmdlet, where the CorrelationID equals the tracking ID mentioned in the error.

get-azurermlog -correlationid 6df0fffb -detailedoutput

This will produce output which you can investigate and determine where the error lies.

In my case, I needed to create a Core quota increase request with Azure support, as my subscription had reached it’s limit.