Azure NSG discovery

During deployment of some resources with an Azure virtual network which has subnets with network security groups (NSG) applied, I made a new discovery that I didn’t previously know. It makes sense in the context of how Azure applies NSG rules, but it doesn’t align with a traditional understanding of firewall ACLs across a subnet.

Communication within subnet

If you apply a Deny rule that has a lower priority than the default 65000 “Allow Vnet inbound”, it will also deny resources within that subnet from communicating with each other.

I discovered this while applying a “Deny inbound” rule in order to restrict lateral movement between subnets, not intending to restrict traffic within a subnet.

For example, I have a “management” subnet, with an NSG applied. Inside this subnet is an AD domain controller, and a member server. I apply a Deny rule for any source, after my “allow incoming” rules have been applied to let other subnets talk to this domain controller.

Now I find that my domain controller cannot reach my member server, despite it residing within the same subnet.

While I do not want to allow service tag “VirtualNetwork” incoming access (again, to restrict lateral movement), I do want “everything inside this subnet can talk to everything inside this subnet”. As such I had to create a specific rule for this behavior.

Azure IaaS Deny outbound considerations

As a general practice, outbound Internet access should be denied except for approved destinations. This is referenced in NIST 800-41 as a “deny by default” posture.

Achieving this within Azure Infrastructure as a Service in a practical and economical way without breaking a large amount of services is quite difficult at the moment.

If Outbound Internet is fully denied, some of the commonly used services of Azure will cease to work:

  • Azure Backup
  • Log Analytics
  • Azure State Configuration (DSC)
  • Azure Update Management
  • Azure Security Center
  • Windows Update

Some of these are not as difficult to solve – Service Tags on NSG rules can allow Azure services where they have been defined by Microsoft. As of Ignite 2018 in late September, there are new service tags covering entire regions, or all of Azure (“AzureCloud”). This means you can allow most of those services above to function and still deny general Internet outbound.

Additional Service Tags for Windows Update, or custom definitions are supposed to be coming in the future, but this doesn’t fully resolve the problem.

What if your application has a GIS component, and it needs to reach *.arcgisonline.com? What if your users have a legitimate reason to access a particular website? It isn’t good enough to just resolve that IP address one time and add it to an NSG.

What is really needed is a method to allow access to a fully qualified domain name (FQDN), particularly with wildcard support.

Here are some possible solutions:

Implement a 3rd Party network virtualization appliance (NVA)

This is the most common response that I see recommended to the outbound problem. Unfortunately, it is really expensive, and overkill if you’re only address this one particular problem. One has to consider high availability of the resources, as well as management of them since you’re just adding more IaaS into your environment, which is what we’re all trying to get away from when we’re using the cloud isn’t it?

Some vendors may not support wildcard FQDN in it’s ACLs (Barracuda CloudGen last I checked), which means you can’t support things like Windows Update where no published IP list exists.

If the implementation is anything like SonicWALL’s method, it will have difficulty being reliable – this relies upon the SonicWALL using the same DNS server as the client (calling it ‘sanctioned’) which may or may not be true in your Azure environment with the use of Azure DNS or external providers.

Implement Azure Firewall

Azure Firewall is new on the scene and released to General Availability as of late September 2018. It supports the use of FQDN references in application rules, and while I haven’t personally tested it, the example deployment template is shown to allow an outbound rule to *microsoft.com.

Confusingly, their documentation states that FQDN tags can’t be custom created, but I believe this just references groups of FQDN, not individual items.

Azure Firewall solves the problem of deploying more IaaS, and it’s natively highly available. However it again isn’t cheap, at $1.25/hour USD it is a high price to pay for just this one feature.

Wait until FQDN support exists in an NSG rule

It has been noted on the Microsoft feedback site that NSG rules containing FQDN is a roadmap item, but since this hasn’t received the “Planned” designation yet, I expect it is very far down the roadmap; particularly considering this feature is available in the Azure Firewall.

Build something custom – Azure Function or runbook which resolves DNS and adds it to your NSG

I’ve toyed with the idea of building a custom Azure Function or Automation runbook which can resolve a record and add it to an NSG. I’ll have a post on the Function side of this coming soon that describes how it would work, and the limitations that made me discard the idea.

Realistically, this isn’t a long-term viable solution as it doesn’t solve the wildcard problem.

Utilize an outbound transparent proxy server

This method involves trusting some other source to proxy your outbound traffic and depending on that source, gives a large amount of flexibility to achieve the outbound denial without breaking your services.

This could be an IaaS resource running Squid or WinGate (a product I’m currently testing for this purpose), or it could be an external 3rd party service like zScaler which specializes in access control of this nature.

To make this work, your proxy must be able to be identified by some kind of static IP to allow it through the NSG, but after that the whitelisting could happen within the proxy service itself.

I see this as the most viable method of solving the problem until either FQDN support exists for NSG, or Azure Firewall pricing comes down with competition from 3rd party vendors.

Azure Support Plan discovery

I learned some things about Azure Support Plans recently.

A co-worker was tasked with adding an Azure Support Plan to a new subscription that was being created. So they went to portal.azure.com, clicked “Help and Support”, and then from the drop-down selected the new subscription ID.

Then they clicked Change Plan and added a Standard support plan. Upon investigation, this appeared to add a support plan to ALL subscriptions, which was scary. We don’t want to be charged 10x$100USD per month!

Working through a support case with Microsoft, I learned the following about support plans that cleared things up.

A support plan is tied to an Azure Account – the account that subscriptions are created under. The support plan is effectively its own subscription, not tied to individual subscriptions themselves. However, you won’t see it displayed this way in portal.azure.com.

It’s not until you log into https://account.windowsazure.com/Subscriptions that you see it itemized by itself and are able to view billing history for the support plan.

So if I have 10 subscriptions, and they are all created under one Azure Account (you can see this on the “Properties” page of your Subscriptions blade), then only a single support plan is needed.

This also clarifies why Microsoft’s instructions on removing a support plan describe to “Go to the portal, and click ‘cancel subscription'”.

When I thought that this was a per-subscription basis, that made me afraid we’d cancel our actual subscription. Knowing now that this is it’s own subscription, that text makes a lot more sense.

Intel Network Connections (ProSet) update or removal

Short-version: be very careful removing the Intel Network Connections software!

I’m working on a task to update NIC drivers and firmware on a Dell PowerEdge R630. It was decided during this process to remove the Intel ProSet software along with it’s generated sub-interfaces for VLAN purposes, and instead utilize switch-port configuration.

I went to uninstall Intel(R) Network Connections from appwiz.cpl, which contains ProSet. On this server, its version was 19.3.0.0. However, I quickly received an error stating that I needed to remove any Teams and VLANs. No problem, I was going to do that anyways so I went into the advanced properties of my NICs and removed the VLAN sub-interface. I wasn’t using any Intel teaming, but rather LFBO within Windows Server.

When I tried again, it still gave me the same error. I decided to try “Modify” on the uninstaller instead, and remove ProSET only.

This was successful, but then when I tried to install the latest version, it said it couldn’t update with this error:
Product: Intel(R) Network Connections — The installed version of Intel(R) Network Connections is not supported for upgrades. You must uninstall it before installing this version.

So I tried the full uninstall again, and this time it worked! Great, I’m thinking “Now I can proceed with the latest version”.

Except it removed ALL NIC config, including IP settings and advanced properties like Jumbo Frames and RSS buffer sizes.

Thankfully I had good documentation of the previous setup, and that should be your biggest takeaway if you’re reading this: always consider “what’s the worst that could happen” and then plan to mitigate that thing.

For future hosts, I left the old version of the software installed and manually updated drivers from the .INF files. Interesting to note, if you already have a higher version installed like Version 22.7, this behavior isn’t seen – you can install an update on top of it without having to uninstall.

 

 

 

Project Honolulu and Server Core

By now many IT administrators have heard of Project Honolulu from Microsoft. I must admit, when I heard the headline and initial talk about it, I book-marked the info intending to come back but didn’t really dig into it. I thought, “oh a revamped Server Manager.”

My perspective is all changed now.

I’m watching the “Windows Server: What’s new and what’s next” session from Ignite 2017. I have been following Aidan Finn’s blogging of Ignite sessions including this one in particular for the “What’s new” session, and recall reading his notes about Server Core being in semi-annual channel, but not Server GUI, so “you better learn some PowerShell to troubleshoot your networking and drivers/firmware”.

But now having watched the session myself, it makes Microsoft’s vision and a path forward here very clear to me.

Using Server Core allows an organization to reduce their surface area for vulnerability, streamline the size/frequency of Windows Updates, optimize performance and scalability on hardware, and stay up to date on the Windows Server cadence.

Project Honolulu makes using Server Core viable. This is the answer to the Windows system administrator saying “Server Core just doesn’t give me the visibility I need into my servers”. As Jeff Woolsey walks through the functions of Project Honolulu, it is obvious that THIS is where the visibility will be; no more RDP into individual servers to manage their roles, devices, and settings. No more MMC windows to open Event Viewer and Shares and other applets.

Now I’m excited.