Azure Site Recovery and Backups

I’m working on a specific test case of Azure Site Recovery and came across an error, which identified a gap in my knowledge of ASR and Hyper-V Replica.

I have ASR configured to replicate a group of VMs at 5 minute intervals. My initial replication policy for this proof-of-concept was configured to hold recovery points for 2 hours, with app-consistent snapshots every 1 hour.

In practice, what I have seen for potential select-able recovery points is one every 5 minutes going back to the latest application-consistent recovery point, and then any additional app-consistent recovery points within the retention time set (2 hours):

Because the underlying mechanism is Hyper-V Replica, this corresponds to the options for Recovery Points visible in Hyper-V Manager:

Hyper-V will perform the .HRL file replication to Azure every 5 minutes as configured, but it will also utilize the Hyper-V integration components to trigger in-guest VSS for the application-aware snapshot at 1 hour intervals. This means the RPO in general is up to 5 minutes, but for application-aware RPO it is 1 hour.

 

In addition to replication, I am backing up a VM with Quest Rapid Recovery. The test was to ensure that both protection methods (Disaster Recovery and Backup) do not conflict with each other. Rapid Recovery is running an incremental snapshot every 20 minutes, and on about 40% of them the following events are received in the Application Log for the VSS service:

Volume Shadow Copy Service error: The I/O writes cannot be flushed during the shadow copy creation period

Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{a9dca4cb). hr = 0x80070016, The device does not recognize the command.

 

Quest has a KB article about this issue, which says to disable the Hyper-V integration component for backup in order to avoid a timing conflict when the host uses the Volume Shadow Copy requestor service. The problem is, disabling this prevents ASR from getting an application-aware snapshot of the virtual machine, which it will begin to throw warnings about after a few missed intervals:

These problems make sense though – for every hourly attempt of Hyper-V to take an application-aware snapshot using VSS, Rapid Recovery finds that writer in use and times-out waiting for it. There isn’t a way to configure when in an hour Hyper-V takes the snapshot, but I’ve begun tweaking my Rapid Recovery schedule to not occur on rounded intervals like :00 or :10, but rather :03 or :23 in an attempt to avoid conflicts with the VSS timing. So far this hasn’t been as effective as I’d hoped.

The other alternative is to disable application-aware snapshots if they’re not needed. If it is just flat files or an application that doesn’t natively tie into VSS, the best you can expect is a crash-consistent snapshot and you should configure your ASR replication policy accordingly, by setting that value to 0. In this manner you can still retain multiple hours of recovery points, they’ll just ALL be crash-consistent.

 

Azure Site Recovery setup errors

While setting up an Azure Site Recovery proof of concept, errors were encountered; at first with associating the replication policy and then afterwards with updating the authentication service.

The background is connecting SCVMM with a Server 2012 R2 Hyper-V Cluster to replicate to Azure. During the final steps of the “Prepare Infrastructure” phase, you need to associate a replication policy. This failed at the following step:

The text of the error was:

Error ID
10003
Error Message
Protection couldn't be configured for cloud/site POC-ASR.
Provider error
Provider error code: 31408

Provider error message:

	Failed to fetch the version of Microsoft Azure Recovery Services Agent installed on the Hyper-V host server . Error: An internal error has occurred trying to contact the  server: : .

WinRM: URL: [http://:5985], Verb: [INVOKE], Method: [GetStringValue], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/StdRegProv]

Check that WS-Management service is installed and running on server .

Provider error possible causes:
	It is possible that Registry provider of WMI is corrupted.

Provider error recommended action:
	Build the repository using MOF compiler and retry the operation.

This occurred right before I was distracted by other items so I didn’t directly troubleshoot. When I came back to the Azure Portal (in a fresh session) I had a surprising new message greeting me at the Recovery Services Vault blade:

This was very odd, since I had just installed the latest version of the Site Recovery provider on my VMM host, as well as the MARS agent on my Hyper-V hosts. But when I clicked “Update Now” it listed my VMM host and displayed a new button to “Update Authentication Service”.

This almost immediately error-ed out:

Error ID
635
Error Message
Updating authentication service information for server -  failed.
Provider error
Provider error code: 31437

Provider error message:

	Failed to fetch the version of Microsoft Azure Site Recovery Agent installed on the Hyper-V host(s) '' as the host is not reachable.

Provider error possible causes:
	
      1. Windows Management Instrumentation service crashed.
      2. Windows Remote Management (WinRM) service is not running.
      3. Required services may not be running on the Hyper-V host(s)''.
  
Provider error recommended action:
	
      Ensure that
      1. A firewall is not blocking HTTPS/HTTPS traffic on the Hyper-V host.
      2. If the server is running windows Server 2008 R2, ensure that KB 982293 is installed on it. Refer to https://aka.ms/kblink982293 for more details.
      3. The Hyper-V Virtual Machine Management service is running.
      4. Ensure that the Windows Management Instrumentation service is running on the Hyper-V host(s).
      5. Ensure that the Windows Remote Management (WinRM) service is running on the Hyper-V host(s).
      6. Verify that CredSSP authentication is enabled on the service configuration of the Hyper-V host(s). To enable the CredSSP on the service configuration, run the following command on the Hyper-V host, from an elevated command line: winrm set winrm/config/service/auth @{CredSSP="true"}.
      7. The Provider version running on the server is up-to-date. Download and install the latest Microsoft Azure Site Recovery Provider.
      8. If the error persists, retry the operation and contact support.
    

I validated all the components in the list here, checked the referenced articles, ensured WMF was updated to 5.1, to no avail.

I finally stumbled upon this post on the Microsoft forums where a check was done against WMI for the object “StdRegProv”, which is mentioned in the original error from the replication policy. Turns out this was my problem too! When I ran the WMI query it returned an error of “Exception calling “GetStringValue” : “Provider not found “” on 3 of my 4 Hyper-V hosts:

$hklm = 2147483650
$key = "Software\Microsoft\Windows\CurrentVersion\Uninstall\Windows Azure Backup"
$value = "DisplayVersion"
$wmi = get-wmiobject -list "StdRegProv" -namespace root\cimv2
($wmi.GetStringValue($hklm,$key,$value)).svalue

I ran the mofcomp command, and then when I ran the last line of the previous query ($wmi.GetStringValue) it returned a value instead of an error.

cd c:\windows\system32\wbem
mofcomp regevent.mof

Following this, the “Update Authentication Service” job completed successfully, and I was able to associate my replication policy without further problems.

 

AppAssure Agent service not started

I have found in recent experience that the Quest RapidRecovery/AppAssure agent service frequently fails to start within the default Windows service startup timer (30 seconds).

I don’t believe this is simply my environment, as Quest has a knowledge article specifically addressing the problem.

This issue poses a problem especially after Windows Updates maintenance windows, because our hundreds of virtual machines will reboot, leaving our monitoring system with hundreds of alerts of stopped services, and many failed backups.

Unfortunately, Quest’s recommendation to simply increase the services startup timeout is system-wide, and not something I was willing to apply in full across my entire environment.

My first thought was to utilize the Recovery options with the Windows service, but upon quick inspection its clear this is only for failed services, and a service that isn’t started does not fall into that category.

Instead I decided to set a GPO with a service preference, with service action set to “Start Service”. This way, every time Group Policy is refreshed, if the service isn’t running it will be started.

The GPO itself has a bit of logic to address the AppAssure/Rapid Recovery upgrade we’re going through right now, which results in a service name change for the agent.

To accomplish this, I created two separate service preferences, each with some item-level targetting for registry matching to ensure it only applies if that service does exist (to avoid spamming Event Logs). This takes advantage of the fact that the AppAssure Agent doesn’t record a product version in the registry:

AppAssure Service

  • Service Name = AppAssureAgent
  • Match if KeyExists HKLM\Software\AppRecovery\Agent
  • AND if not ValueExists HKLM\Software\AppRecovery\Agent\ProductVersion

Rapid Recovery Service

  • Service Name = RapidRecoveryAgent
  • Match if ValueExists HKLM\Software\AppRecovery\Agent\ProductVersion

Project Honolulu and Server Core

By now many IT administrators have heard of Project Honolulu from Microsoft. I must admit, when I heard the headline and initial talk about it, I book-marked the info intending to come back but didn’t really dig into it. I thought, “oh a revamped Server Manager.”

My perspective is all changed now.

I’m watching the “Windows Server: What’s new and what’s next” session from Ignite 2017. I have been following Aidan Finn’s blogging of Ignite sessions including this one in particular for the “What’s new” session, and recall reading his notes about Server Core being in semi-annual channel, but not Server GUI, so “you better learn some PowerShell to troubleshoot your networking and drivers/firmware”.

But now having watched the session myself, it makes Microsoft’s vision and a path forward here very clear to me.

Using Server Core allows an organization to reduce their surface area for vulnerability, streamline the size/frequency of Windows Updates, optimize performance and scalability on hardware, and stay up to date on the Windows Server cadence.

Project Honolulu makes using Server Core viable. This is the answer to the Windows system administrator saying “Server Core just doesn’t give me the visibility I need into my servers”. As Jeff Woolsey walks through the functions of Project Honolulu, it is obvious that THIS is where the visibility will be; no more RDP into individual servers to manage their roles, devices, and settings. No more MMC windows to open Event Viewer and Shares and other applets.

Now I’m excited.

Azure PowerShell DNS – Modify TTL

A brief note on modifying TTL on an Azure DNS record set. This is changed on the record set, not the record itself.

The Azure DNS PowerShell docs don’t make it explicitly clear how to do this, however using “help set-AzureRmDnsRecordSet -examples” gave me a clue on how to achieve it.

The key is the middle line here:

$rs = Get-AzureRmDnsRecordSet -Name "msoid" -RecordType CNAME -ZoneName "domain.com" -ResourceGroupName "DNS"
$rs.TTL = 3600
Set-AzureRmDnsRecordSet -RecordSet $rs