Windows Server 2012 Windows Update Error 0x80240440

Jeff Miles 8 Comments

I have begun setting up a new server for a branch office, and have decided to use Windows Server 2012 on it; thanks Software Assurance! This way I can utilize the new Hyper-V features when I’m ready, as well as virtualize a domain controller properly.

 

However, I ran into a problem with Windows Update on both the Host and Guest running Server 2012. Windows Update reported an error:

 

 

The windows update log located at %windir%/windowsupdate.log reported this:

+++++++++++  PT: Synchronizing server updates  +++++++++++
  + ServiceId = {9482F4B4-E343-43B6-B170-9A65BC822C77}, Server URL = https://fe1.update.microsoft.com/v6/ClientWebService/client.asmx
WARNING: Nws Failure: errorCode=0x803d0014
WARNING: Original error code: 0x80072efe
WARNING: There was an error communicating with the endpoint at 'https://fe1.update.microsoft.com/v6/ClientWebService/client.asmx'.
WARNING: There was an error sending the HTTP request.
WARNING: The connection with the remote endpoint was terminated.
WARNING: The connection with the server was terminated abnormally
WARNING: Web service call failed with hr = 80240440.
WARNING: Current service auth scheme='None'.
WARNING: Proxy List used: '(null)', Bypass List used: '(null)', Last Proxy used: '(null)', Last auth Schemes used: 'None'.
FATAL: OnCallFailure(hrCall, m_error) failed with hr=0x80240440
WARNING: PTError: 0x80240440
WARNING: SyncUpdates_WithRecovery failed.: 0x80240440
WARNING: Sync of Updates: 0x80240440
WARNING: SyncServerUpdatesInternal failed: 0x80240440
WARNING: Failed to synchronize, error = 0x80240440
WARNING: Exit code = 0x80240440

 

At first I thought this may be related to the “Trusted Sites” within Internet Explorer. I have mine set through GPO, so I added “https://*.update.microsoft.com” to that GPO and then did a “gpupdate /force”, but the error remained.

 

Then I thought to look at my Sonicwall NSA 2400; we have the Application Control enabled, and this has been known to cause strange network connectivity issues even when not expected so I’ve just by default started checking here.

Unsurprisingly this turned out to be the problem. The strange thing is, the AppControl rule that was blocking the traffic isn’t visible in the list of applications; only through the logging did I find it.
If you navigate to the AppControl settings page, use the “Lookup Signature”, for signature # 6:

 

Click on the pencil icon, and you’ll see this screen:

 

Turns out the rule “Non-SSL Traffic over SSL port” is blocking this Windows Update traffic.

Setting the Block option to Disabled for this rule allows Windows Update to work properly.

 

 

Fixing a DFSR connection problem

Jeff Miles 13 Comments

What a week; actually started over the weekend, with a DFSR database crash on a spoke server within my topology. This happened for a replication group that is 1.6TB in size, so the volume check takes quite a long time.
During this time, replication hung so I decided to restart the DFSR service on our hub server. Unfortunately, the restart failed, and the service was hung at “stopping”. So I killed the dfsrs.exe process, and then started the service.

At this point, it tried to repair the DFSR database, but failed so it went into “initial replication”. Initial replication on a 1.6 TB replication group is a thing straight from my nightmares. Compounding the problem is the fact that the hub server then crashed the next night (which I haven’t had time to look into yet) and basically had to restart the process.

That was 3 days ago, and after all this time, I’ve got initial replication finished but a backlog of 10,000 files going to 2 of the spoke servers. That backlog didn’t appear to be moving, and investigating the DFS Replication section of the Event Log revealed:

The DFS Replication service encountered an error communicating with partner SW3020 for replication group swg.ca\files\jobs. 
The service will retry the connection periodically. 
Additional Information: 
Error: 9032 (The connection is shutting down)

 

The steps I took to fix this error:

  • On the hub server, I deleted the individual connections from the hub to the spoke servers for this specific replication group
  • From a domain controller in the hub site, I ran this to ensure those changes reached the branch sites sooner:
    repadmin /syncall /e /A /P
  • Then I re-created the connections for each spoke and re-ran the repadmin command.

Following that, both servers showed this in the DFSR log:

The DFS Replication service failed to communicate with partner SW3020 for replication group swg.ca\files\jobs. The partner did not recognize the connection or the replication group configuration. 
The service will retry the connection periodically. 
Additional Information: 
Error: 9026 (The connection is invalid)

 

So from each spoke server, I ran the following:

dfsrdiag pollad /v /member:hubserver.domain.com         (Replication partner)
dfsrdiag pollad /v /member:hub_site_dc.domain.com      (Domain Controller in hub site)

 

Shortly thereafter I saw this in the logs:

The DFS Replication service successfully established an inbound connection with partner SW3020 for replication group swg.ca\files\jobs.

 

And now replication traffic is flowing properly. Now all I have to do is deal with the more than 500 conflict files this whole ordeal has generated.

 

MD3220i Disk Group Expansion

Jeff Miles Leave a comment

Now that I’ve finished migrating data from one disk group to another on my MD3220i, I needed to break down the old disk group (14x300GB 10k SAS) and then use those disks to expand another disk group from the original purchase (16x300GB 10k SAS) giving one big RAID10 array of 30 disks.

I basically went backwards from adding storage, by:

  • Removing the Cluster Storage Volume (within Failover Cluster Manager)
  • Deleting the Virtual Disk as available cluster storage (within Failover Cluster Manager)
  • Remove the host-to-LUN mapping (within the Dell MD Storage Manager)
  • Delete the Disk Group and Virtual Disk (within the Dell MD Storage Manager)

Following those steps, I had 14 disk available storage to re-allocate.

I then went to the Logical tab, right clicked on the disk group, and selected “Add Free Capacity”

 

This gave me a list of disks to use, but I could only select two; the wizard wouldn’t let me select the full set.

Unfortunately, I chose two and completed the wizard, which went directly into reconfiguring the Virtual Disk. This is unfortunate since it’s a 14 hour operation, and I really don’t want to do that 7 more times (for each pair).

I took a look in the CLI guide for the MD3220i, but there doesn’t appear to be any options for adding free capacity to a disk group. At this point I’m stuck waiting for each disk pair to be added.

 

Using Quick Storage Migration for VHDs with DFSR data

Jeff Miles 1 Comment

As shown in my last post, I recently added some storage to our SAN, and will be moving existing VHD files from our Hyper-V cluster to this new storage.

The unique thing about this is that these VHD’s contain data that is being served with Microsoft DFS and replicated with DFSR. Hopefully word is spreading about DFSR data stores requirements for backup, which include some specific requirements for backup, especially when it comes to snapshots due to the multi-master database DFSR uses. Because of this information, I was a little concerned about the Quick Storage Migration (QSM) and so I started digging.

I eventually came across this blog post that went into detail about how the QSM works; It mentions that it does a snapshot of the VM and creates differencing disks, and eventually the snapshot is merged and VM restarted from that saved state. At this point I was concerned about my DFS data and so I sent an email to the wonderful and always helpful AskDS blog seeking clarification.

 

Here’s the response from Ned Pyle:

My presumption is that this is safe because – from what I can glean – this feature never appears to roll back time to an earlier state as part of its differencing process.

He then went the extra mile and contacted internal Microsoft peers closely related to the QSM feature, who responded:

That’s correct, we don’t revert the machine state to an earlier time. A differencing disk is created to keep track of the changes while the parent vhd is being copied. Once the VHD is copied, the differencing disk is merged into the parent.

 

Based on that I performed a QSM of my 1.6 TB VHD yesterday. It took 12.5 hours to complete, but in the end it was fully successful, with no negative repercussions.

Something interesting to note, is that I had to manually move a different 350GB vhd file to my new storage first, instead of a QSM since I was out of space on the original storage to create the avhd differencing disk. I shut down the VM, transferred the VHD (took about an hour), and then re-pathed it within the VM settings and turned the VM back on.

Following this I received a DFSR error # 2212 that “The DFS Replication service has detected an unexpected shutdown on volume F:”. I’m not sure why this occurred, and I only did the one transfer so I can’t verify that it wasn’t related to some other operation or bad shutdown.

 

Adding a Dell MD1220

Jeff Miles 1 Comment

Last week was storage expansion week for me; my team and I added a Dell Powervault MD1220 to our existing SAN controlled by a MD3220i.

We got a full unit loaded with 24 x 900 GB 10krpm SAS, and I was shocked at how quickly it shipped to me. Thanks Dell!

 

The actual installation was incredibly easy, and we had the storage running within 10 minutes. The technical guidebook said that the MD3220i needed to be powered off to add the expansion, but based on the discussion here and the fact that the MD1220 was brand new and empty, I decided to do it live.

 

After connecting the SAS cables and turning on the power supplies, the MD1220 went through a startup check, and then by the time I got back to my computer with the MD Storage Manager software on it, it showed 19 TB of available space to be allocated.

 

From that point, it was as simple as creating a new disk group and virtual disks. I added that virtual disk to our host mapping group, and the storage was immediately visible to my Hyper-V cluster hosts through iSCSI; I didn’t have to re-do any iSCSI configuration.

 

I set up the new virtual disk as a Cluster Volume within the Hyper-V Cluster, and then downloaded a trial of System Center Virtual Machine Manager 2012 to use for the storage migration of the VHD’s that need to go on this new cluster volume.

 

Once those migrations are done, I’ll remove one of the old cluster volumes, break up corresponding virtual disk within the MD Storage Manager software, and expand other virtual disks with those available hard drives. Hopefully I don’t find anything surprising while doing this; unfortunately this type of operation isn’t well documented from what I can find.