Expand Commvault Disk Library

My Commvault disk library is nearing 90% capacity, which is a growing cause for concern. The disk library is provided by a Dell MD1400 with 12x6TB SAS drives, which was originally configured as a 10 disk RAID6 set with 2 hot spares (~45TB usable) on the PERC H830 card.

As an intermediate step before purchasing an additional disk shelf, I did the following to give a little breathing room.

  1. Using the Dell Open Manage Server Administrator (OMSA), I located one of the hot spare drives and unassigned it.
    Dell OMSA unassign global hot spare
  2. Still within OMSA, I chose the “Reconfigure” option on my Virtual Disk, keeping the RAID level at 6 but adding in the new available capacity.
  3. Then I waited. It took 16 days for the reconfiguration to complete as the RAID6 parity was re-spread/calculated across the set. This left me with a ~50TB virtual disk.
    Dell OMSA virtual disk size
  4. Within Disk Management, I carved the free space into two additional ~3TB volumes, which are mounted within a standard disk library path of L:\DiskLibrary. This size adheres to the recommendations given to us by our professional services partner.
    Disk management showing new volumes
  5. During the original configuration, we used Automated Mount Path Detection, to point to L:\DiskLibrary:
    commvault export storage config example
  6. Due to this, the new mount paths were picked up automatically after about 15 minutes, and appeared within the list of mount paths under the library:
    mount paths on commvault library

Checking the Disk Usage tab on the properties of the library dropped utilization down to 80%, and provided us some time to address the growing data concerns.

 

Exchange Online PowerShell access denied

I am attempting to test aspects of Office 365 Modern Authentication in a UAT environment prior to enabling it within our production Tenant.

Part of this work is testing the Exchange Online PowerShell access, as there is quite amount of automation configured in our environment and we want to ensure it doesn’t break. I’ve read it “shouldn’t”, but that’s a dangerous word to trust.

Until now I’ve been unable to make the PowerShell connection to Exchange Online in our UAT environment, receiving the following during my attempts:

New-PSSession : [outlook.office365.com] Connecting to remote server outlook.office365.com failed with the following error message:[ClientAccessServer=servername,BackEndServer=servername.prod.outlook.com,RequestId=e6f6b9e7-7c5e-45ec-87fe-59332db1fb95,TimeStamp=8/17/2017 3:16:52 PM] Access Denied For more information, see the about_Remote_Troubleshooting Help topic.

I can use the same account to connect in-browser to http://portal.office.com, and it is set as a Global Administrator in O365, so I know that the account itself has appropriate access.

Interestingly, if I connect with the MFA-supported PowerShell method, with the same account, it connects successfully.

Through testing I’ve determined that using any on-premise account synchronized through Azure AD Connect fails with the same “Access Denied” message, while any cloud-only account connects successfully.

I began to look at our ADFS implementation in UAT since that is a key component for authenticating the on-premise user account. This environment has ADFS 2.0 on Server 2008 R2, which is different than production but shouldn’t be a barrier to connectivity (without MFA).

After comparing the O365 trust configuration and finding no issues, I decided to use the Microsoft Connectivity tool to test. Using the Office 365 Single Sign On test, I saw a failure with this error:

A certificate chain couldn't be constructed for the certificate.
Additional Details
The certificate chain has errors. Chain status = NotTimeValid.

This let me on the path to fixing expired/broken SSL certificates in our UAT ADFS, which I posted about previously here.

Now that the SSL problem is resolved, I attempted to connect to Exchange Online PowerShell again, and was successful!

Looks like this “Access Denied” message was directly related to the expired certificate of the ADFS proxy.

 

 

ADFS 2.0 renew Service Communications certificate

I’ve recently solved a problem with the help of Microsoft Premier Support that didn’t have any references online that I could find.

Looking at the ADFS console under Certificates, the “Service Communications” section had a message of “Certificate not found in store”.

Connecting to the certificate store showed a proper external SSL cert for our UAT ADFS DNS name. Trying the option “Set Service Communications Certificate” in ADFS produced the error:

The Certificate could not be processed.
Error message: Object reference not set to an instance of an object.

This error led me to this discussion on the Microsoft forums, with the following command to attempt:

AddPsSnapin Microsoft.Adfs.PowerShell SetAdfsCertificate CertificateType “Service-Communications” Thumbprintaa bb cc dd …”

However, when I tried to run this command I repeatedly got the following error:

The type initializer for 'Microsoft.IdentityServer.Dkm.ADRepository' threw an exception. Microsoft.IdentityServer.PowerShell.Commands.SetCertificateCommand

The resolution: run PowerShell as the ADFS service account, and then use the command above to set the certificate. After this, I was able to restart the ADFS service and the console displayed the certificate properly.

I also needed to update the certificate on the ADFS proxy in IIS to get a successful result from the Microsoft Remote Connectivity Analyzer.

WordPress 403 error when saving post

While writing my last post, I encountered a strange error with WordPress. I had written up the majority of my post, and went to save the draft and received this error:

403 error from wordpress

I tried copying and pasting my text into a new post, and it still gave the error.

I found I could still type a few words and save a draft, so I began experimenting with the remaining text of my post that hadn’t been saved yet.

Eventually I stumbled across the word “web dot config” like this (can’t type it obviously)

 

When this was included in the body of my post, it would not save. This is a really hard item to come up with accurate search terms for, so I couldn’t find any specific references of anyone else having and solving this problem. I suspect it has something to do with the .htaccess and re-write rules configured within WordPress.

NLB website behind Azure Application Proxy

This is a quick post noting the minimum steps required to get a network load balanced website in IIS accessible through Azure AD Application Proxy. I’ve recently set this up for Deltek iAccess, and after a bunch of minor issues and mis-configurations (which took me hours to find, of course) I thought it worthwhile to compile the information in one spot.

Note, when troubleshooting this process, many different items can be cached which causes problem solving to go down incorrect paths and assumptions. This was perhaps the most challenging part of getting this set up. You’ll likely need to use the following commands regularly, in addition to waiting for AD and DNS sync intervals, and re-testing some minutes after changes are made in the Azure portal.

  • net stop dnscache && net start dnscache
  • ipconfig /flushdns
  • klist purge

Assumptions in this post are that NLB is already configured and working, and the website (or at least a portion of it) uses integrated Windows Authentication (otherwise many of these steps are unnecessary).

  1. Determine your DNS records. We have split dns and so used the same name internally and externally: “webapp.domain.com”
    1. Create a CNAME in internal DNS for “webapp.domain.com” to point to your NLB cluster name
    2. Create a CNAME in external DNS for ‘webapp.domain.com” to point to the provided Azure Application Proxy name; likely something like app-domain.msappproxy.net
  2. In IIS on each web host, set “UseAppPoolCredentials” to True using Configuration Editor (or web config directly) for the path: system.webServer/security/authentication/windowsAuthentication
  3. In IIS on each web host, configure the application pool identity to a domain user service account
  4. Create an SPN delegation in AD to match your DNS name and this service account:
    1. SetSPN -S HTTP/webapp.domain.com domain\username
  5. Use this SPN “HTTP/webapp.domain.com” within the configuration of your Azure AD Application Proxy.
  6. Add this SPN entry on the Delegation tab of the Azure AD proxy connector object in AD. You would search for the domain user service account, and select the HTTP entry matching webapp.domain.com

Things you don’t have to do, discovered after some trial and error:

  • Use a CNAME vs A record for internal DNS; both should work properly
  • Configure “DisableStrictNameChecking” or “BackConnectionHostNames”, as these apply to NTLM only, not Kerberos
  • Grant NTFS permissions to the domain user service account on the web application files
  • Specify a specific hostname within IIS binding configuration
  • Add the NLB cluster server names to your proxy connector delegation tab