Exchange Online PowerShell access denied

I am attempting to test aspects of Office 365 Modern Authentication in a UAT environment prior to enabling it within our production Tenant.

Part of this work is testing the Exchange Online PowerShell access, as there is quite amount of automation configured in our environment and we want to ensure it doesn’t break. I’ve read it “shouldn’t”, but that’s a dangerous word to trust.

Until now I’ve been unable to make the PowerShell connection to Exchange Online in our UAT environment, receiving the following during my attempts:

New-PSSession : [outlook.office365.com] Connecting to remote server outlook.office365.com failed with the following error message:[ClientAccessServer=servername,BackEndServer=servername.prod.outlook.com,RequestId=e6f6b9e7-7c5e-45ec-87fe-59332db1fb95,TimeStamp=8/17/2017 3:16:52 PM] Access Denied For more information, see the about_Remote_Troubleshooting Help topic.

I can use the same account to connect in-browser to http://portal.office.com, and it is set as a Global Administrator in O365, so I know that the account itself has appropriate access.

Interestingly, if I connect with the MFA-supported PowerShell method, with the same account, it connects successfully.

Through testing I’ve determined that using any on-premise account synchronized through Azure AD Connect fails with the same “Access Denied” message, while any cloud-only account connects successfully.

I began to look at our ADFS implementation in UAT since that is a key component for authenticating the on-premise user account. This environment has ADFS 2.0 on Server 2008 R2, which is different than production but shouldn’t be a barrier to connectivity (without MFA).

After comparing the O365 trust configuration and finding no issues, I decided to use the Microsoft Connectivity tool to test. Using the Office 365 Single Sign On test, I saw a failure with this error:

A certificate chain couldn't be constructed for the certificate.
Additional Details
The certificate chain has errors. Chain status = NotTimeValid.

This let me on the path to fixing expired/broken SSL certificates in our UAT ADFS, which I posted about previously here.

Now that the SSL problem is resolved, I attempted to connect to Exchange Online PowerShell again, and was successful!

Looks like this “Access Denied” message was directly related to the expired certificate of the ADFS proxy.

 

 

ADFS 2.0 renew Service Communications certificate

I’ve recently solved a problem with the help of Microsoft Premier Support that didn’t have any references online that I could find.

Looking at the ADFS console under Certificates, the “Service Communications” section had a message of “Certificate not found in store”.

Connecting to the certificate store showed a proper external SSL cert for our UAT ADFS DNS name. Trying the option “Set Service Communications Certificate” in ADFS produced the error:

The Certificate could not be processed.
Error message: Object reference not set to an instance of an object.

This error led me to this discussion on the Microsoft forums, with the following command to attempt:

AddPsSnapin Microsoft.Adfs.PowerShell SetAdfsCertificate CertificateType “Service-Communications” Thumbprintaa bb cc dd …”

However, when I tried to run this command I repeatedly got the following error:

The type initializer for 'Microsoft.IdentityServer.Dkm.ADRepository' threw an exception. Microsoft.IdentityServer.PowerShell.Commands.SetCertificateCommand

The resolution: run PowerShell as the ADFS service account, and then use the command above to set the certificate. After this, I was able to restart the ADFS service and the console displayed the certificate properly.

I also needed to update the certificate on the ADFS proxy in IIS to get a successful result from the Microsoft Remote Connectivity Analyzer.

NLB website behind Azure Application Proxy

This is a quick post noting the minimum steps required to get a network load balanced website in IIS accessible through Azure AD Application Proxy. I’ve recently set this up for Deltek iAccess, and after a bunch of minor issues and mis-configurations (which took me hours to find, of course) I thought it worthwhile to compile the information in one spot.

Note, when troubleshooting this process, many different items can be cached which causes problem solving to go down incorrect paths and assumptions. This was perhaps the most challenging part of getting this set up. You’ll likely need to use the following commands regularly, in addition to waiting for AD and DNS sync intervals, and re-testing some minutes after changes are made in the Azure portal.

  • net stop dnscache && net start dnscache
  • ipconfig /flushdns
  • klist purge

Assumptions in this post are that NLB is already configured and working, and the website (or at least a portion of it) uses integrated Windows Authentication (otherwise many of these steps are unnecessary).

  1. Determine your DNS records. We have split dns and so used the same name internally and externally: “webapp.domain.com”
    1. Create a CNAME in internal DNS for “webapp.domain.com” to point to your NLB cluster name
    2. Create a CNAME in external DNS for ‘webapp.domain.com” to point to the provided Azure Application Proxy name; likely something like app-domain.msappproxy.net
  2. In IIS on each web host, set “UseAppPoolCredentials” to True using Configuration Editor (or web config directly) for the path: system.webServer/security/authentication/windowsAuthentication
  3. In IIS on each web host, configure the application pool identity to a domain user service account
  4. Create an SPN delegation in AD to match your DNS name and this service account:
    1. SetSPN -S HTTP/webapp.domain.com domain\username
  5. Use this SPN “HTTP/webapp.domain.com” within the configuration of your Azure AD Application Proxy.
  6. Add this SPN entry on the Delegation tab of the Azure AD proxy connector object in AD. You would search for the domain user service account, and select the HTTP entry matching webapp.domain.com

Things you don’t have to do, discovered after some trial and error:

  • Use a CNAME vs A record for internal DNS; both should work properly
  • Configure “DisableStrictNameChecking” or “BackConnectionHostNames”, as these apply to NTLM only, not Kerberos
  • Grant NTFS permissions to the domain user service account on the web application files
  • Specify a specific hostname within IIS binding configuration
  • Add the NLB cluster server names to your proxy connector delegation tab

Server 2016 VM freeze up

I recently deployed a couple Server 2016 virtual machines within my environment, and have been having an issue with them freezing up after periods of inactivity. Symptoms would be inaccessible on the network, locked up from the Hyper-V console (i.e. unresponsive to the Ctrl+Alt+Del command), and not responding to any shutdown commands (from Hyper-V or command line).

I initially thought this might be an incompatibility with the hypervisor, as it is still Server 2012 R2, or perhaps a missing hotfix/update but after some research this doesn’t seem to be the case.

Yesterday I finally hit on a lead sourced from this discussion thread, which indicates the problem originates from the Pagefile being sourced on a separate VHDX.

Turns out this is exactly how I configure my VMs, so that if I ever decide to protect or Hyper-V replica the VM I can exclude it.

The resolution for this particular issue is to:

  • Right click Start Menu
  • Choose “System”
  • Click “Advanced System Settings”
  • Under Startup and Recovery, click “Settings”
  • Change the Write debugging information dropdown to “None”

The implications of this setting are that if Windows crashes due to unexpected failure, it will not create a memory dump file. More detail can be found here.

Azure Automation – Distribution Group

I’ve been wanting to expand my skills and abilities within the Azure platform recently, and this week a great opportunity arose for this very thing.

We provide dynamic distribution groups within Office 365 for staff, based upon AD attributes populates with DirSync. The primary example here is our “SubCompany All Staff” list, which is created with this query filter:

New-DynamicDistributionGroup -alias SubCompanyAllStaff -Name "SubCompany All Staff" -RecipientFilter { ((RecipientType -eq 'UserMailbox') -and (Company -like 'SubCompanyName*')) }

Now we have executive staff who are transitioning from this SubCompany to a parent company, which is causing an issue. Their Company attributes are no longer “SubCompanyName”, but they wish to remain on the distribution list to receive communications. However, IT does not (will not) maintain a manual membership list for this type of group. We could create an “exceptions” group, add these staff in, mail-enable the group and make sure it’s part of the query filter above, but this means every time someone changes company, we have to remember to add them into the exception group.

A better way is to filter on the ProxyAddress attribute; anyone who has an @subcompany.com email should be part of the distribution group. Those who switch companies maintain legacy @subcompany.com addresses as aliases. For example:

Get-Recipient -Filter {EmailAddresses -like *@subcompany.com*}

However, while the Get-Recipient command works here, this type of filtering is NOT supported within a dynamic group recipient filter; it returns no results.

This leads me to the solution: use an Azure Automation runbook to run the Get-Recipient command to populate a static group on a schedule.

The Query

First, I needed a functioning PowerShell query. I initially arrived at this here:

# Add users matching criteria to list.
 
$users = Get-Recipient -Filter {EmailAddresses -like "*subcompany.com*" -and RecipientTypeDetailsValue -eq 'UserMailbox'}
 
foreach($user in $users) { Add-DistributionGroupMember -Identity CompanyTestAll@company.onmicrosoft.com -member $user.WindowsLiveID }
 
# Remove those who no longer have an alias
 
$users = Get-DistributionGroupMember -Identity "Company TestAll" | Get-Recipient -Filter {EmailAddresses -notlike "*@subcompany.com*"} 
 
foreach ($user in $users) {Remove-DistributionGroupMember -Identity "Company TestAll" -member $user.WindowsLiveID -confirm:$false}

While this did perform what I was after, it was messy. I wanted this to run on a schedule, so the first ‘addition’ command would effectively error out on the majority of the group members. When I tried to run the second command as my service account in the run book, it threw a large amount of errors about the users not being found on the domain controller.

After a lot of back-and-forth testing, I had this command that works perfectly:

Update-DistributionGroupMember "Company TestAll" -Members (Get-Recipient -Filter {EmailAddresses -like "*subcompany.com*" -and RecipientTypeDetailsValue -eq 'UserMailbox'}|select -ExpandProperty Alias) -confirm:$false

 

Automation

Having never looked at Azure Automation before, I started reading. This blog post was a helpful start, and I went through quite a bit of the docs.microsoft.com site reading about Runbooks, PowerShell vs. PowerShell Workflow, credential access, and so on.

I created an Automation account that is intended to be specific to Office 365 scripts, along with a corresponding Resource Group. Since this isn’t going to touch any other Azure resources, I did not create a RunAs account with it.

Next I began looking at the best way to connect to Exchange Online. I quickly found this module which looked very promising. However, after importing it as an Asset in the account, the example given of “Set-ModuleCredential” wasn’t found as a valid cmdlet, and I couldn’t see it in the .psm1 file either. Since there were many other examples of connecting to Exchange, I moved on.

The next problem I found was that when I tried to do the standard RemoteSession to connect, testing the runbook would fail without any meaningful error message; just 3 attempts and then “suspended”. Based on that behavior, I found this on the feedback portal for Azure, which is exactly what was happening. The method provided by Rune Myrhaug works great:

# Pull credential from Automation assets
$credObject = Get-AutomationPSCredential -Name "svc_auto_azure"
 
#Connect to Exchange Online
$ExchangeOnlineSession = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/PowerShell-LiveID -Credential $credObject -Authentication Basic -AllowRedirection
Import-Module (Import-PSSession -Session $ExchangeOnlineSession -AllowClobber -DisableNameChecking) -Global

I created a service account in Active Directory, synchronized it to O365 with DirSync, and then used the Exchange Admin portal to ensure it had appropriate roles to manage Distribution Groups. I added this account in as a credential asset, and created a daily schedule as well.

The final step was to publish the runbook and associate it with my schedule, and now I have an automated quasi-dynamic distribution group!

I can forsee this potentially being useful for groups that are not built along attribute filters. Perhaps something like a SharePoint list that is populated by administrative staff or department heads, which is then consumed in the runbook and applied to a distribution group.