Hosting Exchange 2010 and Issues With Duplicate Contacts

When you are creating a hosted Exchange system using the Exchange 2010 On Premises product (not the /hosting version of the product) it is likely that if two or more of your customers create a mail contact in the global address list (GAL) for the same external email recipient they will see some issues with email addressing.

For example, you are hosting Exchange for northwind.com and fineartschool.net within one Exchange organization. Both these companies have a professional relationship with greg@fabrikam.com and so want to create a contact for him in the GAL. The first of your clients to create the contact will be successful, but any future client receives the following error when they attempt to create the contact:

New-MailContact -Name "Greg (Fabrikam)" -ExternalEmailAddress greg@fabrikam.com -OrganizationalUnit FineArtSchool
The proxy address "SMTP:greg@fabrikam.com" is already being used by "isp.corp/Hosted/Northwind/Greg (Fabrikam)". Please choose another proxy address.
    + CategoryInfo          : NotSpecified: (…) :ADObjectId) [New-MailContact], ProxyAddressExistsException
    + FullyQualifiedErrorId : B333D21C,Microsoft.Exchange.Management.Recipient
   Tasks.NewMailContact

The work around is to specify a unique proxy address, as the default proxy address (the contacts actual email address) is already being used:

New-MailContact -Name "Greg (Fabrikam)" -ExternalEmailAddress greg@fabrikam.com -OrganizationalUnit FineArtSchool -PrimarySmtpAddress greg@fineartschool.net

Of course Greg’s email address is greg@fabrikam.com (his external email address) and not greg@fineartschool.net (his proxy or primary SMTP address so far as Fine Art School have configured) and if this client sends an email to Greg and they select Greg from the GAL it will go to his external email address but will look like it has gone to his proxy address. That is, Greg will receive the email but if he looks at the address it was sent to it will say greg@fineartschool.net.

Send an email to two people in external organizations, one being greg@fabrikam.com, and hit Reply All and Greg will appear as greg@proxyaddress and not greg@fabrikam.com. Emails in reply will go to Greg via the hosting company and not direct to Greg. This also has the side affect of showing presence (from Microsoft Lync) as being unavailable as the email is using the wrong email address.

The underlying problem is that though the email is being delivered to the external address (targetAddress attribute in Active Directory) it is being stamped with the primary SMTP address (proxyAddresses in Active Directory) in the P2 header. The P2 header is used to generate the Reply address.

So how do you fix this? The obvious way at first glance is to modify active directory and change the proxyAddresses value back to the correct value – but this does not work (as two objects cannot have the same proxy address). Regardless of the fact that the two mail contacts both have the same targetAddress and proxyAddresses, Exchange Transport detects a problem and reports the error “More than one Active Directory object is configured with the recipient address greg@fabrikam.com. Messages to this recipient will be deferred until the configuration is corrected in Active Directory” in the event log on the first Hub Transport server that sees the message.

So without writing your own transport agent, you need to route all outbound email via an Edge Transport server and configure the Address Rewriting agent. You need to create an address rewrite rule for every contact that is created within your hosted organization once the second contact is created. So in your mail contact provisioning application you need to trap the duplicate proxy address error above, reissue the mail contact creation step, this time with a unique primary SMTP address in the hosted clients domain and then at the same time make an address rewrite rule on your Edge Transport server.

New-AddressRewriteEntry -Name "Greg - Fabrikam - HosterFineArtSchool" -InternalAddress greg@fineartschool.net -ExternalAddress greg@fabrikam.com -OutboundOnly $true

Note that rewrite rules are cached for four hours, so unless you restart the MSExchangeTransport service your rewrite rules will not take effect until four hours have gone by.

Running Offline Web Applications from IIS Server

A feature of HTML 5 based applications is the ability to ensure that applications can still run even if internet connectivity is not present. How to do this is covered on the W3.org website.

A requirement of offline access is the creation of the offline cache manifest file. This manifest file is listed in the HTML tag on the page as such:

<html manifest="offline.appcache">

And a page is saved to the web server with the same name (offline.appcache in this example). This .appcache file follows the conventions described in the above W3.org web page, but this page needs to be served from the web server with a specific MIME type (text/cache-manifest). If the web server is IIS 5.0 or later then it will only serve content that has been listed as a valid MIME type in Windows. If you used a shared hosted webserver then making that change is probably impossible – so from IIS 7.0 or later you can add your own MIME type in the admin UI or modify the web.config file in the root of your web server to add this MIME type. This is just a text file that you upload and so requires no access to the IIS admin application (again, typically something you do not get with  shared hosted web server).

Note: In the example given below, the web.config file changes two properties. If you have an existing web.config file then merge these changes into your file and do not replace your file.

The web.config file needs to be as follows:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <staticContent>
            <mimeMap fileExtension=".appcache" mimeType="text/cache-manifest" />
        </staticContent>
    </system.webServer>
    <location path="offline.appcache">
        <system.webServer>
            <staticContent>
                <clientCache cacheControlMode="DisableCache" />
            </staticContent>
        </system.webServer>
    </location>
</configuration>

The two changes set in this web.config file are, firstly, mimeMap in the staticContent section of system.webServer. This adds the .appcache extension as text/cache-manifest. The second change is clientCache in staticContent section of system.webServer (but this time in a location section, limiting the effect of the setting to the named file – offline.appcache). This change stops the web server or client from caching the page, ensuring that the web server always serves the latest copy of the page.

Upload web.config and your appcache manifest file, along with any page that needs to be viewed offline (or indeed any page that you want to speed up loading for, by causing the pages to be cached on the client) and check that when you browse to the .appcache file directly in a HTML 5 aware browser it is visible. If you get a 404 error on this page then you have not set the MIME type or uploaded the correct web.config file.

HTTPS Load Balancer Issues with Exchange 2010 SP2

When you install Service Pack 2 (and maybe SP1 too) on Exchange 2010 it resets the SSL flag on the root directory of the IIS website. You might have removed this setting for a number of reasons, mainly to do with having a HTTP to HTTPS redirect, but it can also be removed if you are doing SSL Offloading to a load balancer and that load balancer checks the state of the client access server by doing HTTP requests for the root home page. The Citrix Netscaler is one such load balancer that has this as a default setting.

The configuration documentation for the Citrix Netscaler (found here) does not discuss changing the load balancer to use a different directory on IIS to monitor the availability of the site, so when you install SP2 for Exchange 2010 and that update resets the root directory to require SSL, your load balancer thinks the site is offline and does not pass through any traffic!

To fix this issue in the short term, just uncheck the Require SSL option on the root of the Default Web Site on each of your Client Access Servers. Your load balancer should notice within a few seconds and service will resume, for example the Citrix Netscaler checks the root directory via the monitor properties every five seconds for a HTTP success code (and not a HTTPS success code!).

To fix this issue in the long term you should make a new virtual directory on each server covered by the load balancer and get the load balancer to look at this directory to determine if the service is up or down rather than looking at the root directory. Your virtual directory will not be reconfigured by future Exchange service packs (or indeed any other application that you are load balancing that might reset the SSL option on the root directory).

To complete these steps do the following:

1. Create a folder in the inetpub directory called “monitor” or similar (in the examples below the folder is called “netscaler_monitor”).

2. Place an index.htm file in this folder that is a very simple webpage that when browsed returns the page. If you want to make the page more complex to include code (so that issues with the code are picked up by the load balancer then this is fine). A simple page would look like the following:

<html><head>
<title>Netscaler Monitor for Exchange 2010</title>
</head><body>
<p>This page returns a success code to the netscalers if IIS is running. This page must always work over HTTP and never require an SSL connection.</p>
</body></html>

3. In IIS require SSL and then uncheck require SSL – this forces a setting into the IIS config file (applicationHost.config) that says that this folder must always be over HTTP and not require SSL. If you do not do this then this folder will take the setting from the parent folder, and as we have already seen, this will cause the monitor folder to require SSL when you apply the service pack.

This SSL change will result in the following configuration at the bottom of applicationHost.config, which can be added directly to the config file rather than in IIS Manager.

    <location path="Default Web Site/netscaler_monitor">
        <system.webServer>
            <security>
                <access sslFlags="None" />
            </security>
        </system.webServer>
    </location>
</configuration>

4. Update your load balancer so that it has a new monitor for checking the service state on the managed machine. This monitor would be something like the following for a Citrix Netscaler, each load balancer being different. This monitor checks HEAD /netscaler_monitor/ and expects to get back a 200 status code. You need to change the folder name to match, but ensure the / is before and after the folder name.

5. Change the configuration for each client access server in the load balancer so that it uses the new monitor rather than the default HTTP monitor.

6. Save your changes to the load balancer. The next time you service pack Exchange 2010 the resetting of the SSL flag on the root directory will not cause you any issues.

Shadow Redundancy Promotion Disabled After Exchange Server 2010 Service Pack 2

One of the new features added in Exchange Server 2010 Service Pack 1 was called Shadow Redundancy Promotion.
Service Pack 1 creates a setting in EdgeTransport.exe.config to control the behaviour of this setting, and it defaults to a value of “False”, so that the feature is turned off.
Enabling Shadow Redundancy Promotion is done by setting the ShadowRedundancyPromotionEnabled setting to “True” in EdgeTransport.exe.config (found in c:\program files\microsoft\exchange server\v14\bin) and restarting the MSExchangeTransport service.
When you install Service Pack 2 for Exchange Server 2010 it resets this value to false and therefore disables this setting. This is not documented (at the time of writing) in the read me for the service pack.
This does mean of course that if you have chosen to enable Shadow Redundancy within your SP1 Exchange 2010 organization, this will get turned off again when you apply the service pack. Note that other settings in EdgeTransport.exe.config are not reset on applying the service pack.
When the service pack resets the value, it does make a comment in the config file to sort of indicate what it has done. It notes the following text above the line that disables Shadow Redundancy.

<!--Exchange task Set-AppConfigValue updated value for dictionary key ShadowRedundancyPromotionEnabled from True to False at 12/5/2011 5:09:54 AM—>

It does not say it has turned your configuration off again, but that is the effect of the change that the service pack installation makes.

Exchange 2010 SP2 Prerequisites

To upgrade Exchange 2010 SP1 to SP2 requires the addition of an additional Windows server role that was not listed on the RTM and SP1 requirements.

Therefore if you try an upgrade you will get errors about "IIS 6 WMI Compatibility” not being installed.

To install this feature quickly on Windows Server 2008 R2, bring up PowerShell and enter the following commands:

Import-Module ServerManager
Add-WindowsFeature Web-WMI

You can then proceed with the SP2 installation. This role feature is required for Mailbox and CAS roles and specifically for the new OWA Mini feature that Service Pack 2 introduces.

If you are running Exchange 2010 on Windows 2008 (not the R2 version) then the above PowerShell command does not work. Instead you need to install the prerequisites with the following (from an administrative command prompt):

ServerManagerCmd -install Web-WMI

But is this really the only new prerequisite? If you take a look in the exchange-typical.xml file found in the service pack download (in the scripts folder) then you will see a long list of prerequisites, and more than just Web-WMI is new since Service Pack 1. The full list of prerequisites are:

NET-Framework, RSAT-ADDS, Web-Server, Web-Basic-Auth, Web-Windows-Auth, Web-Metabase, Web-Net-Ext, Web-Lgcy-Mgmt-Console, WAS-Process-Model, RSAT-Web-Server, Web-ISAPI-Ext, Web-Digest-Auth, Web-Dyn-Compression, NET-HTTP-Activation, RPC-Over-HTTP-Proxy, Web-WMI

But are they all needed? Only having Web-WMI missing from an SP1 install stops the service pack from being installed, but I suppose that for your own sanity you would install all the prerequisites even if they appear not to be!

Therefore if you run Windows 2008 R2 it is best to run (from PowerShell):

Import-Module ServerManager
Add-WindowsFeature NET-Framework, RSAT-ADDS, Web-Server, Web-Basic-Auth, Web-Windows-Auth, Web-Metabase, Web-Net-Ext, Web-Lgcy-Mgmt-Console, WAS-Process-Model, RSAT-Web-Server, Web-ISAPI-Ext, Web-Digest-Auth, Web-Dyn-Compression, NET-HTTP-Activation, RPC-Over-HTTP-Proxy, Web-WMI

And if you run Windows 2008 then execute as an administrator:

ServerManagerCmd -install NET-Framework RSAT-ADDS Web-Server Web-Basic-Auth Web-Windows-Auth Web-Metabase Web-Net-Ext Web-Lgcy-Mgmt-Console WAS-Process-Model RSAT-Web-Server Web-ISAPI-Ext Web-Digest-Auth Web-Dyn-Compression NET-HTTP-Activation RPC-Over-HTTP-Proxy Web-WMI

Or, skip all this and just install the service pack from the command line (administrative rights of course) with the following command to install all the required updates and prerequisites:

setup /m:upgrade /InstallWindowsComponents

Blog updated 6th Dec to cover the installation on Windows 2008 and to point out the difference for Windows 2008 R2

Exchange Log Truncation Failure in a DAG

Today I visited a client who had noticed that no log files had ever been removed after any backup within Exchange 2010 SP1. It was fortuitous that they had enough log disk space for about eight months of log generations. The disadvantage was that we were four months into this time period, so it was a ticking clock, and that the nightly incremental backups were taking longer and longer.

They were getting the following error in their backup datacentre:

Unable to communicate with the Microsoft Exchange Information Store service to coordinate log truncation for database 'name' due to an RPC communication failure. Error 3355379671 Extended Error: 0 and Event ID 2136 for the MSExchangeRepl service in the Application event log.

What the error does not clearly say is that the Microsoft Exchange Replication service (MSExchangeRepl) on the server in the DR site (a passive node in the DAG) needs to communicate via RPC to the Microsoft Exchange Information Store service on the server holding the active node of the database.

In the case of my client, the Exchange team is not the same people as the network team or indeed the firewall team, and these teams are in different countries. In the case of the network for this client, the Replication network for the DAG had been opened to allow RPC traffic, but the MAPI (Client) network had not.

When Exchange in the DR site needed to check which logs it could truncate (a process it performs every 15 minutes), it needs to talk to the Microsoft Exchange Information Store service on the server holding the active copy of the database, and name resolution was returning (as expected) the IP address of the server on the MAPI/Client network. This network blocked RPC between servers and so (as one of the many issues they now attribute to this problem) logs could not be truncated and Event ID 2136 was posted once per database on the passive node in the DR site. The two servers in the primary site could RPC each other, so this log is not repeated in the primary site.

To solve this log growth problem without waiting for a response from the firewall team, we added a record to the hosts file on the passive server to override DNS name resolution, and within 15 minutes 2TB of log files instantly disappeared on all servers. Name resolution was reverted to DNS and the firewall team contacted.

Office 365 Hybrid Coexistence and Edge Server

One of the delights in my job is when Microsoft give me a call and ask me how something works in one of their products! Such a call came today and it involved get Office 365 hybrid coexistence working with an Edge Server.

Exchange Server Deployment Assistant does not have the answer to this issue; it always refers to a Hub Transport server within the organization. But it was an interesting challenge, as the last time I tried to get this working I got a “a local loop detected” error.

In brief, its is possible to use Edge Server as the coexistence server between your organization and your Office 365 tenant, you just need to make sure that everything you would create on the Hub Transport is either configured on the Edge Server (receive connectors and certificates) or created within the organization and replicated to the Edge Server via EdgeSync (send connectors, remote domains, accepted domains).

The reason for “a local loop detected” error was down to having the service domain for the Office 365 tenant configured as Internal Relay (as it needs to be) and not having a Send Connector on the Edge Server that pointed to the Office 365 infrastructure.

Update 26 Oct 2011

In the above I wrote that the Office 365 tenant domain as configured on-premises needs to be internal relay. This is only required if your on-premises org contains Exchange 2003. If the minimum version of Exchange installed on-premises is 2007 then the domain can be Authoritative. This is because with 2007+ you can forward emails to an authoritative domain using contact or remote-mailbox objects and a send connector for the domain.