I was asked a question recently on the Microsoft Certified Master course for Exchange 2010 and was told that the answer was not clearly written up on the internet. So I thought I would write this blog post. The question was based on the idea that Domain Secure worked from a Hub Transport server in the classroom lab but not when mail flow went via an Edge server.
Domain Secure is end to end security, it cannot have anything in the middle – i.e. it cannot go via an Exchange Edge server, an Exchange 2013 Frontend Server or a third party SMTP relay.
The SMTP client in the connection (the send connector host) needs to connect to the SMTP server (the receive connector host) and swap certificates and prove the other side is who the other side say they are – i.e. mutual authentication. Also the domains must match the TLS list in TransportConfig (TLSSendDomainSecureList and TLSReceiveDomainSecureList). Therefore anything in the middle will offer a different certificate and so Domain Secure fails.
If there is a middle party and you want to do mutual authentication (i.e. swap certs to prove who you are), with one party offering their cert and not the cert of the final recipient domain (i.e. mail.messaging.microsoft.com or postini.com etc.) then use TLSAuthLevel and the DomainValidation option on the send connector (an SP1 addition to Exchange 2010). No green ticky ticky though.
Edge can do Domain Secure though. But Edge needs to be the starting point, i.e. the host of the send connector. So configure Domain Secure on the Edge (i.e. set the certificates and correct firewall settings) and ensure that the send connector for Domain Secure has the Edge server as the source. Ensure Edge Domain Secure receive connector is the target for inbound as well if you want it to work both ways. And of course you need working EdgeSync so hubs can deliver to Edge so that Edge can deliver emails for you.