Dave Stork's IMHO

Mail enabled Public Folder Recipient not found 16 March 10 06:46 AM | dmstork | 0 Comments

Recently I transitioned an Exchange 2003 Server to Exchange 2010. For the client redirection, I mostly wait two week before decommissioning the old Exchange server. Mailboxes, mail flow, remote access and the default Public Folder are all transitioned to the new server. So, in the event of a failure of the old server, the organization would not notice it.

Eventually most or all clients have been redirected, the new server has proven its stability and nothing stands in the way of decommissioning that old server.

Symptoms

So, the decommissioning in this case was troublesome. The setup exited halfway with an error and after that the setup could not remove all of Exchange, because it could not find the items to remove. Even the setup.log couldn't help me any further. Eventually I decided to manually remove Exchange 2003. The server itself would also be decommissioned; any leftovers on the server would be resolved eventually.

The new environment worked without any problems after that. But around 24 hours later, the administrators noticed that the mail enabled Public Folders couldn't receive any mail and the sender would get the following NDR:

554 5.6.0 STOREDRV.Deliver.Exception:ObjectNotFoundException;
Failed to process message due to a permanent exception with message The Active Directory user wasn't found.

Every mail enabled Public Folder was affected, even new mail enabled folders after this issue first arose. The folders where present in the "Public Folder" Address list, GAL and so forth. Even after re-generating the Offline Address book, the issue persisted. I have tested this via Outlook 2003/2007 and 2010 (beta) and via Outlook Web App. No mailbox enabled user had this issue and as far as we know no mail user had related issues with receiving (SMTP) mail.

I'm not sure of the exact steps, but when I tried to make a recipient with the same SMTP address as the troubled Public Folder, Exchange would give the same mail address. In other words, Exchange knew of the existence of that recipient, but couldn't find it.

A search on the internet wasn't successful in finding a relevant knowledge base article or such, although others seem to have the same issue or related issues. The pattern I noticed was that recently an Exchange (2003) server was decommissioned. See here and here. This post describes the same issue, but only has a work intensive workaround

Both first posts are related to Public Folder replication, but as this replication uses SMTP this issue could have the same cause.

Solution

As stated in the comments in both first posts, the resolution was to remove the (empty) Server container via ADSIedit. In my case it was the server container of the administrative group recently containing the Exchange 2003 server. In my case the location was:

CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,
DC=domain,DC=local

I assume that the failed setup may have had a role in this, although I still find it curious that an empty container could have such a detriment effect on mail enabled Public Folders.

Possibly clues to the underlying reasons can be found in the 24 hour frame and actions taken by the setup of perhaps Exchange 2003 or specific behavior of Exchange 2010 regarding recipients. I have never experienced this issue with Exchange 2007.

Conclusion

Check the above mentioned container with ADSIedit after you decommissioned your last Exchange 2000/2003 server. Remove the empty Server container (at you own risk and be sure to have a backup !!).

With the Exchange 2010 Outlook Web App or OWA, it is possible to use Firefox to access your mailbox. Yes, this was always possible but the premium features were only available for Internet Explorer users. As of now, I could only detect one small difference between Firefox and IE namely the S/MIME functionality. Most users or even admins probably don’t know that it exists as it not often implemented.

I am a frequent user of Firefox and prefer it above IE, especially now with Exchange 2010. However, I am annoyed that I always have to enter my login credentials. That’s another benefit of IE: support for Integrated Authentication on Exchange. When logged in on a windows domain computer, why would you have to also log into the Webmail? You are already authenticated.

But… Firefox also supports Integrated Authentication! It is not configured by default, so this way it doesn’t accidentally present AD authentication information to an Internet server. Internet Explorer can be configured to forcibly recognize intranet domain names via Group Policies.

Just type the following in the Firefox addressbar:

about:config

And edit the following values:

network.negotiate-auth.delegation-uris
network.negotiate-auth.gsslib
network.negotiate-auth.trusted-uris

Just add the internal domain or the FQDN of your Exchange (CAS) server. The change is implemented instantly, but remember this only works on Windows domain computers residing in the same domain or forest as your Exchange Server.

Now I’m investigating whether these settings can be configured centrally via GPO’s or scripts. But that is another challenge as Firefox uses configuration files (prefs.js in the user profile) and no registry settings. If you have figured this out, let me know!

Further Reading:
Mozilla Firefox: Integrated Authentication

Exchange 2010: Configure Integrated Windows Authentication

Filed under: Exchange 2010

Mailbox Replication Service unexpectedly quits when moving mailboxes from other Exchange server 17 February 10 06:03 AM | dmstork | 0 Comments

I’ve just completed a transition from Exchange 2003 directly to Exchange 2010. This is a supported scenario, but I did ran into some trouble. Luckily, I also have a solution.

Scenario

The installation of the Exchange 2010 server and upgrading of several items of the Exchange 2003 environment went without any hick-ups. The project was actually ahead of schedule and just when you think you could go home early…

Exchange 2010 now calls a mailbox move “Local Move Request”. The Exchange Mailbox Replication Service will then handle the mailbox move from one database to another. The source database can be Exchange 2003, 2007 or 2010. There are some cool benefits with this new approach, but I digress.

Symptoms

Very quickly after I requested a move of all 2003 mailboxes to the new Exchange 2010 server, the move of the mailboxes seemed to stall. I could see it with this Exchange Management Shell command:

Get-MoveRequest | ft Alias,Status,PercentComplete

Around 20-25%, this percentage suddenly dropped to 0%. After that it didn’t start up again (which it should).

Some research in the eventviewer showed this message:

Source: Service Control Manager
Event ID: 7031
Description:
The Microsoft Exchange Mailbox Replication service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 5000 milliseconds: Restart the service.

Sure enough the Mailbox Replication service started again after five seconds. But after a while it crashed again and again. There where no other relevant errors or warnings which where related to this issue. I decided to increase the logging level of the Replication Service with this cmdlet:

Get-EventlogLevel -Identity "MSExchange Mailbox Replication\*" | Set-EventlogLevel -Level Expert

Note: decrease it again after you don’t need it anymore with “–Level Lowest”.

The expert logging increased my insight in what was happening:

Source: MSExchangeIS
Event ID: 9660
Description:
User JOSH (/o=Test/ou=First Administrative Group/cn=Recipients/cn=JOSH) failed to log on because their mailbox is in the process of being moved.

and

Source: MSExchange Mailbox Replication
Event ID: 1101
Description:
Mailbox move for 'Test.nu/Test/DOCENT/JOSH' (b0de20a6-72ee-47ef-8123-123e123e123e) encountered a transient failure. The operation will be retried (1 out of 60). Error code: -2147467259 MapiExceptionMailboxInTransit: Unable to open message store. (hr=0x80004005, ec=1292)

The above message appear several times. This is happening after the replication service crashed and is probably the lock put in place as with the normal mailbox move lock. Not really surprising. It is nice to see that the replication service is willing to try 60 times.

Source: MSExchange Mailbox Replication
Event ID: 1104
Description:
Mailbox Replication service started initial seeding stage for 'Test.nu/Test/DOCENT/JOSH(b0de20a6-72ee-47ef-8123-123e123e123e). Total number of messages in mailbox: 854 (103.7 MB (108,744,421 bytes)).

And then it goes ahead again, but very soon the replication service crashes again and this starts from scratch (events not shown). There were no further clues on what was happening. Some mailboxes went over perfectly though, others didn’t but another time they went over without a hitch. I did get the feeling that with more mailboxes moving at one time the chance of the service crashing increased, but because of time constraints this was not further investigated.

There were no antivirus clients interfering, no antispam, the servers were in the same forest and site within the same subnet. Both servers were virtualized via vSphere. Yes, for the Exchange 2003 server not a supported situation but the problem was with the Exchange 2010 server which is supported.

Even so the virtual machines were not on the same node and the VMWare tools were up-to-date. Also the Windows Servers and Exchange servers were up-to-date, even with Exchange 2010 Update Rollup 1.

Solution

At this time I decided to call Microsoft Support. After some checks with ADSIedit, disabling firewalls, changing the type of NIC and other checks, Microsoft advised running a integrity check of the Exchange 2003 mailbox store. As I had ran out of ideas, this looked like a good suggestion. So I started ISINTEG on the Exchange 2003 server with:

exchsrvr\bin\isinteg -s <servername> -fix -test alltests

After about half an hour this summary appears:

. . . . . SUMMARY . . . . .
Total number of tests : 21
Total number of warnings : 222
Total number of errors : 0
Total number of fixes : 541
Total time : 0h:33m:26s

Yep, that database was bad…

After that the mailbox move requests completed without any relevant problems.

(I did have a lot of errors described in KB940012 resulting in the move request CompletingWithWarnings, but they are not a problem.)

Conclusion

My troubleshooting focused on the new Exchange 2010 installation, because it’s Mailbox Replication Service repeatedly unexpectedly quits. I also had to take the source server in consideration (something with assumptions…).

Although the problem ultimately was caused by a corrupted Exchange 2003 mailbox store, I hope that Microsoft will make the Mailbox Replication Service somewhat more robust or let it generate relevant error messages. It would have saved me quite some time troubleshooting.

Anyway, If you are not familiar with the Exchange environment and it’s history, it is a good idea to also check the integrity of the source exchange database before moving mailboxes. You can check it with ESEUTIL and ISINTEG:

ESEUTIL /G <databasename.edb>

ISINTEG -s <servername> -verbose -test alltests

/me makes mental note ;-)

Windows Server 2008 R2

Obviously the Exchange mode is something that is only relevant with Exchange 2000/2003 and nothing with this release of Windows.

The compatibility with domain controllers though, is another story. Only Exchange 2007 Service Pack 1 with Rollup update 9 for SP1 (and higher) will work correctly with Windows 2008 R2 domain controllers. You could exclude the specific Domain Controller, when you have other DC’s in your domain/site with the Exchange Server. But I would not recommend it as this can impact the availability of any DC for Exchange and directly the availability of Exchange, next to the extra administrative effort you will have to take. I would only do this (and have done this) during transitional phases.

The same requirement is valid for the Schema Master and Global Catalog, so only Exchange 2007 SP1 with rollup 9 and higher. You also can exclude the specific Global Catalogs, but it comes with the same warning label as with Domain Controllers mentioned above. The Schema Master is a Forest FSMO role, so you can have only one in your AD forest and Exchange organization. No cheating possible here. ;-)

The Domain and Forest Functional Level of Windows Server 2008 R2 feels a bit weird to me, we didn’t have this with Windows 2003 R2. So, there’s probably something really happening here. For a overview of the new possibilities of DFL/FFL 2008 R2, look at this page.

Both the DFL/FFL of Windows 2008 R2 are only supported by Exchange 2007 SP1 with Rollup update 9 and higher. Exchange 2000 already couldn’t work with a domain with exclusive Windows 2008 (R2) server, so this is a logical progression. Exchange 2003 could work with Windows 2008 DC’s, but not with 2008 R2 DC’s, and this logically excludes both 2008 R2 DFL/FFL.

When we revisit our compatibility matrices and add Windows 2008 R2 to it, we get:

Conclusion

Exchange and Active Directory on their own can be complex to maintain. But combining the specific requirements and (im)possibilities, it can be a maze. Especially for those who have to maintain and transition their multi domain forest to newer versions of Exchange and Windows and you have a variety of different versions of Windows and Domain Functional Levels.

But there is one relatively safe route: Windows Server 2003 with at least SP2. This is the golden version of Windows for Exchange 2000 to 2010. The Domain Controller Global Catalog function and the FSMO role of Schema Master can al reside on this version and all versions of Exchange can function with it. For the Domain Functional Level 2003 is the safest level, when considering Exchange transition from 2000 to eventually 2010. The 2003 Forest Functional level also works with al version of Exchange.

For transition paths regarding only Exchange co-existence check this post.

I hope that you find these blog posts helpful in transitioning your Exchange organization to any versions available within any Active directory environment. If you have questions or comments on these posts (or just to say hi), feel free to contact me!

Disclaimer: the information here and in previous posts on this subject is presented as-is and could be subject to change. The information on Exchange 2010 is in some cases still in writing and could contain facts that have changed or will change. Recheck those facts before implementing Exchange 2010 when it has reached general availability, which will be announced at TechEd 2009 in Berlin and other events.

Exchange 2010 is Code Complete and on its way to General Availability

There will be more information on TechEd Berlin, but unfortunatly I will not be attending. Luckily, video of the launch will be available via http://www.thenewefficiency.com/

So, until then I gonna delve into documentation and further testing of RC1!

PS. the documentation on Exchange 2010 is still subject to changes and there are some things that are not correct. For instance, it is mentioned that you will need a Exchange 2003 Front-end server when transition from an 2003 Exchange only organization to Exchange 2010. This is not correct (checked it at Technet Live).

Filed under: Exchange 2010

The Road to Exchange 2010: Active Directory and Exchange Part 2 28 September 09 08:08 PM | dmstork | 0 Comments

In part 1 of this series, I discussed the requirements of Exchange Mode, Domain Controllers and the Schema Master in combination of various versions of Exchange.

In this post I will discuss the placement of Global Catalogs (GC), the Domain Functional Level (DFL) and Forest Functional level (FFL) required. In part 3 I will discuss the compatibility of Exchange with Windows Server 2008 R2.

Global Catalog

The Global Catalog is a specialized option for any Domain Controller, it contains a copy of all important attributes of objects of all domains across the whole forest. Placement is important especially with multiple domains within the same Forest. Any Domain Controller can be activated as Global Catalog and when placed properly it can speed up AD queries and logon times significantly.

Exchange also relies on the Global Catalog, but only since Exchange 2007 there are minimum requirements to be met. This version requires at least Windows 2003 with Service Pack 1 in any Active Directory site where there is an Exchange 2007 server. It can co-exist with older versions (within a site), but you will have to exclude them. According to the System Requirements of Exchange 2010, it will have almost the same minimum requirement: Windows Server 2003 with Service Pack 2. The difference with Exchange 2007 is that 2010 cannot co-exist with lower versions of Windows Domain Controllers. I consider this a low impact difference from the GC requirements of Exchange 2007 as it those DC’s already should be upgraded to SP2 or beter yet replaced by 2008 DC’s, in my opinion.

Exchange 2000 has no specific minimum requirement. But as it cannot communicate with Windows 2008 Domain Controllers and thus Global Catalogs, this will mean you will require GC’s with Windows 2000 or 2003 (any patch level) per site and have to hard link to them (this is not mentioned in the Supportability Matrix). Exchange 2003 is the only one which can communicate with all versions of Domain Controllers and thus Global Catalog (from 2000 to 2008).

The resulting table actually doesn’t differ much from the Domain Controller table:

The green v in a cell means that Exchange can communicate with a domain exclusively with whose versions or mixed with other versions that have a green v. A red x indicates that it cannot communicate when this version is present within the site, domain or forest. The yellow – represents that it cannot communicate with that version, but it can operate as long as the minimum Domain Controller version requirement has been met (within a site or domain).

Domain Functional Level

The Domain Functional Level (DFL) determines the Active Directory Features of a domain. Roughly with each major Server Operating System there is also a new DFL and with it new features and enhanced security. You can however have a lower DFL then the OS level. As of Windows Server 2000, there are the following DFL’s:

Windows 2000 Mixed
Windows 2000 Native
Windows Server 2003 Interim
Windows Server 2003 (sometimes Native is added)
Windows Server 2008

Note: The newest DFL of Windows Server 2008 R2 will be discussed separately.

The Windows 2000 Mixed and Windows 2003 Interim are the only ones which support Windows NT Domain Controllers. Each Native mode requires that all domain controllers in that domain are at the minimum OS level, meaning that for instance DFL Windows 2000 Native requires all DC’s to be at least Windows 2000.

And again Exchange 2007 required the DFL to be minimum of Windows 2000 Native (also excluding Windows 2003 Interim), which meant that you could not transition to Exchange 2007 when there were NT Domain Controllers in the Domain. Note that although the DFL could be 2000 Native, you still had to have a minimum of one 2003 SP1 Domain Controller (for the Global Catalog and of course Schema Master). For the 2008 DFL you will need to have Exchange 2007 SP1 or higher. Exchange 2010 obviously can work at the 2008 DFL (like duh!).

So, what about Exchange 2000? As it cannot operate in a domain with exclusive Windows 2008 Domain Controllers, we can deduce it will not operate at the 2008 DFL. Exchange 2003 on the other hand can operate with exclusive Windows 2008 Domain Controllers but it still cannot operate at 2008 DFL.

To summarize: the Domain Functional Levels of 2000 Native and 2003 (excluding 2000 Mixed and 2003 Interim) are the DFL’s compatible with all Exchange servers from 2000 to 2010. The 2008 DFL is only supported from Exchange 2007 SP1 and higher. I’ve made a table:

Forest Functional Level

The Forest Functional Level is determining the security and features of your whole forest containing all your domains. The following Forest Functional Levels FFL) are possible:

Windows 2000
Windows 2003 Interim
Windows 2003
Windows 2008

Note: The newest FFL of Windows Server 2008 R2 will be discussed separately.

The Windows 2000 FFL can harbor all Domain Controllers from NT upwards. As there are no requirements for Exchange, you could theoretically have a multi domain forest with Windows NT or Windows 2000 and still run Exchange 2007 in a separate domain within the same forest as long other minimum requirements are met (Schema Master and DFL). However for Exchange 2010 a minimum of a 2003 Forest Functional Level is required, as specified in the System Requirements.

The 2003 Interim is for transitions from NT domains directly to 2003 Domain Controllers and no support for Windows 2000. As this would also imply Exchange transitions from Exchange 5.5, this would be a useless situation for Exchange 2007 and 2010 as they no longer support direct transitions from 5.5.

The Windows 2003 FFL, supports only 2003 Domain Controllers and only Domains at the 2003 DFL. All Exchange versions from 2000 to 2010 can support this level. With this level the added bonus of a Forest Trust is possible, so a resource forest could be implemented separating you Exchange organization from your Active Directory with active user accounts.

The 2008 FFL is newer than any Exchange version currently released (excluding Exchange 2010 in any beta, RC and RTM form), so this will probably result in some incompatibilities. And indeed, Exchange 2000 and 2003 cannot function in a 2008 FFL. Although Exchange 2007 RTM cannot be installed on a Windows 2008 server, it does operate in a 2008 FFL. The matrix for the FFL will look like:

This concludes Part 2. In Part 3 we will discuss Windows 2008 R2 and Exchange.

Exchange Mode

The first question points to a left-over from Exchange 5.5, which needed Mixed mode to co-exist with 2000 and 2003. Most organizations can safely upgrade this mode, which is a pre requirement of Exchange 2007 and 2010 when co-existing with or transitioning from an older version of Exchange (respectively 2000/2003 and 2003 only). It’s been a while since I’ve seen 5.5 in production so I expect little real life impact, but still something that has to be done.

You can check and change the mode in Exchange System Manager, select the organization, click Properties, go to the General tab and under Operations Mode the current status is shown. Click Change Mode to change it (duh!).

Domain Controllers

With Exchange 2000/2003 and Windows 2000/2003 there was no question about it. It didn’t matter which OS your domain controller was, it just worked. But then came Windows 2008 and even later Windows 2008 R2, both with Read Only Domain Controllers (RODC) and Read Only Global Catalogs (ROGC). Oh, and lest not forget the Schema Master role…

Will Exchange 2000 work with a Windows 2008 DC? Short answer: No. This means that if you exclusively have Windows 2008 DC’s you can’t run Exchange 2000. This also means that before upgrading all of your Domain Controllers to 2008 or 2008 R2, you will have to upgrade your Exchange server(s) first. However, Exchange 2000 (with SP3) can co-exist with 2008 DC’s. For each AD Site with Exchange 2000 you will have to have a 2000/2003 DC to which the Directory Service Access is hard linked.

What about Exchange 2003? As from Service Pack 2, Exchange 2003 can interact with Windows 2008 Domain Controllers. Even when all domain controllers in the forest are 2008 and if there are Read-Only DC’s and GC’s present. It will ignore them as long as there are writeable DC’s and GC’s present and when it can detect them automatically. So don’t hard link them to read-only servers!

Exchange 2007 can work with Windows 2000 DC’s, only not exclusively. There has to be at least one Windows 2003 SP1 DC in every domain where Exchange 2007 will be installed. Although Exchange 2007 cannot be installed on Windows 2008, it can interact with 2008 DC’s.

The upcoming Exchange 2010 is a bit more demanding than Exchange 2007, it cannot be used in an environment with Windows 2000 domain controllers.

For a visual representation:

Schema Master

The Schema Master is one of the Flexible Single Master Operation (FSMO) roles in Active Directory. This DC is the only one which can update the directory schema, which in turn defines possible objects and its attributes. In each AD Forest there is only one Schema Master. As far as I know, Exchange 2007 was the first version which explicitly demanded that the Schema Master was on a Windows Server 2003 with Service Pack 1 or higher. Exchange 2010 will have this same pre requirement.

How do you find out which Domain Controller holds this FSMO role? Well, there are several ways but the most easy one is with DCDIAG. If it’s necessary to transfer it, I mostly use the Active Directory Schema MMC Snap-in.

Note this: If your Schema Master is running Windows 2003 SP1 (or SP2), your Active Directory can support Exchange 2000, 2003, 2007 and 2010 if we disregard any other requirements. That is a useful fact to know when transitioning.

Will Exchange still be able to run when you have a Windows 2008 Schema Master? The Schema Master role is always placed on a Domain Controllers. As Exchange 2000 cannot directly interact with 2008 DC’s, we can logically conclude that Exchange 2000 will not work with a Windows 2008 Schema Master.

And Exchange 2003? Exchange 2003 can work in a forest exclusively with Windows 2008 DC’s, so it will have a Windows 2008 Schema Master and the deduced answer would be yes! The requirements for Exchange 2007 and 2010 explicitly state that they support Windows 2008 DC’s, and with that also an 2008 Schema Master.

And visually represented, that gives us:

This concludes part 1. In part 2 of this post, I will continue with the different requirements for the Global Catalog, Domain and Forest Functional level. In part 3 I will also discuss the requirements and limitations of Windows 2008 R2 servers in combination with Exchange.

*** edit: updated 28/9/2009 with corrections of some errors/typo’s and added compatibility matrices

Dave Stork's IMHO

Symptoms

Solution

Conclusion

Further Reading

Scenario

Symptoms

Solution

Conclusion

Further reading:

Windows Server 2008 R2

Conclusion

Exchange 2010 is Code Complete and on its way to General Availability

Global Catalog

Domain Functional Level

Forest Functional Level

Exchange Mode

Domain Controllers

Schema Master

Search

This Blog

Tags

Navigation

Archives

Syndication