Mar 24, 2014

In this blog, I want to change tack a little bit, and talk about an issue that has cropped up a number of times in the Exchange Server 2013 courses, as well as in production systems, and that is a situation where the Information Store index becomes corrupted. It manifests itself in a number of ways: emails getting dropped into the Drafts folder instead of being sent, DAG creation and problems with replication, etc. This issue does tend to appear more often in class, purely because when in class, we are constrained by limited resources and time, and therefore, tend to gloss over some of the recommended guidelines. In the “real” world, exchange servers should not be regularly bounced, nor should they be snapshotted or checkpointed, and certainly not reverted. However, from the feedback I have been receiving, it is something that a number of exchange administrators are also encountering in the field, particularly the smaller environments that only have one or two exchange servers.

Cause of issues

As mentioned, the issue here is that the Information Store index is corrupted, and this can easily be confirmed by viewing the 'ContentIndexStatus' of the relevant Exchange mailbox database in the EAC under servers/databases, or via the following PowerShell command:

Get-MailboxDatabaseCopyStatus | FL Name,ContentIndexState

Normally, this should return a value of 'Healthy,' but if the database index is corrupted it will show either 'Failed' or 'FailedAndSuspended.'

Fixes

First and foremost, check your services. That is usually my first port of call for many troubleshooting scenarios, and one I consider mandatory if the exchange server has recently been started/restarted. Make sure that all services (particularly the exchange services) marked as 'Automatic' are started and running. Larger environments with multiple copies of the database can usually get by with initiating a 'reseed' of the database from one of the other copies, but if you only have a single copy of the database then previous versions of Exchange had a PowerShell script called 'ResetSearchIndex.ps1' that fixed everything for you. However, that is currently unavailable in Exchange Server 2013. So now, I have 3 possible fixes for you, depending upon the severity. Try each in turn until you get the ContentIndexStatus showing 'Healthy.'

Case 1 Check your permissions. This is something that is not setup correctly in the course images, but we easily corrected it by creating an AD DS Security group called 'ContentSubmitters' (used by the indexing services), and in the security tab listing the Administrators and Network Service accounts with Full Control permissions. We then restarted the Microsoft Exchange Search and the Microsoft Exchange Search Host Controller services. If that does not work, try the next tip.

Case 2 This is really a repeat of my original 'check your services' fix, and particularly applies if your email messages are getting stuck in the Drafts folder:

1. Type 'Test-ServiceHealth' and check all the required services are running. If they are not, start them.
2. Restart the 'MSExchangeSubmission', 'MSExchangeDelivery' and 'MSExchangeTransport' services:

   Restart-Service MSExchangeSubmission Restart-Service MSExchangeDelivery Restart-Service MSExchangeTransport

3. If the messages are still not being delivered, you can try restarting the 2 Search services listed in case 1 and/or try restarting the Microsoft Exchange Active Directory Topology service from the services console, and recheck all other dependent services have started. If not it may be time for a bit of database surgery, as in ….

Case 3 This is a rework of the missing PowerShell script 'ResetSearchIndex.ps1.' I am not sure why Microsoft did not include this in Exchange Server 2013, but I did check and the registry pointers to the database locations have been changed from previous versions, so I guess it is possible we might see an updated script in due course.

1. Stop the Microsoft Exchange Search Host Controller service (HostControllerService). You can use the PowerShell command:

   Stop-Service HostControllerService

2. Delete the index directory and all its contents from each "Failed..." Database. This is the big long GUID ID directory in each Mailbox Database directory (under C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database.....\). You need to delete the index directories of all "Failed" databases before the doing the next step.
3. Start the Microsoft Exchange Search Host Controller service (HostControllerService). Again, you can use the PowerShell command:

   Start-Service HostControllerService

4. Wait a few minutes for the index to be rebuilt. If you use the 'Get-MailboxDatabaseCopyStatus,' command as listed above, you will find the status will firstly show "Crawling" during the rebuild(s) and eventually give a "Healthy" status.

DISCLAIMER:

There always has to be one, right? The above fixes has been found to work fine in the Exchange Server training courses so far, but you should always test things out in your own test labs, and of course, you all have up to date backups for your production environments, don’t you?