Why items may get 'stuck' in a Pending State - Enterprise Vault (EV) Basic Archive Process.

There may be various reasons for mail items to be 'stuck' in a pending state.  Rather than give specific causes, it is best to understand the logic of the process. From there, it can be better identified where the 'points of failure' may occur.  To review a number of known causes, please see Related Documents at the end of this article.

Note: "Pending State" refers to mail items in the middle of being processed for archival.  The Message Class of these items are altered to IPM.Note.EnterpriseVault.PendingArchive.  The Result of this change is the 'envelope icon' of the message is changed to a 'vault' icon with a clock.

When archiving a message, a message will go through the following process:

1.  For Mailbox Archiving, the Mailbox Archive Task will populate the A3(Run Now of Task) or A5 (Scheduled Run of Task) queues with Mailbox object references.
     For Journal Archiving the Journal Task connects to the Journal mailbox every 60 seconds to determine if there are items to process (J3 Queue)

Notes:
   a. Message Queuing (MSMQ) => Private Queues can be found in Computer Management => Services and Applications => Message Queuing.
   b. In Windows 2008, MSMQ is located under Server Manager => Features => Message Queuing.
   c. There are individual Queues per Task, each with their own responsibilities. (e.g. If EV has 2 Mailbox Archiving Tasks, there will be 2 sets of 'A' queues, A1-A7)
   d. If a Task is deleted and recreated, the old queues are not automatically removed.  Also, if there are objects in these orphaned queues, they will not be processed.
   e. It is always recommended to refresh the page for the most current information.
   f. Expanding the Queue Folder (on Left) - Queue Messages and selecting an item within the queue can provide details on when the item was placed in the queue.

2.  Once the Mailbox object reference is accessed, the Archive Task/Journaling Task worker threads begin scanning the mailbox for eligible items.

3.  A call is made to Exchange, via MAPI, to change the Message Class of the Eligible Items from IPM.Note to IPM.Note.EnterpriseVault.PendingArchive.
   a. Journal Archiving will change the items to be processed to a Pending State and add references to these items to the J2 Queue to be processed

Notes: To determine the exact Message Class of an item, it is possible to add the Message Class field as a column in Outlook.

4.  A check of the EV Server Storage Service status is confirmed to be in Read-Write Mode
   a. EV 7.5 and below: 
   HKLM\Software\KVS\EnterpriseVault\Storage\EnableArchive = 1)
   b. EV 8.0 and above:
   A check in SQL is made to confirm EV Storage is not in a backup state

5. Items that are changed to a Pending State will be added to the Storage Archive Message Queue

Notes:
   a. At this stage, the copy is sent to the Storage Device
   b. Confirm that Enterprise Vault is not in a Read-only State.  See article TECH63337 under Related Documents for details.
   c. Additional Details on the Queues and their purposes can be found within the Enterprise Vault Administrator's Guide.

6.  Once added to Storage, the following are updated:
   EV 2007 and below:
   a.  In the associated Vault Store database, a Unique entry (Saveset ID) for the item is added to the Saveset table.
   b.  A similar entry is added to the Vault Store's Database - JournalArchive Table.
       i.  If Safety copies are set to be removed After backup, an entry is also added to the WatchFile table.
   c.  The unique Transaction Identifier (IdTransaction) is added to the original Pending Item in the mailbox.

Notes:
    a. See Article 000028261 and TECH51119 in Related Documents for details on mass canceling of pending items.

    EV 8.0 and above:

7.  The item, when being processed for Storage, is also read and indexed for searching purposes. Once the item is indexed successfully, the entry within the JournalArchive table is modified ('Indexcommited' is changed from 0 to 1).

8.  If the Vault Store is configured to 'Remove Safety copies after backup', the processing of the item will hold until a backup is performed on the storage device.  
Once the StorageFileWatch process identifies that the stored item ‘archive bit’ has been updated, a call to the JournalArchive Table will be made to change the corresponding Saveset entry to show that the items have been backed up ('BackupComplete' is changed from 0 to 1).

   a. How the Vault Store handles the Safety copy of items being archived is located in the VAC (Vault Admin Console).
       i. Open the VAC
       ii. Expand to Enterprise Vault => Directory on <SERVERNAME> => <SiteName> => Vault Stores.
       iii. Right-click on the Vault Store => Properties => Setting: "Remove safety copies:"

Notes:
   a. If running EMC Centera Storage, the 'backup' process would be considered automatic as this would instead be the Centera's internal Replication process.
   b. If running EMC Centera Storage with Collections enabled, items will stay as BackupComplete = 0 until the items have been collected from the Collection location.
   c. Journal items will commonly have this value already set (It is most common for a Journal Vault Store Database to be set to remove Safety copies immediately after archiving).
   d. There is a difference in how items are processed in SQL, based on the setting of "Remove safety copies" within the vault Store Database properties.
       i. When set to "After backup", a reference to the item is placed in the WatchFile table and the JournalArchive table (There is a constraint between these tables).
       ii. When set to "Immediately after archive", a reference to the item is only placed in the JournalArchive table and the WatchFile table is no longer scanned.
 
Important Note:  When making the decision to alter the method of removing safety copies from "After backup" to "Immediately after archive" it is necessary to disable the schedule for the Task (Mailbox Archiving) or set the Task for Report Mode (Journal Archiving). This will confirm that additional items are not added to these tables.  Once both of these tables are finished processing all outstanding items, it is then safe to change this functionality. If this option is changed with outstanding items in the WatchFile table, items associated with these entries will not complete the Post Process and be locked in a Pending State.  This can also cause items from being able to be post processed and pending items to build up.
 
9.  Once the BackupComplete value is set to 1, it is identified that the item(s) can now go through 'Post Processing'.  

Notes:
   a. These items are removed from the JournalArchive Table and the WatchFile Table once Indexcommited and BackupComplete are both 1.
   b. As of EV 6.0 SP3, this process was changed slightly.  Initially EV would start the Post Process once an item was both Indexed (Indexcommited = 1) and Backed up (BackupComplete = 1).  Post EV 6.0 SP3, Indexing is no longer a requirement for the item to begin Post Process.

10.  Once the items are registered as 'backed up', a list is constructed and these items are then referenced in the Message queues, within the A1/J1 Queue.

11.  As the worker threads of the Archiving Task/Journal Task pull each item from the A1/J1 queue, EV enters the Post Process stage where EV makes a call back to the original mailbox to 'strip' the item, removing the "safety copy", and change the Message Class of the item from IPM.Note.EnterpriseVault.PendingArchive to IPM.Note.EnterpriseVault.Shortcut.

Notes:
   a. Task Controller Service must be started and the Archiving Task/Journal Task must be running)
   b. Journal Archiving will not create a Shortcut but will rather be instructed to remove the item after archival.

Examples for troubleshooting:
- If the JournalArchive and WatchFile Tables are empty, it would be safe to observe that there is nothing waiting for backup or indexing. Check the Storage Archive Queue for backlogged messages.
- If nothing is in the A1/J1 queues, it is possible that Enterprise Vault is waiting for prerequisites to start the post process. Check the JournalArchive Table for confirmation.  

Recommendations to monitor Archiving Process:

JournalArchive Table
Perform the following Select scripts to attain a list of items still waiting to be processed.

A)  SELECT * FROM JournalArchive WHERE Indexcommited = '0'
- How many items are waiting to be indexed.

B)  SELECT * FROM JournalArchive WHERE BackupComplete = '0'
- How many items are waiting to be backed up.

*An Alternative would be the following to simply return a count of the number of items, rather than a list:

SELECT COUNT(*) FROM JournalArchive WHERE Indexcommited = '0'
SELECT COUNT(*) FROM JournalArchive WHERE BackupComplete = '0'

Message Queues

Check the 'A' Queues and 'J' Queues to confirm if there are any items left to be processed.

The most common are:

EV 7.5 and Prior

A1 = Post Processing
A3 = Mailbox list for 'Run Now' archiving
A5 = Mailbox list for Scheduled archiving
A6 = Synchronize Request

J1 = Journal Post Processing
J2 = Individual Items to be Processed

Storage Archive = List of items to be processed to Storage

EV 8.0 and above

A1 = Post Processing
A3 = Mailbox list for 'Run Now' archiving
A5 = Mailbox list for Scheduled archiving
A7 = Synchronize Request

J1 = Journal Post Processing
J2 = Individual Items to be Processed

Storage Archive = List of items to be processed to Storage

To temporarily see that items are successfully being processed through a Queue, it is possible to enable Journaling on a Queue:
   a. Expand Message Queuing
   b. Right-click the Queue => Properties => General Tab
   c. Under the Journal section, select Enabled*
 
Objects 'Journaled' for the Queue are referenced in the Journal messages sub-folder under the Queue.

Notes:
   a. If the Storage Service is located on an alternate EV Server, check the Outgoing Queues for items pending when being sent to the Storage Server.
   b. When enabling Journaling against a queue, these items will not be removed automatically and will take MSMQ storage space.  
   c. Do not keep the Journaling option enabled for a long period of time.  Once finished observing the results, disable this option and purge the Journal messages queue.
   d. Prior to Windows 2003 SP2, MSMQ Storage was set to 8 gigabytes (GB). In Windows 2003 SP2, this default was changed to 1 GB.  See Related Documents for further details on this limitation (Event Warning 3249).

Storage

   a. Check after the Backup to confirm the Archive bits have been 'flipped'.
   b. For EMC Centera Storage, the 'backup' process would be considered automatic as this would instead be the Centera's internal Replication process. Confirm replication is processing properly.
   c. For EMC Centera Storage, when Collections are enabled*, confirm that the Collections location is not backlogged and no Storage-based errors are occurring in the Event Log.
   d. For other Non-NTFS based Storage devices that do not utilize an "Archive" Attribute (IE. NetApp filers), please see instructions on the use of the IgnoreArchivebitTrigger.txt under Related Documents.

Notes:
   a. The Collection status is controlled per Vault Store Partition.  This value is set in the VAC => Vault Stores => <Vault Store Name> => <Partition Name> => Properties => Collections Tab.
   b. Once Collections are enabled on a Partition, the feature cannot be disabled.

Event Logs
Commonly the best source for any errors that may have occurred the night before.

Final Note: Post EV 6.0, regarding article 000028261, Solution 3, it is possible to set the Journal Policy - Advanced - Pending shortcut timeout to a specific day count (Ex. 1 day).  

Do not set this value to a time interval other than 0 in order to perform a single 'sweep' to Cancel pending items.