Applying Legal Holds or exporting items appears to stall with the error "Problem with object properties."

Problem

Applying Legal Holds appears to stall.  The DA Client display of the Case properties shows the error "Cause:Problem with object properties. Property: ItemID. Reason: The identifier specified was not formatted correctly.".  A dtrace of the Accelerator Service shows the error codes "-2147220730" and "0x80070057".

Also, attempting to export appears to stall with all other items completed. The DA Client display of the export properties will show the error "Failed to retrieve: Problem with object properties.  Property: Id. Reason: One of Id or CustomIdentifier or SequenceNum must be set"

Error Message

- Error displayed on the properties page of the DA Case as viewed through the DA Client:
Legal Hold Thread - worker thread (6)
Case ID: 83
Number of tries: 2
Cause:Problem with object properties. Property: ItemId. Reason: The identifier specified was not formatted correctly.

- Sample from dtrace log of the AcceleratorService, formatted for easier reading:
EV-H {LEGALHOLD.EN_US} Exception: An internal failure occurred.
Internal Error: 'The parameter is incorrect. [0x80070057]'.
Info:{C2.EN_US} An COM error occurs [adding] holds,
Error Code: -2147220730
Diag:
HRESULT: 80040306
Type:System.Runtime.InteropServices.COMException ST:
at KVS.EnterpriseVault.Interop.IHolds.PlaceHolds()|
at KVS.Accelerator.LegalHold.LegalHold.AddReleaseHolds(Int32 CaseID, String GroupId, String ConsumerId, Boolean addHolds, LegalHoldSyncStatus legalHoldSyncStatus)

- DA export details page will show a table view similar to the following:

Cause

This issue is caused by a KVSSaveSetID that is not formatted properly or is missing in the tblIntDiscoveredItems table for the DA Case in which the hold or export processing appears to be stuck.  The KVSSaveSetID field is expected to be formatted in character sets of 15 - 18 - 1 - 32, with each set separated by the tilde (~) character (i.e., 201201106229769~201201102103070000~Z~E025F93E4725FE4B298AAE9042E38B81). Any format other than this or no contents of this field is considered invalid.

The cause of missing KVSSaveSetID and other indexed metadata has been traced to improperly indexed items where multiple instances of the indexed metadata exists within beginning <value> and ending </value> XML anchors in the Search Results XML files returned by the EV Indexing Engine's search processing to the DA server.  For example, the following line contains a snippet from a Search Results XML files where there exists duplicated indexed metadata surrounded by the <value> and </value> XML anchors.  Note that the line has been formatted to allow easier reading.  Normally, each line displayed below would be part of a single line that can contain up to 1000 search results within the XML file.
<RESULT NUMBER="29">
<cont><value>Please find attached newsletter issue #9 This news contains articles for our products. Contents of this bulletin:   New Pro</value><value>Please find attached newsletter issue #9 This news contains articles for our products. Contents of this bulletin:   New Pro</value></cont>
 <ssid><value>201406255281379~201406250202250000~Z~109E5A18EEE54373C7A58A8196667A41</value><value>201406255281379~201406250202250000~Z~109E5A18EEE54373C7A58A8196667A41</value></ssid>
 <locn><value>Inbox</value><value>Inbox</value></locn>
 <snum><value type="integer">138655493</value><value type="integer">138655493</value></snum>
 <auth><value>USER1,TEST</value><value>USER1,TEST</value></auth>
 <reto><value>USER2,TEST</value><value>USER2,TEST</value></reto>
 <nrto><value>1</value><value>1</value></nrto>
 <nrcc><value>0</value><value>0</value></nrcc>
 <nrbc><value>0</value><value>0</value></nrbc>
 <renv><value>test_user3@evlab.local</value><value>test_user3@evlab.local</value></renv>
 <nren><value>1</value><value>1</value></nren>
 <subj><value>Newsletter #9</value><value>Newsletter #9</value></subj>
 <cnid><value>F7060671A35B47E6871D6A6B52A7103F</value><value>F7060671A35B47E6871D6A6B52A7103F</value></cnid>
 <impo><value type="integer">1</value><value type="integer">1</value></impo>
 <date><value type="dateTime">2014-06-25T02:02:25Z</value><value type="dateTime">2014-06-25T02:02:25Z</value></date>
 <mdat><value type="dateTime">2014-06-25T02:09:40Z</value><value type="dateTime">2014-06-25T02:09:40Z</value></mdat>
 <size><value type="integer">724</value><value type="integer">724</value></size>
 <dtyp><value>MSG</value><value>MSG</value></dtyp>
 <natc><value type="integer">1</value><value type="integer">1</value></natc>
 <customProperty set="Vault" name="MsgType"><value>EXCH</value><value>EXCH</value></customProperty>
 <customProperty set="Vault" name="MsgDirection"><value>1</value><value>1</value></customProperty>
</RESULT>

For a comparison example, the following line contains a snippet from a Search Results XML files where the metadata is returned in the expected format.  Note that the line has been formatted to allow easier reading.  Normally, each line displayed below would be part of a single line that can contain up to 1000 search results within the XML file as noted with the above example.
<RESULT NUMBER="28">
 <cont>Hi Test User1 and Test User2,   We are keeping the accumulated run time of the system in a dedicated log file.  From the previous system runs, the longes</cont>
 <ssid>201406255280925~201406250156060000~Z~107FEE0A90808F2003407240CB07DF01</ssid>
 <locn>Inbox</locn>
 <snum>138651829</snum>
 <auth>USER3, TEST</auth>
 <reto><value>USER1,TEST</value><value>USER2,TEST</value></reto>
 <nrto>2</nrto>
 <recc><value>USER4,TEST</value><value>USER5,TEST</value><value>USER6,TEST</value></recc>
 <nrcc>3</nrcc>
 <nrbc>0</nrbc>
 <renv><value>test_user4@evlab.local</value><value>test_user5@evlab.local</value><value>test_user6@evlab.local</value></renv>
 <nren>3</nren>
 <subj>RE: FAT and shipment date (Daily Update)</subj>
 <cnid>E738CF26081349C1B7DDAA7C3A2149D6</cnid>
 <impo>1</impo>
 <date>2014-06-25T01:56:06Z</date>
 <mdat>2014-06-25T02:02:05Z</mdat>
 <size>81</size>
 <dtyp>MSG</dtyp>
 <natc>0</natc>
 <customProperty set="Vault" name="MsgType">EXCH</customProperty>
 <customProperty set="Vault" name="MsgDirection">2</customProperty>
</RESULT>

As can be seen in the above example, the only lines where the <value> and </value> anchors are expected are in the message's To and Cc fields when multiple recipients in each of those lines are present.  These anchors are also expected in the Bcc field when multiple recipients are present.

The data in the "Search Results" XML files are translated into "Search Items to Commit" XML files that are sent to the DA Customer Database for processing.  The translation of the results that have the unexpected <value> and </value> anchors causes the information within those anchors to be replaced with blanks in all fields except for the <date> and <mdat> fields where the date and time of the search are entered.  These entries are then processed into the DA customer database with missing KVSSaveSetID information along with missing or incorrect other indexed metadata.

When Legal Holds are to be applied to items captured in a DA Case, a procedure runs to identify the items on which to apply holds in batches of 1000 by default.  This procedure obtains the KVSSaveSetID of each item and the VaultID set in DA for the archive in which the item is stored.  The improperly formatted or missing KVSSavesetID causes the procedure to loop and repeat the same listing of KVSSavesetIDs that need to have holds applied.

When exports are run in DA, a procedure runs to obtain the listing of all items to be exported.  Like the Legal Hold procedure, this export procedure obtains the KVSSavesetID and VaultID of each item to be exported.  Any item with an improperly formatted or missing KVSSavesetID entry will initially fail the export with the error information noted above.  The export process will retry such an item up to 3 times by default before marking it permanently in the export as failed.  All other items in the export will be processed normally.  If the export is configured to export to PST files, then all attempts to export the one item to an MSG file prior to moving it to a PST file will have to complete before the export processing will continue on to move items into PST files.  The export will eventually finish with a status of 'Finished with errors'.

Workaround for applying Legal Holds:

Set the LegalError code for the item with the improperly formatted KVSSaveSetID to 427 (Item Deleted) using the following steps:

--- On a workstation running the Discovery Accelerator (DA) Client using an account with access to the Case

1) Note the CaseID of the DA Case where the Legal Holds appear to be stuck using the steps in Technical Article TECH72127, "How to obtain the CaseID and SearchID in Enterprise Vault (EV) Compliance Accelerator (CA) or Discovery Accelerator (DA)" (see the Related Articles section below).

--- On the Discovery Accelerator (DA) server

Important: Backup the Discovery Accelerator Customer database prior to completing the following steps.

2) Record a 5 minute long dtrace log of the AcceleratorService while attempting to apply holds.

3) Review the dtrace log for instances of the following lines:

4) Count and note the number of instances of "CContentManagementAPI::get_Hold" following the line with the CaseID obtained in Step 1 above (i.e., CaseID 115 in the sample line above).

5) Close the DA  Client.

6) Stop the Customer's Background Task (CBT)  through the EVBAAdmin site or stop the Enterprise Vault Accelerator Manager Service (EVAMS).

--- On the SQL Server hosting the DA Customer database

7) Open a 'New Query' window focused on the DA Customer database in SQL Server Management Studio while logged onto the SQL Server using an account with permissions to modify the DA Customer database contents (such as the Vault Service Account).

8) Run the following query in the window, replacing XX with the CaseID obtained in Step 1 and displaying the output to either the grid below the query window or to a file:

9) Review the output and scroll down the rows to the number obtained in Step 4 (i.e., Row 155).

10) Confirm the KVSSaveSetID  is not formatted properly (i.e., contains the Subject field contents instead of a valid KVSSaveSetID).

11)  Note the DiscoveredItemID of the item with the improperly formatted KVSSaveSetID.

12) Check for additional instances of the message in the tblIntDiscoveredItems table by running the following SQL query in the query window, replacing 'message subject' with the Subject field content of the item found in Step 9 and replacing XX with the CaseID obtained in Step 1:

13) Note the number of items returned with the same VaultID as the message with the improperly formatted KVSSavesetID.  This number will be compared against another count in Step 25 to determine if there is only one instance of the item with an improperly formatted KVSSavesetID entry.

--- On the Enterprise Vault Indexing Service server

14) Log on as the Vault Service Account (VSA).

15) Open the Vault Administration Console.

16) Navigate to the Archives folder.

17) Navigate to the Archives sub-folder containing the archive in which the items were found (for example, Exchange Journal).

18) Right click on the archive in which the items were found and select the Properties option.

19) Ensure the VSA has at least Read permission granted, adding that permission if needed, then click the OK button to close the properties.

20) Right click on the archive again, then select the Search Archive option.

21) Click the Browser search link in the upper left corner of the search criteria pane.  This action will launch Internet Explorer to the 'search.asp' search criteria page.

22) Select the archive to be searched in the Vault field using the drop down arrow to display the available archives to which the VSA has at least Read permission, then click on the archive to select it.

23) Enter into the Subject field the contents of the Subject field from the result of the SQL Query of Step 12.

24) Change the inclusion option from contains any of to contains all of, then click the Search button in the lower left of the page.

25) Count the number of items found by the search and compare this number to the number of items noted in Step 13.

26) If the count of items in Step 25 is 1 less than the number from Step 13, the item with the incorrectly formatted KVSSaveSetID does not exist in the Vault Store and can safely be ignored.

27) If the count is equal to the number from Step 13, for each item returned

   a) Click the link of the message.  This will display a preview of the HTML rendering of the message in the browser window.

  b) Highlight the URL in the Address field and copy it into Notepad.  This will provide a line with the VaultID (labeled as 'vault=') and KVSSavesetID (labeled as 'SaveSetID=' in the line) for the item.

  c) Locate the portion of the URL in NotePad that begins with SaveSetID and place a carriage return at the beginning of SaveSetID to place that information on a separate line.

  d) In the SaveSetID line, locate each instance of %7E and replace these characters with the tilde (~).  There should be 3 instances to replace.  For example:

SaveSetID=790000000000000%7E200805291331590000%7E0%7E1E4261319490410080C9BF4F043F9A7 would become

SavesetID=790000000000000~200805291331590000~0~1E4261319490410080C9BF4F043F9A7

  e) Save the NotePad document, but keep it open until all of the SaveSetIDs have been obtained.

  f) Return to Step 27a) for the next item until each item has been processed to obtain its SaveSetID.

--- On the SQL Server hosting the DA Customer database

28)  Compare the SaveSetIDs in the NotePad document with the KVSSaveSetID entries from the SQL query of Step 12.

29) All of the SaveSetIDs should match the KVSSaveSetIDs except for the one with the improperly formatted KVSSaveSetID.

30) If the number of KVSSaveSetIDs is one more than the SaveSetIDs AND the SaveSetIDs match the remaining KVSSaveSetIDs, run the following SQL statement in the query window, replacing XXX  with the DiscoveredItemID  noted in Step 11:

31) If the number of SaveSetIDs matches the number of items returned in the query of Step 13, note the SaveSetID that does not have a matching KVSSaveSetID as this is the SaveSetID that should be in that item's KVSSaveSetID field.

  a) Run the following SQL statement to correct the KVSSaveSetID for the item, replacing XXX with the DiscoveredItemID noted in Step 11 and replacing the word "actualsavesetid" with the SaveSetID noted in Step 31:

32) Repeat Steps 8 and 9 to ensure the item is no longer listed, or is listed with a properly formatted KVSSaveSetID, indicating a successful change of the LegalStatus from 420 to 427 or correction of its KVSSaveSetID.

Note: Additional rows may exist in the tblIntDiscoveredItems table with improperly formatted KVSSaveSetID field contents.  A review of the query output below the item identified in Step 9 should be performed to look for any other such rows.  Any other such rows should have their DiscoveredItemID noted and used in the SQL statement of Step 12 for each such item.

--- On the Discovery Accelerator (DA) server

33)  Start the CBT through the EVBAAdmin site or start EVAMS.

--- On a workstation running the Discovery Accelerator (DA) Client using an account with access to the Case

34)  Open the DA  Client.

35)  Change to the Cases tab.

36)  Click in the left panel on the Case where the holds were stuck.

37)  Monitor the properties to confirm at least 1 'Number of errors:', the 'Remaining items:' is decreasing and the 'Number of holds' is increasing.

There is no need for a workaround for the export processing as the item with a missing or improperly formatted KVSSavesetID will eventually be marked as failed in the export and the remaining items to be exported will complete the export processing.


Workaround to correct the entries in the "Search Items to Commit" XML files where the "Search Results" XML Files contain unexpected multiple instances of <value> and </value> tagged indexed metadata:

1. Run the following query against the Accelerator Customer database to identify the Archives that contain items with invalid SSID entries:

WITH BlankSSID AS (
SELECT DiscoveredItemID, VaultID
FROM tblIntDiscoveredItems
WHERE KVSSaveSetID = '')
SELECT DISTINCT KVSVaultName, tv.KVSVaultEntryID, tvs1.Name AS 'VaultStoreName', tvs2.VaultServer
FROM tblVaults AS tv
JOIN BlankSSID ON tv.VaultID = BlankSSID.VaultID
JOIN tblVaultStore AS tvs1 ON tv.VaultStoreID = tvs1.VaultStoreID
JOIN tblVaultServers AS tvs2 ON tv.VaultServerId = tvs2.VaultServerId
ORDER BY tvs2.VaultServer, tvs1.Name, KVSVaultName;

2. Enable the following Diagnostic Settings in the Accelerator Client under Configuration | Settings | Diagnostics, and restart the EVAMS as prompted:
- Save XML Search Items to Commit
- Save XML Search Results

3. Run an open Accelerator Search targeting one of the listed Archives from the step 1 query results (it is not necessary to Accept the Search). Upon Search completion, review the Search Results XML file(s) and find all instances of <ssid><value>. The starting anchor <ssid><value> is an invalid SSID anchor and causes the KVSSavesetID to be blank in the Search Item To Commit XML file as well as in tblSearchItems and tblIntDiscoveredItems. The correct anchor should be <ssid> by itself, without <value> immediately following it.

4. For each item with a starting SSID anchor of <ssid><value>, find the <snum> anchor and note the value - this is the ISN. Note: There may be multiple instances of the <ssid><value> and <snum> entries - each instance should contain the same information.

5. Review each of the Search Items to Commit XML files, look for KVSSaveSetID="", and note the XML file names for each file where KVSSaveSetID="" is found. The file name will be in the format Items_S<SearchID>_V<VaultID>_I<IndexVolumeID>_C<BatchNumber>_B<ArchiveName>.xml. For example, if the file name is Items_S584_V2_I785_C2_BJournalMailbox02.xml, the IndexVolumeID is 785.

6. Edit the following query to add the affected IndexVolumeID values as indicated and run the query against the EnterpriseVaultDirectory database:

SELECT a.RootIdentity AS Archive_RootIdentity,
r.VaultEntryID,
a.ArchiveName,
iv.IndexVolumeIdentity,
ISNULL(irpe.IndexRootPath,'None') AS Location_of_Index_Volume,
ISNULL(iv.FolderName,'None') AS Index_Volume_Folder_Name,
ISNULL(iv.FirstItemSequenceNumber, -1) AS First_Item_Sequence_Number,
ISNULL(iv.HighestItemSequenceNumber, -1) AS Highest_Item_Sequence_Number
FROM EnterpriseVaultDirectory.dbo.Archive AS a
JOIN EnterpriseVaultDirectory.dbo.Root AS r ON a.RootIdentity = r.RootIdentity
JOIN EnterpriseVaultDirectory.dbo.IndexVolume AS iv ON r.RootIdentity = iv.RootIdentity
JOIN EnterpriseVaultDirectory.dbo.IndexRootPathEntry AS irpe ON iv.IndexRootPathEntryID = irpe.IndexRootPathEntryID
WHERE iv.IndexVolumeIdentity IN (2, 3) ---Set the EV IndexVolumeID values here
ORDER BY a.ArchiveName, iv.IndexVolumeIdentity;

7. Confirm the ISNs obtained above are within the ISN ranges listed in the query output. These Index Volumes will need to be Rebuilt. A Rebuild is the only way to correct the SSIDs listed with the incorrect starting <ssid><value> and ending </value></ssid> anchors and associated data.

8. Reject the Accelerator Search. Repeat steps 3 - 8 for any remaining Archives and corresponding Index Volumes.

Solution

For applying Legal Holds where items exist with missing or malformed SavesetID information, this issue has been addressed in the following releases:

Enterprise Vault 11.0.1 Cumulative Hotfix 4 Release
https://www.veritas.com/docs/000097959

Enterprise Vault 12

The following link contains information about Enterprise Vault 12
https://www.veritas.com/docs/000099905

For correcting the entries in the "Search Items to Commit" XML files where the "Search Results" XML Files contain unexpected multiple instances of <value> and </value> tagged indexed metadata:

This issue has been addressed in the following release:

Enterprise Vault 12.1

The following link contains information about Enterprise Vault 12.1
https://www.veritas.com/docs/000107766

Note: For information on how to obtain Enterprise Vault 12 see: How to obtain the license key and installation download for Veritas products