Unable to retrieve items in Accelerator Review or Exports

Problem

Clicking on an item in the Enterprise Vault Compliance Accelerator (CA) or Discovery Accelerator (DA) Review Set does not display the item preview. The message 'An error has occurred' is displayed in the Preview pane. Similar behaviour is seen when attempting to Export the item. One or more of the Event Log entries below may also be seen in the EV Event Logs on the Accelerator server. A DTrace of the AcceleratorService process may show lines as listed below.

Error Message

Enterprise Vault Event Log entries on the Accelerator server:

Log Name: Veritas Enterprise Vault
Source: Accelerator Service Processor
Event ID: 375
Level: Error
Description:
APP AS - Customer ID: X - An Error has occured when retrieving Item
SaveSetID: . . .
VaultID: . . .
Format: MUD:U
sAttachmentID: 0.
System.Runtime.InteropServices.COMException (0x800706BA): Creating an instance of the COM component with CLSID {F019A230-FF92-11D1-8C20-0000F87502DE} from the IClassFactory failed due to the following error: 800706ba The RPC server is unavailable. (Exception from HRESULT: 0x800706BA).
   at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
   at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic)
   at System.Activator.CreateInstance(Type type)
   at KVS.Accelerator.Common.EVStorageOnline.LoadStorageOnline(Boolean bForceReload)
   at KVS.Accelerator.Common.EVStorageOnline.GetOnlineAttachment(String savesetId, String vaultId, String attachmentNum, String format, Object& oBytes, Object& oMimeTag, Object& oExtension, Object& oTitle, Object& oMsgStructure, Object& oLargeFileSize)
   at KVS.Common.VaultItem.Item.GetOnlineAttachment(String sSaveSetID, String sAttachmentID, String sFormat, String sVaultID, Object& oBytes, Object& oMimetag, Object& oExtension, Object& oTitle, Object& oStructure, Object& oLargeFileSize)
   at KVS.Common.VaultItem.Item.GetItemFromSTGOnline(String itemID, String sSaveSetID, String sAttachmentID, String sFormat, String sVaultID, MessageType msgType)

V-437-375

Log Name: Veritas Enterprise Vault
Source: Accelerator Service Processor
Event ID: 375
Level: Error
Description:
APP AS - Customer ID: X - An Error has occured when retrieving Item
SaveSetID: . . .
VaultID: . . .
Format: XML
sAttachmentID: 0.
System.Runtime.InteropServices.COMException (0x80040300): Enterprise Vault is not running.
   at KVS.EnterpriseVault.Interop.IItem.Get(Int32 DetailLevel)
   at KVS.Common.VaultItem.Item.CallEcmAPI(String sSaveSetID, String sVaultID, String sAttachmentID, String sFormat)

V-437-375

Log Name: Veritas Enterprise Vault
Source: Accelerator Service Processor
Event ID: 375
Level: Error
Description:
APP AS - Customer ID: X - An Error has occured when retrieving Item
SaveSetID: . . .
VaultID: . . .
Format: XML
sAttachmentID: 0.
System.Runtime.InteropServices.COMException (0x80040300): Enterprise Vault is not running.
   at KVS.Common.VaultItem.Item.CallEcmAPI(String sSaveSetID, String sVaultID, String sAttachmentID, String sFormat)
   at KVS.Common.VaultItem.Item.GetEcmArchivedItem(String sSaveSetID, String sVaultID, String sAttachmentID, String sFormat)
   at KVS.Common.VaultItem.Item.GetItemFromECM(String itemID, String sSaveSetID, String sAttachmentID, String sFormat, String sVaultID, MessageType msgType)

V-437-375

Log excerpts from a DTrace of the AcceleratorService process:

[1234]    (AcceleratorService)    <5678>    EV:H    {CItem::PerformItemGet} HRXEX fn trace : Error [0x800706ba], [d:\builds\16_\ev\v-m-s\sources\source\storage\evstorageapi\item.cpp, lines {792,1352}, built Nov  1 13:11:51 2018].

[1234]    (AcceleratorService)    <5678>    EV:M    CItem::Get com_error raised Error [0x800706ba] (err=800706ba)

[1234]    (AcceleratorService)    <5678>    EV:M    CItem::Get Id: . . . ArchiveId:. . . hr: The RPC server is unavailable.  (0x800706ba)

[1234]    (AcceleratorService)    <5678>    EV:H    {CItem::Get} HRXEX fn trace : Unexpected fn exit path, returning error [0x800706ba], [d:\builds\16_\ev\v-m-s\sources\source\storage\evstorageapi\item.cpp, lines {728,769}, built Nov  1 13:11:51 2018].

[1234]    (AcceleratorService)    <5679>    EV-H    {-} Exception: Error VaultItem_ItemGet2 Info:{AcceleratorEvent} {C1} An Error has occured when retrieving Item|SaveSetID: . . .|VaultID: . . .|Format: XML|sAttachmentID: 0.|System.Runtime.InteropServices.COMException (0x80040300): Enterprise Vault is not running.
|   at KVS.EnterpriseVault.Interop.IItem.Get(Int32 DetailLevel)
|   at KVS.Common.VaultItem.Item.CallEcmAPI(String sSaveSetID, String sVaultID, String sAttachmentID, String sFormat) Diag: Type:System.Exception ST: Inner:None

[1234]    (AcceleratorService)    <5679>    EV-H    {Item} Exception: Enterprise Vault is not running. Info:{C1} ECM cache access error Diag:HRESULT: 80040300 Type:System.Runtime.InteropServices.COMException ST:   at KVS.Common.VaultItem.Item.CallEcmAPI(String sSaveSetID, String sVaultID, String sAttachmentID, String sFormat)
|   at KVS.Common.VaultItem.Item.GetEcmArchivedItem(String sSaveSetID, String sVaultID, String sAttachmentID, String sFormat) Inner:None

[1234]    (AcceleratorService)    <5679>    EV-H    {-} Exception: Error VaultItem_ItemGet2 Info:{AcceleratorEvent} {C1} An Error has occured when retrieving Item|SaveSetID: . . .|VaultID: . . .|Format: XML|sAttachmentID: 0.|System.Runtime.InteropServices.COMException (0x80040300): Enterprise Vault is not running.
|   at KVS.Common.VaultItem.Item.CallEcmAPI(String sSaveSetID, String sVaultID, String sAttachmentID, String sFormat)
|   at KVS.Common.VaultItem.Item.GetEcmArchivedItem(String sSaveSetID, String sVaultID, String sAttachmentID, String sFormat)
|   at KVS.Common.VaultItem.Item.GetItemFromECM(String itemID, String sSaveSetID, String sAttachmentID, String sFormat, String sVaultID, MessageType msgType) Diag: Type:System.Exception ST: Inner:None

[1234]    (AcceleratorService)    <5679>    EV-H    {RBO_ItemContent} Exception: Enterprise Vault is not running. Info:{C1} RBO method failed Diag:HRESULT: 80040300 Type:System.Runtime.InteropServices.COMException ST:
   at KVS.Common.VaultItem.Item.GetItemFromECM(String itemID, String sSaveSetID, String sAttachmentID, String sFormat, String sVaultID, MessageType msgType)|   
   at KVS.Common.VaultItem.Item.GetItem(String itemID, String sSaveSetID, String sAttachmentID, String sFormat, String sVaultID, MessageType msgType)|   
   at KVS.Accelerator.RBO.RBO_ItemContent.GetContent(Int32 proxyID, EFormat format, ItemIDType idType, Int64 itemID, Int32 attachmentID, Int64 offset, Int32 count) Inner:None

Cause

The Event Log entries and the DTrace Log entries show the same symptoms of the EV server not running. However, the services on the EV server may be running and no errors seen in the Event Logs on the EV server. Therefore the real issue is the EV server cannot be contacted via normal DCOM communications.

From the Event Log entries, the COM component with CLSID {F019A230-FF92-11D1-8C20-0000F87502DE} corresponds to the EV StorageOnlineOpns.exe process. When an item retrieval is initiated, the StorageOnlineOpns.exe process on the Accelerator server attempts to contact the StorageOnlineOpns.exe on the EV Storage server via the EV server to retrieve the item.

The following hex codes also confirm the EV server cannot be contacted via normal DCOM communications:
- 0x800706BA: RPC server is unavailable (RPC_S_SERVER_UNAVAILABLE).
- 0x80040300: Service is not running or cannot be contacted

Solution

One or more of the following items can contribute to this issue:

1. Miss-matched EV API/binaries on the Accelerator server.
Verify the EV API or the EV binaries installed on the Accelerator server match the EV server, both in major version and in minor version/release update. For example, if the EV server is running 12.5.2, the EV API or EV binaries installed on the Accelerator server should also be 12.5.2. If the EV API or EV binaries installed on the Accelerator server are 12.5.0, this would be considered a miss-match.

2. EV and/or Accelerator server not being rebooted after Windows Updates.
If Windows Updates have been installed on the Accelerator and/or EV server(s) and the servers have not been rebooted, reboot the servers at the earliest opportunity.

3. Incorrect or missing DNS records
If the EV server's DNS entries are incorrect or do not exist, DCOM communications from the Accelerator server will not be able to connect to the EV server. To test for the correct DNS entries, simply ping the EV server from the Accelerator server using the EV server's hostname (NetBIOS name), fully qualified domain name (FQDN) and IP address. Determine which tests are successful.

To ping the EV server:
- Log in to the Accelerator server with an account having local administrative rights on the server. Typically the Vault Service Account has these rights.
- Open a command prompt.
- Edit the following command to replace <server> with the hostname of the EV server and execute the command:
    ping <server>
- Repeat the ping command using the FQDN and then the IP address of the target EV server.

Review each ping attempts' response using the hostname, FQDN and IP address, and determine which is not successful. A successful response, indicating the target server is online and can be contacted via normal network communications using the name specified, will list 'Reply from' and list the IP address of the target server, the packet size and the response time. There are 3 common unsuccessful responses, each having its own remediation steps, as below:

Request Timed Out: The response from the target server was not received within the specified timeout. If this response is seen along with 'Reply from' responses, this usually indicates the target server can be contacted by normal network communications, however there is excessive network traffic at the time the ping command was run. If these combined responses are seen every time the ping command is run, the network could be very heavily used or congested. If the 'Request Timed Out' is the only response, this could indicate a physical network connectivity issue or an incorrect TCP/IP configuration on the target server.

Destination Host Unreachable: Typically seen when the target server is on a different subnet and the ping request cannot find a route to the target server's subnet. This is most often caused by a missing gateway or a gateway that has outdated routing tables on the server executing the ping request. Please contact your network team for assistance if this response is seen.

Ping request could not find host: The IP address of the target server using the specified name could not be determined. This can be due to the name being incorrectly entered or can be due to the name not being listed in DNS. If the target server name used is correct, the most likely cause is a missing or incorrectly configured DNS entry for the target server. Please contact your network team to review the DNS entries for the EV server.

To bypass any possible DNS issues until they can be resolved, it may be required to add a HOSTS file entry on the Accelerator server pointing to the EV server. These entries can be removed after the DNS issues are resolved. To add a HOSTS file entry:

- Log in to the Accelerator server with an account having local administrative rights on the server. Typically the Vault Service Account has these rights.
- Use Windows Explorer to navigate to C:\Windows\System32\Drivers\etc\.
- Use Notepad to open the HOSTS file.
- Enter the IP address and the FQDN of the EV server at the bottom of the file. Note the syntax examples as provided at the top of the file.
- Save the file and reboot the server to make the change effective.