Error: "CreateProcess() failed for ( ..\program files\veritas\Cluster Server\bin\RVGPrimary\monitor.pl), Error (2)" reported for RVGPrimary resource in VCS.

Problem

For Veritas Cluster Server (VCS) on the Windows platform, there are instances wherein the RVGPrimary resource fails to probe, preventing the resource from moving to an online state.

Error Message

The following error messages are reported in the %VCS_HOME%\log\RVGPrimary_A.log

YYYY/MM/DD HH:MM:SS VCS WARNING V-16-2-13179 Thread(xxxx) CreateProcess() failed for ( <install_path>:\program files\veritas\Cluster Server\bin\RVGPrimary\monitor.pl), Error (2)

YYYY/MM/DD HH:MM:SS VCS ERROR V-16-2-13038 Thread(xxxx) Resource<RVGPrimary_resource_name>: Error executing the program( <install_path>:\program files\veritas\Cluster Server\bin\RVGPrimary\monitor.pl).

Cause

The monitor.pl is a perl script and requires a perl.exe module to execute it.

This perl.exe module is available at the <install_path>\Program Files\Veritas\VRTSPerl\bin path

The issue occurs if

a) The perl.exe at \Program Files\Veritas\VRTSPerl\bin is missing or corrupt.

AND/OR

b) The environment variable has an incorrect value for the VRTSPERL_HOME system variable. 

AND/OR

c) The perl.exe execution is blocked by an antivirus software/Group Policy Object (GPO)

Solution

To address this issue, ensure :

1) The perl.exe is present at <install_path>\Program Files\Veritas\VRTSPerl\bin 

2) Verify the size for that is not showing 0 Kb. 

If so, replace perl.exe from a working setup to the problem node(s).

3) Ensure the Antivirus software has the exclusions listed at Setting up anti-virus and firewall software for InfoScale for Windows 

4) Ensure there are no GPOs restricting the execution of  <install_path>\Program Files\Veritas\VRTSPerl\bin\perl.exe

5) Browse to This PC -> Properties ->Advanced System Settings -> Environment variables...->System variables  and verify the System variable VRTSPERL_HOME exists and shows the correct value for <install_path>\Program Files\Veritas\VRTSPerl\

Sample output below will be captured from the RVGPrimary_A.log when Debug is enabled 

YYYY/MM/DD HH:MM:SS VCS DBG_AGDEBUG V-16-50-0 Thread(7620) command_line is ("<Install_path>:\program files\veritas\VRTSperl\;<Install_path>:\program files\veritas\VRTSperl\bin\perl.exe" "<Install_path>:\program files\veritas\Cluster Server\bin\RVGPrimary\monitor.pl" "<RVGPrimary_resource_name>" "Vvr_Rvg" "1" "1" "" "0" "0" "0") 

As seen above, there are duplicate entries for the VRTSperl path. (<Install_path>:\program files\veritas\VRTSperl\)

Sample entry for a healthy entry (assuming the Veritas Infoscale install is done on the E:\ drive)

6) Restart the HAD service for the change in the System variable to be applied.  

   hastop -local -force
   hastart

7)  Repeat on all cluster nodes that are having the issue.