Veritas InfoScale™ 7.3.1 Troubleshooting Guide - Linux

If a unit service is active and the corresponding process is stopped outside of systemd, the service cannot be started again using 'systemctl start'

While a unit service is in the 'active (running)' or 'active (exited)' state, the corresponding process may be stopped outside of the systemd context. In that case, you cannot start the service again using the systemctl start command.

In the following example, the HA process (had) is not running, but the unit service appears to be in the active (running) state:

( root@localhost-vm1 ):[ ~ ]#systemctl status vcs
vcs.service - VERITAS Cluster Server (VCS)
Loaded: loaded (/opt/VRTSvcs/bin/vcs; enabled)
Active: active (running) since Sat 2017-04-22 00:10:24 IST;
 2 days ago
Process: 6671 ExecStart=/opt/VRTSvcs/bin/vcs start 2>&1
 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/vcs.service
|-7724 /usr/lib/fs/vxfs/vxfsckd -p /var/adm/cfs/vxfsckd-pid
'-8046 /usr/sbin/vxnotify -g crsdg2324 -icfspdvACLMSk
Apr 22 00:16:30 localhost-vm1 Had[6753]: VCS ERROR V-16-2-13066 
(localhost-vm1) Agent is calling clean for resou...leted.
Apr 22 00:16:34 localhost-vm1 Had[6753]: VCS ERROR V-16-2-13066 
(localhost-vm0) Agent is calling clean for resou...leted.
Apr 22 00:17:31 localhost-vm1 AgentFramework[6835]: VCS ERROR
 V-16-2-13006 Thread(140152660154112) Resource(cssd): ...time.
Apr 22 00:17:31 localhost-vm1 Had[6753]: VCS ERROR
 V-16-2-13006 (localhost-vm1) Resource(cssd): clean procedure
 ... time.
Apr 22 00:17:35 localhost-vm1 Had[6753]: VCS ERROR
 V-16-2-13006 (localhost-vm0) Resource(cssd): clean procedure
 ... time.
Apr 22 00:22:37 localhost-vm1 AgentFramework[6835]: VCS ERROR
 V-16-2-13078 Thread(140152660154112) Resource(cssd) -...mpts.
Apr 22 00:22:37 localhost-vm1 AgentFramework[6835]: VCS ERROR
 V-16-2-13071 Thread(140152660154112) Resource(cssd): ...t(0).
Apr 22 00:22:37 localhost-vm1 Had[6753]: VCS ERROR V-16-1-54031
 Resource cssd (Owner: Unspecified, Group: cvm) is ...t-vm24
Apr 22 00:22:44 localhost-vm1 Had[6753]: VCS ERROR V-16-1-54031
 Resource cssd (Owner: Unspecified, Group: cvm) is ...t-vm23
Apr 24 21:51:11 localhost-vm1 systemd[1]: Started VERITAS
 Cluster Server (VCS).
Hint: Some lines were ellipsized, use -l to show in full.

( root@localhost-vm1 ):[ ~ ]# ps -ef | grep had
root 13735 10930 0 21:52 pts/2 00:00:00 grep --color=auto had

( root@localhost-vm1 ):[ ~ ]#/opt/VRTSvcs/bin/vcs status
had is stopped

In this case, systemd cannot process the systemctl start vcs command.

Recommended action

To work around this issue, run the following commands sequentially:

( root@localhost-vm1 )[ ~ ] # systemctl stop vcs
...

( root@localhost-vm1 )[ ~ ] # systemctl start vcs
...

Or, run the restart command as follows:

( root@localhost-vm1 )[ ~ ] #systemctl restart vcs

Verify that unit service is running:

( root@localhost-vm1 ):[ ~ ]# systemctl status vcs
vcs.service - VERITAS Cluster Server (VCS)
Loaded: loaded (/opt/VRTSvcs/bin/vcs; enabled)
Active: active (running) since Mon 2017-04-24 21:57:49 IST;
 3s ago
Process: 15339 ExecStop=/opt/VRTSvcs/bin/vcs stop 2>&1
 (code=exited, status=0/SUCCESS)
Process: 15356 ExecStart=/opt/VRTSvcs/bin/vcs start 2>&1
 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/vcs.service
|- 7724 /usr/lib/fs/vxfs/vxfsckd -p /var/adm/cfs/vxfsckd-pid
|- 8046 /usr/sbin/vxnotify -g crsdg2324 -icfspdvACLMSk
|-15394 /opt/VRTSvcs/bin/had
'-15400 /opt/VRTSvcs/bin/hashadow
Apr 24 21:57:49 localhost-vm1 systemd[1]: Starting VERITAS
 Cluster Server (VCS)...
Apr 24 21:57:49 localhost-vm1 vcs[15356]: Starting VCS: [ OK ]
Apr 24 21:57:49 localhost-vm1 systemd[1]: Started VERITAS
 Cluster Server (VCS).
Apr 24 21:57:49 localhost-vm1 Had[15394]: VCS NOTICE
 V-16-1-10619 'HAD' starting on: localhost-vm1
Apr 24 21:57:49 localhost-vm1 Had[15394]: VCS NOTICE
 V-16-1-10620 Waiting for local cluster configuration status
Apr 24 21:57:49 localhost-vm1 Had[15394]: VCS NOTICE
 V-16-1-10625 Local cluster configuration valid
Apr 24 21:57:49 localhost-vm1 Had[15394]: VCS NOTICE
 V-16-1-11034 Registering for cluster membership
Apr 24 21:57:49 localhost-vm1 Had[15394]: VCS NOTICE
 V-16-1-11035 Waiting for cluster membership
Apr 24 21:57:54 localhost-vm1 Had[15394]: VCS INFO
 V-16-1-10077 Received new cluster membership
Apr 24 21:57:54 localhost-vm1 Had[15394]: VCS NOTICE
 V-16-1-10086 System localhost-vm1 (Node '1') is in Regular
 Membership - Membership: 0x2
Apr 24 21:57:54 localhost-vm1 Had[15394]: VCS NOTICE
 V-16-1-10073 Building from local configuration
Apr 24 21:57:55 localhost-vm1 Had[15394]: VCS NOTICE
 V-16-1-10066 Entering RUNNING state
Apr 24 21:57:55 localhost-vm1 Had[15394]: VCS NOTICE
 V-16-1-50311 VCS Engine: running with security OFF
Apr 24 22:01:22 localhost-vm1 Had[15394]: VCS INFO
 V-16-1-10077 Received new cluster membership
Apr 24 22:01:22 localhost-vm1 Had[15394]: VCS NOTICE
 V-16-1-10086 System localhost-vm0 (Node '0') is in Regular
 Membership - Membership: 0x3

Verify that the corresponding services are started:

( root@localhost-vm1 ):[ ~ ]# /opt/VRTSvcs/bin
/vcs status
had (pid 15394) is running...

( root@localhost-vm1 ):[ ~ ]# ps -ef | grep had
root 15394 1 0 21:57 ? 00:00:00 /opt/VRTSvcs/bin/had
root 15400 1 0 21:57 ? 00:00:00 /opt/VRTSvcs/bin/hashadow