NetBackup™ Web UI Kubernetes Administrator's Guide
- Overview of NetBackup for Kubernetes
- Deploying and configuring the NetBackup Kubernetes operator
- Deploy service package on NetBackup Kubernetes operator
- Port requirements for Kubernetes operator deployment
- Upgrade the NetBackup Kubernetes operator
- Delete the NetBackup Kubernetes operator
- Configure NetBackup Kubernetes datamover
- Configure settings for NetBackup snapshot operation
- Troubleshooting NetBackup servers with short names
- Managing image groups
- Deploying certificates on NetBackup Kubernetes operator
- Managing Kubernetes assets
- Managing Kubernetes intelligent groups
- Protecting Kubernetes assets
- Recovering Kubernetes assets
- Troubleshooting Kubernetes issues
- Error during the primary server upgrade: NBCheck fails
- Error during an old image restore: Operation fails
- Error during persistent volume recovery API
- Error during restore: Final job status shows partial failure
- Error during restore on the same namespace
- Datamover pods exceed the Kubernetes resource limit
- Error during restore: Job fails on the highly loaded cluster
- Custom Kubernetes role created for specific clusters cannot view the jobs
- Openshift creates blank non-selected PVCs while restoring applications installed from OperatorHub
- NetBackup Kubernetes operator become unresponsive if PID limit exceeds on the Kubernetes node
- Failure during edit cluster in NetBackup Kubernetes 10.1
- Restore from snapshot fails for large sized PVC
- Restore of namespace file mode PVCs to different file system partially fails
- Restore from backup copy fails with image inconsistency error
NetBackup Kubernetes operator become unresponsive if PID limit exceeds on the Kubernetes node
In Linux systems, there is an initd or system process running as PID 1 to reap zombie processes. Containers that do not have such an initd process would keep spawning zombie processes.
After certain time period these zombie processes accumulates and then reaches the max limit of PIDs set on the Kubernetes node.
In NetBackup Kubernetes operator, nbcertcmdtool spawns child processes to carry out certificate-related operations. On completion of the operation, the processes get orphan and are not reaped. Eventually it hits the max PID limit and NetBackup Kubernetes operator becomes unresponsive.
Error message: login pod/nbukops-controller-manager-67f5498bbb-gn9zw -c netbackupkops -n nbukops ERRO[0005] exec failed: container_linux.go:380: starting container process caused: read init-p: connection reset by peer a command that is terminated with exit code 1.
Recommended actions:
To fix the PID limit exceed issue, you can use the Initd script. Initd script acts as parent process or entry point script to the controller pod.
As a parent process it attaches zombie process to itself after the child process completion to terminate the persistent zombie process. It also helps you to shut down the container gracefully. Initd script is available in NBUKOPs build version 10.0.1.
Use the following steps to remove the existing nbcertcmdtool zombie processes:
Describe the NetBackup operator pod and find the Kubernetes node on which the controller pod is running. Run the command:
kubectl describe -c netbackupkops <NB k8s operator pod name> -n <namespace>
Log on to the Kubernetes node, run the command:
kubectl debug node/nodename
Terminate the nbcertcmdtool zombie processes, run the command:
ps -ef | grep "\[nbcertcmdtool\] <defunct>" | awk '{print $3}' | xargs kill -9
Note:
These steps terminate all the zombie processes for that worker node. But it resolves the issue temporarily. For a permanent solution, you must deploy a new KOps build with Initd script.