NetBackup™ Deployment Guide for Kubernetes Clusters
- Introduction
- Section I. Deployment
- Prerequisites for Kubernetes cluster configuration
- Deployment with environment operators
- Deploying NetBackup
- Preparing the environment for NetBackup installation on Kubernetes cluster
- Recommendations of NetBackup deployment on Kubernetes cluster
- Limitations of NetBackup deployment on Kubernetes cluster
- Primary and media server CR
- Configuring NetBackup IT Analytics for NetBackup deployment
- Managing NetBackup deployment using VxUpdate
- Migrating the cloud node for primary or media servers
- Deploying NetBackup using Helm charts
- Deploying MSDP Scaleout
- Deploying MSDP Scaleout
- Prerequisites for AKS
- Prerequisites for EKS
- Installing the docker images and binaries
- Initializing the MSDP operator
- Configuring MSDP Scaleout
- Using MSDP Scaleout as a single storage pool in NetBackup
- Configuring the MSDP cloud in MSDP Scaleout
- Using S3 service in MSDP Scaleout for AKS
- Enabling MSDP S3 service after MSDP Scaleout is deployed for AKS
- Deploying Snapshot Manager
- Verifying Cloud Scale deployment
- Section II. Monitoring and Management
- Monitoring NetBackup
- Monitoring MSDP Scaleout
- Monitoring Snapshot Manager
- Managing the Load Balancer service
- Managing MSDP Scaleout
- Managing PostrgreSQL DBaaS
- Performing catalog backup and recovery
- Setting key parameters in Cloud Scale deployments
- Section III. Maintenance
- MSDP Scaleout Maintenance
- PostgreSQL DBaaS Maintenance
- Upgrading
- Uninstalling
- Troubleshooting
- Troubleshooting AKS and EKS issues
- View the list of operator resources
- View the list of product resources
- View operator logs
- View primary logs
- Socket connection failure
- Resolving an issue where external IP address is not assigned to a NetBackup server's load balancer services
- Resolving the issue where the NetBackup server pod is not scheduled for long time
- Resolving an issue where the Storage class does not exist
- Resolving an issue where the primary server or media server deployment does not proceed
- Resolving an issue of failed probes
- Resolving token issues
- Resolving an issue related to insufficient storage
- Resolving an issue related to invalid nodepool
- Resolving a token expiry issue
- Resolve an issue related to KMS database
- Resolve an issue related to pulling an image from the container registry
- Resolving an issue related to recovery of data
- Check primary server status
- Pod status field shows as pending
- Ensure that the container is running the patched image
- Getting EEB information from an image, a running container, or persistent data
- Resolving the certificate error issue in NetBackup operator pod logs
- Pod restart failure due to liveness probe time-out
- NetBackup messaging queue broker take more time to start
- Host mapping conflict in NetBackup
- Issue with capacity licensing reporting which takes longer time
- Local connection is getting treated as insecure connection
- Primary pod is in pending state for a long duration
- Backing up data from Primary server's /mnt/nbdata/ directory fails with primary server as a client
- Storage server not supporting Instant Access capability on Web UI after upgrading NetBackup
- Taint, Toleration, and Node affinity related issues in cpServer
- Operations performed on cpServer in environment.yaml file are not reflected
- Elastic media server related issues
- Failed to register Snapshot Manager with NetBackup
- Pods unable to connect to flexsnap-rabbitmq post Kubernetes cluster restart
- Troubleshooting AKS-specific issues
- Troubleshooting EKS-specific issues
- Troubleshooting AKS and EKS issues
- Appendix A. CR template
Updating database certificate in DBaaS
AKS-specific
- Launch an Azure CLI pod into the AKS cluster using the following command:
$ kubectl run az-cli --image=mcr.microsoft.com/azure-cli --command sleep infinity
Note:
Access to Azure Key Vault is restricted to specific subnets. Passwords stored in Azure Key Vault can be easily updated from a pod running in AKS.
- Exec into the Azure CLI pod as follows:
$ kubectl exec -it az-cli -- /bin/ash
- From Azure CLI pod, log into Azure account:
$ az login --scope https://graph.microsoft.com//.default
- (Optional) Create a key vault policy to allow the current user to retrieve the database credential.
Obtain the name of your resource group, key vault and ID of the current user by using the following respective commands:
Resource group name:
$ RESOURCE_GROUP=<resource_group_name>
Key vault name:
$ KEY_VAULT_NAME=$(az keyvault list --resource-group $RESOURCE_GROUP --resource-type vault | jq -r '.[].name')
Current user ID name:
$ USER_ID=$(az account show | jq -r '.user.name')
Create a key vault access policy as follows:
$ az keyvault set-policy -n $KEY_VAULT_NAME --upn $USER_ID --resource-group $RESOURCE_GROUP --secret-permissions all
- Store the updated certificate in key vault as follows:
$ DB_CERT_URL='https://cacerts.digicert.com/DigiCertGlobalRootCA.crt.pem'
$ DB_CERT_VALUE=$(curl -L "${DB_CERT_URL}" -s | awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}')
$ az keyvault secret set --vault-name $KEY_VAULT_NAME --name dbcertpem --value "$DB_CERT_VALUE"
- (Optional) Delete the key vault access policy created in step 4 above:
$ az keyvault delete-policy -n $KEYVAULT --upn $USER_ID
- Exit from the azure CLI pod:
$ exit
- Delete the az CLI pod:
$ kubectl delete pod az-cli
- (Applicable only for an existing cloudscale deployment) Restart the primary pod:
$ kubectl rollout restart "statefulset/${PRIMARY}" --namespace "${NAMESPACE}"
In the above command,
NAMESPACE is the namespace containing your NetBackup deployment
PRIMARY is the name of primary pod's stateful set
Use the following command to obtain NAMESPACE and PRIMARY:
$ kubectl get --namespace "${NAMESPACE}" primaryserver -o jsonpath='{.items[0].status.attributes.resourceName}'
EKS-specific
- Obtain the validity of AWS certificate as follows:
$ wget https://truststore.pki.rds.amazonaws.com/us-east-1/us-east-1-bundle.pem
$ openssl x509 -text -in ./us-east-1-bundle.pem
For example the certificate details are displayed as follows:
Certificate: Data: Version: 3 (0x2) Serial Number: c7:34:67:36:92:50:ae:75 Signature Algorithm: sha256WithRSAEncryption Issuer: C=US, L=Seattle, ST=Washington, O=Amazon Web Services, Inc., OU=Amazon RDS, CN=Amazon RDS Root 2019 CA Validity Not Before: Aug 22 17:08:50 2019 GMT Not After : Aug 22 17:08:50 2024 GMT Subject: C=US, L=Seattle, ST=Washington, O=Amazon Web Services, Inc., OU=Amazon RDS, CN=Amazon RDS Root 2019 CA... - Use lambda function to update the certificate.
LAMBDA_ARN is the ARN of the password changing lambda function. This can be obtained from the lambda function page on AWS console.
REGION=$(aws configure get region || true) DB_CERT_URL=https://truststore.pki.rds.amazonaws.com/${REGION}/${REGION}-bundle.pem DB_PROXY_CERT_URL=https://www.amazontrust.com/repository/AmazonRootCA1.pem DB_CERT_VALUE=$(curl -L "${DB_CERT_URL}" -s | awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}') DB_PROXY_CERT_VALUE=$(curl -L "${DB_PROXY_CERT_URL}" -s | awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}') DB_CUM_CERT_VALUE="$DB_CERT_VALUE$DB_PROXY_CERT_VALUE" tmp_file=$(mktemp /tmp/dbcert.XXXXXX.json) aws lambda invoke \ --function-name "$LAMBDA_ARN" \ --cli-binary-format raw-in-base64-out \ --payload "{\"dbcertpem\": \"${DB_CUM_CERT_VALUE}\"}" \ $tmp_file > /dev/null 2>&1 - Restart the primary pod:
$ kubectl rollout restart "statefulset/${PRIMARY}" --namespace "${NAMESPACE}"
In the above command,
NAMESPACE is the namespace containing your NetBackup deployment
PRIMARY is the name of primary pod's stateful set
Use the following command to obtain NAMESPACE and PRIMARY:
$ kubectl get --namespace "${NAMESPACE}" primaryserver -o jsonpath='{.items[0].status.attributes.resourceName}'