NetBackup™ Deployment Guide for Azure Kubernetes Services (AKS) Cluster

Last Published:
Product(s): NetBackup & Alta Data Protection (10.0.0.1)
  1. Introduction to NetBackup on AKS
    1.  
      About NetBackup deployment on Azure Kubernetes Services (AKS) cluster
    2.  
      Required terminology
    3.  
      User roles and permissions
    4.  
      About MSDP Scaleout
    5.  
      About MSDP Scaleout components
    6.  
      Limitations in MSDP Scaleout
  2. Deployment with environment operators
    1. About deployment with the environment operator
      1.  
        Prerequisites
      2.  
        Contents of the TAR file
      3.  
        Known limitations
    2.  
      Deploying using the deploy.sh file
    3.  
      Deploying the operators manually
    4.  
      Deploying NetBackup and MSDP Scaleout manually
    5.  
      Configuring the environment.yaml file
    6.  
      Uninstalling NetBackup environment and the operators
    7.  
      Applying security patches
  3. Assessing cluster configuration before deployment
    1.  
      How does the Config-Checker utility work
    2.  
      Config-Checker execution and status details
  4. Deploying NetBackup
    1.  
      Preparing the environment for NetBackup installation on AKS
    2.  
      Recommendations of NetBackup deployment on AKS
    3.  
      Limitations of NetBackup deployment on AKS
    4. About primary server CR and media server CR
      1.  
        After installing primary server CR
      2.  
        After Installing the media server CR
    5.  
      Monitoring the status of the CRs
    6.  
      Updating the CRs
    7.  
      Deleting the CRs
    8.  
      Configuring NetBackup IT Analytics for NetBackup deployment
    9.  
      Managing NetBackup deployment using VxUpdate
    10.  
      Migrating the node pool for primary or media servers
  5. Upgrading NetBackup
    1.  
      Preparing for NetBackup upgrade
    2.  
      Upgrading NetBackup operator
    3.  
      Upgrading NetBackup application
    4.  
      Procedure to rollback when upgrade fails
  6. Deploying MSDP Scaleout
    1.  
      Deploying MSDP Scaleout
    2.  
      Prerequisites
    3.  
      Installing the docker images and binaries
    4.  
      Initializing the MSDP operator
    5.  
      Configuring MSDP Scaleout
    6.  
      Using MSDP Scaleout as a single storage pool in NetBackup
    7.  
      Configuring the MSDP cloud in MSDP Scaleout
  7. Upgrading MSDP Scaleout
    1.  
      Upgrading MSDP Scaleout
  8. Monitoring NetBackup
    1.  
      Monitoring the application health
    2.  
      Telemetry reporting
    3.  
      About NetBackup operator logs
    4.  
      Expanding storage volumes
    5.  
      Allocating static PV for Primary and Media pods
  9. Monitoring MSDP Scaleout
    1.  
      About MSDP Scaleout status and events
    2.  
      Monitoring with Azure Container insights
    3.  
      The Kubernetes resources for MSDP Scaleout and MSDP operator
  10. Managing the Load Balancer service
    1.  
      About the Load Balancer service
    2.  
      Notes for Load Balancer service
    3.  
      Opening the ports from the Load Balancer service
  11. Performing catalog backup and recovery
    1.  
      Backing up a catalog
    2.  
      Restoring a catalog
  12. Managing MSDP Scaleout
    1.  
      Adding MSDP engines
    2.  
      Adding data volumes
    3. Expanding existing data or catalog volumes
      1.  
        Manual storage expansion
    4.  
      MSDP Scaleout scaling recommendations
    5. MSDP Cloud backup and disaster recovery
      1.  
        About the reserved storage space
      2.  
        Cloud LSU disaster recovery
    6.  
      MSDP multi-domain support
    7.  
      Configuring Auto Image Replication
    8. About MSDP Scaleout logging and troubleshooting
      1.  
        Collecting the logs and the inspection information
  13. About MSDP Scaleout maintenance
    1.  
      Pausing the MSDP Scaleout operator for maintenance
    2.  
      Logging in to the pods
    3.  
      Reinstalling MSDP Scaleout operator
    4.  
      Migrating the MSDP Scaleout to another node pool
  14. Uninstalling MSDP Scaleout from AKS
    1.  
      Cleaning up MSDP Scaleout
    2.  
      Cleaning up the MSDP Scaleout operator
  15. Troubleshooting
    1.  
      View the list of operator resources
    2.  
      View the list of product resources
    3.  
      View operator logs
    4.  
      View primary logs
    5.  
      Pod restart failure due to liveness probe time-out
    6.  
      Socket connection failure
    7.  
      Resolving an invalid license key issue
    8.  
      Resolving an issue where external IP address is not assigned to a NetBackup server's load balancer services
    9.  
      Resolving the issue where the NetBackup server pod is not scheduled for long time
    10.  
      Resolving an issue where the Storage class does not exist
    11.  
      Resolving an issue where the primary server or media server deployment does not proceed
    12.  
      Resolving an issue of failed probes
    13.  
      Resolving token issues
    14.  
      Resolving an issue related to insufficient storage
    15.  
      Resolving an issue related to invalid nodepool
    16.  
      Resolving a token expiry issue
    17.  
      Resolve an issue related to inconsistency in file ownership
    18.  
      Resolve an issue related to KMS database
    19.  
      Resolve an issue related to pulling an image from the container registry
    20.  
      Resolving an issue related to recovery of data
    21.  
      Check primary server status
    22.  
      Pod status field shows as pending
    23.  
      Ensure that the container is running the patched image
    24.  
      Getting EEB information from an image, a running container, or persistent data
    25.  
      Resolving the certificate error issue in NetBackup operator pod logs
  16. Appendix A. CR template
    1.  
      Secret
    2.  
      MSDP Scaleout CR

MSDP Scaleout CR

  • The CR name must be less than 40 characters.

  • The MSDP credentials stored in the Secret must match MSDP credential rules.

    See Deduplication Engine credentials for NetBackup

  • MSDP CR cannot be deployed in the namespace of MSDP operator. It must be in a separate namespace.

  • You cannot reorder the IP/FQDN list. You can update the list by appending the information.

  • You cannot change the storage class name. The storage class must be backed with Azure disk CSI storage driver "disk.csi.azure.com".

  • You cannot change the data volume list other than for storage expansion. It is append-only and storage expansion only. Up to 16 data volumes are supported.

  • Like the data volumes, the catalog volume can be changed for storage expansion only.

  • You cannot change or expand the size of the log volume by changing the MSDP CR.

  • You cannot enable NBCA after the configuration.

  • Once KMS and the OST registration parameters set, you cannot change them.

  • You cannot change the core pattern.

MSDP Scaleout CR template:

# The MSDPScaleout CR YAML
apiVersion: msdp.veritas.com/v1
kind: MSDPScaleout
metadata:
  # The CR name should not be longer than 40 characters.
  name: sample-app
  # The namespace needs to be present for the CR to be created in.
  # It's not allowed to deploy the CR in the same namespace with MSDP 
operator.
  namespace: sample-namespace
spec:
  # Your ACR URL where the docker images can be pulled from by the 
AKS cluster on demand
  # The allowed length is in range 1-255
  # It's optional for BYO. The code does nt check the presence or 
validation.
  # User needs to specify it correctly if it's needed.
  containerRegistry: sample.azurecr.io
  #
  # The MSDP version string. It's the tag of the MSDP docker images.
  # The allowed length is in range 1-64
  version: "sample-version-string"
  #
  # Size defines the number of Engine instances in MSDP Scaleout.
  # The allowed size is between 1-16
  size: 4
  #
  # The IP and FQDN pairs are used by the Engine Pods to expose the MSDP 
services.
  # The IP and FQDN in one pair should match each other correctly.
  # They must be pre-allocated.
  # The item number should match the number of Engine instances.
  # They're not allowed to be changed or re-ordered. New items can be 
appended for scaling out.
  # The first FQDN is used to configure the storage server in NetBackup, 
automatically if autoRegisterOST is enabled,
  # or manually by the user if not.
  serviceIPFQDNs:
    # The pattern is IPv4 or IPv6 format
  - ipAddr: "sample-ip1"
    # The pattern is FQDN format. `^[a-z][a-z0-9-.]{1,251}[a-z0-9]$`
    fqdn: "sample-fqdn1"
  - ipAddr: "sample-ip2"
    fqdn: "sample-fqdn2"
  - ipAddr: "sample-ip3"
    fqdn: "sample-fqdn3"
  - ipAddr: "sample-ip4"
    fqdn: "sample-fqdn4"
  #
  # Optional annotations to be added in the LoadBalancer services for the 
Engine IPs.
  # In case we run the Engines on private IPs, we need to add some 
customized annotations to the LoadBalancer services.
  # See https://docs.microsoft.com/en-us/azure/aks/internal-lb
  # It's optional. It's not needed in most cases if we're 
with public IPs.
  # loadBalancerAnnotations:
  #   service.beta.kubernetes.io/azure-load-balancer-internal: "true"
  #
  # SecretName is the name of the secret which stores the MSDP credential.
  # AutoDelete, when true, will automatically delete the secret specified 
by SecretName after the
  # initial configuration. If unspecified, AutoDelete defaults to true.
  # When true, SkipPrecheck will skip webhook validation of the MSDP 
credential. It is only used in data re-use
  # scenario (delete CR and re-apply with pre-existing data) as the 
secret will not take effect in this scenario. It
  # can't be used in other scenarios. If unspecified, SkipPrecheck 
defaults to false.
  credential:
    # The secret should be pre-created in the same namespace which has 
the MSDP credential stored.
    # The secret should have a "username" and a "password" key-pairs 
with the corresponding username and password values.
    # Please follow MSDP guide for the rules of the credential.
    #   https://www.veritas.com/content/support/en_US/article.100048511
    # A secret can be created directly via kubectl command or with the 
equivalent YAML file:
    #   kubectl create secret generic sample-secret --namespace 
sample-namespace \
    #   --from-literal=username=<username> --from-literal=password=
<password>
    secretName: sample-secret
    # Optional
    # Default is true
    autoDelete: true
    # Optional
    # Default is false.
    # Should be specified only in data re-use scenario (aka delete and 
re-apply CR with pre-existing data)
    skipPrecheck: false
  #
  # Paused is used for maintenance only. In most cases you don't need 
to specify it.
  # When it's specified, MSDP operator stops reconciling the corresponding 
MSDP-X (aka the CR).
  # Optional.
  # Default is false
  # paused: false
  #
  # The storage classes for logVolume, catalogVolume and dataVolumes should 
be:
  #   - Backed with Azure disk CSI driver "disk.csi.azure.com" with the 
managed disks, and allow volume
  #     expansion.
  #   - The Azure in-tree storage driver "kubernetes.io/azure-disk" is not 
supported. You need to explicitly
  #     enable the Azure disk CSI driver when configuring your AKS cluster, 
or use k8s version v1.21.x which
  #     has the Azure disk CSI driver built-in.
  #   - In LRS category.
  #   - At least Standard SSD for dev/test, and Premium SSD or Ultra Disk 
for production.
  #   - The same storage class can be used for all the volumes.
  #   -
  #
  # LogVolume is the volume specification which is used to provision a 
volume of an MDS or Controller
  # Pod to store the log files and core dump files.
  # It's not allowed to be changed.
  # In most cases, 5-10 GiB capacity should be big enough for one MDS or 
Controller Pod to use.
  logVolume:
    storageClassName: sample-azure-disk-sc1
    resources:
      requests:
        storage: 5Gi
  #
  # CatalogVolume is the volume specification which is used to provision a 
volume of an MDS or Engine
  # Pod to store the catalog and metadata. It's not allowed to be changed 
unless for capacity expansion.
  # Expanding the existing catalog volumes expects short downtime of the 
Engines.
  # Please note the MDS Pods don't respect the storage request in 
CatalogVolume, instead they provision the
  # volumes with the minimal capacity request of 500MiB.
  catalogVolume:
    storageClassName: sample-azure-disk-sc2
    resources:
      requests:
        storage: 600Gi
  #
  # DataVolumes is a list of volume specifications which are used to 
provision the volumes of
  # an Engine Pod to store the MSDP data.
  # The items are not allowed to be changed or re-ordered unless for 
capacity expansion.
  # New items can be appended for adding more data volumes to each 
Engine Pod.
  # Appending new data volumes or expanding the existing data volumes 
expects short downtime of the Engines.
  # The allowed item number is in range 1-16. To allow the other MSDP-X 
Pods (e.g. Controller, MDS) running
  # on the same node, the item number should be no more than "<the maximum 
allowed volumes on the node> - 5".
  # The additional 5 data disks are for the potential one MDS Pod, one 
Controller Pod or one MSDP operator Pod
  # to run on the same node with one MSDP Engine.
  dataVolumes:
    - storageClassName: sample-azure-disk-sc3
      resources:
        requests:
          storage: 8Ti
    - storageClassName: sample-azure-disk-sc3
      resources:
        requests:
          storage: 8Ti
  #
  # NodeSelector is used to schedule the MSDPScaleout Pods on the specified 
nodes.
  # Optional.
  # Default is empty (aka all available nodes)
  nodeSelector:
    # e.g.
    # agentpool: nodepool2
    sample-node-label1: sampel-label-value1
    sample-node-label2: sampel-label-value2
  #
  # NBCA is the specification for MSDP-X to enable NBCA SecComm 
for the Engines.
  # Optional.
  nbca:
    # The master server name
    # The allowed length is in range 1-255
    masterServer: sample-master-server-name
    # The CA SHA256 fingerprint
    # The allowed length is 95
    cafp: sample-ca-fp
    # The NBCA authentication/reissue token
    # The allowed length is 16
    # For security consideration, a token with maximum 1 user allowed and 
valid for 1 day should be sufficient.
    token: sample-auth-token
  #
  # KMS includes the parameters to enable KMS for the Engines.
  # We support to enable KMS in init or post configuration.
  # We don't support to change the parameters once they have been set.
  # Optional.
  kms:
    # As either the NetBackup KMS or external KMS (EKMS) is configured or 
registered on NetBackup master server, then used by
    # MSDP by calling the NetBackup API, kmsServer is the NetBackup master 
server name.
    kmsServer: sample-master-server-name
    keyGroup: sample-key-group-name
  #
  # autoRegisterOST includes the parameter to enable or disable the 
automatic registration of
  # the storage server, the default disk pool and storage unit when MSDP-X 
configuration finishes.
  autoRegisterOST:
    # If it is true, and NBCA is enabled, the operator would register the 
storage server,
    # disk pool and storage unit on the NetBackup primary server, when the 
MSDP CR is deployed.
    # The first Engine FQDN is the storage server name.
    # The default disk pool is in format "default_dp_<firstEngineFQDN>".
    # The default storage unit is in format "default_stu_<firstEngineFQDN>".
    # The default maximum concurrent jobs for the STU is 240.
    # In the CR status, field "ostAutoRegisterStatus.registered" with value 
True, False or Unknown indicates the registration state.
    # It's false by default.
    # Note: Please don't enable it unless with NB_9.1.2_0126+.
    enabled: true
  #
  # CorePattern is the core pattern of the nodes where the MSDPScaleout 
Pods are running.
  # It's path-based. A default core path "/core/core.%e.%p.%t" will be 
used if not specified.
  # In most cases, you don't need to specify it.
  # It's not allowed to be changed.
  # Optional.
  # corePattern: /sample/core/pattern/path
  #
  # tcpKeepAliveTime sets the namespaced sysctl parameter 
net.ipv4.tcp_keepalive_time in Engine Pods.
  # It's in seconds.
  # The minimal allowed value is 60 and the maximum allowed value is 1800.
  # A default value 120 is used if not specified. Set it to 0 to disable 
the option.
  # It's not allowed to change unless in maintenance mode (Paused=true), 
and the change will not apply until the Engine Pods get restarted
  # For AKS deployment in P release, please leave it unspecified or specify 
it with a value smaller than 240.
  # tcpKeepAliveTime: 120
  #
  # TCPIdleTimeout is used to change the default value for Azure Load 
Balancer rules and Inbound NAT rules.
  # It's in minutes.
  # The minimal allowed value is 4 and the maximum allowed value is 30.
  # A default value 30 minutes is used if not specified. Set it to 0 to 
disable the option.
  # It's not allowed to change unless in maintenance mode (Paused=true), 
and the change will not apply
  # until the Engine Pods and the LoadBalancer services get recreated.
  # For AKS deployment in P release, please leave it unspecified or specify 
it with a value larger than 4.
  # tcpIdleTimeout: 30