Veritas NetBackup™ for OpenStack Administrator's Guide

Last Published:
Product(s): NetBackup (9.1)
  1. Introduction
    1.  
      About NetBackup for OpenStack
    2. NetBackup for OpenStack Architecture
      1.  
        Backup as a Service
      2.  
        Main Components
      3.  
        Service Endpoints
      4.  
        Network topology
  2. Deploying NetBackup for OpenStack
    1. Requirements
      1. System requirements NetBackup for OpenStack VM
        1.  
          Software Requirements
    2. NetBackup for OpenStack network considerations
      1.  
        Existing endpoints in OpenStack
      2.  
        OpenStack endpoints required by NetBackup for OpenStack
      3.  
        Recommendation: Provide access to all OpenStack Endpoint types
      4.  
        Backup target access required by NetBackup for OpenStack
      5.  
        Example of a typical NetBackup for OpenStack network integration
      6.  
        Other examples of NetBackup for OpenStack network integrations
    3. Preparing the installation
      1.  
        Tenant Quotas
      2.  
        AWS S3 eventual consistency
      3.  
        NetBackup for OpenStack Cluster
    4. Spinning up the NetBackup for OpenStack VM
      1. Creating the cloud-init image
        1.  
          Needed tools
        2.  
          Providing the Metadata
        3.  
          Creating the image file
      2.  
        Spinning up the NetBackup for OpenStack appliance
      3.  
        Uninstalling cloud-init after first start
    5. Installing NetBackup for OpenStack Components
      1. Installing on RHOSP
        1. 1. Prepare for deployment
          1.  
            1.1] Select "backup target" type
          2.  
            1.2] Copy nbos-cfg-scripts to the undercloud
          3.  
            1.3] If backup target type is "Ceph based S3" with SSL:
        2.  
          2] Upload NetBackup for OpenStack puppet module
        3. 3] Update overcloud roles data file to include NetBackup for OpenStack services
          1.  
            3.1] Add NetBackup for OpenStack Datamover API Service to role data file
          2.  
            3.2] Add NetBackup for OpenStack Datamover Service to role data file
        4.  
          4] Prepare NetBackup for OpenStack container images
        5.  
          5] Provide environment details in nbos_env.yaml
        6.  
          6] Deploy overcloud with NetBackup OpenStack environment
        7. 7] Verify deployment
          1.  
            7.1] On Controller node
          2.  
            7.2] On Compute node
          3.  
            7.3] On the node with Horizon service
        8. 8] Additional Steps on NetBackup for OpenStack Appliance
          1.  
            8.1] Change the nova user ID on the NetBackup for OpenStack Nodes
        9.  
          9] Troubleshooting for overcloud deployment failures
    6. Configuring NetBackup for OpenStack
      1. Details needed for the NetBackup for OpenStack Appliance
        1.  
          NetBackup for OpenStack Cluster information
        2.  
          OpenStack Credentials information
        3. Backup Storage Configuration information
          1.  
            Using the NFS protocol
          2.  
            Using the S3 protocol
          3.  
            Policy Import
      2. Advanced settings
        1.  
          Setup NetBackup for OpenStack and Datamover API endpoints.
        2.  
          Set up an external database
        3.  
          Define the NetBackup for OpenStack service user password
      3.  
        Starting the configurator
    7. Post Installation Health-Check
      1.  
        Verify the NetBackup for OpenStack Appliance services are up
      2.  
        Check the NetBackup for OpenStack pacemaker and NGINX cluster
      3.  
        Verify API connectivity of the NetBackup for OpenStack Appliance
      4.  
        Verify the nbosdm services are up and running
      5.  
        Verify that the NFS Volume is correctly mounted
    8. Uninstalling from RHOSP
      1.  
        Clean NetBackup for OpenStack Datamover API service
      2.  
        Clean NetBackup for OpenStack Datamover Service
      3.  
        Clean NetBackup for OpenStack haproxy resources
      4.  
        Clean NetBackup for OpenStack Keystone resources
      5.  
        Clean NetBackup for OpenStack database resources
      6.  
        Revert overcloud deploy command
      7.  
        Revert back to original RHOSP Horizon container
      8.  
        Destroy the NetBackup for OpenStack VM Cluster
    9. Install workloadmgr CLI client
      1.  
        About the workloadmgr CLI client
      2.  
        Installing the workloadmgr client
  3. Configuring NetBackup OpenStack Appliance
    1.  
      Reconfigure the NetBackup for OpenStack Cluster
    2. Configuring the NetBackup master server details
      1.  
        About security management and certificates in NetBackup
    3.  
      Change NetBackup for OpenStack dashboard password
    4.  
      Reset NetBackup for OpenStack dashboard password
    5.  
      Reinitialize NetBackup for OpenStack
    6.  
      Download NetBackup for OpenStack logs
  4. Configuring NetBackup Master Server
    1.  
      License for OpenStack plug-in for NetBackup
    2.  
      Allow NetBackup for OpenStack VM on NetBackup master server
    3. About launching the OpenStack Horizon UI from the NetBackup web UI
      1.  
        Adding the OpenStack Horizon instance on NetBackup web UI
      2.  
        Creating the custom role for NetBackup for OpenStack administrator
      3.  
        Launching the Horizon UI from the NetBackup web UI
  5. NetBackup for OpenStack policies
    1.  
      About policies
    2.  
      List of policies
    3.  
      Create a policy
    4.  
      Policy overview
    5.  
      Edit a policy
    6.  
      Delete a policy
    7.  
      Unlock a policy
    8.  
      Reset a policy
  6. Performing backups and restores of OpenStack
    1.  
      About snapshots
    2.  
      List of snapshots
    3.  
      Creating a snapshot
    4.  
      Snapshot overview
    5.  
      Delete snapshots
    6.  
      Snapshot Cancel
    7.  
      About restores
    8.  
      List of Restores
    9.  
      Restores overview
    10.  
      Delete a Restore
    11.  
      Cancel a Restore
    12.  
      One-Click Restore
    13.  
      Selective Restore
    14.  
      Inplace Restore
    15. Required restore.json for CLI
      1.  
        General required information
      2. Selective Restore required information
        1.  
          Information required in instances
        2.  
          Information required in network topology restore or network mapping
        3.  
          Full selective restore example
      3. Inplace Restore required information
        1.  
          Information required in instances
        2.  
          Network mapping information required
        3.  
          Full Inplace restore example
    16.  
      About file search
    17.  
      Navigating to the file search tab in Horizon
    18. Configuring and starting a file search in Horizon
      1.  
        Choose the VM the file search shall run against
      2.  
        Set the file path
      3.  
        Define the Snapshots to search in
    19.  
      Start the File Search and retrieve the results in Horizon
    20.  
      Doing a CLI File Search
    21.  
      About snapshot mount
    22. Create a File Recovery Manager Instance
      1.  
        Steps to apply on CentOS and RHEL cloud-images
    23.  
      Mounting a snapshot
    24.  
      Accessing the File Recovery Manager
    25.  
      Identifying mounted snapshots
    26.  
      Unmounting a snapshot
    27.  
      About schedulers
    28.  
      Disable a schedule
    29.  
      Enable a schedule
    30.  
      Modify a schedule
    31.  
      About email notifications
    32.  
      Requirements to activate email Notifications
    33.  
      Activate/Deactivate the email Notifications
  7. Performing Backup Administration tasks
    1. NBOS Backup Admin Area
      1.  
        Access the NBOS Backup Admin area
      2.  
        Status overview
      3.  
        Policies tab
      4.  
        Usage tab
      5.  
        Nodes tab
      6.  
        NBOSDM tab (NetBackup for OpenStack Datamover Service)
      7.  
        Storage tab
      8.  
        Audit tab
      9.  
        Policy Attributes tab
      10. Settings tab
        1.  
          Email Settings
        2.  
          Disable or enable Job Scheduler
    2. Policy Attributes
      1.  
        List the available policies
      2.  
        Create policy attributes
      3.  
        Edit the policy attribute
      4.  
        Assign/Remove a policy
      5.  
        Delete a policy
    3. Policy Quotas
      1.  
        Work with Policy Quotas via Horizon
      2. Work with Policy Quotas via CLI
        1.  
          List available Quota Types
        2.  
          Show Quota Type Details
        3.  
          Create a Quota
        4.  
          List allowed Quotas
        5.  
          Show allowed Quota
        6.  
          Update allowed Quota
        7.  
          Delete allowed Quota
    4. Managing Trusts
      1.  
        List all trusts
      2.  
        Show a trust
      3.  
        Create a trust
      4.  
        Delete a trust
    5. Policy import and migration
      1.  
        Import policies
      2.  
        Orphaned policies
      3.  
        Reassigning policies
    6. Disaster Recovery
      1.  
        Disaster Recovery Process
      2.  
        Mount-paths
    7. Example runbook for disaster recovery using NFS
      1.  
        Scenario
      2.  
        Prerequisites for the disaster recovery process
      3. Disaster recovery of a single policy
        1. Copy the policy directories to the configured NFS Volume
          1.  
            Mount NFS B2 Volume to a NetBackup for OpenStack VM
          2.  
            Identify the policy on the NFS B2 Volume
          3.  
            Copy the policy
        2. Make the Mount-Paths available
          1.  
            Identify the base64 hash values
          2.  
            Create and bind the paths
        3. Reassign the policy
          1.  
            Add admin-user to required domains and projects
          2.  
            Discover orphaned policies from NFS-Storage of Target Cloud
          3.  
            List available projects on Target Cloud in the Target Domain
          4.  
            List available users on the Target Cloud in the Target Project that have the right backup trustee role
          5.  
            Reassign the policy to the target project
          6.  
            Verify that the policy is available at the desired target_project
        4. Restore the policy
          1.  
            Prepare the selective restore by getting the snapshot information
          2.  
            Prepare the selective restore by creating the restore.json file
          3.  
            Run the selective restore
          4.  
            Verify the restore
        5. Clean up
          1.  
            Delete the policy
          2.  
            Remove the database entry
          3.  
            Remove the admin user from the project
      4. Disaster recovery of a complete cloud
        1. Reconfigure the Target NetBackup for OpenStack installation
          1.  
            Add NFS B2 to the NetBackup for OpenStack Appliance Cluster
          2.  
            Add NFS B2 to the NetBackup for OpenStack Datamovers
        2. Make the Mount-Paths available
          1.  
            Identify the base64 hash values
          2.  
            Create and bind the paths
        3. Reassign the policy
          1.  
            Add admin-user to required domains and projects
          2.  
            Discover orphaned policies from NFS-Storage of Target Cloud
          3.  
            List available projects on Target Cloud in the Target Domain
          4.  
            List available users on the Target Cloud in the Target Project that have the right backup trustee role
          5.  
            Reassign the policy to the target project
          6.  
            Verify that the policy is available at the desired target_project
        4. Restore the policy
          1.  
            Prepare the selective restore by getting the snapshot information
          2.  
            Prepare the selective restore by creating the restore.json file
          3.  
            Run the selective restore
          4.  
            Verify the restore
        5. Reconfigure the Target NetBackup for OpenStack installation back to the original one
          1.  
            Delete NFS B2 from the NetBackup for OpenStack Appliance Cluster
          2.  
            Delete NFS B2 from the NetBackup for OpenStack Datamovers
        6. Clean up
          1.  
            Remove the database entry
          2.  
            Remove the admin user from the project
  8. Troubleshooting
    1. General Troubleshooting Tips
      1. What is happening where
        1. NetBackup for OpenStack cluster
          1.  
            During a backup process
          2.  
            During a restore process
        2.  
          nbosdmapi
        3.  
          nbosdm
      2.  
        Everything on the Backup Target happens as user nova
      3.  
        NetBackup for OpenStack Trustee Role
      4.  
        OpenStack Quotas
    2.  
      Using the workloadmgr CLI tool on the NetBackup for OpenStack Appliance
    3. Health check of NetBackup for OpenStack
      1.  
        On the NetBackup for OpenStack Cluster
      2.  
        The nbosdmapi service
      3.  
        The datamover service
    4. Important log files
      1.  
        On the NetBackup for OpenStack Nodes
      2.  
        NetBackup for OpenStack Datamover service logs on RHOSP
      3.  
        NetBackup for OpenStack Datamover service logs on Ansible OpenStack
  9.  
    Index

On the NetBackup for OpenStack Cluster

wlm-workloads

This service runs and is active on every NetBackup for OpenStack node.

[root@Upstream ~]# systemctl status wlm-workloads
● wlm-workloads.service - workloadmanager workloads service
   Loaded: loaded (/etc/systemd/system/wlm-workloads.service; enabled; 
   vendor preset: disabled)
   Active: active (running) since Wed 2020-06-10 13:42:42 UTC; 1 weeks 
   4 days ago
 Main PID: 12779 (workloadmgr-wor)
    Tasks: 17
   CGroup: /system.slice/wlm-workloads.service
           ├─12779 /home/stack/myansible/bin/python 
/home/stack/myansible/bin/workloadmgr-workloads 
--config-file=/etc/workloadmgr/workloadmgr.conf
           ├─12982 /home/stack/myansible/bin/python 
/home/stack/myansible/bin/workloadmgr-workloads 
--config-file=/etc/workloadmgr/workloadmgr.conf
           ├─12983 /home/stack/myansible/bin/python 
/home/stack/myansible/bin/workloadmgr-workloads 
--config-file=/etc/workloadmgr/workloadmgr.conf
           ├─12984 /home/stack/myansible/bin/python 
/home/stack/myansible/bin/workloadmgr-workloads 
--config-file=/etc/workloadmgr/workloadmgr.conf
           [...]

wlm-api

This service runs and is active on every NetBackup for OpenStack node.

[root@Upstream ~]# systemctl status wlm-api
● wlm-api.service - Cluster Controlled wlm-api
   Loaded: loaded (/etc/systemd/system/wlm-api.service; disabled; 
   vendor preset: disabled)
  Drop-In: /run/systemd/system/wlm-api.service.d
           └─50-pacemaker.conf
   Active: active (running) since Thu 2020-04-16 22:30:11 UTC; 
   2 months 5 days ago
 Main PID: 11815 (workloadmgr-api)
    Tasks: 1
   CGroup: /system.slice/wlm-api.service
           └─11815 /home/stack/myansible/bin/python /home/stack/
           myansible/bin/workloadmgr-api --config-file=/etc/
           workloadmgr/workloadmgr.conf
wlm-scheduler

This service runs and is active on every NetBackup for OpenStack node.

[root@Upstream ~]# systemctl status wlm-scheduler
● wlm-scheduler.service - Cluster Controlled wlm-scheduler
   Loaded: loaded (/etc/systemd/system/wlm-scheduler.service; disabled; 
   vendor preset: disabled)
  Drop-In: /run/systemd/system/wlm-scheduler.service.d
           └─50-pacemaker.conf
   Active: active (running) since Thu 2020-04-02 13:49:22 UTC; 2 months 
   20 days ago
 Main PID: 29439 (workloadmgr-sch)
    Tasks: 1
   CGroup: /system.slice/wlm-scheduler.service
           └─29439 /home/stack/myansible/bin/python /home/stack/myansible
           /bin/workloadmgr-scheduler --config-file=/etc/workloadmgr/
           workloadmgr.conf
wlm-cron

This service is controlled by pacemaker and runs only on the master node

[root@Upstream ~]# systemctl status wlm-cron
● wlm-cron.service - Cluster Controlled wlm-cron
   Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; 
   vendor preset: disabled)
  Drop-In: /run/systemd/system/wlm-cron.service.d
           └─50-pacemaker.conf
   Active: active (running) since Wed 2021-01-27 19:59:26 UTC; 6 days ago
 Main PID: 23071 (workloadmgr-cro)
   CGroup: /system.slice/wlm-cron.service
           ├─23071 /home/stack/myansible/bin/python3 /home/stack/
myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr
/workloadmgr.conf
           └─23248 /home/stack/myansible/bin/python3 /home/stack/
myansible/bin/workloadmgr-cron --config-file=/etc/workloadmgr/
workloadmgr.conf

Feb 03 19:28:43 nbosvm1-ansible-ussuri-ubuntu18-vagrant workloadmgr-cron
[23071]: ● wlm-cron.service - Cluster Controlled wlm-cron
Feb 03 19:28:43 nbosvm1-ansible-ussuri-ubuntu18-vagrant workloadmgr-cron
[23071]: Loaded: loaded (/etc/systemd/system/wlm-cron.service; disabled; 
vendor preset: disabled)
Feb 03 19:28:43 nbosvm1-ansible-ussuri-ubuntu18-vagrant workloadmgr-cron
[23071]: Drop-In: /run/systemd/system/wlm-cron.service.d
Feb 03 19:28:43 nbosvm1-ansible-ussuri-ubuntu18-vagrant workloadmgr-cron
[23071]: └─50-pacemaker.conf
Feb 03 19:28:43 nbosvm1-ansible-ussuri-ubuntu18-vagrant workloadmgr-cron
[23071]: Active: active (running) since Wed 2021-01-27 19:59:26 UTC; 
6 days ago
Feb 03 19:28:43 nbosvm1-ansible-ussuri-ubuntu18-vagrant workloadmgr-cron
[23071]: Main PID: 23071 (workloadmgr-cro)
Feb 03 19:28:43 nbosvm1-ansible-ussuri-ubuntu18-vagrant workloadmgr-cron
[23071]: CGroup: /system.slice/wlm-cron.service
Feb 03 19:28:43 nbosvm1-ansible-ussuri-ubuntu18-vagrant workloadmgr-cron
[23071]: ├─23071 /home/stack/myansible/bin/python3 /home/stack/myansible/
bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf
Feb 03 19:28:43 nbosvm1-ansible-ussuri-ubuntu18-vagrant workloadmgr-cron
[23071]: ├─23248 /home/stack/myansible/bin/python3 /home/stack/myansible/
bin/workloadmgr-cron --config-file=/etc/workloadmgr/workloadmgr.conf
Feb 03 19:28:43 nbosvm1-ansible-ussuri-ubuntu18-vagrant workloadmgr-cron
[23071]: └─27145 /usr/bin/systemctl status wlm-cron
Pacemaker Cluster Status

The pacemaker cluster controls and watches the VIP on the NetBackup for OpenStack Cluster. It also controls on which node the wlm-api and wlm-scheduler service runs.

[root@Upstream ~]# pcs status
Cluster name: NetBackup for OpenStack

WARNINGS:
Corosync and pacemaker node names do not match (IPs used in setup?)

Stack: corosync
Current DC: nbosvm1-ansible-ussuri-ubuntu18-vagrant (version 
1.1.23-1.el7_9.1-9acf116022) - chapterition with quorum
Last updated: Wed Feb  3 19:20:02 2021
Last change: Wed Jan 27 20:00:12 2021 by root via crm_resource on 
nbosvm1-ansible-ussuri-ubuntu18-vagrant

1 node configured
6 resource instances configured

Online: [ nbosvm1-ansible-ussuri-ubuntu18-vagrant ]

Full list of resources:

 virtual_ip     (ocf::heartbeat:IPaddr2):       Started nbosvm1-ansible-
ussuri-ubuntu18-vagrant
 virtual_ip_public      (ocf::heartbeat:IPaddr2):       Started nbosvm1-
ansible-ussuri-ubuntu18-vagrant
 virtual_ip_admin       (ocf::heartbeat:IPaddr2):       Started nbosvm1-
ansible-ussuri-ubuntu18-vagrant
 virtual_ip_internal    (ocf::heartbeat:IPaddr2):       Started nbosvm1-
ansible-ussuri-ubuntu18-vagrant
 wlm-cron       (systemd:wlm-cron):     Started nbosvm1-ansible-ussuri-
ubuntu18-vagrant
 Clone Set: lb_nginx-clone [lb_nginx]
     Started: [ nbosvm1-ansible-ussuri-ubuntu18-vagrant ]

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
Mount availability

The NetBackup for OpenStack Cluster needs access to the Backup Target and should have the correct mount at all times.

[root@Upstream ~]# df -h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               3.8G     0  3.8G   0% /dev
tmpfs                  3.8G   38M  3.8G   1% /dev/shm
tmpfs                  3.8G  427M  3.4G  12% /run
tmpfs                  3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/vda1               40G  8.8G   32G  22% /
tmpfs                  773M     0  773M   0% /run/user/996
tmpfs                  773M     0  773M   0% /run/user/0
10.10.2.20:/upstream  1008G  704G  254G  74% /var/NetBackupOpenStack-mounts/
                                             MTAuMTAuMi4yMDovdXBzdHJlYW0=
10.10.2.20:/upstream2  483G   22G  462G   5% /var/NetBackupOpenStack-mounts/
                                             MTAuMTAuMi4yMDovdXBzdHJlYW0y