The EMM_DATA.db file can grow very large when running NetBackup 8.2

Problem

When NetBackup is under heavy load, some master servers have observed an increase in the size of the NetBackup database file called EMM_DATA.db. On some Master Servers, the increase in size is considerable; many tens of gigabytes.

Under normal operations, the various NetBackup database tables will grow with use. Under normal circumstances, all daily operations are successful and no corrective action needs to be taken.

However, in some cases, the substantial growth of EMM_DATA.db can cause a failure during the validation phase of the catalog backup, thus resulting in a failed catalog backup.

Error Message

N/A: EMM_DATA.db file has grown very large

Cause

NetBackup Engineering is currently working with Sybase Engineering to understand more about why the EMM_DATA.db namespace is growing. This is what we know so far.

When data is added to the database, it consumes space in the form of table pages. As data is removed, empty table pages become free pages and those free pages can be consumed again by future table page requests. At some point, on busy systems, although data is removed from the tables, the empty table pages fail to be freed for re-use. When this occurs, new table page requests are satisfied by growing the database file itself (EMM_DATA.db) because there are not enough internal free pages to satisfy the requests.

On very busy systems, a lot of transitory resource allocation data is added to and removed from tables within the EMM_DATA.db file. When empty table pages fail to become freed, rapidly occurring resource requests associated with jobs starting and stopping, consume more and more new space within EMM_DATA.db causing the excessive growth being observed.

Under lighter loads, the Sybase database does not fail to free empty table pages.

Solution

Several options are available depending on the situation.

Option A:

If negative symptoms are not resulting from the large file size, corrective action is not necessary.

Option B:

If the validation phase (third child job) of a catalog backup is failing, this is usually caused by a memory constraint relative to the size of the database objects within the file. The validation is performed by a second dbsrv17 process which is configured to use an upper limit of 1 GB of RAM. Very large objects may require more than 1 GB of RAM to successfully complete database validation. To increase the upper limit, modify the following file and increase the "-ch" value in increments of 1024 until a value is found which allows the catalog backup to succeed.

     UNIX/Linux: /usr/openv/var/global/server.conf.forbackup
     Windows: <install_path>\Veritas\NetBackupDB\CONF\server.conf.forbackup

Note: The default content is: -c 100M -ch 1024M -cl 100M
Note: Modification of this file does not require a restart of NetBackup services.

Option C:

In some cases it may be necessary to periodically shrink the size of the NetBackup database files in order to reclaim disk space. These are the steps to perform that operation.

Note: It is recommended that jobs not be running during this procedure because the rebuild process locks the relational database, and this may result in jobs failing due to timeouts while trying to access the database. At a minimum suspend scheduling to prevent new jobs from being scheduled.

1. Prevent NetBackup from starting scheduled jobs.

     UNIX/Linux: /usr/openv/netbackup/bin/admincmd/nbpemreq -suspend_scheduling
     Windows: <install_path>\NetBackup\bin\admincmd\nbpemreq -suspend_scheduling

2. Perform a backup of the database files (this is a precaution).

     UNIX/Linux: /usr/openv/db/bin/nbdb_backup -online <destination_directory>
     Windows: <install_path>\NetBackup\bin\nbdb_backup -online <destination_directory>

3. Perform the rebuild.

     UNIX/Linux: /usr/openv/db/bin/nbdb_unload -rebuild -verbose
     Windows: <install_path>\NetBackup\bin\nbdb_unload -rebuild -verbose

4. Perform the reorganize.

     UNIX/Linux: /usr/openv/db/bin/nbdb_admin -reorganize -verbose
     Windows: <install_path>\NetBackup\bin\nbdb_admin -reorganize -verbose

5. Resume processing of scheduled jobs.

     UNIX/Linux: /usr/openv/netbackup/bin/admincmd/nbpemreq -resume_scheduling
     Windows: <install_path>\NetBackup\bin\admincmd\nbpemreq -resume_scheduling

Option D:

Veritas Engineering has written an Emergency Engineering Binary (EEB) which removes allocation requests (associated with new jobs) from the Sybase database. Instead, allocation requests will utilize a SQLite database which is dedicated to tracking this information.

This change relieves load from the Sybase database and will allow it to function properly.

For customers who are experiencing excessive growth in EMM_DATA.db, applying this EEB is recommended.

However, applying the EEB will not reduce the size of the database once it has experienced growth.

As part of the EEB implementation process, it is recommended (although not required) to follow the steps to rebuild / reorganize the database as described in step C above.

For EEB checksum and installation instructions, see Using the NetBackup Emergency Engineering Binary (EEB) installer

Installing the EEB

1. Obtain the EEB associated with ET 3996459v2

2. Stop all Active and Queued NetBackup jobs, but leave NetBackup services up and running
    Note: This may require suspending scheduling or disabling policies

3. Execute the following command to release any reservations

     UNIX/Linux: /usr/openv/netbackup/bin/admincmd/nbrbutil -resetAll
     Windows: <install_path>\NetBackup\bin\admincmd\nbrbutil -resetAll

4. Perform a rebuild / reorganize (recommended)

5. Stop NetBackup services

6. Install the EEB associated with ET 3996459v2

7. Start NetBackup services

8. NetBackup is now able to continue normal operations

Things to know about the EEB

1. The new SQLite database file is located here

     UNIX/Linux: /usr/openv/netbackup/db/rb.db
     Windows: <install_path>\Veritas\NetBackup\db\rb.db

2. The rb.db file itself is not packaged within the EEB. It is automatically spawned when the NetBackup Resource Broker service first starts after EEB installation.

3. The size of the new rb.db file will vary from system to system, but is not expected to grow beyond 1 GB in size.

4. This file is not backed up by a catalog backup. The file is not needed for a catalog recovery.

5. All NetBackup Master Server configurations are supported.

6. If the EEB is to be installed onto a clustered Master Server, be sure to install it onto all nodes.