Problem
An Oracle RMAN PROXY backup was taken using a snapshot that was mounted and read by an off-host alternate client.
The image is cataloged in NetBackup and can be listed, but a restore from the image reports that not all of the backupset pieces can be found and fails with status 5.
Error Message
The dbclient debug log from the restore shows the query to determine which image contains the needed backupset piece that Oracle wants. The querying is using the correct date range and client name. Notice that the Oracle server process performing the restore is running as user=oracle group=oinstall.
16:56:07.456 [27668] <2> sbtpcrestore: INF - entering
...snip...
16:56:07.457 [27668] <2> int_GetBfsDateRange: INF - RMAN file name = bk_myid_s6717_p38_t836188287
...snip...
16:56:07.470 [27668] <2> logconnections: BPRD CONNECT FROM 3.3.3.3.51268 TO 1.1.1.1.1556 fd = 12
16:56:07.470 [27668] <2> bsa_bplist: start_date = Mon Jan 6 02:31:27 2014
16:56:07.470 [27668] <2> bsa_bplist: end_date = Wed Jan 8 02:31:27 2014
16:56:07.470 [27668] <2> bsa_bplist: Request = oracle oinstall myclient myclient myclient NONE 0 3 999 1388997087 1389169887 4 4 1 0 1 0 4 1015 503 3 0 C C C C C 0 2 0 0 0
16:56:07.470 [27668] <4> bsa_bplist: Filepath = /
The image information, returned from the master server, for the backup should contain pairs of rows, first one for the physical file name followed by one for the logical backupset piece name. Notice that file system01.dbf is correctly paired with piece /bk_myid_s6717_p38_t836188287. None of these files or pieces are for user=oracle or group=oinstall, but all of them are readable other/world which is why they were returned. However, the three preceding piece names lack file names. The physical file name for some of the later piece names is also missing. The master server reports status 0 indicating that it returned all file information matching the search criteria.
16:56:07.711 [27668] <2> dbc_get_string: Output = ... 1 -1 9 -r--r--r-- root root 0 ... /PROXY_FI_BACKUP
16:56:07.711 [27668] <2> dbc_get_string: Output = ... 53002297 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p1_t836188287
16:56:07.711 [27668] <2> dbc_get_string: Output = ... 59289711 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p22_t836188287
16:56:07.711 [27668] <2> dbc_get_string: Output = ... 64307365 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p30_t836188287
16:56:07.711 [27668] <2> dbc_get_string: Output = ... 64307369 14004 1 -rw-r--r-- 1015 504 1912610816 ... /db/sys001/oradata/SID/system01.dbf
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 68042971 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p38_t836188287
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 68042975 14001 1 -rw-r--r-- 1015 504 419438592 ... /db/idx001/oradata/SID/tax_ndx1_01.dbf
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 68862225 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p47_t836188287
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 68862227 14001 1 -rw-r--r-- 1015 504 246677504 ... /db/idx001/oradata/SID/efl_ctrl_ndx01.dbf
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 69344053 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p52_t836188287
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 69344057 14000 1 -rw-r--r-- 1015 504 136323072 ... /db/dat001/oradata/SID/aud2006_tbl01.dbf
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 69610347 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p59_t836188287
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 69610349 14000 1 -rw-r--r-- 1015 504 104865792 ... /db/dat001/oradata/SID/audh_tbl01.dbf
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 69815199 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p68_t836188287
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 69917651 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p75_t836188287
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 69979143 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p79_t836188287
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 69989435 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p85_t836188287
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 1 -1 9 -r--r--r-- root root 0 ... /PROXY_FI_BACKUP
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 7 14000 1 -rw-r--r-- 1015 504 18944008K ... /db/dat001/oradata/SID/trn2007_tbl01.dbf
16:56:07.712 [27668] <2> dbc_get_string: Output = ... 37888057 -1 9 -r--r--r-- root root 0 ... /bk_myid_s6717_p6_t836188287
...snip...
16:56:07.781 [27668] <2> dbc_get_string: Output = EXIT STATUS 0
The missing file information for the 3 preceding pieces causes the rows for file system01.dbf and piece bk_myid_s6717_p38_t836188287 to not be read as a pair and they are discarded. This prevents the returned rows from being a valid match for the piece requested by RMAN, and the restore fails.
16:56:10.092 [27668] <4> bsa_QueryFile: INF - None of the objects returned matched the query file: /bk_myid_s6717_p38_t836188287
16:56:10.092 [27668] <2> xbsa_QueryObject: INF - leaving (17)
16:56:10.092 [27668] <2> int_FindBackupImage: INF - bk_myid_s6717_p38_t836188287 not found
16:56:10.092 [27668] <2> int_FindBackupImage: INF - leaving
16:56:10.093 [27668] <16> int_AddToFileList: ERR - Backup file <bk_myid_s6717_p38_t836188287> not found in NetBackup catalog
16:56:10.093 [27668] <2> sbtpcrestore: INF - leaving
...snip...
16:56:10.093 [27668] <2> sbterror: INF - Error=7502: Backup file <bk_myid_s6717_p38_t836188287> not found in NetBackup catalog
16:56:10.446 [27668] <2> sbtpcend: INF - entering
...snip...
16:56:10.446 [27668] <2> sbtpcend: INF - --- end of proxy copy restore ---
The bplist output using the same search criteria, excluding owning user and group, confirms that physical files are cataloged for all 10 of the pieces. For some reason, the top 3 and bottom 3 are not being returned to dbclient. Notice the mix of owning group and modes, they are not consistent. Also, the UID and GIDs are stored where a user or group name would normally be stored.
$ bplist -C myclient -t 4 -k mypolicy -X -s 1389074400 -e 1389099600 -l -R /
...snip...
-rw-r----- 1015 503 26501128K Dec 01 2012 /db/dat001/oradata/SID/trn2011_tbl_01.dbf
-rw-r----- 1015 503 3143688K Dec 01 2012 /db/idx002/oradata/SID/trn2011_ndx2_01.dbf
-rw-r----- 1015 503 2508808K Jan 28 2011 /db/idx003/oradata/SID/trn2009_ndx3_01.dbf
-rw-r--r-- 1015 504 1912610816 Jan 07 02:30 /db/sys001/oradata/SID/system01.dbf
-rw-r--r-- 1015 504 419438592 Jan 07 02:30 /db/idx001/oradata/SID/tax_ndx1_01.dbf
-rw-r--r-- 1015 504 246677504 Jan 07 02:30 /db/idx001/oradata/SID/efl_ctrl_ndx01.dbf
-rw-r--r-- 1015 504 136323072 Jan 28 2011 /db/dat001/oradata/SID/aud2006_tbl01.dbf
-rw-r--r-- 1015 504 104865792 Jan 07 02:30 /db/dat001/oradata/SID/audh_tbl01.dbf
-rw-r----- 1015 503 52436992 Jan 28 2011 /db/dat001/oradata/SID/aud2008_tbl_01.dbf
-rw-r----- 1015 503 31465472 Jan 28 2011 /db/dat001/oradata/SID/aud2008_ndx_01.dbf
-rw-r----- 1015 503 5251072 Jan 02 09:47 /db/dat001/oradata/SID/aud2014_tbl01.dbf
The cat_convert output for the backup image shows that there are physical file names for each backupset piece correctly cataloged. Notice the ownership and modes match the bplist output above.
$ cat_convert -dump mypolicy_1389083745_FULL.f
num ... data
...snip...
1 ... /db/dat001/oradata/SID/trn2011_tbl_01.dbf 33184 1015 503 1367351296 1389083487 1354349341 1354349341
0 ... /bk_myid_s6717_p1_t836188287 292 root root 0 1389086993 1389086993 1389086993
...snip...
2 ... /db/idx002/oradata/SID/trn2011_ndx2_01.dbf 33184 1015 503 2145394688 1389083489 1354349341 1354349341
0 ... /bk_myid_s6717_p22_t836188287 292 root root 0 1389087435 1389087435 1389087435
...snip...
3 ... /db/idx003/oradata/SID/trn2009_ndx3_01.dbf 33184 1015 503 1495277568 1389083489 1296233339 1296233339
0 ... /bk_myid_s6717_p30_t836188287 292 root root 0 1389087741 1389087741 1389087741
...snip...
4 ... /db/sys001/oradata/SID/system01.dbf 33188 1015 504 1912610816 1389083490 1389083425 1389083425
0 ... /bk_myid_s6717_p38_t836188287 292 root root 0 1389087917 1389087917 1389087917
...snip...
5 ... /db/idx001/oradata/SID/tax_ndx1_01.dbf 33188 1015 504 419438592 1389083490 1389083425 1389083425
0 ... /bk_myid_s6717_p47_t836188287 292 root root 0 1389087928 1389087928 1389087928
...snip...
6 ... /db/idx001/oradata/SID/efl_ctrl_ndx01.dbf 33188 1015 504 246677504 1389083490 1389083425 1389083425
0 ... /bk_myid_s6717_p52_t836188287 292 root root 0 1389087936 1389087936 1389087936
...snip...
7 ... /db/dat001/oradata/SID/aud2006_tbl01.dbf 33188 1015 504 136323072 1389083490 1296233326 1296233326
0 ... /bk_myid_s6717_p59_t836188287 292 root root 0 1389087940 1389087940 1389087940
...snip...
8 ... /db/dat001/oradata/SID/audh_tbl01.dbf 33188 1015 504 104865792 1389083490 1389083425 1389083425
0 ... /bk_myid_s6717_p68_t836188287 292 root root 0 1389087946 1389087946 1389087946
...snip...
9 ... /db/dat001/oradata/SID/aud2008_tbl_01.dbf 33184 1015 503 52436992 1389083490 1296233338 1296233338
0 ... /bk_myid_s6717_p75_t836188287 292 root root 0 1389087948 1389087948 1389087948
...snip...
10 ... /db/dat001/oradata/SID/aud2008_ndx_01.dbf 33184 1015 503 31465472 1389083491 1296233338 1296233338
0 ... /bk_myid_s6717_p79_t836188287 292 root root 0 1389087949 1389087949 1389087949
...snip...
11 ... /db/dat001/oradata/SID/aud2014_tbl01.dbf 33184 1015 503 5251072 1389083491 1388677676 1388677676
0 ... /bk_myid_s6717_p85_t836188287 292 root root 0 1389087950 1389087950 1389087950
The file system information (ls –l) from the source client, which is also the destination client, matches the image catalog, but with the UID and GIDs mapped correctly to names.
-rw-r--r-- 1 oracle dba 136323072 Jan 12 13:53 /db/dat001/oradata/SID/aud2006_tbl01.dbf
-rw-r----- 1 oracle oinstall 31465472 Jan 12 13:53 /db/dat001/oradata/SID/aud2008_ndx_01.dbf
-rw-r----- 1 oracle oinstall 52436992 Jan 12 13:53 /db/dat001/oradata/SID/aud2008_tbl_01.dbf
-rw-r----- 1 oracle oinstall 5251072 Jan 12 13:53 /db/dat001/oradata/SID/aud2014_tbl01.dbf
-rw-r--r-- 1 oracle dba 104865792 Jan 12 13:53 /db/dat001/oradata/SID/audh_tbl01.dbf
-rw-r--r-- 1 oracle dba 246677504 Jan 12 13:53 /db/idx001/oradata/SID/efl_ctrl_ndx01.dbf
-rw-r--r-- 1 oracle dba 419438592 Jan 12 13:53 /db/idx001/oradata/SID/tax_ndx1_01.dbf
-rw-r----- 1 oracle oinstall 3219136512 Jan 12 13:53 /db/idx002/oradata/SID/trn2011_ndx2_01.dbf
-rw-r----- 1 oracle oinstall 2569019392 Jan 12 13:53 /db/idx003/oradata/SID/trn2009_ndx3_01.dbf
-rw-r--r-- 1 oracle dba 1912610816 Jan 12 13:53 /db/sys001/oradata/SID/system01.dbf
The bppllist output for the backup policy shows that the backup policy is using an alternate client which is reading the files from disk and providing the meta data for each file to the media and master servers. That host is not able to resolve UID=1015 to oracle, nor GIDs 504 and 503 to dba and oinstall respectively.
Policy Name: mypolicy
...snip...
Policy Type: Oracle (4)
Active: yes
Perform Snapshot Backup: yes
Perform Offhost Backup: yes
...snip...
Use Alternate Client: yes
Alternate Client Name: myAltClient
...snip...
Enable Instant Recovery: no
...snip...
Client/HW/OS/Pri: myclient Solaris Solaris9 0 0 0 0 ?
Cause
For Oracle RMAN PROXY backups, Oracle provides a list of the files to be backed up and then a backupset piece name to be associated with each file. The NetBackup dbclient and bpbkar processes coordinate to ensure that both the physical (file name) and the logical (piece name) are cataloged and available at restore time. The cataloged information also includes user, group, and modes for each physical file and the user, group, and modes associated with the Oracle server process performing the backup.
At restore time, the user and group of the Oracle server process performing the restore are compared to the cataloged values to ensure that the requesting process is authorized to restore the possibly sensitive data that was backed up.
Where possible, NetBackup resolves the UID and GID to user and group names and uses them for comparison, if resolution is not possible, the raw IDs are used.
There are two inconsistencies present on the hosts performing this backup and restore.
- The files to be backed up have a mix of group ownership and modes. Process running as group 503 can restore all the files, but group 504 can only restore some of the files.
- The files were cataloged at backup time using user 1015 and group 504 or 503 because the alternate client (myAltClient) cannot resolve the UID and GIDs associated with the files to oracle, dba, and oinstall respectively.
As a result the restore process running as user=oracle and group=oinstall only has access to the files that are other/world readable and therefore fails because it cannot access the files with modes of 0640 and owned by user=1015 and group=503.
Solution
Always ensure that the process that is performing the backup or restore, in this case the Oracle server process that is loading dbclient, has the correct permissions to access the files.
To prevent this situation from affecting future backups make at least one, and preferably both of the following changes.
1) Ensure all the files to be backed up have modes that allow read access to other/world, or at least have consistent modes and ownership so they are all processed in the same way.
2) Ensure that the backup client, alternate client, and restore client hosts can resolve the UIDs and GIDs associated with the files and the Oracle server process to the same user and group names. This topic is covered in the last bullet on page 150 of the NetBackup 7.1 for Oracle Administrator's Guide:
Configuration requirements for snapshot backups with NetBackup for Oracle
...snip...
- The user identification and group identification numbers (UIDs and GIDs) associated with the files to be backed up must be available to both the primary client and the alternate backup client. The UID on the primary client and the alternate backup client must be the same. Similarly, the GID on the primary client and the alternate backup client must be the same.
The inconsistent nature of the currently existing backup images is not correctible and any restore of these older images will need to run with the correct user and/or group to access the files as previously preserved by the backup.
Applies To
NetBackup 7.1, but could affect any version
Related Knowledge Base Articles
Was this content helpful?
Translated Content
Please note that this document is a translation from English, and may have been machine-translated. It is possible that updates have been made to the original version after this document was translated and published. Veritas does not guarantee the accuracy regarding the completeness of the translation. You may also refer to the English Version of this knowledge base article for up-to-date information.