How to troubleshoot core files for AIX NetBackup support cases

# cd/<location_of_core_file>

#file core

core: AIX core file fulldumplargedata 32-bit, bpdbjobs

 

Another way to determine which binary created the core is to use the lquerypv command -

The program that caused the coredump is listed on the 6E0 line. For example:  

#lquerypv -h core 6b0 64

 

000006B0   7FFFFFFFFFFFFFFF 7FFFFFFF FFFFFFFF  |................|

000006C0   00000000000007D0 7FFFFFFF FFFFFFFF  |................|

000006D0   0012000017C2F688 00000000 0000000F  |................|

000006E0   627064626A6F6273 00000000 00000000  | bpdbjobs........|

000006F0   0000000000000000 00000000 00000000  |................|

00000700   0000000000000000 00000000 000027F0  |..............'.|

00000710   000000000000007C 00000000 000027F0  |.......|......'.|

2. Start the debugger.
dbx and adb commands can be used to get information from a core file, these commands are both part ofbos.adt.debug fileset.

To verify that bos.adt.debug file is installed run the following command -

# lslpp -l | grepbos.adt.debug
 bos.adt.debug              6.1.1.0  COMMITTED  BaseApplication Development

The GNU debugger, gdb, can also be used to analyze the core files. RPM for gdb needs to be installed separately.
To verify that gdb RPM is installed run the following command-

# rpm -q gdb

gdb-2.9.aix51.020209-4

 

# dbx<path_to_binary> <path_to_core> 2>&1 | tee /tmp/dbx.txt

(dbx) thread will displaya list of threads

For each thread run:

(dbx) thread current<thread_number>

(dbx) where

To exit run:

(dbx) quit

 

 

 

# adb<path_to_binary> <path_to_core> 2>&1 | tee /tmp/adb.txt

$c will display the stacktrace

$q will quit

 

 

# /usr/bin/gdb<path_to_binary> <path_to_core> 2>&1 | tee /tmp/gdb.txt

At the (gdb) prompt gather some informationregarding the core file.

(gdb)bt

(gdb)info threads

(gdb)thread apply all where

(gdb)info stack

(gdb)info registers

(gdb)info target

(gdb)quit

3. Include the /tmp/dbx.txt, /tmp/adb.txt  or /tmp/gdb.txt files with any open support case where the analysis of the core file is necessary.

 

To enableunique core file names in AIX 5.x run:

 

If multiple core files are generated the core file will be overwritten by the last entry then enable uniquecore file naming.
# chcore -n on
Core files will now be generated in the format core.<pid>.<ddhhmmss>.  The process id <pid>and the day, hours, minutes, seconds <ddhhmmss> will be included in the file name.  This will prevent one core from overwriting another.

To verify the change took place run:
# lscore
compression: off
path specification: off
corefile location: not set
naming specification:on

 

To enable unique core file names in AIX4.3.3:
If multiple core files are generated the core file will be overwritten by the last entry then enable unique core file naming.  Unique core file naming requires the environment variable CORE_NAMING be exported with a value of true.

Core file size limit:
1. All versions of AIX have a limit for the size of a core file.
#ulimit -a |grepcore
coredump(blocks)    2097151

This can be changed, or set to unlimited by running:
# ulimit -cunlimited


2. Enable the full coredump
# chdev -l sys0 -afullcore=true
sys0 changed


To check the setting run-
# lsattr -D -c sys -a fullcore-H
attribute deflt description          user_settable
fullcore  false Enable full CORE dump True


For more information on core files in the various versions of AIX see the IBM Website:
  https://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.files/doc/aixfiles/core.htm