Status 89: Shared memory error. Linux RedHat Media Server is not processing backup jobs.

Problem

Status 89: Shared memory error.  Linux RedHat Media Server is not processing backup jobs.

Error Message

Log Snippet from BPTM on Media Server

<16> mpx_setup_shm: could not allocate enough shared memory for backup buffers, No space left on device 07:59:49.665 [9290] <2> send_brm_msg: ERROR 89

or BPBRM on Media Server

<16> bpbrm get_shared_mpx_mem: Could not get shared memory for bpbrm child process communication, No space left on device (28)

Cause

Linux OS is issuing an error (ENOMEM)

• No memory could be allocated for segment overhead.
• Shared memory limit (SHMMAX)  OR  shared memory segments (SHMMNI) kernel setting has been reached.

In the case of    " No space left on device (28)" , implies space or inode shortage
Verify with: 

•  #  df -h
•  #  df -i

To view active shared memory and number shared memory segments (SHMMNI)

• # ipcs -m
• # ipcs -m | wc -l

To view server SHMMAX limit -

•  #  cat /proc/sys/kernel/shmmax
•  #  grep shmmax /etc/sysctl.conf  

To view server SHMMNI limit -

•  #  grep shmmni /etc/sysctl.conf  

Solution

Before adjusting Linux Server Shared Memory settings, check if any old processes are hung and holding memory resources. Also check if server has not been rebooted recently.

• ps -ef --sort=start_time | tac        ( 'tac' reverses output, oldest listed last)
• /usr/bin/uptime

Reboot server to release system resources.

Tuning Shared Memory settings - Check server memory and usage

• # cat /proc/meminfo | grep MemTotal
• # free -m  (megabytes)
• # free -b   (bytes)

Increase shared memory setting

NOTE: (Set  <number_value> in bytes to 50% or more of the server physical memory) 

• echo <number_value>  >  /proc/sys/kernel/shmmax     (change value without reboot)

Edit /etc/sysctl.conf  file     (change persists after reboot)

• echo "kernel.shmmax = <number_value>"  >>  /etc/sysctl.conf      

Increase shared memory segments (ID's)

• echo 8092  >  /proc/sys/kernel/shmmni         (default is 4096)

        Edit /etc/sysctl.conf  file      (change persists after reboot)

• echo "kernel.shmmni = 8092"  >>  /etc/sysctl.conf      

Then run command to pick up the changes from /etc/sysctl.conf

•  sysctl -p

If status 89 persists, more server physical RAM memory may be needed and the SHMMAX value increased further.

Workaround:
Use one or more of the following...

1. Decrease value in NetBackup buffer files
          echo 8 > /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS
          echo 8 > /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS_DISK
2. Drop the number of concurrent jobs running on the media server.
   • Policy
     • Reduce 'Max jobs per policy' value
   • Schedule
     • Decrease 'Media Multiplexing' value
   • Tape storage Units
     • Reduce multiplexing per drive setting
   • Disk storage Units
     • Decrease 'maximum concurrent jobs'