NetBackup for OpenStack Datamover API (NBOSDMAPI) service timeout in the haproxy connection.

Problem

NBOSDMAPI service in the haproxy connection may timeout due to slow response time in the highly used environments.

Error Message

HTTP 504 timeout errors, that may result in the following messages:

Found disk corruption during snapshot, resetting the policy.

Cause

The default haproxy configuration works fine with most of the environments. When timeout issue with the NBOSDMAPI is observed, update the haproxy configuration.

Solution

For RHOSP :-

On each controller note, update the haproxy.cfg file.

This file is located at /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

Add the following configuration under listen nbosdmapi section.

Restart haproxy container after updating the haproxy.cfg file.

  retries 5
  timeout http-request 10m 
  timeout queue 10m 
  timeout connect 10m 
  timeout client 10m 
  timeout server 10m 
  timeout check 10m 
  balance roundrobin 
  maxconn 50000

For example,

listen nbosdmapi
  bind <ip>:8784 transparent
  bind <ip>:8784 transparent
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  option httpchk
  option httplog
  retries 5 
  timeout http-request 10m 
  timeout queue 10m 
  timeout connect 10m 
  timeout client 10m 
  timeout server 10m 
  timeout check 10m 
  balance roundrobin 
  maxconn 50000
  server overcloud-controller-0.internalapi.localdomain <ip>:8784 check fall 5 inter 2000 rise 2

You can also fix this issue before deployment of NBOS components. Update the nbosdmapi.yaml file.

The file is located at /home/stack/nbos-cfg-scripts/redhat-director-scripts/rhosp16.1/services/nbosdmapi.yaml

Add the following configuration in the haproxy section:

tripleo::haproxy::nbosdmapi::options:
            'retries': '5'
             'maxconn': '50000'
             'balance': 'roundrobin'
             'timeout http-request': '10m'
             'timeout queue': '10m'
             'timeout connect': '10m'
             'timeout client': '10m'
             'timeout server': '10m'
             'timeout check': '10m'

For information about NBOS deployment, see Veritas NetBackup™ for OpenStack Administrator's Guide

For OpenStack Ansible :- Update the user_variables.yml file.

The file is located at /etc/openstack_deploy/user_variables.yml

Add the following configuration at the end of the file:

haproxy_balance_alg: roundrobin
haproxy_timeout_client: 10m
haproxy_timeout_server: 10m
 

For example,

haproxy_extra_services:
  - service:
      haproxy_service_name: nbosdm_service
      haproxy_backend_nodes: "{{ groups['nbosdmapi_all'] | default([]) }}"
      haproxy_ssl: "{{ haproxy_ssl }}"
      haproxy_port: 8784
      haproxy_balance_type: http
      haproxy_balance_alg: roundrobin
      haproxy_timeout_client: 10m
      haproxy_timeout_server: 10m
      haproxy_backend_options:
   - "httpchk GET / HTTP/1.0\\r\\nUser-agent:\\ osa-haproxy-healthcheck"

Reconfigure haproxy for NBOSDMAPI.

cd /opt/openstack-ansible/playbooks

openstack-ansible haproxy-install.yml

For Kolla :-

On each controller note, update the haproxy.cfg file.

This file is located at /etc/kolla/haproxy/haproxy.cfg

Add the following configuration under defaults section.

  retries 5

  timeout http-request 10m

  timeout queue 10m

  timeout connect 10m

  timeout client 10m

  timeout server 10m

  timeout check 10m

  balance roundrobin

  maxconn 50000

Restart haproxy container after updating the haproxy.cfg file.

For example:

global

    chroot /var/lib/haproxy

    user haproxy

    group haproxy

    daemon

    log 10.210.129.102:5140 local1

    maxconn 40000

    nbproc 1

    stats socket /var/lib/kolla/haproxy/haproxy.sock group kolla mode 660

defaults

    log global

    option redispatch

    retries 3

    timeout http-request 10s

    timeout queue 1m

    timeout connect 10s

    timeout client 1m

    timeout server 10m

    timeout check 10s

    balance roundrobin

    maxconn 10000

listen stats

   bind 10.210.129.102:1984

   mode http

   stats enable

   stats uri /

   stats refresh 15s

   stats realm Haproxy\ Stats

   stats auth openstack:7BJvIThiCcJQGesRjBEXtD5p1ifq8QB7l9bdb302

frontend status

    bind 10.210.129.102:61313

    bind 10.210.129.100:61313

    mode http

    monitor-uri /