Configure application consistent backups for Kubernetes

Description

Some pods will be running applications, such as databases, which require additional procedures to obtain application consistent backups.  

Application consistent backups require a mechanism to understand the application metadata, its state in memory, and the persistent data that resides on persistent storage. To achieve a healthy state during restore, an application consistent backup across all these Kubernetes resources will help streamline the recovery process.  These procedures are not required if only a crash consistent backup is required.

The application will have vendor-documented steps to pause Input and Output (I/O) operations in order to perform an application consistent snapshot.    This will vary from one application to another, so the custom nature of these procedures is important.  The content of these procedures is the customer’s responsibility.

For protecting Kubernetes workloads with NetBackup, the method to achieve application consistent snapshots is to apply application pod annotations that leverage backup hooks.  Kubernetes annotations are simply metadata which can be applied to any Kubernetes resources. Hooks within Kubernetes are user-defined actions and can be any command or multiple commands.  Within your Kubernetes infrastructure, apply these annotations and hooks to any application pod that requires a quiesced state.

Backup hooks are used for both pre (before the snapshot) and post (after the snapshot) processing.   In the context of data protection, this usually means that a pre-backup hook will be calling the quiesce procedure or command, and the post-backup hook will be calling the un-quiesce procedure or command.  Each set of hooks will specify the command, as well as the container where it will be applied.  It is important to note that the commands are not executed within a shell on the containers, so the full command string including the directory will be used in the examples below.

Command behaviour can be controlled with "on-error" hook which is optional. The default value of on-error hook would be Fail. Valid values are Fail and Continue. If the value is set to "Fail" and the user-specified command fails, the backup snapshot does not proceed.

Identify the applications that require application consistent backups and apply the annotation with a set of backup hooks as part of the configuration for Kubernetes data protection.  

To add an annotation to a pod, use the Kubernetes User Interface (UI). Alternatively, use the kubectl annotate function on the Kubernetes cluster console for a specific pod or label.  The methods to apply annotations may vary depending on the distribution, therefore the examples below will focus on the kubectl command, based on its wide availability in most distributions. Additionally, annotations can be added to the base Kubernetes objects, such as the deployment or replicaset resources to ensure the annotations are included in any newly deployed pods. Annotations can be updated by the Kubernetes administrator dynamically.

Labels are key-value pairs which are attached to Kubernetes Objects such as Pods, or Services. Labels are used as attributes for objects that are meaningful and relevant to the user. Labels can be attached to objects at creation time and subsequently added and modified at any time. Kubernetes offers integrated support for using these labels to query objects and perform bulk operations on selected subsets. Each object can have a set of key-value labels defined. Each Key must be unique for a given object. 

As an example of formatting and syntax of the label metadata:

"metadata": {"labels": {"key1":"value1","key2":"value2"}}

Either specify the pod name specifically, or a label that applies to the desired group of pods.  If multiple annotation arguments are used, specify the correct JSON formatting, such as a JSON array: '["item1","item2","itemn"]'

# kubectl annotate pod [ {pod_name} | -l {label=value}] -n {the-pods-namespace_name}   [annotation syntax - see below]

This method can also be combined with && to join multiple commands if some applications require multiple commands to achieve the desired result.  The commands specified are not provided by Veritas, and the user must manually customize the application pod.  Replace {values} with the actual names used in your environment.

Note: kubectl commands should all be on one line.  Take care when copying and pasting the examples below.
Prior to 10.1.1, below annotations should be used:

pre.hook.backup.velero.io/command

pre.hook.backup.velero.io/container

pre.hook.backup.velero.io/on-error

post.hook.backup.velero.io/command

post.hook.backup.velero.io/container

post.hook.backup.velero.io/on-error

After upgrading to NetBackup 10.1.1, update the annotations to these new pre and post backup hooks that now include the "netbackup” prefix:

netbackup-pre.hook.backup.velero.io/command

netbackup-pre.hook.backup.velero.io/container

netbackup-pre.hook.backup.velero.io/on-error

netbackup-post.hook.backup.velero.io/command

netbackup-post.hook.backup.velero.io/container

netbackup-post.hook.backup.velero.io/on-error

MongoDB Example Using Pod Name

For example, the commands to lock and unlock a MongoDB 4.2.23 database:  

# mongo --eval "db.fsyncLock()"
# mongo --eval "db.fsyncUnlock()"

This translates into the following single command to set both the pre and post backup hooks for MongoDB.  Note the special syntax to escape special characters as well the brackets ( [ ] ), single and double quotes and commas (,) used as part of the JSON format:

# kubectl annotate pod {mongodb-pod-name} -n {mongodb namespace}  netbackup-pre.hook.backup.velero.io/command='["/bin/bash",  "-c", "mongo --eval  \"db.fsyncLock()\""]'  netbackup-pre.hook.backup.velero.io/container={mongodb-pod-name} netbackup-pre.hook.backup.velero.io/on-error=Fail netbackup-post.hook.backup.velero.io/command='["/bin/bash","-c","mongo --eval  \"db.fsyncUnlock()\""]' netbackup-post.hook.backup.velero.io/container={mongodb-pod-name} netbackup-post.hook.backup.velero.io/on-error=Fail

MySQL Example Using Labels

The commands to quiesce and un-quiesce the MySQL database:

# mysql -uroot -ppassword -e "flush tables with read lock"
# mysql -uroot -ppassword -e "unlock tables"

This translates into the following single command to set both the pre and post backup hooks for MySQL.  In this example, we used a label instead of a pod name, so the label can annotate multiple pods at once.  Note the special syntax to escape special characters as well the brackets ( [ ] ), single and double quotes and commas (,) used as part of the JSON format:

# kubectl annotate pod -l label=value -n {mysql namespace}  netbackup-pre.hook.backup.velero.io/command='["/bin/bash", "-c", "mysql -uroot -ppassword -e \"flush tables with read lock\""]' netbackup-pre.hook.backup.velero.io/container={mysql container name}  netbackup-post.hook.backup.velero.io/command='["/bin/bash", "-c", "mysql -uroot -ppassword -e \"unlock tables\""]'  netbackup-post.hook.backup.velero.io/container={mysql container name}

Postgres Example Using Labels

 The commands to quiesce and un-quiesce the PostgreSQL database:

# psql -U postgres -c "SELECT pg_start_backup('tagvalue');"
# psql -U postgres -c \"SELECT pg_stop_backup();"

This translates into the following single command to set both the pre and post backup hooks for Postgres.  In this example, we used a label instead of a pod name, so the label can annotate multiple matching pods at once.  Labels can be applied to any Kubernetes object, and in this case, we are using them to provide another way to modify a specific container and select only certain pods.  Note the special syntax to escape special characters as well the brackets ( [ ] ), single and double quotes and commas (,) used as part of the JSON format:

# kubectl annotate pod -l app=app-postgresql -n {postgres namespace}  netbackup-pre.hook.backup.velero.io/command='["/bin/bash", "-c", "psql -U postgres -c \"SELECT pg_start_backup(quote_literal($EPOCHSECONDS));\""]'     netbackup-pre.hook.backup.velero.io/container={postgres container name}    netbackup-post.hook.backup.velero.io/command='["/bin/bash", "-c", "psql -U postgres -c \"SELECT pg_stop_backup();\""]' netbackup-post.hook.backup.velero.io/container={postgres container name}

NGINX Application Example Without Container Hook

The commands to quiesce and un-quiesce the Nginx application:

# /sbin/fsfreeze --freeze /var/log/nginx
# /sbin/fsfreeze --unfreeze /var/log/nginx

This translates into the following single command to set both the pre and post backup hooks for NGINX.  In this example, we will omit the container hooks, and this will modify the first container that matches the pod name by default. Note the special syntax to escape special characters as well the brackets ( [ ] ), single and double quotes and commas (,) used as part of the JSON format:

# kubectl annotate pod {nginx-pod-name} -n {nginx namespace} netbackup-pre.hook.backup.velero.io/command='["/sbin/fsfreeze", "--freeze", "/var/log/nginx"]'  netbackup-post.hook.backup.velero.io/command='["/sbin/fsfreeze", "--unfreeze", "/var/log/nginx"]' 

Cassandra Example

The commands to quiesce and un-quiesce the Cassandra database:

# nodetool flush
# nodetool verify

This translates into the following single command to set both the pre and post backup hooks for Cassandra.  Note the special syntax to escape special characters as well the brackets ( [ ] ), single and double quotes and commas (,) used as part of the JSON format: 

# kubectl annotate pod {cassandra-pod} -n {Cassandra namespace}  netbackup-pre.hook.backup.velero.io/command='["/bin/bash", "-c", "nodetool flush"]'  netbackup-pre.hook.backup.velero.io/container={cassandra-pod}  netbackup-post.hook.backup.velero.io/command='["/bin/bash", "-c", "nodetool verify"]' netbackup-post.hook.backup.velero.io/container={cassandra-pod} 

Notes:
The examples are provided as an initial guide only, and specific requirements for each workload must include collaboration between the backup administrator, the workload administrator and the Kubernetes administrator.

The default timeout value for the command to return an exit status is 30 seconds, but can be changed with these hooks as annotations to the pods:

netbackup-pre.hook.backup.velero.io/timeout=#in-seconds#
netbackup-post.hook.backup.velero.io/timeout=#in-seconds#

For example:

netbackup-pre.hook.backup.velero.io/timeout=90s
netbackup-post.hook.backup.velero.io/timeout=120s