NetBackup™ for Kubernetes Administrator's Guide

Configure application consistent backup

Some pods that are running applications, such as databases, which require additional procedures to obtain application consistent backups.

Application consistent backups require a mechanism to understand the application metadata, its state in memory, and the persistent data that resides on the persistent storage. To achieve a healthy state during restore, an application consistent backup across all these Kubernetes resources helps streamline the recovery process. These procedures are not required if only a crash consistent backup is required.

The application has vendor-documented steps to pause Input and Output (I/O) operations to perform an application consistent snapshot. This varies from one application to another, so the custom nature of these procedures is important. The content of these procedures is the customer's responsibility.

For protecting Kubernetes workloads with NetBackup, the method to achieve application consistent snapshots is to apply application pod annotations that leverage backup hooks. Kubernetes annotations are simply metadata which can be applied to any Kubernetes resources. Hooks within Kubernetes are user-defined actions and can be any command or multiple commands. Within your Kubernetes infrastructure, apply these annotations and hooks to any application pod that requires a quiesce state.

Backup hooks are used for both pre (before the snapshot) and post (after the snapshot) processing. In the context of data protection, this usually means that a netbackup-pre-backup hook calls a quiesce procedure or command, and the netbackup-post-backup hook calls an un-quiesce procedure or command. Each set of hooks specifies the command, as well as the container where it is applied. Note that the commands are not executed within a shell on the containers. Thus, a full command string with the directory is used in the given examples.

Identify the applications that require application consistent backups and apply the annotation with a set of backup hooks as part of the configuration for Kubernetes data protection.

Add an annotation to a pod, use the Kubernetes User Interface (UI). Alternatively, use the kubectl annotate function on the Kubernetes cluster console for a specific pod or label. The methods to apply annotations may vary depending on the distribution, therefore the following examples focuses on the kubectl command, based on its wide availability in most distributions.

Additionally, annotations can be added to the base Kubernetes objects, such as the deployment or replica set resources to ensure the annotations are included in any newly deployed pods. The Kubernetes administrator can update annotations dynamically.

Labels are key-value pairs which are attached to the Kubernetes objects such as Pods, or Services. Labels are used as attributes for objects that are meaningful and relevant to the user. Labels can be attached to objects at creation time and subsequently added and modified at any time. Kubernetes offers integrated support for using these labels to query objects and perform bulk operations on selected subsets. Each object can have a set of key-value labels defined. Each Key must be unique for a given object.

As an example of formatting and syntax of the label metadata:

"metadata": {"labels": {"key1":"value1","key2":"value2"}}

Either specify the pod name specifically, or a label that applies to the desired group of pods. If multiple annotation arguments are used, then specify the correct JSON format, such as a JSON array: '["item1","item2","itemn"]'# kubectl annotate pod [ {pod_name} | -l {label=value}] -n {the-pods-namespace_name} [annotation syntax - see following]

This method can be combined with && to join multiple commands if some applications require multiple commands to achieve the desired result. The commands specified are not provided by Veritas, and the user must manually customize the application pod. Replace {values} with the actual names used in your environment.

After upgrading to NetBackup 10.2, update the annotations to these new netbackup-pre and netbackup-post backup hooks that now include the "netbackup" prefix:

netbackup-pre.hook.back.velero.io/command
netbackup-pre.hook.backup.velero.io/container
netbackup-post.hook.back.velero.io/command
netbackup-post.hook.backup.velero.io/container

Following are the commands to lock and unlock a MongoDB 4.2.23 database:

# mongo --eval "db. fsyncLock ()"

# mongo --eval "db.fsyncUnlock()"

This translates into the following single command to set both the pre and post backup hooks for MongoDB. Note the special syntax to escape special characters as well the brackets ([]), single and double quotes and commas (,) used as part of the JSON format:

# kubectl annotate pod {mongodb-pod-name} -n {mongodb namespace} netbackup-pre.hook.back.velero.io/command='["/bin/bash", "-c", "mongo --eval \"db.fsyncLock()\""]' netbackup-pre.hook.backup.velero.io/container={mongodb-pod-name} netbackup-post.hook.backup.velero.io/command='["/bin/bash","-c","mongo --eval \"db.fsyncUnlock()\""]' netbackup-post.hook.backup.velero.io/container={mongodb-pod-name}

Following are the commands to quiesce and un-quiesce the MySQL database:

# mysql -uroot -ppassword -e "flush tables with read lock"

# mysql -uroot -ppassword -e "unlock tables"

This translates into the following single command to set both the pre and post backup hooks for MySQL. In this example, we used a label instead of a pod name, so the label can annotate multiple pods at once. Note the special syntax to escape special characters as well the brackets ([]), single and double quotes and commas (,) used as part of the JSON format:

# kubectl annotate pod -l label=value -n {mysql namespace} netbackup-pre.hook.backup.velero.io/command='["/bin/bash", "-c", "mysql -uroot -ppassword -e \"flush tables with read lock\""]' netbackup-pre.hook.backup.velero.io/container={mysql container name} netbackup-post.hook.backup.velero.io/command='["/bin/bash", "-c", "mysql -uroot -ppassword -e \"unlock tables\""]' netbackup-post.hook.backup.velero.io/container={mysql container name}

Following are the commands to quiesce and un-quiesce the PostgreSQL database:

# Psql -U postgres -c "SELECT pg_start_backup('tagvalue');"

# psql -U postgres -c \"SELECT pg_stop_backup();"

This translates into the following single command to set both the pre and post backup hooks for Postgres. In this example, we used a label instead of a pod name, so the label can annotate multiple matching pods at once. Labels can be applied to any Kubernetes object, and in this case, we are using them to provide another way to modify a specific container and select only certain pods. Note the special syntax to escape special characters as well the brackets ([]), single and double quotes and commas (,) used as part of the JSON format:

# kubectl annotate pod -l app=app-postgresql -n {postgres namespace} netbackup-pre.hook.backup.velero.io/command='["/bin/bash", "-c", "psql -U postgres -c \"SELECT pg_start_backup(quote_literal($EPOCHSECONDS));\""]' netbackup-pre.hook.backup.velero.io/container={postgres container name} netbackup-post.hook.backup.velero.io/command='["/bin/bash", "-c", "psql -U postgres -c \"SELECT pg_stop_backup();\""]' netbackup-post.hook.backup.velero.io/container={postgres container name}

Following are the commands to quiesce and un-quiesce the Nginx application:

# /sbin/fsfreeze --freeze /var/log/nginx

# /sbin/fsfreeze --unfreeze /var/log/nginx

This translates into the following single command to set both the pre and post backup hooks for NGINX. In this example, we will omit the container hooks, and this will modify the first container that matches the pod name by default. Note the special syntax to escape special characters as well the brackets ([]), single and double quotes and commas (,) used as part of the JSON format:

# kubectl annotate pod {nginx-pod-name} -n {nginx namespace} netbackup-pre.hook.backup.velero.io/command='["/sbin/fsfreeze", "--freeze", "/var/log/nginx"]' netbackup-post.hook.backup.velero.io/command='["/sbin/fsfreeze", "--unfreeze", "/var/log/nginx"]'

Following are the commands to quiesce and un-quiesce the Cassandra database:

# nodetool flush

# nodetool verify

This translates into the following single command to set both the pre and post backup hooks for Cassandra. Note the special syntax to escape special characters as well the brackets ([]), single (''), and double quotes ("") and commas (,) used as part of the JSON format:

# kubectl annotate pod {cassandra-pod} -n {Cassandra namespace} netbackup-pre.hook.backup.velero.io/command='["/bin/bash", "-c", "nodetool flush"]' netbackup-pre.hook.backup.velero.io/container={cassandra-pod} netbackup-post.hook.backup.velero.io/command='["/bin/bash", "-c", "nodetool verify"]' netbackup-post.hook.backup.velero.io/container={cassandra-pod}