Performance tuning and Throttling configuration in M365 OneDrive plugin of Backup Exec

Description
 

BackupExec M365 OneDrive plugin is designed and developed taking into consideration wide variety of user data which the plugin must protect. 

A OneDrive user can have:

•    Very small or very large files
•    Different types of files, e.g., simple text files, videos, and images. 
•    Large number of files and folders. 

The plugin ensures that all this data is backed up and restored with minimum throttling and with acceptable performance.
 
All the best practices mentioned by Microsoft are followed by the plugin. The details can be found here: https://docs.microsoft.com/en-us/graph/throttling  

The plugin has built-in scale up and scale down functionality. Important use cases where this tuning will help is in scenarios where there are many user OneDrive accounts which need to be backed up and total data size is also huge (hundreds of GBs to TBs). Although the default tuning is likely to work in most scenarios, some specific tuning may be required on a case-by-case basis.

Tuning Scenarios
 

Slow backup job speed
Throttling errors during backup
Errors related to properties restore during restore

Default path of the file where these tuning parameters are located is - 
C:\Program Files\Veritas\Backup Exec\SCFPlugins\O365\o365_settings.conf
For OneDrive refer to the section named:  [onedrive]

Listed below are the scenarios where tuning of the default parameters may be required. 
 

Note: This tuning must be done under the guidance of Veritas Support only otherwise it may result in undesired performance/issues. 

Slow backup job speed
 

If a less than desired backup job rate is observed, then a couple of parameters in the o365_settings.conf file can be changed for trial. If the new values work, they can be retained for subsequent job runs. 
Refer to the section named:  [onedrive] within the configuration file.

 # Maximum number of requests in a download batch  
#backup_max_requests_per_batch = 6 
The value of backup_max_requests_per_batch can be set to a little higher value. This will ensure that more requests are sent to OneDrive during backup.  

# Wait time between download batches. 
#backup_waitime_between_batches = 5 
The value of backup_waitime_between_batches can be reduced so that there is less wait time between different batch requests. 

Note: Changing these values could potentially increase throttling from Microsoft. Also, while setting these parameters, we should take into consideration the parallel streams configured in the backup job.
 
If a single OneDrive account is getting backed up, then you can set aggressive values of these parameters. 
However, if there are huge OneDrive accounts getting backed up in parallel (e.g., parallel streams set to 10 or more), then we may want to set these values to a more conservative level. 
 

Throttling errors during backup

If there is a lot of throttling seen during consecutive backup job runs, then the following parameters can be tuned in o365_settings.conf located at C:\Program Files\Veritas\Backup Exec\SCFPlugins\O365 folder.
Refer to the section named:  [onedrive] within the configuration file.

Note: Throttling may also depend on other factors like -

1. The time of the day when the backup is run if it co-incides with other activities that may affect the backup. 
2. How many jobs are running in parallel at that time.

It is advisable to check the throttling behavior for a few runs before changing these parameters.

# Wait time between download batches. 
#backup_waitime_between_batches = 5 
The value of backup_waitime_between_batches can be increased so that there is more time between requests and throttling can be reduced. 

# Default number of attempts to retry a failed request. 
#backup_max_retry_attempts = 5 
If there are many requests which fail after retries and this is seen in multiple job runs then the above parameter value can be increased to try more number of times. 

# Maximum number of requests in a download batch  
#backup_max_requests_per_batch = 6 
The value of backup_max_requests_per_batch can be set to a lower value so that maximum requests per batch are reduced and we see less throttling. 

Errors related to properties restore during restore
 

When an item is restored by OneDrive plugin, the plugin also restores properties and permissions of that item. This means multiple API calls for a single item and this may result in throttling especially if there are many items or OneDrives getting restored together. 

If throttling errors or errors during properties restore are seen, then one way to mitigate the problem is try to restore the OneDrives one by one in separate jobs instead of a single job.  

Other options to reduce throttling in restore are to disable properties and permissions restore. 
(Following parameters can be tuned in o365_settings.conf located at C:\Program Files\Veritas\Backup Exec\SCFPlugins\O365 folder. Refer to the section named:  [onedrive] within the configuration file.)

# Enable/disable properties restore. By default it is enabled. 
#restore_in_place_and_redirected_item_properties = 1  
The above property can be used to disable properties restore. This will reduce the number of API calls in item restore and may speed up restore. Please note that disabling this will result in just item data restore and no properties (e.g. modified by, modified time etc.) will be restored. 
 
# Enable/disable permissions restore. By default it is enabled. 
#restore_disable_inplace_item_permissions = 0 
The above property can be used to disable permissions restore. This will help reduce the number of API calls. Disabling this will not restore any permissions of the item and user will have to assign the permissions manually. Disable this only in case there are many throttling errors seen and there is no other way out.