Performance tuning and throttling configuration in M365 Exchange plugin of Backup Exec

Description
 

Performance of backup and restore of M365 Exchange Online depends on the network and the amount of throttling faced from M365 during those operations. M365 Exchange Online throttles requests to maintain optimum performance for various use cases. Backup Exec also faces the same throttling when sending requests to M365 Exchange Online. Backup Exec has followed Microsoft's recommended best practices to deal with the throttling.  

Considerations before running a backup job 

Due to the nature of the workload, various operations, especially the first full backup, can take time, it is important to plan and size your job before starting the operation. Factors such as number of mailboxes, type and size of mailboxes, backup job window should be considered before the running the job. 

Listed below are some recommendations that may help speed up the backup job, especially the first full job.

1.    Disable EWS throttling - 

M365 has a self-service mechanism by which EWS throttling can be temporarily relaxed. It is advisable that throttling be temporarily disabled for the initial full backup. After the first full is completed, this setting can be reverted. 
  

2.    Different type of mailboxes should be backed up in separate jobs 

It is recommended that different type of mailboxes - User Mailboxes, Archive Mailboxes and Group Mailboxes be backed up in different jobs with different non-overlapping schedules. 

Tuning Scenarios
 

Listed below are the scenarios where default parameters can be changed in the o365_settings.conf file. If the new values work, they can be retained for subsequent job runs. 

Default path for this file is: 
C:\Program Files\Veritas\Backup Exec\SCFPlugins\O365\o365_settings.conf

For Exchange refer to the section named: [exchange]

Note: The changes affects all backup and restore operations on the BE server.

Items failing due to throttling errors during backup
Items failing due to time out errors during backup
Items failing due to throttling/batch processing errors during restores
Slow backup when EWS throttling is disabled from Exchange Online
Slow restore when EWS throttling is disabled from Exchange Online

Items failing due to throttling errors during backup

Items throttled during backup are picked up in the next backup. Try to ensure that backups happen when there are no other users accessing the mailboxes. Reducing the number of requests and increasing the number of retries may help in reducing the errors.    

These parameters can be decreased -  Make sure to remove # after changing the value for the new value to take effect.
 
# Number of threads for enumerating folder items during backup. Microsoft recommended value. (minimum - 2, maximum - 5) 
#backup_sync_requests = 5

# Number of threads for backing up items. (minimum - 2, maximum - 15) 
#backup_download_requests = 5  

# Number of items to download per request during backup (minimum - 5, maximum - 100) 
#backup_items_per_download_request = 80 
 
The parameters can be increased - Make sure to remove # after changing the value for the new value to take effect.

# Number of retries when requests fail during backup (minimum - 0, maximum - 10) 
#backup_retries_for_requests = 5  

Items failing due to time out errors during backup
 

If Backup fail with the error - "V-79-57344-43290 - Error in SaaS Connector Framework - A network error has occurred.
A network error occurred during the operation. Check the network or firewall settings. 
End of file or no input: message transfer interrupted or timed out (300 sec max recv delay) (300 sec max send delay) (-1)". 
 

There can be multiple reasons for this behaviour: 

•    Intermittent network errors. 
•    Excessive throttling from Exchange Online. 

Please check for any networking issues which connecting with Exchange Online services. Changing the following values may help. 
 
These parameters can be decreased - Make sure to remove # after changing the value for the new value to take effect.

# Minimum number of items to download per request to fallback to on network timeout errors during backup (minimum - 1, maximum - 20) 
#backup_min_items_per_download_request = 5 

# Maximum total size of all items to download per request during backup (minimum - 1, maximum - 500) 
#backup_max_size_per_download_request_mb = 20 
 
These parameters can be increased -  Make sure to remove # after changing the value for the new value to take effect.

# Network send timeout for requests sent during backup (minimum - 0, maximum - 300) 
#backup_gsoap_send_timeout_secs = 300 

# Network receive timeout for requests sent during backup (minimum - 0, maximum - 300) 
#backup_gsoap_recv_timeout_secs = 300
 

Items failing due to throttling/batch processing errors during restores. 
 

For the failed items, users will have to selectively run the restores for those items to be restored. To avoid throttling failures in any new restores, try to ensure that restore happen when there are no other users accessing the mailboxes targeted for restore or there are no active backup for those mailboxes. 
 

Reducing the number of requests and increasing the number of retries may help in reducing the errors. 
 
These parameters can be decreased -  Make sure to remove # after changing the value for the new value to take effect.

# Maximum number of items to upload in a single request during restore (minimum - 1, maximum - 200) 
#restore_max_items_per_upload_batch = 100 

# Maximum total size of all items to upload per request during restore (minimum - 1, maximum - 50) 
#restore_max_payload_per_batch_mb = 5 

# Number of threads to upload items during restore (minimum - 1, maximum - 10) 
#restore_max_item_upload_tasks = 3  

# Maximum number of items to verify in a single request during restore (minimum - 1, maximum - 200) 
#restore_max_items_per_verification_batch = 50 

# Number of threads to verify items during restore (minimum - 1, maximum - 10) 
#restore_max_item_verification_tasks = 2  
 
This parameter can be increased -  Make sure to remove # after changing the value for the new value to take effect.

# Number of retries when requests fail during restore (minimum - 0, maximum - 10) 
#restore_max_retries = 5 

Slow backup when EWS throttling is disabled from Exchange Online  

When EWS throttling is disabled from Exchange Online, changing the following settings may increase backup speeds. 

These paramters can be increased -  Make sure to remove # after changing the value for the new value to take effect.

# Number of threads for enumerating folder items during backup. Microsoft recommended value. (minimum - 2, maximum - 5) 
# backup_sync_requests = 5 

# Number of threads for backing up items. (minimum - 2, maximum - 15) 
# backup_download_requests = 5  

# Number of items to download per request during backup (minimum - 5, maximum - 100)
# backup_items_per_download_request = 80 

# Maximum total size of all items to download per request during backup (minimum - 1, maximum - 500) 
# backup_max_size_per_download_request_mb = 20 

# Maximum total size of all items (downloaded and across all outstanding requests) during backup 
# backup_max_total_items_download_size_mb = 100 (minimum - 10, maximum - 2000) 
 
Note: Changing above settings will have no effect when EWS throttling is enabled on Exchange Online and may even make job performance worse. Hence, when EWS throttling is enabled on Exchange Online, above settings should be reverted to original values. 

Slow restore when EWS throttling is disabled from Exchange Online
 

When EWS throttling is disabled from Exchange Online, changing the following settings may increase restore speeds. 

These parameters can be increased -  Make sure to remove # after changing the value for the new value to take effect.

# Maximum number of items to upload in a single request during restore (minimum - 1, maximum - 200) 
#restore_max_items_per_upload_batch = 100 

# Maximum total size of all items to upload per request during restore (minimum - 1, maximum - 50) 
#restore_max_payload_per_batch_mb = 5 

# Number of threads to upload items during restore (minimum - 1, maximum - 10) 
#restore_max_item_upload_tasks = 3  

# Maximum number of items to verify in a single request during restore (minimum - 1, maximum - 200) 
#restore_max_items_per_verification_batch = 50 

# Number of threads to verify items during restore (minimum - 1, maximum - 10) 
#restore_max_item_verification_tasks = 2  
 
Note: Changing above settings may have no effect when EWS throttling is enabled on Exchange Online and may even make job performance worse. So, when EWS throttling is enabled on Exchange Online, above settings should be reverted to original values.