Search <book_title>...

InfoScale™ Operations Manager 9.0 User's Guide

Last Published: 2025-04-14

Product(s): InfoScale & Storage Foundation (9.0)

Platform: AIX,HP-UX,Linux,Solaris,VMware ESX,Windows

About file system deduplication

The deduplication feature in Storage Foundation enables customers to use file system deduplication to optimize existing primary storage. Enabling deduplication at the file system layer results in storage savings and avoids complex and expensive appliances typically associated with file deduplication.

Deduplication is performed without needing any application changes and with minimal overhead. Deduplication does not change the file extension, allowing users and applications to use files normally, without performance impact.

Before setting up deduplication for a file system, evaluate whether the nature of the data makes it a good candidate for deduplication.

The following are good candidates for deduplication:

Virtual machine boot image files (vmdk files)
User home directories
File systems with multiple copies of files

The following might not be the best candidates for deduplication, as they have little or no duplicate data:

Databases
Media files, such as JPEG, MP3, and MOV

The VxFS deduplication feature works as follows. It eliminates duplicate blocks used by your data by comparing blocks across the file system. When the deduplication feature finds a duplicate block, it removes the space used and instead creates a pointer to the common block. If the duplicate file is changed, thus making the files no longer share the same block, then that changed block is saved to disk instead of the pointer.

The deduplication process performs the following tasks:

Scans the file system for changes
Fingerprints the data
Identifies duplicates
Eliminates duplicates after verifying the duplicates

The space consumed by the deduplication database is a function of the amount of data in the file system and the deduplication chunk size. On Linux or Solaris, Arctera recommends a chunk size of 4k for SFCFSHA, where multiple copies of virtual machine images are accessed over NFS. For all other datasets, Arctera recommends a chunk size of 16k or higher. More information is available on deduplication chunk size.

The deduplication feature has the following limitations:

A full backup of a deduplicated Veritas File System (VxFS) file system can require as much space in the target as a file system that has not been deduplicated. For example, if you have 2 TB of data that occupies 1 TB worth of disk space in the file system after deduplication, this data requires 2 TB of space on the target to back up the file system, assuming that the backup target does not do any deduplication. Similarly, when you restore such a file system, you must have 2 TB on the file system to restore the complete data. However, this freshly restored file system can be deduplicated again to regain the space savings. After a full file system restore, Arctera recommends that you remove any existing deduplication configuration and that you reconfigure deduplication.
Deduplication is limited to a volume's primary fileset.
Deduplication does not support mounted clone and snapshot mounted file system.
After you restore data from a backup, you must deduplicate the restored data to regain any space savings provided by deduplication.
If you use the cross-platform data sharing feature to convert data from one platform to another, you must remove the deduplication configuration file and database and re-enable deduplication after the conversion.
You cannot use the FlashBackup feature of NetBackup in conjunction with the data deduplication feature, because FlashBackup does not support disk layout Version 8 and 9.