Data Deduplication Really a Big Deal?

Data deduplication is dramatically improving IT economics by minimizing storage footprint requirements, backup windows, and network bandwidth consumption in distributed enterprises and datacenter locations alike.  Many vendors now have backup solutions out on the market that offer deduplication features.  Worth it?

In real-world environments, deduplication is accelerating backup and recovery efficiency and driving down IT costs.  The real driver is the ability to keep pace with the near doubling of storage growth annually.  This growth is fueled by new applications, the proliferation of virtualization, creation of electronic document stores and document sharing, and the retention or preservation of digital records.  With many budgets under pressure, the need to curb data growth is one of the top priority items - reduce capital and operating costs.  From a physical perspective, many data center managers are also dealing with infrastructure concerns in terms of power, cooling, floor space and DR.  Deduplication is a technology that not only aids in accelerating storage efficiency by reducing cost but also alleviates physically constrained data centers.

Deduplication also addresses challenges associated with management, backup, and network inefficiency.  As data grows, there is an increasingly disproportionate relationship between the number of IT personnel and the amount of storage requiring management.  Deduplication reduces the data footprint, keeping this ratio in balance.  In addition, as the gap between server processing power and disk continues to widen, many companies are looking for ways to improve performance throughout their environment over a WAN, within disk storage subsystems, and across limited backup windows.  Data deduplication technology can optimize available physical and virtual infrastructure by sending less data over local or remote network links.  It can also improve service level response times and help meet shrinking backup windows.  Deduplication also makes use of random access media, improving recovery times, data security, and reliability.

So how does it work?  Data deduplication is most often associated with subfile comparison processes.  This is different from single-instance storage (SIS), which compares data at the file or object level.  Subfile deduplication examines a file and breaks it up into "segments".  These smaller segments are then evaluated for the occurrence of redundant data content across multiple systems and locations. Deduplication is also different from compression, which reduces the footprint of a single object rather than across files or pieces of a file.  Additionally, deduplication data can also be compressed for further space savings.  Like a Unitrends product.

Some of the benfits of deduplication:  drive down cost, improve backup and recovery, change the economics of IT, and reduce the carbon footprint.  So given these benefits, it would seem it makes sense.  I would recommend your due diligence on the vendor and product selection.  Not all work as advertised, nor worth the investment.  I would also start with your remote offices and focus on the WAN sites first and then proceed from there. 

If you are still on tape for your backups, I would focus on getting to a new media first.  Reliability of tape solutions is not acceptable to the enterprise, nor is the lack of security.  I will have a future discussion around D2D, Tape, and D2D2D type backups.  Keep checking this blog each day.