Need an Air Gap? Call a Plumber

Recommendations for a Secure Storage Project

I have spent the last several years of my career attempting to alert my fellow sumos, partners and customers about the growing cybersecurity threat known as ransomware (see previous post here). Most people (certainly in the cybersecurity and information technology space) now understand the threat better, and some have even taken precautions, updating their security appliances, educating their employees and re-evaluating their backup systems.

I have also waxed poetic that an organization’s backup systems could be cause for concern, since criminals often seek out and destroy an org’s backup data first, to eliminate any possibility of recovery and thus increasing the likelihood of having to pay ransom, typically in untraceable bitcoin.

Recognizing this risk, some organizations have even gone so far as to evaluate the need for a supplemental, redundant backup system that would serve as a sort of “secure enclave”, should the primary data and backup systems also be compromised.

While this is the right idea, I have noticed some troubling trends regarding strategies for improving cybersecurity posture with a separate, redundant backup system. I have also noticed a general misunderstanding regarding terms and nomenclature for this wild west of cybersecurity. This post is designed to explain the most important terms to know, what to look for in a secure, tertiary storage environment and some recommendations for evaluation criteria.

Talk Like an Expert- The Terms to Know

Air Gap

An Air Gap. Useful for passing a health inspection.

An air gap is a plumbing term, referring to the unobstructed vertical space between the water outlet (like a faucet) and the flood level of a fixture (like a kitchen sink). This provides backflow safety, which protects the water source from contamination. “Air gap” is NOT a useful term regarding secure data access and storage.

Unless the data is stored on removable media (like tape or CD) and stored offline on a shelf, there is no air gap. Furthermore, a true air-gapped computer system also wouldn’t be very useful for recovery, testing or patch management. Any product that is network-attached cannot be “air-gapped”, and even vendors that have adopted this term are still careful to call it an “operational” air gap, which is acknowledging that it is certainly not an air gap.  Remember, if you need an air gap, call a plumber. Typically when customers are referring to an air gap solution, they actually mean a strong set of security features that I will describe below in greater detail. So what terms should we be using?


FlashArray Storage Snapshots

Now we’re getting somewhere! Immutable storage means that the data cannot be altered, updated or changed in any way. Storage snapshots are a superb example of immutable data storage. Snapshots create a frozen copy of data that is impervious to change. This typically provides a DVR capability to revert back to a point in time before ransomware encrypted the data. Please note that network storage appliances (that are not protected with snapshots) are NOT immutable. Conceivably, the files could be opened and changed. More importantly, even immutable storage can still be destroyed and eradicated, which is where WO/RM comes in⇣

Write Once, Read Many (WO/RM)

WO/RM storage describes an indestructible quality that means data cannot be overwritten, deleted or removed by any user, even a mighty administrator. WO/RM storage has been available for decades dating back to ROM (read-only memory). Tape systems also offered WO/RM varieties and many of you fondly remember CD-R and DVD-R, which allowed for a single-write operation, then prevented any further overwrites to that optical disk. Now that network-attached storage is preferred for backup applications, WO/RM is not a common standard feature found on most appliances but make no mistake; WO/RM is a CRITICAL feature to demand in any highly secure application, such as a secure, hacker-proof storage environment.

Legal Hold

Legal hold is a notification sent from an organization’s legal team to an IT team (and probably relevant employees), instructing them not to delete electronically stored information. Similar to WO/RM, legal hold requires that data be preserved in a tamper-proof and indestructible way. Legal hold differs from WO/RM, in that a legal hold request typically requires the preservation of data to be applied retroactively. To guarantee extended retention of data, legal hold must be applied to an individual’s or organization’s data, often for an indefinite period of time.

Multifactor Authentication (MFA)

MFA is an authentication system that requires more than one distinct authentication factor for successful authentication, typically to gain access to a secure management system. MFA can be implemented through an authentication platform (such as Okta or Duo), or can be implemented with local credentials, and authenticated through an SMS text code, or using a popular authentication app, such as Google’s “Authenticator”. MFA is quite possibly the single most important safeguard against unwanted access to critical applications and security systems.

Now that we know the game, let’s play. Below are suggested starter evaluation criteria for anyone evaluating a secure storage solution ⇣

Example Evaluation Criteria


We recommend that customers looking for secure storage solutions for protection against ransomware should research both modern on-premises and cloud-based storage systems that incorporate a combination of immutable architecture and WO/RM (write-once, read many) technology. Together, these indestructible features create a bedrock of data that cannot be compromised by external threats such as ransomware, or even internal threats such as sabotage.

These devices absolutely must also use strong access controls with multifactor authentication (MFA), preferably with separate credentials from the primary domain (such as Microsoft Active Directory). Ideally, select a system that provides local user authentication with multifactor support.

In this ultra-secure application, we want as much risk isolation as possible, which includes separation of hardware and software development cycles. Another recommendation is to consider only products with entirely different hardware and software from what is currently used for primary storage and backup. One of our customers told us that, during a routine service event, their current vendor’s service technician accidentally reformatted the wrong backup storage system, causing data loss and an outage. While this was unintentional, this event caused the customer to research secure storage for protection against risks like ransomware and sabotage, and the customer evaluated only storage products and vendors that were different from their current provider, in order to minimize the risk of exposure to their secure environment.

Finally, we strongly recommend evaluating only those systems that provide high-performance to be able to meet more demanding SLAs for recovery. Ransomware typically inflicts maximum pain by encrypting as much data as possible, which would require quick recovery of potentially all your data. Pre-ransomware era backup and storage technologies are typically based on slower, low-cost components. We recommend all-flash technologies that can perform at-scale, and allow for easy testing. A system that can perform a near instantaneous restore of all data can also account for the contingency that even the primary storage is unavailable, and thus run indefinitely, until the compromised storage is back online.

Better yet, start to demand these secure storage features in your primary storage and backup systems, and reduce the need to rely on such ultra-secure redundant devices in the first place.

Greener Pastures

After 5 ½ years at Nimble Storage, I recently made the difficult decision to leave for greener pastures-  As of today, my stellar account executive Bryan and I are Cohesians!  I wish my friends back at HPE | Nimble all the best.  I am grateful to you all for the experience and I look forward to watching as you duel it out in the Coliseum that is the primary storage marketplace – This time watching from the stands.  I’ve got my popcorn ready!

Why Cohesity

I hope it goes without saying I wanted to work for a company that shares my professional values around winning the right way, customer focus, and something our founder, Mohit Aron says, “Stay humble and keep learning.”  That statement itself is pretty humble from the lead architect behind the google file system (GFS), along with founding one of technology’s most successful startups in recent years, Nutanix.

Of course I also needed to work for a company I believe has luminescent technology, something that can actually help organizations reach their objectives easier and faster.  I joined Cohesity because I firmly believe that the simplest solution always has the lowest TCO, and managing data has to get radically simpler. 

The Problem with Today’s Storage Offerings

The primary storage market has undergone a serious transformation in the past 5 years, thanks largely to NAND flash, which is THE game changer in primary storage today.  The transition is well under way and primary storage providers offer terrific choices for customers looking to upgrade their old primary production storage systems with a flashy new storage array or hyperconverged appliances.

Are newer storage systems faster?  You bet!  More efficient?  Certainly.  Simpler?  Like, completely eliminating islands of backup/file/object/cloud storage?  Eh, not really.  While primary storage today is faster and more efficient than ever before, I noticed that many of my customer’s most insistent demands were not being addressed.  These needs include (but are not restricted to):

  • Comprehensive consolidation of data silos
  • Modernized data protection and ransomware strategy
  • Improved operational visibility (think dark data)
  • App mobility (from site à site, site à cloud or cloud à cloud)
  • Ability to scale up/out
  • Tech refresh & lifecycle management

A system for managing data storage that is truly simple would address most of these needs, but the problem is that massive data fragmentation has led to dark silos of fractured infrastructure that is vulnerable to threats, immobile, inefficient and impossible to extract any value from.  No commercially available platform can really address all these needs, at least not until now.

We are entering into a new era where data growth is exponential, and merely updating new storage media and protocols has done very little to solve these newer fragmentation, mobility and visibility difficulties.  Historically, attempting to address these large-scale customer needs with a single service, vendor or application is like trying to strap up another horse to pull your buggy…  It might go a little faster but it’s more complex and it will never be a car.

Back it Up, Back it Up – Beep, Beep, Beep

Take for example, data protection.  Let’s say your company puts out an RFP for a complete backup solution.  One particular vendor offers data protection software for backup & DR.  After a nice demo, this starts to sound pretty good but upon further examination, a complete solution would require a server OS to run the software (like Windows), a general purpose files system (like NTFS), and a disk appliance sized to your best projections for several years.

Even within just this one area of secondary data, we have given an example of fragmentation of GUI’s, multiple vendor relationships, support contracts and so on.  Even worse, at large scale, this type of traditional backup architecture will require multiple proxy servers and disk silos to spread the load, further amplifying the fragmentation for larger enterprise organizations.

While at first, this backup software looked promising to address a critical need for data protection, now this solution looks far too complex and limited.  This type of typical backup solution does nothing to collapse other silos of storage such as file/object, test/dev & analytical workloads, and prolonged exposure will give you… Confusion!

Managing data is way too complex.  As my peer Dimitris says on his excellent blog post here, “storage should be easy to consume.” I whole-heartedly agree with his thesis statement.  I believe what Cohesity offers is something fundamentally new. 

After a few of my most-respected peers moved to Cohesity, I had to rub the magic lamp.  Out popped a blue Robin Williams and Poof, I’m here!  Now let me introduce you to our enchanted potion (ok, I’m done with magic jokes.)

Cohesity addresses these needs with a platform we suitably named DataPlatform®.  Cohesity DataPlatform® is a scale-out solution that consolidates all your secondary workloads, including backup and recovery, files and objects, test/dev and analytics in a single, cloud-native solution.  I look forward to telling you all about Cohesity in follow-on posts about what DataPlatform® does and how it does it, but for now- just know that it’s really cool.

If you read this far, thank you! Bryan and I plan to have many more posts and pics as we tour the mid-south spreading green goodness and making all our customer’s data dreams come true so stay tuned, and as always, #GoGreen!