Recommendations for a Secure Storage Project
I have spent the last several years of my career attempting to alert my fellow sumos, partners and customers about the growing cybersecurity threat known as ransomware (see previous post here). Most people (certainly in the cybersecurity and information technology space) now understand the threat better, and some have even taken precautions, updating their security appliances, educating their employees and re-evaluating their backup systems.
I have also waxed poetic that an organization’s backup systems could be cause for concern, since criminals often seek out and destroy an org’s backup data first, to eliminate any possibility of recovery and thus increasing the likelihood of having to pay ransom, typically in untraceable bitcoin.
Recognizing this risk, some organizations have even gone so far as to evaluate the need for a supplemental, redundant backup system that would serve as a sort of “secure enclave”, should the primary data and backup systems also be compromised.
While this is the right idea, I have noticed some troubling trends regarding strategies for improving cybersecurity posture with a separate, redundant backup system. I have also noticed a general misunderstanding regarding terms and nomenclature for this wild west of cybersecurity. This post is designed to explain the most important terms to know, what to look for in a secure, tertiary storage environment and some recommendations for evaluation criteria.
Talk Like an Expert- The Terms to Know
Air Gap
An air gap is a plumbing term, referring to the unobstructed vertical space between the water outlet (like a faucet) and the flood level of a fixture (like a kitchen sink). This provides backflow safety, which protects the water source from contamination. “Air gap” is NOT a useful term regarding secure data access and storage.
Unless the data is stored on removable media (like tape or CD) and stored offline on a shelf, there is no air gap. Furthermore, a true air-gapped computer system also wouldn’t be very useful for recovery, testing or patch management. Any product that is network-attached cannot be “air-gapped”, and even vendors that have adopted this term are still careful to call it an “operational” air gap, which is acknowledging that it is certainly not an air gap. Remember, if you need an air gap, call a plumber. Typically when customers are referring to an air gap solution, they actually mean a strong set of security features that I will describe below in greater detail. So what terms should we be using?
Immutable
FlashArray Storage Snapshots
Now we’re getting somewhere! Immutable storage means that the data cannot be altered, updated or changed in any way. Storage snapshots are a superb example of immutable data storage. Snapshots create a frozen copy of data that is impervious to change. This typically provides a DVR capability to revert back to a point in time before ransomware encrypted the data. Please note that network storage appliances (that are not protected with snapshots) are NOT immutable. Conceivably, the files could be opened and changed. More importantly, even immutable storage can still be destroyed and eradicated, which is where WO/RM comes in⇣
Write Once, Read Many (WO/RM)
WO/RM storage describes an indestructible quality that means data cannot be overwritten, deleted or removed by any user, even a mighty administrator. WO/RM storage has been available for decades dating back to ROM (read-only memory). Tape systems also offered WO/RM varieties and many of you fondly remember CD-R and DVD-R, which allowed for a single-write operation, then prevented any further overwrites to that optical disk. Now that network-attached storage is preferred for backup applications, WO/RM is not a common standard feature found on most appliances but make no mistake; WO/RM is a CRITICAL feature to demand in any highly secure application, such as a secure, hacker-proof storage environment.
Legal Hold
Legal hold is a notification sent from an organization’s legal team to an IT team (and probably relevant employees), instructing them not to delete electronically stored information. Similar to WO/RM, legal hold requires that data be preserved in a tamper-proof and indestructible way. Legal hold differs from WO/RM, in that a legal hold request typically requires the preservation of data to be applied retroactively. To guarantee extended retention of data, legal hold must be applied to an individual’s or organization’s data, often for an indefinite period of time.
Multifactor Authentication (MFA)
MFA is an authentication system that requires more than one distinct authentication factor for successful authentication, typically to gain access to a secure management system. MFA can be implemented through an authentication platform (such as Okta or Duo), or can be implemented with local credentials, and authenticated through an SMS text code, or using a popular authentication app, such as Google’s “Authenticator”. MFA is quite possibly the single most important safeguard against unwanted access to critical applications and security systems.
Now that we know the game, let’s play. Below are suggested starter evaluation criteria for anyone evaluating a secure storage solution ⇣
Recommendations
We recommend that customers looking for secure storage solutions for protection against ransomware should research both modern on-premises and cloud-based storage systems that incorporate a combination of immutable architecture and WO/RM (write-once, read many) technology. Together, these indestructible features create a bedrock of data that cannot be compromised by external threats such as ransomware, or even internal threats such as sabotage.
These devices absolutely must also use strong access controls with multifactor authentication (MFA), preferably with separate credentials from the primary domain (such as Microsoft Active Directory). Ideally, select a system that provides local user authentication with multifactor support.
In this ultra-secure application, we want as much risk isolation as possible, which includes separation of hardware and software development cycles. Another recommendation is to consider only products with entirely different hardware and software from what is currently used for primary storage and backup. One of our customers told us that, during a routine service event, their current vendor’s service technician accidentally reformatted the wrong backup storage system, causing data loss and an outage. While this was unintentional, this event caused the customer to research secure storage for protection against risks like ransomware and sabotage, and the customer evaluated only storage products and vendors that were different from their current provider, in order to minimize the risk of exposure to their secure environment.
Finally, we strongly recommend evaluating only those systems that provide high-performance to be able to meet more demanding SLAs for recovery. Ransomware typically inflicts maximum pain by encrypting as much data as possible, which would require quick recovery of potentially all your data. Pre-ransomware era backup and storage technologies are typically based on slower, low-cost components. We recommend all-flash technologies that can perform at-scale, and allow for easy testing. A system that can perform a near instantaneous restore of all data can also account for the contingency that even the primary storage is unavailable, and thus run indefinitely, until the compromised storage is back online.
Better yet, start to demand these secure storage features in your primary storage and backup systems, and reduce the need to rely on such ultra-secure redundant devices in the first place.
