The TAU Programming Languages and Systems Seminar - Architecture for Breaking Cost/Performance/Resiliency Tradeoffs in Enterprise Storage System

Abstract:

VAST Data was founded on one idea: customers should no longer be forced to choose between a series of compromised architecture and application outcomes. Flash, at its fundamental level, has broken the long standing tradeoff between performance and capacity - everything scales linearly with flash. Furthermore, new technologies such as persistent memory and data-center scale low-latency storage fabrics can help flash express this linearity while also driving down costs when applied through a new type of storage architecture.

The journey to this concept required a great deal of contrarian thinking and for us to break decades-old conventions. Fortunately, we had the benefit of a new foundation to build upon - new technologies that enabled VAST's architects to simplify the problem set in order to exponentially expand the gains. The results are very counter-intuitive:

·         Flash is the only media that can be used to bring the cost of storage under what people pay today for HDD-based systems.

·         Low-endurance QLC flash can be used for even the most transactional of workloads.

·         Storage computing can be disaggregated from storage media to enable greater simplicity than shared-nothing and hyper-converged architectures.

·         New local decodable codes can reduce overhead to only 2% while enabling levels of resiliency 10 orders of magnitude more than classic RAID. 

·         A new compression scheme that can compress data much better than existing compression and deduplication algorithm while preserving flash media speed.

·         Customers can build shared storage architectures that can compose and assign dedicated performance and security isolation to tenants on the fly.

·         NFS and S3 can be used for applications that up until now required a level of performance that could only come from block storage.

·         One scalable storage system can be 'universal' and can enable a diverse array of workloads and requirements.

When we understood that absolute performance is not the ultimate target, we realized that the aggregate flash performance across petabytes to exabytes of resilient, affordable flash capacity will truly enable the modern computing agenda. IOPS and bandwidth are now a byproduct of flash capacity, and everything just becomes fast, scalable and cheap.