The Storage Gorilla

A year or two ago, I got a call from one of our sales guys, who was looking for a solution to compete with a technology I’d never heard of, for a pretty unusual requirement.  The customer was doing (very) high-performance video effects work, and the sales guy was looking at potential replacements for an existing solution, the customer having mentioned in passing that while the equipment worked extremely well, she had some concerns around support for what was a non-mainstream product.

Sales guy, having jumped on this opening like a starving weasel, wanted to understand which of the vendors would be the best solution to compete with the incumbent, a company called DataDirect Networks.

Having sat down and looked at the set of requirements provided and done a bit of research into DDN, my response was that nothing currently on the market would do what the customer wanted, at the price they were prepared to pay.  Sure, a DMX or USP-V could be configured to provide what were, quite frankly, outlandish performance requirements, but even preliminary configurations were pricing at around double what the customer claimed to have paid for the DDN system they already had.

“Are you sure they’re actually getting this level of performance?”

The above became my catchphrase for the next few days until the sales guy, disheartened by conversations around solid state disks, huge cache requirements and the need to mortgage a kidney to pay for it all, went off to look for some slightly lower hanging fruit and I was left with the feeling that I should find out more about this DDN bunch.

So who are DDN?

DataDirect Networks (DDN) can be characterised as the most successful unknown niche storage vendor on the market.  A privately held company founded in 1988 and based in Chatsworth California, DDN are the manufacturers of an expanding product range, based around a parallel processing architecture.

Basically, imagine if we took the XIV capabilities and rather than using SATA disk, we attach higher-speed SAS disks.  At this point, we get something which can power “40 of the 100 fastest computing environments in the world”.  At the time I looked at DDN, their architecture was capable of a sustained 2.8GB/s (bytes, not bits) – this has since been expanded to 6GB/s.

That’s.  21.  Terabytes.  Per.  Hour.

Ouch

Put simply, this is about what I’d expect to get practically out of an enterprise system costing twice as much and smothered in solid state technology.  DDN achieve this with spinning disks and a Clariion-style footprint. And, as I keep coming back to, a lower price-tag.  With the implementation of new architectures and solid state disks, DDN are claiming that 10GB/s is now possible.

So what’s the secret?

The DDN product range is based around the Silicon Storage Appliance (S2A) and Storage Fusion architectures.  As with the IBM XIV, these arrays use a massively parallel processing architecture to ensure that sequential data is processed at effectively wire speeds.  Unlike mid-range arrays such as the EMC Clariion or Hitachi AMS, data is not funnelled to a single cache to speed up disk access – all error checking is done by the individual S2A processing complexes and passed directly to disk.

So what can DDN do for me?

Basically, DDN have opted for more flexibility than IBM have with XIV.  Rather than a single disk type, DDN provide a range of disk types and shelf configurations.  The performance option is similar to XIV’s front-loaded, 12 disks per shelf design, providing a small capacity footprint, but achieving the promised high-performance.  This is the configuration which is used in the supercomputing environments which make up DDN’s more interesting case studies.

The high-capacity solution on the other hand, uses a top-loading, densely packed shelf, allowing 60 disks per enclosure – with 2TB SATA drives, this would be capable of around 2 petabytes in a single rack.  This is marketed as a high-performance VTL and archiving product, rather than as a compute platform.  More recently, the company has moved from building block storage arrays, to also providing scalable file-system storage, based on their own NAS designs.

So where can I buy one?

In the UK at least, DDN arrays tend to turn up as part of larger vendors’ solutions.  IBM have long sold the S2A arrays as part of their supercomputing portfolio and have integrated the capacity versions into their SONAS product. 

Update – I’m told on good authority that the DDN products are also resold outside of the suporcomputing area as the DCS9900.  I’d be interested to hear from any IBM customers (or anyone else) who are using DDN, but particularly in this area, as I’ve not seen a case-study or use-case for non BlueGene implementations of DDN.

HP have recently announced that they will sell DDN as their top-end NAS product.  It appears from the announcements that, rather than bolt the storage array onto their own NAS offerings as IBM have done, HP will sell DDN’s own NAS offering.

Conclusion – so is this a competitor to XIV?

Strangely no, not really.  At least not yet.

At least at the present time, DDN tend to sell to pretty specific use-cases.  While XIV is now sold as a general computing platform, the DDN arrays tend to be sold for big number-crunching and media-serving environments.  The head of the company was quoted as saying that, in non-media serving environments, the S2A architecture would be “horrible” compared to competitors such as the EMC Clariion. 

But then, as IBM showed by marketing the XIV in the early days as a web serving platform only, things can change (P45 for the head of marketing, hey?).  It may be that DDN forms the upper and lower tiers of a storage environment, with the middle taken up by a general computing platform such as Clariion or XIV. 

The diagram below shows a possible solution of this type:

My own feeling is that the only thing stopping DataDirect Networks from becoming the next generation of EMC or HP storage is the fact that it is still a private company.  If DDN ever take the IPO route, I predict they will last about 5 minutes before the first takeover attempts (hostile or not) begin. 

This might not be a bad thing.  XIV is currently gaining IBM market share based on a combination of usability, cost and performance which I’ve not seen traditional storage architectures begin to touch.  DDN for their part, would get the same kind of boost that XIV received from IBM – a properly global reach, greater R&D spend and far greater brand awareness, at the cost of identity.

Depressed by 10 years of reselling LSI seconds, IBM went for a complete makeover, and stole a march on the competition by purchasing the technology most likely to shake up the market.  Other storage vendors will have difficulty developing an answer to the XIV architecture in a reasonable time – purchasing the DDN architectures would give an immediate route to replace existing dual-processor architectures (and an existing user-base of weird and wonderful customers). 

The attempts may already have begun.  Commentators have pointed out that the HP reseller deal fills no gaps in the HP product range – they already have large scale NAS products so DDN is effectively redundant within their portfolio.  I’d suggest that this is an attempt by HP to get an understanding of the practical reality of the DDN products, with a view to understanding whether it’s a credible product set for HP to buy up – possibly as a replacement for the rebadged HDS arrays currently sold in the enterprise space. 

Either way, I’m keeping an eye on DDN – in any case, reading their customer case studies is always going to be entertaining.  A selection below gives a sample of customers:

• TimeWarner Cable – Video on Demand
• Pacific Title & Art – The post-production effects house for Batman – The Dark Knight movie
• Microsoft Studios – Xbox Live Media & Data Serving
• CCTV – Chinese State Television, Media Serving for Coverage of the Beijing Olympics
• Shutterfly – Photo sharing site serving up to 1 billion photos at present (growth from 100 million in 2 years)
• Slide.com
• Kodak Easyshare
• National Center for Data Mining (UIC)
• Northwestern University & John Hopkins University – Winner, 7th Annual Bandwidth Challenge (2006)
• CalTech, CERN, University of Florida and the University of Michigan – 2nd, 7th Annual Bandwidth Challenge
• Indiana University – 3rd place, Bandwidth Challenge at Supercomputing 2006
• Lawrence Livermore National Laboratories,
• Sandia National Laboratories,
• NASA and NASA Ames
• Argonne National Laboratory