DDN On Files, Objects And AI Training And Inference
Blocks&Files, Monday, February 24th, 2025
DDN has been supplying high speed file access with its EXAScaler Lustre parallel file system software for many years. This supports NVIDIA GPUDirect protocol. It has just announced its Infinia 2.0 fast access object storage and this will support GPUDirect for objects.
Interview By Chris Mellor
Led by VAST Data and Glen Lockwood, a Microsoft Azure AI supercomputer architect, there is a move to file-based consign AI training on NVIDIA GPUs to history and use object storage in future. How does DDN view this concept? We discussed this with by James Coomer, SVP For Products at DDN, and the following Q and A has been edited for brevity.
Blocks & Files: Where would you start discussing file and object storage use and AI?
James Coomer: There's three main points. Firstly, it's not really as if all large scale AI systems can, out of the box, use S3 and we try it, right? So we've got the fastest S3 in the planet with Infinia. The 'general object storage challenge' is it's really not mature.
So the fact is, just ask any one of these SuperPODs, ask Nvidia, ask the largest language model developers on the planet. They're using file systems and they're using parallel file systems. So are they all wrong? I'm not sure they are. So firstly, the current state of play isn't what they're stating. If you were starting from scratch today, could you design an IO model which could avoid the use of a parallel filesystem and try to do things in parallel?