Google Cloud Enhances AI/Ml Workflows With Hierarchical Namespace In Cloud Storage
InfoQ, May 14,2025
On March 17, 2025, Google Cloud introduced a hierarchical namespace (HNS) feature in Cloud Storage, aiming to optimize AI and machine learning (ML) workloads by improving data organization, performance, and reliability.​
AI/ML processes, particularly during model training, involve frequent checkpointing to save the model's state. Traditional flat namespace storage systems handle folder renames by rewriting or deleting each object individually, which can be time-consuming and error-prone. With HNS, Cloud Storage now supports atomic folder-level operations, enabling faster and more reliable checkpointing.
According to their blog post, Google's benchmarking indicates that hierarchical namespace buckets can accelerate checkpoint writes by up to 20 times compared to flat buckets. This improvement is achieved through a new RenameFolder API that performs metadata-only operations, completing tasks in a fraction of the time required by flat namespace buckets.