Scalable, dual-to-large-scale parallel metadata services

Spearheaded by PEAK:AIO with close technical collaboration from Los Alamos National Laboratory, and published with support from the Linux Foundation.

Lattice is not just a faster MDS. It is an attempt to make the MDS itself more ephemeral, horizontally scalable, and free from traditional state constraints.

We invite the community to co-develop the shared Protocol State Plane, richer placement intelligence, and the next generation of open scale-out pNFS storage. 

 

THE FOUNDATION

PEAK:AIO began on FreeBSD to leverage a slow, but established pNFS Flex Files model.

THE CRUCIBLE

With LANL and CMU, PEAK:AIO learned what was needed to push pNFS from entry-level deployments to the larger scale demands of AI, and HPC.

THE OUTCOME

Simplified a parallel file system to basic plug-n-play and created profiles (Strict, AI, HPC) to manage metadata staleness, DS attribute handling, and pre-created file pools etc.

pNFS-Lattice Evolution

The target: Ephemeral metadata fleets

Lattice is a scale-out metadata and coordination architecture for NFSv4.2/pNFS and future distributed storage personalities.

Initiated by PEAK:AIO, shaped through close technical collaboration with Los Alamos National Laboratory (LANL), and launched in collaboration with the Linux Foundation, Lattice is intended to provide a modern open foundation for the next generation of distributed metadata, protocol coordination, and pNFS data services.

The project began on the FreeBSD platform to understand pNFS from first principles, from entry-level deployment through to large-scale adoption. That work now informs Lattice's design goal: a scalable user-space MDS runtime above a shared metadata authority plane, with a future shared protocol state plane able to support both static and more ephemeral MDS instances.